Noah Kantrowitz

Jenkins Wrangling for Fun & Profit

2017-12-05T05:00:00Z

While there have been many new developments in CI/testing tools, Jenkins is still a mainstay. And to be fair to the Jenkins team, it has come a tremendous way in the past few years. The new Pipelines system is more flexible than anything I’ve used before, and the Blue Ocean UI is a big graphical and UX upgrade. My team at SAP started using Jenkins long before I arrived, but over the years we have slowly accumulated some complaints about how it was working and how we managed things.

This post is going to be a (very) long-form dive into how we set things up and why. I do not think this is going to work out of the box for (almost) anyone else, but the hope is that this will provide a blueprint for others to build their own solutions on the same general ideas.

The Problems

Before I launch into what we did, let’s list out the issues we had with our current set up so we are all on the same page. Leaving aside some details that exist only for PCI, we have a Jenkins server deployed by Chef. Some plugins were installed originally by the Chef cookbook, but most were installed and upgraded by hand since then. Jobs were mostly created via a custom CLI tool that talks to the Jenkins API, but then updated by hand (or more often, not) after that.

So we spent some time in a meeting room with a whiteboard and came up with a few top-level problems:

Upgrades are too unpredictable, both for Jenkins and individual plugins.
Jenkins configuration is (mostly) not versioned, ditto for job configs.
Job configs can bitrot over time with no easy way to update them other than one at a time.
No pull-request builds, and overall existing builds are quite slow.

While this applied to a lot of different use cases for Jenkins, the one I chose to tackle first was Chef cookbook testing, but with a clear eye towards building a solution which will grow to include other use cases as we want to move them off the old Jenkins server.

The Shape

After a bunch of research, the overall shape of the goal came together fairly quickly. We’ll go through each of these in excruciating detail later on, but to start let’s break it down into bullet points:

Deployed on Kubernetes, because this is the way we’re trying to move everything.
Build a container image containing Jenkins, all the plugins, and configuration.
Use Helm for managing the deployment (and rollback if needed).
Manage the configuration via Jenkins groovy as much as possible.
Use the “organization folder” system in Jenkins to auto-detect projects.
Use shared pipeline libraries to keep the per-repository config low.
Build a container image for the cookbook testing environment that has all needed gems pre-installed.
Work out a way to test cookbooks on top of Kubernetes pods.

Kubernetes?

We did briefly look over non-Kubernetes deployment options like building a new Chef cookbook or using a dedicated Nomad cluster, but with the continued rise of Kubernetes as an operations platform it seemed like a good idea to use this as an internal experiment for running “real” (but not customer facing) services on Kubernetes. In the months since that choice, I think we have only seen the industry move even more behind Kubernetes as the next dominant platform so this seems to have been the right move. If you already have a heavy investment in Mesos or Nomad then perhaps just ignore the Kubernetes-specific bits of this.

Within the Kubernetes ecosystem there are a few tools/patterns for managing deployment of complex applications (i.e. things that need more than just a pod). While “folder full of YAML + kubectl” and ksonnet are nice from a simplicity point-of-view, the rollback capabilities of Helm made it the clear choice in my mind.

The deployment of Kubernetes and Helm themselves are out-of-scope for this post; there are numerous guides for Kubernetes and setting up Helm is mostly just helm init. Our production cluster is currently on AWS and set up with Kops, but your mileage may vary if you aren’t on AWS. If you just want to play with the stuff in this post, I would highly recommend starting with a hosted cluster option like Google GKE, Azure AKS, or the newly announced Amazon EKS.

The Jenkins Container

The first stop on our journey is building a Docker image for the Jenkins server. Going line by line so we can talk about it as we go:

FROM jenkins/jenkins:2.92-alpine

We’re starting from the existing Jenkins Docker Hub images. This determines which version of Jenkins gets used so doing a Jenkins upgrade consists of changing this line, building a new container, pushing it to our registry, and then upgrading the Helm release. This is using the Jenkins weekly release, so we try to keep this bumped roughly once a week, though if it ends up a few weeks behind that’s totally fine.

COPY saml-idp-metadata.xml /metadata.xml
COPY plugins.txt /plugins.txt
COPY style.css /style.css

Next we copy some base files. The SAP internal authentication system uses SAML (please hold your sighs for the end) so we store the IdP metadata in a file to use in the Jenkins config. The plugins.txt looks like:

kubernetes:1.1
workflow-aggregator:2.5
workflow-job:2.15
credentials-binding:1.13
git:3.6.4
blueocean:1.3.3
github-oauth:0.28.1
matrix-auth:2.2
saml:1.0.4

and gets used later down by the install-plugins.sh script that comes in the base image. The style.css file is a few minor tweaks on top of the theme for things it gets wrong or we didn’t like:

// The theme overwrites this so we need to fix.
.glyphicon {
  font-family: 'Glyphicons Halflings' !important;
}

// Theme has no icon for this.
.icon-github-branch {
  background-image: url('/static/ccf6b398/plugin/github-branch-source/images/24x24/github-branch.png');
}

// Force all icons in that bar to be grayscale.
.icon-md {
  filter: grayscale(1);
}

Overall files that aren’t expected to change often.

RUN mkdir -p /usr/share/jenkins/ref/secrets && \
    # Why is this not the default?
    echo false > /usr/share/jenkins/ref/secrets/slave-to-master-security-kill-switch && \
    # Install all our plugins so they are baked in to the image.
    /usr/local/bin/install-plugins.sh < /plugins.txt && \
    # Install a nicer default theme to make it look shiny for non-BlueOcean.
    mkdir /usr/share/jenkins/ref/userContent && \
    curl --compressed http://jenkins-contrib-themes.github.io/jenkins-neo-theme/dist/neo-light.css > /usr/share/jenkins/ref/userContent/neo-light.css.override && \
    cat /style.css >> /usr/share/jenkins/ref/userContent/neo-light.css.override

Then the meat of the Dockerfile. A poorly documented feature of the Jenkins Docker image is that all files under /usr/share/jenkins/ref are used to seed the creation of the JENKINS_HOME folder during startup. Normally these files are only copied over the first time, but if they end in .override it is copied every time Jenkins starts (with the .override trimmed off).

First we set the poorly named slave-to-master-security-kill-switch file which makes it so JNLP builders don’t get admin access to the Jenkins server because we don’t want rogue builds to take down the universe if possible.

Next we install all the Jenkins plugins. It should be noted that the Docker layer cache can sometimes bite you here. Because we only list the top-level plugins we want (the script handles finding all dependencies), if we want to upgrade an internal dependency but nothing else has changed, might need to manually zap the cached layer image. Given that Jenkins itself releases weekly anyway (meaning we change the FROM image and invalidate the whole cache), it’s not hugely likely that this will an operational problem, but be aware.

After that we set up some custom CSS theming. While Blue Ocean does have a nicely refreshed UI, the default post-login landing page is still the normal UI and we wanted to spruce that up a bit, both for aesthetic reasons and to make it easier to tell at a glance which Jenkins server you are looking at. Neo-light seemed the nicest of the themes that still worked, but you can change or ignore this part as you wish.

COPY config.groovy /usr/share/jenkins/ref/init.groovy.d/zzz_alti-jenkins.groovy.override
COPY plugin/target/alti-jenkins-plugin.hpi /usr/share/jenkins/ref/plugins/alti-jenkins-plugin.hpi.override

And finally we copy over two more files. These change more often than the plugins.txt or Jenkins version (at least so far during development, hopefully that will change over time) so they go at the end. The config Groovy code gets put in place to be run automatically at startup, with the weird zzz thing because Jenkins alpha-sorts the hook scripts if there is more than one and we want to be last. The config itself is big and complex so we’ll cover that further down.

The alti-jenkins plugin is a bit of an experiment, currently all it does is add the <link> tags to the HTML for the theme CSS and sets a few security HTTP headers. This could probably be replaced with the simple-theme plugin instead, but I would like to add more stuff to it (ex. custom job health metric that ignores failed PR builds), so we’re leaving it for now.

And with that, we have a Jenkins Docker image. docker build -t ourrepo.com/alti_jenkins:2.92 ., docker push ourrepo.com/alti_jenkins:2.92, ???, profit. As a built artifact this encompasses the Jenkins release, all plugins used, and the configuration code. Just about everything we could ask for.

We’ll talk about the config.groovy in just a moment, but because all secrets (or any other configuration you want to hide) is coming in at run-time from Kubernetes, this image doesn’t actually contain anything that needs to be hidden. If you aren’t running your own registry already, you could push this up to a public Docker Hub account instead.

`config.groovy`

This was the bulk of my time on the project, a slowly expanding config script that started with some basics and now encompasses the entire setup process. I will lead off with the fact that I am neither a Jenkins nor Groovy expert so I’m sure this code can be improved, for example I only learned very late in my writing that import is optional in Groovy if you use the fully-qualified class name. With that in mind, let’s go line by line again:

import static jenkins.model.Jenkins.instance as jenkins

The most important import, the Jenkins object singleton. We use this a ton, so put it in a magic global.

import com.cloudbees.plugins.credentials.*
import com.cloudbees.plugins.credentials.common.*
import com.cloudbees.plugins.credentials.domains.*
import com.cloudbees.plugins.credentials.impl.*
import hudson.security.GlobalMatrixAuthorizationStrategy
import hudson.security.Permission
import hudson.util.Secret
import jenkins.branch.OrganizationFolder
import jenkins.install.InstallState
import jenkins.plugins.git.GitSCMSource
import org.csanchez.jenkins.plugins.kubernetes.ContainerEnvVar
import org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate
import org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud
import org.csanchez.jenkins.plugins.kubernetes.PodTemplate
import org.csanchez.jenkins.plugins.kubernetes.ServiceAccountCredential
import org.jenkinsci.plugins.github_branch_source.BranchDiscoveryTrait
import org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator
import org.jenkinsci.plugins.github_branch_source.OriginPullRequestDiscoveryTrait
import org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl
import org.jenkinsci.plugins.saml.SamlEncryptionData
import org.jenkinsci.plugins.saml.SamlSecurityRealm
import org.jenkinsci.plugins.workflow.libs.LibraryConfiguration
import org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever

println "--> configuring alti_jenkins"

As mentioned, I didn’t really know how Groovy imports worked when starting this so this is mostly not needed but I haven’t cleaned it up yet.

try {

By default, if an init hook script fails, Jenkins prints an error to the log and keeps on truckin’. The whole config code is inside a try/catch so we can at least attempt to not let it continue starting if we might have failed to configure something important (like, say, authentication). This will only catch runtime errors though, if there is a syntax error in the script, that will still result in Jenkins starting as per usual.

  //////// CONFIG
  def secretsRoot = System.getenv('JENKINS_SECRETS') ?: '/var/jenkins_secrets'
  def downwardRoot = System.getenv('DOWNWARD_VOLUME') ?: '/etc/downward'
  println "--> Loading configuration from from secrets:$secretsRoot and downward:$downwardRoot"
  def githubUser = new File("$secretsRoot/github-user").text.trim()
  def githubUserToken = new File("$secretsRoot/github-token").text.trim()
  def samlPass = new File("$secretsRoot/saml-pass").text.trim()
  def samlKeystore = "$secretsRoot/saml-keystore"
  def developmentMode = new File("$secretsRoot/development-mode").text.trim() == 'true'
  def kubeNamespace = new File("$downwardRoot/namespace").text.trim()
  def admins = [
    'nkantrowitz', // Noah Kantrowitz
    'etc', // Someone else
  ]
  def githubOrg = 'MyOrg'
  def librariesRepo = "$githubOrg/jenkins-pipeline-libs"
  def agentVersion = '3.10-1-alpine'

  // Parse the labels test.
  def labels = [:]
  new File("$downwardRoot/labels").eachLine {
    def parts = it.split('=')
    labels[parts[0]] = parts[1][1..-2]
  }

Next up, loading and parsing a bunch of configuration data. We’ll look at the pod configuration later on, but this is mostly reading from either a Kubernetes Secret volume (for secrets) or a Downward API volume (for metadata about the pod we are running inside of). And then a few hardcoded values that don’t change often enough to be exposed outside of the file/image like the name of the GitHub organization.

  //////// GENERAL SETTINGS
  // Bypass the setup wizard because this script defines all of our config.
  // This is _supposed_ to be handled by /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state
  // but that doesn't seem to be working. See https://github.com/jenkinsci/docker#script-usage.
  if (!jenkins.installState.isSetupComplete()) {
    println '--> Neutering SetupWizard'
    InstallState.INITIAL_SETUP_COMPLETED.initializeState()
  }
  // Disable CLI over the remoting protocol for security.
  jenkins.getDescriptor("jenkins.CLI").get().enabled = false
  // More security, disable old/unsafe agent protocols.
  jenkins.agentProtocols = ["JNLP4-connect", "Ping"] as Set
  // Enable CSRF.
  jenkins.crumbIssuer = new hudson.security.csrf.DefaultCrumbIssuer(true)
  // Disable execution on the main server.
  jenkins.numExecutors = 0

Then some baseline global configuration. This disables the “welcome to Jenkins” setup wizard, sets some security stuffs, and turns off job execution on the Jenkins server itself because we want all jobs to run inside Kubernetes workers.

  //////// AUTHENTICATION
  if (new File(samlKeystore).exists()) {
    // Configure the SAML plugin.
    println '--> Configuring SAML authentication realm'
    def realm = new SamlSecurityRealm(
      new File('/metadata.xml').text, // String idpMetadata,
      'display_name', // String displayNameAttributeName,
      '', // String groupsAttributeName,
      SamlSecurityRealm.DEFAULT_MAXIMUM_AUTHENTICATION_LIFETIME, // Integer maximumAuthenticationLifetime,
      'uid', // String usernameAttributeName,
      'email', //  String emailAttributeName,
      '', // String logoutUrl,
      null, // SamlAdvancedConfiguration advancedConfiguration,
      new SamlEncryptionData( // SamlEncryptionData encryptionData,
        samlKeystore, // String keystorePath,
        Secret.fromString(samlPass), // Secret keystorePassword,
        Secret.fromString(samlPass), // Secret privateKeyPassword,
        'saml-key' // String privateKeyAlias
      ),
      'lowercase' // String usernameCaseConversion,
    )
    jenkins.securityRealm = realm
  }
  else {
    println '--> Not configuring SAML'
    // TODO This shoud set up some fallback realm.
  }

Ahh the fabled dev TODO, I swear I’ll get back to that someday. This section is setting up the authentication system for logging in to Jenkins, the “security realm” in official parlance. For our production servers we’re using our company-wide SAML SSO system because if I can ever not have to store passwords, I’ll take that option in a heartbeat. If you don’t have a a similar internal SSO system, I would recommend looking at the GitHub OAuth plugin, but you can always use the internal login form realm if needed. If you are using SAML, the attribute configuration options are likely to be different for you, but the rest should look pretty similar. For unknown reasons, the Jenkins SAML plugin refers to the SP signing key as “encryption data”, but it definitely is the signing key. You can also see here is where we read back in the IdP metadata we embedded in the Jenkins container image up above.

  //////// AUTHORIZATION
  if (developmentMode) {
    // Turn off authorization in case hacking on SAML configs leads to lockout.
    // As it says in the values.yaml, do not do this in production.
    println "--> Configuring Unsecured authorization strategy. THIS BETTER NOT BE PROD."
    def unsecured = new hudson.security.AuthorizationStrategy$Unsecured()
    jenkins.authorizationStrategy = unsecured
  }
  else {
    // Configure matrix auth and ACLs.
    println "--> Configuring Matrix authorization strategy."
    def authz = new GlobalMatrixAuthorizationStrategy()
    [
      "hudson.model.Hudson.Read",
      "hudson.model.Item.Build",
      "hudson.model.Item.Cancel",
      "hudson.model.Item.Discover",
      "hudson.model.Item.Read",
      "hudson.model.Item.Workspace",
      "hudson.model.Run.Replay",
      "hudson.model.Run.Update",
    ].each {
      // Use the string form because I'm lazy and don't want to import all the things.
      authz.add(it + ":authenticated")
    }
    // Admins always get all permissions. Hopefully I won't regret this.
    Permission.getAll().each { perm ->
      admins.each { user ->
        authz.add(perm, user)
      }
    }
    jenkins.authorizationStrategy = authz
  }

While the authentication configuration (or security realm if you want to call it that) determines how users log in, the authorization strategy decides what they can do once logged in. For general use, we’re using the relatively-standard Matrix authorizer from the plugin of the same name. If you need more complex access controls you might want to look at the similarly more complex role-strategy authorizer. But here we only really have three bits of authorization config, first is the developmentMode setting coming in from Secret volume. If that is set we entirely disable authorization. This in here for times where I need to hack on the authentication config or if I’m offline somewhere and don’t have access to the corporate SAML servers. Otherwise for the Matrix we set up one entry for the generic “logged-in user” group to give them some minimal, read-only permissions, which is enough for normal users to view builds and force them to re-run if they end up with a flaky test (though hopefully they will fix it soon after). For admins, we create one row for each administrator giving them every permission available in Jenkins. Because this is rebuilt from scratch every time the configuration script runs, it means that is we do manage to break the permissions settings via “accidental” clicking in the web configuration GUI, it will at least get automatically restored as soon as we restart the container.

  //////// GITHUB CONFIG
  // Create the credentials used to access GitHub.
  def creds = CredentialsProvider.lookupCredentials(StandardUsernamePasswordCredentials, jenkins)
  def cred = creds.findResult { it.description == "GitHub access token" ? it : null }
  if (cred) {
    println "--> Updating existing GitHub access token credential ${cred.id}"
    def newCred = new UsernamePasswordCredentialsImpl(
      cred.scope,
      cred.id,
      cred.description,
      githubUser,
      githubUserToken)
    SystemCredentialsProvider.instance.store.updateCredentials(Domain.global(), cred, newCred)
  }
  else {
    println '--> Creating GitHub access token credential'
    cred = new UsernamePasswordCredentialsImpl(
      CredentialsScope.GLOBAL,
      java.util.UUID.randomUUID().toString(),
      "GitHub access token",
      githubUser,
      githubUserToken)
    SystemCredentialsProvider.instance.store.addCredentials(Domain.global(), cred)
  }

Next we construct a Jenkins credential with our GitHub access token. This was read in up at the top from the Secret volume, and here we either create a new credential if one isn’t found, or update the existing one. Again, the goal is convergent behavior so every time Jenkins start, it tries to match the persistent state to the desired state.

  //////// GLOBAL LIBRARIES
  def retriever = new SCMSourceRetriever(new GitSCMSource(
    "pipeline",
    "https://github.com/${librariesRepo}.git/",
    cred.id,
    "*",
    "",
    false))
  def pipeline = new LibraryConfiguration("pipeline", retriever)
  pipeline.defaultVersion = "master"
  pipeline.implicit = true
  pipeline.includeInChangesets = false
  jenkins.getDescriptor("org.jenkinsci.plugins.workflow.libs.GlobalLibraries").get().setLibraries([pipeline])

This configures the shared pipeline libraries. We’ll show the library code further down, but roughly this allows having a centralized repo with Groovy snippets that can be used by the per-repo Jenkinsfiles. In practical terms, this actually contains almost all of the pipeline logic, the final Jenkinsfile is currently always exactly one line long, calling one of the global presets. The implicit setting means Jenkinsfiles don’t have to explicitly include the library, and disabling includeInChangesets means that a new version of the library won’t trigger every job to build (though that would certainly be a nice load test).

  //////// CLOUUUUUD (NOT BUTT)
  // Register the Kubernetes magic secret.
  creds = CredentialsProvider.lookupCredentials(ServiceAccountCredential, jenkins)
  if (creds.isEmpty()) {
    println '--> Creating Kubernetes service account credential'
    kubeCred = new ServiceAccountCredential(
      CredentialsScope.GLOBAL,
      java.util.UUID.randomUUID().toString(),
      "Kubernetes service account")
    SystemCredentialsProvider.instance.store.addCredentials(Domain.global(), kubeCred)
  }
  else {
    kubeCred = creds[0]
  }

Starting in on setting up the Kubernetes support in Jenkins. The plugin declares a special credential type that effectively loads on the fly from the pod’s service account. But we still need to actually create that stub secret to feed into the rest of config. I guess if we weren’t using a service account this would be different, but service account are what the cool kids do.

  // Configure the cloud plugin.
  println '--> Configuring Kubernetes cloud plugin'
  def cloud = new KubernetesCloud('kubernetes')
  cloud.serverUrl = 'https://kubernetes.default'
  cloud.namespace = kubeNamespace
  cloud.jenkinsUrl = "http://${labels['app']}:8080"
  cloud.jenkinsTunnel = "${labels['app']}-agent:50000"
  cloud.credentialsId = kubeCred.id
  def podTemplate = new PodTemplate()
  podTemplate.name = 'default'
  podTemplate.label = "${labels['release']}-agent"
  def containerTemplate = new ContainerTemplate('jnlp', "jenkins/jnlp-slave:$agentVersion")
  containerTemplate.workingDir = '/home/jenkins'
  containerTemplate.command = ''
  containerTemplate.args = '${computer.jnlpmac} ${computer.name}' // Single quotes are intentional.
  containerTemplate.envVars.add(new ContainerEnvVar('JENKINS_URL', cloud.jenkinsUrl))
  containerTemplate.resourceRequestCpu = '200m'
  containerTemplate.resourceLimitCpu = '200m'
  containerTemplate.resourceRequestMemory = '256Mi'
  containerTemplate.resourceLimitMemory = '256Mi'
  podTemplate.containers.add(containerTemplate)
  cloud.addTemplate(podTemplate)
  jenkins.clouds.clear()
  jenkins.clouds.add(cloud)

Then the actual cloud plugin configuration. This aims the plugin at the same cluster as Jenkins is running inside of, and uses the pod’s service account credentials as mentioned above. Then we set up a pod and container template for the JNLP worker. This handles the communication with Jenkins once the worker pod launches, but we’ll add additional containers to it in our pipeline libraries to do the actual heavy lifting of the build. Those resource limits are based on the Helm community chart for Jenkins and I’m not yet sure if they reflect reality when Jenkins is under heavy load.

  //////// PROJECT FOLDER
  println '--> Creating organization folder'
  // Create the top-level item if it doesn't exist already.
  def folder = jenkins.items.isEmpty() ? jenkins.createProject(OrganizationFolder, 'MyName') : jenkins.items[0]
  // Set up GitHub source.
  def navigator = new GitHubSCMNavigator(githubOrg)
  navigator.credentialsId = cred.id // Loaded above in the GitHub section.
  navigator.traits = [
    // Too many repos to scan everything. This trims to a svelte 265 repos at the time of writing.
    new jenkins.scm.impl.trait.WildcardSCMSourceFilterTrait('*-cookbook', ''),
    // We have a ton of old branches so try to limit to just master and PRs for now.
    new jenkins.scm.impl.trait.RegexSCMHeadFilterTrait('^(master|PR-.*)'),
    new BranchDiscoveryTrait(1), // Exclude branches that are also filed as PRs.
    new OriginPullRequestDiscoveryTrait(1), // Merging the pull request with the current target branch revision.
  ]
  folder.navigators.replace(navigator)

This part I’m very proud of. The traditional way to automate Jenkins job creation is the venerable Job DSL plugin. Job DSL uses its own Groovy scripting API to create and manage jobs totally separately from the Jenkins Groovy scripting framework, usually using a single “seed job” to create all the others. This special DSL does (easily) support this use case, but I wanted to try and avoid it. Having one fewer plugin to worry about, as well as a more tightly integrated configuration seemed worth a bit of extra exploration. Building the job configuration in pure Jenkins Groovy turned out to be pretty straightforward other than being entirely undocumented, almost all of the default values are actually what you want in this case. This block of code will create the top-level folder and set up the GitHub folder source. As mentioned in the comments, the organization scan was being too slow for my tastes so I set up those two filter traits to cut down on the number of things Jenkins will even bother checking for a Jenkinsfile. As I expand past just cookbook testing those will probably go away, and the org scan only runs once a day so it being slow isn’t actually a runtime problem, I was just being impatient in development.

An aside about the GitHub plugin and webhooks. The plugin can automatically configure the organization webhook for you, however I’m not actually doing that here. That would require giving Jenkins an admin-capable token which I prefer not to do. If you’re adapting this config to create dozens of organization folder, maybe consider putting that back in (add navigator.afterSave(folder) after the save() in the next snippet) but barring that I would just configure it manually. You’ll want to set the URL to https://myjenkinsserver.com/github-webhook/ and enable the push, pull request, and repository events.

  println '--> Saving Jenkins config'
  jenkins.save()

We did it! I don’t think a manual save is actually required but it makes life slightly easier if we have to kubectl exec to log in and stare at the config.xml manually.

  println '--> Scheduling GitHub organization scan'
  Thread.start {
    sleep 30000 // 30 seconds
    println '--> Running GitHub organization scan'
    folder.scheduleBuild()
  }

Because we really wanted this to be fully hands-off, we schedule a org scan on startup. This has to wait for a few other startup tasks inside Jenkins, so it runs 30 seconds after this script.

  println "--> configuring alti_jenkins... done"
}
catch(Throwable exc) {
  println '!!! Error configuring alti_jenkins'
  org.codehaus.groovy.runtime.StackTraceUtils.sanitize(new Exception(exc)).printStackTrace()
  println '!!! Shutting down Jenkins to prevent possible mis-configuration from going live'
  jenkins.cleanUp()
  System.exit(1)
}

And then finally the catch for those runtime errors we talked about up at the start. This will print the error to the log (ends up kubectl logs or whatever else you are using) and then tells Jenkins to shut down.

And there you have it. A fully convergent configuration script to set up a working Jenkins based on GitHub and Pipelines and all that jazz. Now we just need to run this sucka’.

Helm Chart

We started out first trying to use the community Helm chart for Jenkins directly, and then making a wrapper chart, but both approaches turned out more complex than we wanted. At this time, my recommendation for charts as complex as Jenkins is to fork them and use them as a starting point for your own chart. As an example, the community chart uses the Jenkins container image directly, and installs the config and plugins in an initContainer, rather than building our image as shown above. We felt this approach left too many moving pieces at deploy time and potentially compromised our ability to do rollbacks if an upgrade didn’t go according to plan. I’m not going to show the entire chart as there is a lot that is going to be specific to my specific requirements but I do want to talk about the core of it.

Deployment

As with most simple applications, the heart of the deployment is a Kubernetes Deployment object, which manages the actual pods.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "alti-jenkins.fullname" . }}
  labels:
    app: {{ template "alti-jenkins.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
    component: "{{ .Release.Name }}-jenkins"

We start with a pretty standard set of metadata for a Helm-created object. Some of these values are used for selectors in the service/ingress side of things, but mostly these are to aid in human debugging and management.

spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      component: "{{ .Release.Name }}-jenkins"

Simple defaults, we only want one pod at a time because Jenkins’ idea of HA is “restart it if it crashes” so leave things at that.

  template:
    metadata:
      labels:
        app: {{ template "alti-jenkins.fullname" . }}
        heritage: "{{ .Release.Service }}"
        release: "{{ .Release.Name }}"
        chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
        component: "{{ .Release.Name }}-jenkins"
      annotations:
        checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

Next the same metadata but for the Pod object this time. The annotation is somewhat standard Helm pattern where we put a checksum of the rendered Secret object in the Deployment so that when the Secret changes, the Deployment will re-roll the pods automatically. This is needed because our config code only looks at the secret data at container startup, so if they change after that it will be ignored. The actual checksum isn’t used for anything, but it changing will trigger Tiller (the server component of Helm) to update the Deployment, which triggers the Pods to re-roll.

    spec:
      serviceAccountName: {{ template "alti-jenkins.fullname" . }}
      imagePullSecrets:
        - name: {{ template "alti-jenkins.fullname" . }}-pull
      securityContext:
        # This is the default gid for the jenkins group in the upstream container.
        fsGroup: 1000

Some general Pod configuration, setting the service account for Jenkins, the image pull secret to talk to our internal registry to download the alti_jenkins image we made before, and setting the GID used for volume mounts down below so that we can lock down the file modes a little bit just in case someone manages to get a shell on the Jenkins container somehow.

      containers:
        - name: {{ template "alti-jenkins.fullname" . }}
          image: "{{ .Values.Server.Image }}:{{ .Values.Server.ImageTag }}"
          {{- if .Values.Server.ImagePullPolicy }}
          imagePullPolicy: "{{ .Values.Server.ImagePullPolicy }}"
          {{- end }}

Basic container set up, nothing too interesting here. The only reason to change the image pull policy is to set it to Always in local development if I’m rebuilding the same image version multiple times before I release it. But I usually do my development in minikube anyway, so I use minikube docker-env to build directly in the Docker daemon used by Kubernetes later on: ( eval "$(minikube docker-env)" && docker build -t myrepo.com/alti_jenkins:2.whatever ).

          ports:
            - containerPort: 8080
              name: http
            - containerPort: 50000
              name: agentlistener

Expose two ports, one for HTTP and the other for JNLP workers. Unlike the community chart, I didn’t see much reason to make these configurable since there is no worry of port collisions or whatever.

          resources:
            requests:
              cpu: "{{ .Values.Server.Cpu }}"
              memory: "{{ .Values.Server.Memory }}"

We don’t yet really have a good idea for what these limits should be from production data under heavy load, so just make them configurable for now.

          volumeMounts:
            - mountPath: /var/jenkins_home
              name: jenkins-home
            - mountPath: /var/jenkins_secrets
              name: jenkins-secrets
              readOnly: true
            - name: downward
              mountPath: /etc/downward
              readOnly: true
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: {{ .Values.Persistence.ExistingClaim | default (include "alti-jenkins.fullname" .) }}
      - name: jenkins-secrets
        secret:
          secretName: {{ template "alti-jenkins.fullname" . }}
          defaultMode: 0440
      - name: downward
        downwardAPI:
          items:
            - path: labels
              fieldRef:
                fieldPath: metadata.labels
            - path: namespace
              fieldRef:
                fieldPath: metadata.namespace

And then the volumes. We need three volumes for three different purposes. The big one is the JENKINS_HOME mount. Despite my best efforts towards immutable configuration, Jenkins still does store a lot of state in the JENKINS_HOME directory, like job history and build artifacts. As such, this needs to be persistent at least over short timescales. If we lost this persistent volume we could still trivially rebuild Jenkins, but we would lose enough history that it might be frustrating. So for now, PVC.

Then the two configuration volumes, a Secret volume and the Downward API volume. As we saw back at the top of the config.groovy, these are used to feed configuration data into the Jenkins config. The defaultMode settings works with fsGroup up above to slightly restrict things, though probably not in a way that really matters but yay for defense in depth.

Secret

Mostly pretty rote, but including it here as an example:

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "alti-jenkins.fullname" . }}
  labels:
    app: {{ template "alti-jenkins.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  artifactory-token: {{ required "Secrets.ArtifactoryToken is required" .Values.Secrets.ArtifactoryToken | b64enc | quote }}
  github-user: {{ required "Secrets.GithubUser is required" .Values.Secrets.GithubUser | b64enc | quote }}
  github-token: {{ required "Secrets.GithubToken is required" .Values.Secrets.GithubToken | b64enc | quote }}
  saml-keystore: {{ required "Secrets.SamlKeystore is required" .Values.Secrets.SamlKeystore | nospace | quote }}
  saml-pass: {{ required "Secrets.SamlPass is required" .Values.Secrets.SamlPass | b64enc | quote }}
  # Not technically secret but convenient to put here because the same kind of code needs them.
  development-mode: {{ printf "%t" .Values.Server.DevelopmentMode | b64enc | quote }}

One thing to note is that the SamlKeystore value is coming in already base64-encoded because it’s a binary file format and it’s vastly easier to store it in the Helm values file (and Tiller storage of the same) as text data. Given that we expect this value to change infrequently (cert rotation, or in case of a security issue), we just put it in base64 up front by hand.

Services

We end up with two services, one each for HTTP and worker traffic.

apiVersion: v1
kind: Service
metadata:
  name: {{ template "alti-jenkins.fullname" . }}
  labels:
    app: {{ template "alti-jenkins.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
    component: "{{.Release.Name}}-jenkins"
spec:
  ports:
    - port: 8080
      targetPort: 8080
      name: http
  selector:
    component: "{{.Release.Name}}-jenkins"

and then:

apiVersion: v1
kind: Service
metadata:
  name: {{ template "alti-jenkins.fullname" . }}-agent
  labels:
    app: {{ template "alti-jenkins.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
    component: "{{ .Release.Name }}-jenkins"
spec:
  ports:
    - port: 50000
      targetPort: 50000
      name: agentlistener
  selector:
    component: "{{ .Release.Name }}-jenkins"
  type: ClusterIP

Ingress

And finally an Ingress to handle TLS in production. We went with an Ingress because we wanted to use kube-lego to automate certificates via LetsEncrypt. If you’re on AWS and want to use ACM instead, you can do that directly via the first Service object above.

{{- if .Values.Server.PublicHostname }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ template "alti-jenkins.fullname" . }}
  labels:
    app: {{ template "alti-jenkins.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
  annotations:
    kubernetes.io/tls-acme: "true"
spec:
  tls:
    - secretName: {{ template "alti-jenkins.fullname" . }}-tls
      hosts:
        - {{ .Values.Server.PublicHostname | quote }}
  rules:
    - host: {{ .Values.Server.PublicHostname | quote }}
      http:
        paths:
          - path: /
            backend:
              serviceName: {{ template "alti-jenkins.fullname" . }}
              servicePort: 8080
{{- end }}

This will only be active on production, for local development I just talk directly to the service via minikube’s minikube service helper command. I’ve only tested with the Nginx ingress controller, but I would imagine it should work with the GCE ingress too, and any other controller supported by kube-lego. In production we installed both nginx-ingress and kube-lego using their community Helm charts directly:

$ helm install -n nginx-ingress stable/nginx-ingress
$ helm install --set config.LEGO_EMAIL=me@example.com \
  --set config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory \
  -n kube-lego stable/kube-lego

Global Pipeline Libraries

Okay, so we have a working, running Jenkins server. Progress! Next step is to get some actual builds on it. Because we have hundreds of cookbooks which should all use the same build logic, we wanted to make sure all of that was kept somewhere centralized. Jenkins’ global libraries system made this very easy, though as I’ve only attacked the cookbook testing use case I don’t actually have very much yet.

An aside about how to structure this: each helper goes in a file named vars/nameOfHelper.groovy. The bit after vars/ is what ends up being the function name for your Jenkinsfiles.

// vars/altiNode.groovy
def call(Closure body) {
  def secretsRoot = System.getenv('JENKINS_SECRETS') ?: '/var/jenkins_secrets'
  def artifactoryToken = new File("$secretsRoot/artifactory-token").text.trim()

  withEnv(['CI=true', "BERKSHELF_PATH=${env.WORKSPACE}/.berkshelf", "ARTIFACTORY_API_KEY=$artifactoryToken"]) {
    node('cookbook') {
      container('alti-pipeline') {
        body()
      }
    }
  }
}

First a utility helper to help avoid too many levels of indentation in other helpers. This is like the build in node {} pipeline step but with some standard stuffs for our testing environment.

// vars/altiCookbook.groovy
def call(Closure body) {
    def altiPipelineVersion = '4.9.2'

    def downwardRoot = System.getenv('DOWNWARD_VOLUME') ?: '/etc/downward'

    // Parse the labels test.
    def labels = [:]
    new File("$downwardRoot/labels").eachLine {
      def parts = it.split('=')
      labels[parts[0]] = parts[1][1..-2]
    }

Next is the interesting one, the cookbook testing pipeline, though currently a very simple one. First we do some configuration stuff like we saw in config.groovy. It is, in fact, a copy-pasta because I couldn’t find a reasonable way to share code between the two contexts and it’s not very long anyway.

    podTemplate(label: 'cookbook', imagePullSecrets: ["${labels['app']}-pull"], containers: [
      containerTemplate(name: 'alti-pipeline', image: "altiscale-docker-dev.jfrog.io/alti_pipeline:${altiPipelineVersion}", alwaysPullImage: false, command: "/bin/sh -c \"trap 'exit 0' TERM; sleep 2147483647 & wait\""),
    ]) {

Then we set up the pod for our job to build in. This will be combined with the podspec we gave the Kubernetes cloud plugin up in the Jenkins configuration so the final pod will end up with two containers, one for the JNLP worker and another with the build environment image. The build environment doesn’t actually have a service to run, so use the sleep wait to keep it busy until the pod is shut down.

        def integrationTests = []
        stage('Check') {
            altiNode {
                checkout scm
                // Check that we have an acceptable version of alti_pipeline, just looks at the major version.
                def gemfile = readFile('Gemfile')
                if(gemfile =~ /gem.*alti_pipeline.*\b${altiPipelineVersion[0]}\./) {
                    echo "Gemfile is compatible with alti_pipeline ${altiPipelineVersion}"
                } else {
                    error "Gemfile is not compatible with alti_pipeline ${altiPipelineVersion}:\n"+gemfile
                }
                // Parse out the integration tests for use in the next stage.
                integrationTests = sh(script: 'kitchen list --bare', returnStdout: true).split()
            }
        }

The first stage is a sanity check. Unlike many Chef shops, we don’t actually use ChefDK (that would have to be a whole ‘nother blog post so just take it as a given) and instead have a Gemfile in each cookbook that points it at our equivalent gem, alti_pipeline. Here we want to make sure that the cookbook’s Gemfile is the same major version as the build image since if it isn’t, the build is very unlikely to work. We also grab the list of all Test Kitchen instances to build in the next stage.

        stage('Test') {
            testJobs = [
                'Lint': {
                    altiNode {
                        checkout scm
                        sh 'rm -f Gemfile Gemfile.lock'
                        sh 'rake style'
                    }
                },
                'Unit Tests': {
                    altiNode {
                        checkout scm
                        try {
                            sh 'rm -f Gemfile Gemfile.lock'
                            sh 'rake spec'
                        } finally {
                            junit 'results.xml'
                        }
                    }
                },
            ]
            integrationTests.each { instance ->
              testJobs["Integration $instance"] = {
                    altiNode {
                        checkout scm
                        sh 'rm -f Gemfile Gemfile.lock'
                        sh "kitchen test --destroy always $instance"
                    }
                }
            }
            parallel(testJobs)
        }

        body()

    }
}

And then finally the actual test bit of the pipeline. This sets up jobs for lint checking, unit tests, and one job each for the integration tests so they can all run in parallel. You can see this uses the altiNode helper from above, instead of the usual node pipeline step. We’re also removing the Gemfile in-place since we’ve already installed all the needed gems in my build environment image and don’t want bundler to even try and activate.

Cookbook Integration Testing

As part of this project I also built a new Test Kitchen driver, kitchen-kubernetes, specifically for running Chef cookbook integration tests on top of Kubernetes. This works similarly to kitchen-docker and kitchen-dokken, but using Kubernetes machinery rather than plain Docker containers. If duplicating this set up for yourself, make sure you remember to include rsync in the job build image (alti_pipeline above) as that is required for kitchen-kubernetes’s file upload system.

Build Environment Image

While most people doing Chef cookbook testing should probably use the chef/chef-dk image, as mentioned before we are not using ChefDK for our environment management. The short version of “why” is that we’re still on Chef 12 but wanted newer versions of a lot of tools, as well as including a lot of our own utility gems. We may yet transition back to ChefDK but for now we needed to create a container image that included a bunch of private gems. Pulling in private gems means including an access token for the repository (careful readers have probably figured out by now that we use Artifactory), but unfortunately build-time secrets are still a notable problem with docker build. There are a few options, short-lived tokens that do get baked in to the image but are already expired by the time anyone could get them, localhost proxies that handle authentication, use of alternative image build systems like Habitat/buildah, but we decided to try and keep it simple and use the new “squash build” feature in Docker.

We decided to use alpine as the base image (shoutout to the great folks at Glider Labs) to minimize the file size. Kubernetes does cache images aggressively, but every little bit helps in improving build performance. The final Dockerfile looks like this:

FROM alpine:latest
ENV VERSION=4.9.2
ENV ALTISCALE_KITCHEN_KUBERNETES=true
ENV ALTISCALE_BERKS_ARTIFACTORY=true
COPY .gemrc /root
RUN set -x && \
    apk --update-cache add build-base ruby-io-console ruby ruby-dev libffi libffi-dev zlib zlib-dev curl git openssh-client rsync && \
    gem install alti_pipeline -v $VERSION && \
    git clone https://github.com/coderanger/kitchen-kubernetes /tmp/kitchen-kubernetes && \
    ( cd /tmp/kitchen-kubernetes && gem build *.gemspec && gem install --local *.gem ) && \
    curl -L -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && \
    chmod +x /usr/local/bin/kubectl && \
    gem sources --clear-all && \
    rm -rf /root/.gemrc /usr/lib/ruby/gems/2.4.0/cache/*.gem /tmp/kitchen-kubernetes && \
    for f in /usr/lib/ruby/gems/2.4.0/gems/*; do rm -rf $f/spec $f/test $f/examples $f/distro $f/acceptance; done && \
    apk del build-base ruby-dev libffi-dev zlib-dev curl

This can be broken down in to four parts. First the base image and some environment variables we want set for all builds. Then copying the gem server credentials, from this point on things become radioactive because we have a secret value in the image. Then the installs, first a bunch of Alpine packages we need, then the top-level alti_pipeline gem, kitchen-kubernetes (from git because I haven’t actually put up a release yet), and kubectl (for use by kitchen-kubernetes). Finally a whole bunch of cleanup. This is mostly to reduce the final image size, removing files and packages we don’t need after image creation. But also we remove the .gemrc, making the image no longer radioactive if built correctly.

Even with this COPY+rm though, we need to make sure to build the image using docker build --squash (which requires experimental features be enabled on the Docker daemon, add --experimental=true to the daemon command line). If built without --squash, the final image would look like it doesn’t have the token, but it would still be visible in the intermediary layer created between the COPY and RUN. Hopefully at some point there will be a better solution for build-time secrets, but for now this is enough to get us a build environment weighing in at around 100MB.

Per-Repo Jenkinsfile

One requirement of this set up is you do need to put a Jenkinsfile in each repository you want to be built. This might be frustrating for some, having to touch every repo when that could potentially be (and in my case, is) hundreds of projects. That said, currently the Jenkinsfile we are adding to each repo is literally altiCookbook { }. So it’s not much in terms of footprint, but you do have to do the legwork, either by hand or via a script using the GitHub API.

To Conclude

As I mentioned at the start, my goal here is to provide a jump start in designing your own Jenkins deployment. I suspect the precise combo of design choices shown here might be literally unique in the world, but most of the bits are very modular and the overall structure should be a starting point for your own specifics.

If you have any questions on any of this code or the design decisions behind it you can reach me at noah@coderanger.net.

React and Patents

2017-09-25T04:00:00Z

Over the past several weeks there has been an evolving discussion about the licensing of React, a popular library for building complex user interfaces. There has been a lot of conflicting information posted about the state of React, what kind of licensing terms are involved, and why all of this matters. I would like to try and set the record straight to the best of my ability. But first:

A Disclaimer

I am not a lawyer and nothing in this post should be considered legal advice. If you have any specific questions about any of this, please talk to a licensed intellectual property attorney in your local jurisdiction. This is for entertainment purposes only. This is just my opinions/knowledge, not those of my employer, Facebook, the Apache Software Foundation, or anyone else.

Okay, with the disclaimer out of the way, let’s rewind, where did all this start?

The Story Up Until Now

While we could start way back with the initial creation of React, that was a long time ago and the part of the story we all care about begins in April 2017. In a somewhat unassuming Jira ticket, one of the Cassandra team asked the Apache Software Foundation legal affairs team if directly integrating with RocksDB was acceptable. I hear you say “RocksDB? I thought this was about React!” well that was brought up soon after in the same ticket, as both RocksDB and React were made available under the same license. This was resolved a few months later in July when the ASF legal team made the call that the license terms used for RocksDB (and therefore React) were incompatible with the requirements on ASF projects. This was quickly picked up by the media and run with headlines like “Apache says ‘no’ to Facebook code libraries”.

We’ll come back to RocksDB later, but with React this kicked off a firestorm of concerned and confused users asking what was going on. There was some discussion for a few weeks as to if React should be changed to a different license, but in mid-August this was closed with Facebook confirming that they felt the current license was appropriate.

Then, unexpectedly, a few days ago (September 22nd) Facebook announced they would be relicensing React and several related projects to use the MIT license.

Okay, so that’s the whole history, let’s look at some major questions that came out of these few months:

Who Owns React (and RocksDB)?

To get it out of the way, code is in general owned by whomever writes it. If you write code as a part of your job, it is owned by your employer, give or take what is in your employment contract. Facebook does require a contributor license agreement (CLA) for their projects, and the CLA grants Facebook some strongly worded rights for copyright use and patent access, but in the end each contributor remains the owner of the copyright in their contribution. This means that while React is managed by Facebook, and they do own a lot of the copyright over their projects, in the end they are actually owned by the collective community of developers that works on them. In the case of RocksDB specifically there is also a large chunk of the code which is owned by Google, as it is an outgrowth of an earlier Google project called LevelDB.

Ownership is generally thought of in terms of copyright ownership, as that’s the more directly relevant bit, but another factor in the control of software is who owns any patents which cover the software. We’ll leave this part for later in this post as it’s a much more complex topic.

What Was The Original License For React (and RocksDB)?

The license used originally by Facebook has two main parts, a copyright license and a patent license. The copyright license half is pretty straightforward, a 3-Clause BSD license. This is part of a family of minimalist copyright licenses that allow use, copying, and modification of the code. Without this license, those rights would remain exclusively with the copyright holders (i.e. the person or company which wrote each section of the code). The BSD license (usually in 2 or 3-clause variants, sometimes 4) is fairly well known and the requirements you must meet to use those licensed rights (preserve the copyright notice and license, don’t use Facebook’s name in endorsements, don’t sue anyone because the code doesn’t work) are quite reasonable.

The second half is a patent license, which says that anyone using the software is allowed to use any Facebook patents that might cover that software so Facebook cannot sue them for patent infringement. The goal here is the same as with the copyright license, making sure users have enough rights to be successful open-source citizens while protecting Facebook and the other rightsholders involved.

Facebook has termed this a “BSD+Patents” license, though this should not be confused with the OSI-approved license also called “BSD+Patent” (or sometimes BSD-2-Clause-Patent) as the two are unrelated.

What Problem Did The ASF Have With BSD+Patents?

As mentioned in the previous section, the use of the BSD license for copyrights was 100% okay with everyone, the issues all revolved around the patent license. This was something created by Facebook specifically, so to start with it was not a well-known quantity in the same way as things like the BSD or MIT license would be. While using standardized licenses (or more specifically, licenses recognized by the Open Source Initiative) does certainly make life easier for open-source lawyers, it isn’t a specific requirement.

The Apache Software Foundation does its best to make the projects under its umbrella be somewhat uniform from a legal perspective. If you get a legal “okay” to use one of them, you can almost certainly use any other ASF project. This is important for ecosystem stability as many ASF projects depend on other ASF projects to function. We’ll get to the Apache-2.0 license in more detail later, but the ASF created it to help in this mission, to provide a unified set of licensing terms for all ASF projects. But the open-source ecosystem is a big place, and many ASF projects want to depend on libraries not from the ASF and thus sometimes under different license terms. So the doctrine used by the ASF legal team is that other licenses are okay as long as they do not put any more requirements on the end user than the Apache-2.0 license does. So for example, pretty much all software in the world depends on zlib, which is distributed under the zlib license, but from the point of view of the ASF crew, that is less restrictive than Apache-2.0 so it doesn’t make life any harder for end users.

The problematic section of the patent license was under what conditions the license would be revoked. While most copyright licenses are given as irrevocable, it is normal for patent licenses to include a termination clause in the case of a lawsuit over the software. This helps to protect the company, if they get sued they can at least prevent you from continuing to benefit from their IP for the duration of the lawsuit. But that said, the Facebook patent license went a lot further: “The license granted hereunder will terminate, automatically and without notice, if you … initiate … any Patent Assertion against Facebook or any of its subsidiaries or corporate affiliates”. So rather than being scoped to just lawsuits over the specific project, any patent suit brought against Facebook would terminate the patent license. The ASF Legal team decided this represented a restriction above and beyond the Apache-2.0 license, and so it could not be used with ASF projects.

It should be repeated that this was a decision by the ASF Legal team for ASF-controlled projects, that’s it. This was not an abstract judgment that Apache-2.0 code can’t be used with BSD+Patents in general, nor was it any comment on the quality of Facebook’s code or on their patent license as being good or bad. It was specifically to ensure the licensing goals of the ASF remained safe, nothing more.

So What Happened To RocksDB?

In response to the original decision from ASF Legal, the RocksDB team quickly switched from BSD+Patents to Apache-2.0. There isn’t a lot of public information on how this change went down or why, but given it happened within a few hours I’m guessing it was considered fairly uncontroversial by everyone involved. As mentioned earlier, there is a lot of code from LevelDB still in use, which is owned by Google and will continue to be made available under the terms of the 3-Clause BSD license.

One slight addendum, since this will come up again with React: I’m not actually 100% what the legal basis for the change of license was. Normally you need approval from all copyright holders before changing a license, but it is possible the Facebook legal team decided the sublicensing requirement of the Facebook CLA made it okay to relicense in this fashion without getting approval. Or it’s possible they just figured everyone would be okay with it, which is probably correct.

Why Did Facebook Want The Broad Termination Clause?

Without getting too far in to the details, one major problem facing the software industry right now is a plethora of patent lawsuits from companies that buy out very broad or wide-reaching patents and then use them to try and extract settlements from a large number of software companies by threating litigation. These are colloquially called “patent trolls”. Regardless of your personal feelings on the issue, Facebook considers this a threat to their business and was using the broad termination clause across all of their open-source software to try and take some wind out of the sails of future trolls as if the troll was using anything under the Facebook patent license, if they filed a lawsuit over one of the troll patents, Facebook could counter-sue using one of theirs.

This defensive measure was deemed important enough to reject the initial request for relicensing React. And speaking personally, I totally understand that impulse. I agree that troll lawsuits are a chilling scourge on our industry and in general I support legal defense machinery to try and reduce them.

Why Did Other People Not Want The Broad Termination Clause?

While you might initially think “but I would never sue Facebook”, things are often more complex than that. For other large companies, this one-sided termination clause effectively means the company as a whole is giving up the right to pursue legitimate patent infringement cases against Facebook. Given the scope and scale of many patent portfolios, this represents an unacceptable risk to many (already risk-averse) legal departments.

For smaller companies, especially VC-backed startups, suing Facebook would likely have never been an option in the first place even if Facebook was very clearly infringing as the legal costs can be immense. However the other shoe dropping is that many startups want to end up getting acquired by larger companies, so all the same issues get pulled back in. While I don’t know of any examples citing React specifically, acquisition deals fall apart all the time over intellectual property concerns so this is something even startups should take seriously.

That said, for some cases it really is okay. I personally used React for some projects, like the dashboard that powers my home information system, when it was under the BSD+Patents license.

So If React Isn’t BSD+Patents Anymore, What Happened?

While the public discussion about relicensing React ended after the first 4 weeks, apparently it continued within Facebook, for which they should be recognized and commended. This resulted in a somewhat surprising announcement that React will switch from BSD+Patents to the MIT license. At the time of writing, I don’t think this change has been made, but they are planning to implement it soon. As with the RocksDB license change, I’m unsure of the exact legal mechanism being used, especially since this is moving to a less restrictive license (i.e. the copyright holders are giving up more rights). But regardless, we can expect this to be reality soon and the rest of this post assumes Facebook will follow through on the license change.

Awesome, This Means Everything Is Fine Now, Right?

Maybe. Here is where things start getting dicey. The MIT license is generally thought of as a copyright license, just like the BSD licenses. With the switch to it, this would mean Facebook is intending to terminate (or at least no longer rely on) their patent license. But that doesn’t make Facebook’s patents go away.

A quick check on Google Patents shows Facebook currently controls around 5200 patents (not even counting their subsidiaries). Without going through every single one of them, there is no good way to know which, if any, would apply to React or related projects. The posture Facebook initially took around their patent license certainly intimated that they think one or more of their patents does apply, but they don’t have to actually show their hand until/unless they actually file a lawsuit against someone the first time.

So if we are now ignoring the Facebook patent license, there are three scenarios. One is that the whole thing was a bluff (perhaps unintentionally) and there are no Facebook patents which cover React, or equivalently Facebook thinks any patents they do have would be indefensible in a post-Alice world. Second, React will now be covered by a copyright license, but all patents covering React (and here we assume there are some) will not be licensed and Facebook can sue anyone using React at any time. Third, Facebook is relying on a somewhat-controversial reading of the MIT license that says it does provide a patent license.

The first case seems unlikely. The second is downright evil and I have more trust in both the Facebook open-source folks and the React team than that. This leaves us with option three, further bolstered by a tweet from one of the React team. This reading of the MIT license revolves around the phrase “including without limitation the rights to use”. Some people claim given the broad wording of this rights grant, it would include both copyrights and patent rights. But this is the aforementioned dicey bit. To the best of my knowledge, an implicit patent grant this vague and outside of a commercial context has never been tested in court. If this is the goal of the Facebook team, it would also mean they are effectively permanently renouncing those patents, as the implied grant would have no termination rules, even for a lawsuit directly about React. I would very much like to believe this case is our reality, but without a more well-known legal footing, I’m very concerned we may have inadvertently created a scenario two where everyone is actually just infringing all the time.

The announcement of the relicensing from Facebook was very notably absent any discussion of patent rights, and all postings I’ve seen from Facebook folks about this have been very carefully worded to avoided specifically stating what Facebook thinks the patent rights situation will be.

But You Said RocksDB Relicensed And Everything Was Fine?

RocksDB switched to the Apache-2.0 license. Like many newer licenses, it includes specific provisions covering both copyright rights and patent rights (and more). This means that patent rights around RocksDB might actually be more restrictive than React, but they are far more explicit and well understood.

Specifically the Apache-2.0 license uses a fairly standard termination clause, where the lawsuit must claim “the Work or a Contribution incorporated within the Work” is what is infringing. As mentioned earlier, this simplifies lawsuits over the project itself while not totally disarming patent litigation about unrelated things. Beyond just the patent licensing, the Apache-2.0 license also makes a much more specific declaration of which copyright-related rights are being shared, covers trademark licensing, and makes the rules for redistribution clearer.

The fact that Facebook chose the MIT license for React instead of Apache-2.0 sends a signal that they may be trying to hedge their bets here in a way the RocksDB team did not.

Okay, Can I Just Use Preact Instead?

Again, maybe. While copyright law has a concept of “derived works” vs. “non-derived works”, patent law (in this case) does not. Preact is (I think) built as a whitebox reimplementation of many React APIs in a way that Facebook doesn’t hold any copyright over it. But patents don’t care about that, if Preact implements an algorithm or process patented by Facebook it doesn’t matter if it was created independently or not. But since we don’t know which of their thousands of patents might apply, there is no good way to say if Preact would be infringing or not.

Tl;Dr What Do I Do?

At this time, I still think it would be wise for companies to avoid React. If at some point in the future Facebook provides more specific guidance about patent rights this could be easily remedied, but only time will tell on that. I do firmly believe that Facebook and the React team are not being intentionally bad actors in all this, and the open-source community should give them the benefit of the doubt when it comes to incomplete explanations, but at the same time we each need to ensure the legal safety of our own companies and projects.

If you found this discussion of intellectual property issues interesting, I’ve got a whole talk about the basics of IP which can you check out the slides for or come see the talk at PyGotham or DevOpsDays Hartford.

As I said at the start, I am not an attorney so if you have specific questions about React or IP law in general I’m happy to do my best to help but please also talk to a lawyer or your company’s legal department if you have one.

Gathering Chef Usage Data

2017-07-26T04:00:00Z

A proposed Chef RFC currently sitting in the review queue revolves around gathering anonymized usage information about Chef and some of the Chef ecosystem tools. I think this proposal needs some more eyeballs on it from the Chef community so hopefully this will encourage some of you to weigh in by commenting on the pull request. Usual disclaimer on time-sensitive posts, if you found this via Google some time in the future, it might not apply anymore.

Why gather user data?

As both a Chef maintainer and a cookbook developer, I have an interest in this user data. A big problem for Chef, as with almost all open-source projects, has been that we know almost nothing about how the things we write actually get used. We have some signaling from people asking questions on Slack or filing bug reports, but this is both very coarse-grained and ignores a huge majority of users. This makes it very hard to know how to allocate time and resources, both of which are in short supply (again, same as in any other open-source project, none of this is unique to Chef).

Having some kind of automated data collection would give us a huge boost in knowing what things are popular and should be given more attention, and which were maybe failed experiments or are broken in some way we haven’t been made aware of. This has happened many times over the years, we maintainers assuming something was no longer used, and finding out only after the release that we were wrong (obligatory xkcd).

As a concrete example, Test Kitchen supports a lot of driver plugins. There are two Docker-based drivers, my kitchen-docker and Sean’s kitchen-dokken. We’ve been trying to steer people towards Dokken as a better user experience for simple use cases, but as the maintainer of kitchen-docker I have no way to judge how common the complex or advanced cases are, or even the rough distribution of users for docker vs dokken. This makes it very hard to know what kinds of things to support in each and (probably) leads to worse software for everyone.

Why not to gather user data?

The main reason to not gather user data is the risk of said data ending up used for nefarious purposes. A simple case might be someone finding a password that was incorrectly pasted in the wrong section of a config file, but more subtle issues can arise like searching for users with old versions of ChefDK which have a known vulnerability in their OpenSSL. To counter this we have two main lines of defense: first is that all data is stored anonymously with a session identifier reset every ten minutes. Second is that we plan to make sure all data is public from the start, so we don’t have to wait for an inevitable leak to realize there is a problem with the data.

Anonymous data is never perfect though, and the rise in public data sets has lead to new advances in de-anonymization. We will do our best to keep on top things, but there will always be some passive risk. Therefore any data collection must be optional with both per-user and per-project configuration, but more on that below. Also it is worth restating here that only workstation commands are being considered for this. chef-client, chef-solo, and the whole of Chef Server do not and will not collect data of any kind.

As a secondary thing, there is almost always a PR storm any time an open-source project brings up this topic. I am probably contributing to this, though hopefully with enough positive to outweigh any trolls I bring in to the discussion. Still, this could end up being a harm to users if we have to fight FUD for a while, so that should be considered. All planning and development of this feature is being done in the open, which also hopefully helps offset the “sky is falling” crowd.

Opt In or Out

As mentioned above, all data collection will be 100% optional, but that brings up the question of how to control it. We will need to pick a default yes/no, as well as how to tell the user about this whole thing.

Being silently disabled by default is effectively a non-option since we would get so little data as to not be worth the effort to build the whole system, and being silently enabled by default would (I think rightly) be viewed by many users as a breach of trust and overstepping our bounds as a project.

I think the best approach is a middle ground, off by default, but the first time you run any instrumented command it asks if you want to allow telemetry with a default of “yes” if you hammer on the enter key. This will mean we don’t get data from TTY-less systems like CI servers, but is otherwise a balance between user expectations and data quality. The opt-out will always be available afterwards via a command like chef telemetry disable or something, or you can manually touch the .chef/no_telemetry file (which can be checked in to source control to permanently disable things for a project).

We would love to hear from you if you have thoughts on this though.

Prior Art

The proposed RFC covers this a bit but I wanted to dive in a bit more on some ways other projects have handled this, mostly just to point out that this can be done without the world ending.

Both Chrome and Firefox (and probably Safari but who knows with them) gather anonymized usages statistics as well as crash reports, and I don’t think I’ve ever seen complaints about either. Firefox crash reports in particular, I know several people that used to be on the team that managed that system and heard from them frequently about how useful it was to the Firefox development team.

Debian’s PopCon, Ubuntu’s Apport, and Fedora’s Retrace projects all gather some level of usage information about packages (the latter two only on crashes) and have been mostly uncontroversial in my experience (other than complaints about apport from Python devs because it does weird stuff to the load path but that’s neither here nor there). PopCon is mostly just for fun, but the error collection has been cited as useful by a bunch of Ubuntu/Fedora projects.

Homebrew added similar tracking via Google Analytics last year, and definitely had some growing pains with it initially. After the initial flurry of fixes to what was being collected, it seems to be going okay, though the Homebrew team has been somewhat gruff in dealing with complaints which might indicate that they get a lot of flak behind the scenes.

Overall it seems like a lot of bigger projects add this kind of tracking to little fanfare, especially desktop applications (like ChefDK).

Next Steps

Hopefully by now I’ve encouraged you to go drop us a line on the pull request. We really do want to make sure we only move forward with this if it is in the best interests of users and to do that we need to know what your interests are. If anyone wants to provide feedback anonymously or has questions not answered here, you can always reach me at noah@coderanger.net.

The Agents Are Coming

2017-02-13T05:00:00Z

No, not Hugo Weaving

Over the past five years, agent programs have been rising in both importance and ubiquity. Apple mostly kicked things off for the modern generation of agents with the release of Siri in 2011. Siri was a voice-based personal assistant, able to send text messages, update calendars, and tell you the weather. From this relatively-primitive nucleus, a whole industry has sprung up. While there are several major archetypes of agents we’ll talk about below, they all share a few major characteristics. The first is a generally conversational nature, you talk to them either via voice or text but in a way you would talk to a human. This is usually far from a perfect simulacrum, but Siri interactions are a far cry from text adventure prompts or command line interfaces. The second unifying feature is some concept of context. Again taking Siri as an example, it knows where you are when you ask for the weather and so “does the right thing”. The third trait is a bit more fuzzy, but most of the current agent developers have tried to give their creations some kind of personality. All together, these ideas bring together a broad swath of AI research and development and have fueled an explosion in agent systems of all types.

Personal Voice Assistants

While Siri is probably the most high profile, all of the tech titans have thrown their hat in the ring of “personal voice assistants” in one way or another. On the heels of Apple’s Siri has been Amazon’s Alexa (or Echo depending on your wake-word). Google also has the somewhat less popular (and less-humanly named) Google Assistant and Microsoft has Cortana, but Siri and Alexa have mostly defined this battlefield with all others playing catch-up. A split has emerged from those two, with Siri “living” in phones (and later, laptops) while Alexa is based in mostly-unmoving physical objects in a home (Echo, Dot, etc). Google has moved to offer both options for their Assistant, but this design schism has certainly polarized much of the marketing of their respective camps, if not the design of the software. While some third-party and open-source teams have tried to offer additional options here (the Jasper project probably being the most complete though I’m personally still hoping Jibo manages to succeed), for the most part the big four companies have this on lock-down. The value of a personal assistant is very related to pervasive access and proximity, and each of the existing phone and computer OS makers has made it clear they have no interest in allowing options other than their own (for reasons that will become clear below).

Business Assistants

With the personal assistant market mostly inaccessible, many companies wanting to cash in on this space have turned to businesses as an untapped market. Rather than integrating with just one person, these agents generally aim to fulfill the same role as a secretary or office manager; scheduling meetings, sending out team-wide updates, and other relatively simple-but-frequently-annoying duties. The two main players I see a lot in this space are Clara from Clara Labs and Amy from X.ai (and yes, feminine names/traits for subservient agents is apparently just a given at this point), but they are proliferating as fast as VCs can write checks it seems. My guess is that travel management will be next on the list for many of these companies, but time will tell. Unfortunately one big problem with a lot of these agents is the open secret that much of their AI is heavily “supervised” by human “trainers”. The labor politics of AI and mechanization are complex to say the least, but further development here is certainly likely to have an impact on the polarization of tech industry workforces, with more of the few remaining entry level jobs being moved into some mix of agent AI and “virtual remote assistant” services.

Chat Bots

Focused more on consumer-facing systems, chat bots have exploded even more recently than the other two. The first wave was the now-ubiquitous “customer support chat” windows on websites. While I’m sure some of these originally used human reps just as they would in a call-center, more and more are moving to either “AI assisted” or straight-up agent AI systems. Nina from Nuance (side note: Nuance has been involved in almost every company I’ve listed here in one way other another, they are probably one of the most important AI companies you’ve never heard of), is one of the most visible but almost every customer support firm has added agent tools to their offering.

More recently we’ve seen several major chat systems build frameworks for companies to interact with customers/users via chat bots. Facebook has been among the most visible, promoting all manner of companies as bot partners. Slack has gone even further, not just promoting bot makers but building a custom venture fund to try and attract people towards integrating with Slack. These kinds of chat bots are definitely the least agent-y of what we’ve seen, but it seems clear that there will be some convergent evolution as things move forward. Rather than acting on the behalf of the user, the bots are projections of a company or service, but beyond that the interaction structure is very similar.

The Near Future

Where are we going from here? I think there are three main areas we are going to see growth in over the short term. The first is improvements in the contextual awareness of agents. The simplest example is that no current voice agent can distinguish multiple voices. The uses for this kind of data would be huge, like saying “play my favorite song” to Alexa or having Siri not pick up on commands from a TV show. More context into our lives will allow more reactive behavior, potentially even moving into proactive systems like knowing to pause Spotify on my phone and start it on my laptop or Echo when I walk into my house.

Next, I think we’ll see a major increase in the reach of agents. Right now Siri can turn my lights on and off, and set calendar items or reminders, but not really a whole lot of real “weight”. Alexa’s shopping integration offers a broader grasp, but we are still a long way from connecting an agent to my Paypal account, or my (hopefully soon to be self-driving) car, or even my email. The more systems an agent is hooked in to, the more potential to cause problems. It will take a lot of time and slow build-up, but seems inevitable that reach will grow over time.

Right now all of these agents really only “exist” while you’re interacting with them. They have no permanent existence, and so can’t take action outside of a specific call-and-response prompt. Of all the limitations on current agent tools, this seems this most restrictive. There is no way to tell my Siri to talk to someone else’s Siri to schedule a meeting, or for me to authorize my Alexa to let someone else stream to my TV. The first inklings of this are starting to appear with push notifications in the same way as for phone apps, but this is only the beginning. For a possible end-game view I turn to a story from Vint Cerf during a talk he gave on the future of the internet in 2015. He talked about how he has security cameras and motion detectors throughout his home already, something which requires the utmost in information security for hopefully obvious reasons. But if there was a fire, would you want the fire department to get access to that data? If they could immediately know where everyone in the house is and what situation they are in, it is a virtual certainty that lives will be saved. At the same time no reasonable (I think I can say this with full bipartisan agreement) person wants to give a government agency a live video feed of their home. So who or what should make the call to enable or disable the data sharing? This is a role that a more autonomous agent system could fill, though only time will tell if we ever get this far.

Beyond the technical improvements, I think the biggest story for agents in the coming years will be their commercialization. Telegram already refers to their agent directory as a “store” (though for now I think all of them are gratis). Apple has recently moved to allow limited third-party development in Siri, and Amazon has been pushing hard on their Alexa APIs. Just as phone apps led to the gold rush that was the iOS and Android app stores, I think we will see a major push towards app-ification of agents or agent functionality/integrations.

The Further Future?

Too move even further into speculation tinged by sci-fi, I think (and hope) there is a possibility of agent systems becoming more of a proxy of ourselves in digital space. We’re already seeing lots of growth in tools for “attention modeling” (eg. Facebook timeline) to help manage the massive influx of information that is part of modern life. I think over time this might fuse with the other functions mentioned above to form a more complete virtual projection. This also harkens back to the view of personal agents as replacing a a human assistant or secretary, but as models of attention and trust get better these avatars can accomplish more by interacting with each other instead of humans. If I ask my agent to find a good time for dinner with a friend, it can just do that rather than the “well what time is good for you” dance. These kinds of avatars have been such a common trope in science fiction that it is hard to even imagine them without taking fairly direct cues from specific stories or instances, but as both our rate of data ingestion and interconnectedness show no sign of slowing, something along these lines seems inevitable.

With that said, a lot will have to change in our social and legal landscape to make this truly viable. Already today we’re beginning to see some backlash against poorly secured hosted services. While Siri has mostly been limited to living in a specific phone, Alexa and Google Assistant are built from the ground up as hosted services like GMail or Amazon Video. These kinds of services have already firmly blurred the line between expectations of privacy versus what is actually a personal space, and things are only likely to get worse. Giving an agent access to all your personal information, habits, and data to allow it to better help you is going to be a powerful motivator but with things like the iCloud photo thefts or NSL data disclosures still fresh in many minds (okay, maybe more the former than the latter but I can dream) I hope to see a push for legal protections for this kind of system or for hosted/shared services more broadly as they do truly become an extension of ourselves.

Why?

So why has all of the this been happening? Why do I think this field is important enough to spend the time to write several thousand words on it? Because if done well I think agent systems will open up whole new worlds of interaction in the same way that the jump from teletype command line interfaces to GUIs and mice did for workstations, or that touchscreens did for phones. We are fundamentally social creatures, and interacting with the world through language is much more intuitive than the forced abstractions we’ve created in the computing world like menu bars and double clicking. Agents, and especially voice agents offer a way to streamline the daily lives of an almost uncountable number of people around the world.

My Todo List

2016-11-29T05:00:00Z

With the US Thanksgiving holiday this past week, I decided to take some time and write down all the projects that are still on my overall todo list. To lead off with a disclaimer: this is not a promise I’ll ever get to any of these or that they will happen in any particular order. This is mostly for my own benefit but hopefully it provides some value to others, if only documenting what still needs fixing. Also this is just my own list and does not reflect the opinions of Chef as a project or Chef Software as a company, and is neither in a specific order nor exhaustive.

If any or all of these projects sound interesting to you and you would like to see them happen sooner, maybe consider checking out my job search page?

Finish Porting Foodcritic Rules to Rubocop-Chef

This is in-progress now but still has a ways to go. You can see the current status in the rubocop-chef README. The goal is to replace as much of Foodcritic as possible with Rubocop so that we can get features like multiple message levels, autocorrect, and a unified configuration file. It will eventually be integrated with the cookstyle project as a one-stop-shop for opinions on how to write Chef cookbooks if you want to use an existing style guide.

Secrets Management Abstraction/API

I’ve talked about this a bunch and it’s very close to the top of my list right now. Basically I want to build a modular API for accessing secrets from Chef code so that cookbooks can be written to use “any” storage system (chef-vault, Hashicorp Vault, S3/KMS, etc). The DSL elements will probably look very similar to my citadel cookbook but with an internal configuration system integrated with Chef and plugable backends.

Deeper Integration Between Chef and HashiVault

I wrote up a lot of words about this, but it is still vaporware. Would be nice to spend some cycles making at least some parts of this exist.

RAM Usage Analysis for `poise-profiler`

This has been sitting half-finished for a while now and needs to get over the finish line. poise-profiler already provides a lot of timing data for Chef performance, I want to add memory usage to that as well to see when a recipe or resource is allocating a large amount of stuff (usually search results) and then not using it.

Meta-Helper Library

A common stumbling block for new and experienced Chef users alike is adding helper methods to Chef. Many of the old patterns in the ecosystem (like Chef::Recipe.send(:include, ...)) are both dangerous and unnecessary. A library cookbook to provide an API to declare various kinds of helper methods could both simplify the process and help document the tradeoffs of the different kinds of helpers.

Implement RFC 33

Chef RFC 33 adds support for a simplified folder structure for cookbooks with the aim of making small wrapper cookbooks or other short one-offs feel less boiler-plate-y. I implemented this long ago, but the branch bit-rotted before we could agree on the RFC so it needs to be started from scratch by now.

Implement RFC 36

Chef RFC 36 adds support for writing Chef recipes and some other Chef-related files in languages other than Ruby (or JSON). Specifically I want to add things like YAML and TOML support for attributes files and maybe some day Python/JavaScript support for recipe code. Similar to 33, there are a few old implementation of this but any new attempt will have to start over.

Implement RFC 75

Chef RFC 75 adds support for assigning more than one policy to a node to allow use of the Policyfile workflow in some situations where having a single compiled object is problematic. I’ve now tried implementing this twice, only to find the approach I used was fatally flawed, but I still think the feature is important and should exist.

Improve Test-Kitchen’s Concurrent Mode

With the surge in Chef releases thanks to the new monthly cadence, CI build times have been rising rapidly. For most of my cookbooks I currently run four test platforms across 18 Chef versions, and Travis only allows for 5 of those to run concurrently at most. Test Kitchen does support its own concurrent execution mode to speed things up, but currently the output ends up intermixed between the concurrent runs making it almost unreadable. For interactive use this can be okay because you just repeat a failed command to see the error message, but in a CI system this isn’t an option. An improved system could buffer the Test Kitchen output to files and then display it all together when an instance completes the command. Not a ton of work, but anything involving threads (which TK’s concurrent mode is built on) is always deceptively difficult.

Use `docker exec` for Kitchen-Docker

Currently the kitchen-docker plugin launches sshd inside the container and then uses the normal Test Kitchen SSH machinery to connect to the new instance. This works okay, but it requires a fair bit of complexity around authentication and sudo access that can sometimes make testing more difficult. In the years since the plugin was first written, Docker has added their own remote execution system in form of docker exec. Switching to this would allow for fewer moving pieces and simplify the default Dockerfile.

Improve Kitchen-Docker Build Performance

The default Dockerfile used by kitchen-docker is unfortunately slow to build sometimes. Much of this is due to the creation of many intermediary layers which could be collapsed together. I’ve already done a lot of this for my own use but it should be ported back out to the rest of the community.

New Cookbook for Supervisord

The current supervisord cookbook is in need of a major upgrade and retrofit. This can take advantage of the improved Python resources from poise-python and the patterns built as part of the new poise-monit cookbook. This would also include adding supervisord support to poise-service, allowing it to be used with any cookbook based around that service abstraction.

Resource-based Java Cookbook

The current Java cookbook has been a source of much frustration in the Chef world. I’ve already got a lot of tools built for managing language runtimes with Chef as part of the poise-python/ruby/javascript work, though undoubtedly Java will bring its own unique complications.

More Providers for Python and Ruby

The poise-python and poise-ruby cookbooks support multiple providers for how to install their respective language runtimes, but they could both stand to add some more options. On the Python side, source installs still need to be added (refactored from the existing source install support for Ruby) and some common PPAs like Deadsnakes should be included. For Ruby, the Brightbox Ruby packages would be a good option for many.

Improved SCL Handling on RHEL

The recent-ish 2.0 release of poise-langauages fixed support for Software Collections on CentOS, but unfortunately broke many RHEL users. It follows the official RedHat instructions to configure the RHSCL repositories through RedHat’s subscription system, but this flat out doesn’t work for most cloud images and development systems. Cross-installing the CentOS SCLs on RHEL seems like a reasonable fix, as well as a more flexible :rhscl provider that works on at least some more of the ways RHEL can be run.

Finish Refactoring Halite Spec Helpers

As part of the Halite project, I built out a lot of helper methods for writing Chef unit tests for complex cookbooks. Most of these are not specifically tied to Halite so I’ve started splitting them out to a new gem: poise-spec. These rest of the helpers need to be split out and then Halite’s helper module should be rewritten to use and extend poise-spec.

Migrate the Chef Documentation

Now that we’ve started to clean up the Sphinx documentation for Chef, I want to move the docs for Chef itself into Chef’s code repository. This will make it easier to require Chef changes to include doc updates in the same way as we currently require tests, as well as making it easier to version the docs alongside Chef. Some complexity will be needed to figure out how to integrate the Chef docs with the rest of the Chef’s product documentation so it can all be published to docs.chef.io but I’m sure we can work through that.

Integrate YARD and Sphinx

The current Chef docs are written using an excellent framework called Sphinx. It has lots of tools and helpers for writing high-quality narrative documentation, but most of it’s reference documentation tools are aimed at the Python and C worlds where Sphinx comes from originally. The gold standard for auto-extracted code/reference documentation in the Ruby world is YARD, and we’ve already begun adding more YARD comment markup throughout Chef and the broader ecosystem. The missing link between them is a way to integrate YARD data into Sphinx docs directly. A related project here is using YARD data to generate cookbook READMEs like other Sphinx documents.

Rewrite Poise Using New APIs

My Poise helper library has aged reasonably but enough has changed in Chef core to make a major upgrade look more and more inviting. This will probably not happen until ~April 2017 (i.e. the Chef 13.0 release), when it will be more reasonable to drop support for Chef <12.5. Notably the new resource property APIs will allow major refactoring of Poise’s helper methods and a lot fewer uses of define_method.

Other Stuff

Outside of the Chef world I’ve got a few projects kicking around in my brain. I’m a strong believer that Kubernetes is going to win the container orchestration wars, but it currently has pretty much nothing for testing and static analysis. Ditto on Terraform for infrastructure management (though there is a kitchen-terraform plugin that looks very interesting for the future). Someone bringing the same “teaching ops folks how to software” approach I’ve aimed at the Chef community is probably going to be needed in the long run.

tl;dr

I’ve got a big backlog of projects in the Chef world (and beyond) and if you want to help make them happen you should hire me.