Yarn

Yarn 1.0 is Here

Thu, 07 Sep 2017 08:00:00 +0000

Workspaces in Yarn

Wed, 02 Aug 2017 08:00:00 +0000

Projects tend to grow over time, and, occasionally, some pieces of a project can be useful elsewhere in other projects. For example, Jest, being a generic testing tool, gave birth to many packages, one of them is jest-snapshot that is now used in other projects like snapguidist and chai-jest-snapshot.

Monorepos

Those who have tried splitting a project into multiple packages know how hard it is to make changes across multiple packages at one time. To make the process easier, some big projects adopted a monorepo approach, or multi-package repositories, which reduces the burden of writing code across packages.

Several projects used every day by JavaScript developers are managed as monorepos: Babel, React, Jest, Vue, Angular.

However, separating pieces of projects into their own folders is sometimes not enough. Testing, managing dependencies, and publishing multiple packages quickly gets complicated and many such projects adopt tools such as Lerna to make working with monorepos easier.

Lerna

Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm. Internally it uses Yarn or the npm CLI to bootstrap (i.e. install all third party dependencies for each package) a project. In a nutshell, Lerna calls yarn/npm install for each package inside the project and then creates symlinks between the packages that refer each other.

Being a wrapper of a package manager, Lerna can’t manipulate the contents of node_modules efficiently:

Lerna calls yarn install multiple times for each package which creates overhead because each package.json is considered independent and they can’t share dependencies with each other. This causes a lot of duplication for each node_modules folder which quite often use the same third-party packages.
Lerna manually creates links between packages that refer each other after installation has finished. This introduces inconsistency inside node_modules that a package manager may not be aware of, so running yarn install from within a package may break the meta structure that Lerna manages.

Issues such as these convinced us, as package manager developers, that we should support multi-package repositories directly in Yarn. Starting with Yarn 0.28, we’re excited to share that we support such repositories under the Workspaces feature.

Introducing Yarn Workspaces

Yarn Workspaces is a feature that allows users to install dependencies from multiple package.json files in subfolders of a single root package.json file, all in one go.

Making Workspaces native to Yarn enables faster, lighter installation by preventing package duplication across Workspaces. Yarn can also create symlinks between Workspaces that depend on each other, and will ensure the consistency and correctness of all directories.

Setting up Workspaces

To get started, users must enable Workspaces in Yarn by running the following command:

yarn config set workspaces-experimental true

It will add workspaces-experimental true to the .yarnrc file in your OS home folder. Yarn Workspaces is still considered experimental while we gather feedback from the community.

Let’s take Jest as an example and set Yarn Workspaces up for that structure. As a matter of fact, it has already been done in a PR, and Jest has been using Yarn to bootstrap its packages for a while.

Jest’s project structure is typical for an Open Source JavaScript monorepo.

| jest/
| ---- package.json
| ---- packages/
| -------- jest-matcher-utils/
| ------------ package.json
| -------- jest-diff/
| ------------ package.json
...

The top-level package.json defines the root of the project, and folders with other package.json files are the Workspaces. Workspaces usually are published to a registry like npm. While the root is not supposed to be consumed as a package, it usually contains the glue code or business specific code that is not useful for sharing with other projects, that is why we mark it as “private”.

The following example is a simplified root package.json that enables Workspaces for the project and defines third-party packages needed for the project build and test environment.

{
  "private": true,
  "name": "jest",
  "devDependencies": {
    "chalk": "^2.0.1"
  },
  "workspaces": [
    "packages/*"
  ]
}

To keep things simple I’ll describe two small Workspaces packages:

jest-matcher-utils Workspace:

{
  "name": "jest-matcher-utils",
  "description": "...",
  "version": "20.0.3",
  "license": "...",
  "main": "...",
  "browser": "...",
  "dependencies": {
    "chalk": "^1.1.3",
    "pretty-format": "^20.0.3"
  }
}

jest-diff Workspace that depends on jest-matcher-utils:

{
  "name": "jest-diff",
  "version": "20.0.3",
  "license": "...",
  "main": "...",
  "browser": "...",
  "dependencies": {
    "chalk": "^1.1.3",
    "diff": "^3.2.0",
    "jest-matcher-utils": "^20.0.3",
    "pretty-format": "^20.0.3"
  }
}

A wrapper like Lerna would first run yarn install for each package.json separately and then run yarn link for packages that depend on each other.

If we used that approach, we would get a folder structure like the following:

| jest/
| ---- node_modules/
| -------- chalk/
| ---- package.json
| ---- packages/
| -------- jest-matcher-utils/
| ------------ node_modules/
| ---------------- chalk/
| ---------------- pretty-format/
| ------------ package.json
| -------- jest-diff/
| ------------ node_modules/
| ---------------- chalk/
| ---------------- diff/
| ---------------- jest-matcher-utils/  (symlink) -> ../jest-matcher-utils
| ---------------- pretty-format/
| ------------ package.json
...

As you see, there is a redundancy of third-party dependencies.

With Workspaces enabled, Yarn can produce a much more optimized dependency structure and when you run the usual yarn install anywhere in the project you’ll get the following node_modules.

| jest/
| ---- node_modules/
| -------- chalk/
| -------- diff/
| -------- pretty-format/
| -------- jest-matcher-utils/  (symlink) -> ../packages/jest-matcher-utils
| ---- package.json
| ---- packages/
| -------- jest-matcher-utils/
| ------------ node_modules/
| ---------------- chalk/
| ------------ package.json
| -------- jest-diff/
| ------------ node_modules/
| ---------------- chalk/
| ------------ package.json
...

Packages like diff, pretty-format and the symlink to jest-matcher-utils were hoisted to the root node_modules directory, making the installation faster and smaller. The package chalk however could not be moved to the root because the root already depends on a different, incompatible version of chalk.

Both of the structures above are compatible, but the latter is more optimal while still being correct regarding the Node.js module resolution logic.

For avid Lerna users this is similar to bootstrapping code via the --hoist flag.

If you run code inside the jest-diff Workspace, it will be able to resolve all its dependencies:

require(‘chalk’) resolves to ./node_modules/chalk
require(‘diff’) resolves to ../../node_modules/diff
require(‘pretty-format’) resolves to ../../node_modules/pretty-format
require(‘jest-matcher-utils’) resolves to ../../node_modules/jest-matcher-utils that is a symlink to ../packages/jest-matcher-utils

Managing dependencies of Workspaces

If you want to modify a dependency of a Workspace, just run the appropriate command inside the Workspace folder:

$ cd packages/jest-matcher-utils/
$ yarn add left-pad
✨ Done in 1.77s.
$ git status
modified: package.json
modified: ../../yarn.lock

Note that Workspaces don’t have their own yarn.lock files, and the root yarn.lock contains all the dependencies for all the Workspaces. When you want to change a dependency inside a Workspace, the root yarn.lock will be changed as well as the Workspace’s package.json.

Integrating with Lerna

Do Yarn Workspaces make Lerna obsolete?

Not at all. Yarn Workspaces are easily integrated with Lerna.

Lerna provides a lot more than just bootstrapping a project and it has a community of users around it that have fine-tuned Lerna for their needs.

Starting with Lerna 2.0.0, when you pass the flag –use-workspaces when running Lerna commands, it will use Yarn to bootstrap the project and also it will use package.json/workspaces field to find the packages instead of lerna.json/packages.

This is how Lerna is configured for Jest:

{
  "npmClient": "yarn",
  "useWorkspaces": true
}

Jest relies on Yarn to bootstrap the project, and on Lerna for running the publish command(s).

What is next?

Yarn Workspaces is the first step of what a package manager could do for managing monorepos as they become a more common solution to code sharing.

At the same time we don’t want to put all the possible monorepo features into Yarn. We want to keep Yarn focused and lean, and this means that Yarn and projects like Lerna will continue working together.

Our next goal is to finalize Yarn 1.0, which is meant to summarize the work we have done on Yarn over the past year, and recognizing how reliable Yarn has become. We’ll also share our thoughts on what we’d like to build next for Yarn then.

Stay tuned.

Let's Dev: A Package Manager

Tue, 11 Jul 2017 08:00:00 +0000

Hello everyone! Today, we’re gonna write a new package manager, even better than Yarn! Ok, maybe not, but at least we’re gonna have some fun, learn how package managers work, and think about what could come next on Yarn.

The devil is in the details

This article omits small details and environment quirks, and focuses on the high-level architecture of a package manager, in an effort to stay succinct. For example, we’re gonna assume that all paths are regular POSIX paths.

That being said, there’s much to say about these compatibility layers, and maybe talking about them could be an interesting follow up! Feel free to tweet at @yarnpkg if you’re interested to know more about them! 😃

To fully understand how things work, we’re gonna go step by step, incrementally, adding or extending a single function at a time. We’ll treat each of those steps as a separate chapter, and you will find an index of all chapters below this paragraph. Don’t worry - they’re all relatively short! Note that ES2017 features will be used all through the article - if you’re unfamiliar with them, we recommend you to take a look at the great books Explore ES6 and/or Understanding ECMAScript 6, and Explore ES2017. Good lecture!

# Chapter 1 - Bravely Download

Or: where we download package tarballs
# Chapter 2 - One Reference to Rule Them All

Or: where we resolve package ranges
# Chapter 3 - Dependencies of Our Dependencies Are Our Dependencies

Or: where we extract dependencies from packages
# Chapter 4 - Super Dependency World

Or: where we do the same thing, but recursively
# Chapter 5 - Links Awakening

Or: where we install our dependencies on the filesystem
# Chapter 6 - Lord of the Optimization

Or: where we try not to install the whole world on our system
# Conclusion - There Really Was a Cake

Or: where we reflect on what we’ve learned

Chapter 1 - Bravely Download

So, where should we start? First we have to think about what a package manager is. Let’s forget the caches, the mirrors, the lockfiles, and all of the fancy command-line stuff, and let’s focus on the very core: a package manager is a download manager. You ask it to download a package, and it happily executes. That’s how we’ll begin our adventure: with a very basic function that simply downloads something from the internet.

import fetch from 'node-fetch';

async function fetchPackage(reference) {

    let response = await fetch(reference);

    if (!response.ok)
        throw new Error(`Couldn't fetch package "${reference}"`);

    return await response.buffer();

}

Nice job! We just have to give this function a URL, and we’ll eventually get the referenced package back! Of course, it only works if you know the exact URL for your package, but it’s a good start. Rome wasn’t built in one day, and our package manager won’t be built with a single function either.

Ok, what’s next? Let’s take a break and look at a classic package.json file to see what we could implement.

{
    "dependencies": {
        "react": "^15.5.4",
        "babel-core": "6.25.0"
    }
}

Oh, right, version ranges! It would be nice if we were able to just pass a version number to our fetcher, and let it convert it to an URL, right? Then let’s do this! To make it easier, we’ll only add support for pinned references (ie. 1.0.0 will be supported, but not ^1.0.0). Finding the right regexp could be tedious, but thankfully we can rely on the excellent semver module, which will handle the bulk of the work for us! That being said, we’ll still need to make a small change to the signature of our fetchPackage function. Instead of using a string to describe a package, we’ll now use a {name, reference} object, where the name is the package name and the reference is the identifier that allows us to unequivocally locate this package. Thanks to this change, we can now write:

import semver from 'semver';

async function fetchPackage({name, reference}) {

    if (semver.valid(reference))
        return await fetchPackage({name, reference: `https://registry.yarnpkg.com/${name}/-/${name}-${reference}.tgz`});

    // ... same code as before

}

What do you think? If we detect that the reference is a semver version, then we convert it to an actual URL located on the Yarn registry. That’s a nice download manager we have here, right? Ok, let’s add a quick support for filesystem paths before we call it a day:

import fs from 'fs-extra';

async function fetchPackage({name, reference}) {

    // In a pure JS fashion, if it looks like a path, it must be a path.
    if ([`/`, `./`, `../`].some(prefix => reference.startsWith(prefix)))
        return await fs.readFile(reference);

    // ... same code as before

}

What do you think? Pretty simple, right?

Chapter 2 - One Reference to Rule Them All

Our fetchPackage function is great, but it has one shortcoming, and a big one: As we said, our function can currently only serve pinned references. Ranges such as ^1.0.0 cannot be served, because they can potentially refer to multiple different versions, each of them having their own tarballs. So, in order to serve them, we’ll need to find a way to extract a unique pinned reference from those ranges. Fortunately, it’s not that hard! See for yourself:

import semver from 'semver';

async function getPinnedReference({name, reference}) {

    // 1.0.0 is a valid range per semver syntax, but since it's also a pinned
    // reference, we don't actually need to process it. Less work, yeay!~
    if (semver.validRange(reference) && !semver.valid(reference)) {

        let response = await fetch(`https://registry.yarnpkg.com/${name}`);
        let info = await response.json();

        let versions = Object.keys(info.versions);
        let maxSatisfying = semver.maxSatisfying(versions, reference);

        if (maxSatisfying === null)
            throw new Error(`Couldn't find a version matching "${reference}" for package "${name}"`);

        reference = maxSatisfying;

    }

    return {name, reference};

}

// getPinnedReference({name: "react", reference: "~15.3.0"})
//     → {name: "react", reference: "15.3.2"}

// getPinnedReference({name: "react", reference: "15.3.0"})
//     → {name: "react", reference: "15.3.0"}

// getPinnedReference({name: "react", reference: "/tmp/react-15.3.2.tar.gz"})
//     → {name: "react", reference: "/tmp/react-15.3.2.tar.gz"}

And … that’s it! If we see a semver range, we just have to query the NPM registry to retrieve the list of all available versions. Once we’ve obtained it, it’s just a matter of selecting the best one (which is made easy thanks to the maxSatisfying function provided by the semver module), and we’re all set.

Note that we don’t need to do anything particular with semver versions, direct URLs, nor filesystem paths, since they’ll always refer to a single package at any given time. So when we encounter them, we can just return them back without doing anything fancy.

Thanks to this function, we can now rest assured that the references we’ll send to our fetchPackage function will always be pinned references! Another day, another great victory for us.

Chapter 3 - Dependencies of Our Dependencies Are Our Dependencies

In Chapter 1 we saw how to make a magic function that would download any package from anywhere, and return it. In Chapter 2, we saw how to convert volatile dependencies into pinned dependencies. That’s a great start! But now we’ll need to resolve a bigger issue: dependencies. See, the Node ecosystem being what it is, most packages rely on other packages in order to work properly. Fortunately, they all agreed on using a single standard to list those dependencies (remember the package.json file we’ve seen above), and so we should be able to make good use of this. Let’s write our function. Given a package, we want it to return the dependencies this package relies on.

Can’t escape the tooling

Even if this article tries to stay focused on the core principle of package managers, we will need some utility function from time to time. When you encounter a symbol imported from ./utilities, just don’t bother understanding how it works under the hood. It’s usually some boring and verbose code. That being said, all sources are available in the repository linked at the end of this post, including the utilities, so if you’re really interested, give it a look later!

// This function reads a file stored within an archive
import {readPackageJsonFromArchive} from './utilities';

async function getPackageDependencies({name, reference}) {

    let packageBuffer = await fetchPackage({name, reference});
    let packageJson = JSON.parse(await readPackageJsonFromArchive(packageBuffer));

    // Some packages have no dependency field
    let dependencies = packageJson.dependencies || {};

    // It's much easier for us to just keep using the same {name, reference}
    // data structure across all of our code, so we convert it there.
    return Object.keys(dependencies).map(name => {
        return { name, reference: dependencies[name] };
    });

}

// getPackageDependencies({name: "react", reference: "15.6.1"})
//     → [{name: "create-react-class", reference: "^15.6.0"},
//        {name: "prop-types", reference: "^15.5.10"}]

What do you think? We’ve even been able to use our very own fetchPackage implementation to get the archive from where we extract the package information! From now on, whatever package people send us, we’ll be able to know what other packages it depends on. That’s a good start, but we’ll now have to expand this ability a bit further: instead of resolving the first level of dependencies only, we’ll want to resolve everything. And that’s what the next chapter is about!

Chapter 4 - Super Dependency World

Time we go full recursion. See, the idea is that before being able to install your packages into your node_modules folder, we’ll first have to “install” them in memory. Why, you say? Well, proceeding this way will allow us to manipulate the tree before actually persisting it on the filesystem. Whether it’s deduplication or hoisting, everything will have to be applied on this tree rather than on the actual disk (which would be really slow otherwise). But we’ll cover that in another chapter! Right now, let’s focus on extracting a complete dependency tree from a single root dependency. Since we’ve already written all the needed pieces (first the function to convert a volatile reference to a pinned reference, then the function to obtain a package dependencies), it will be quick. Let’s get down to it:

async function getPackageDependencyTree({name, reference, dependencies}) {

    return {name, reference, dependencies: await Promise.all(dependencies.map(async (volatileDependency) => {

        let pinnedDependency = await getPinnedReference(volatileDependency);
        let subDependencies = await getPackageDependencies(pinnedDependency);

        return await getPackageDependencyTree(Object.assign({}, pinnedDependency, {dependencies: subDependencies}));

    }))};

}

This one might look hard to digest, but bear with me! We start from a single package with its list of dependencies. Then, for each one of those dependencies, we first resolve the dependency’s reference to become a pinned reference, then fetch its own dependencies, and then repeat the cycle on those sub-dependencies. In the end, we’ll have a tree structure, where each package will be a node that contains its own dependencies!

In order to use this function, we just have to read the initial dependencies from the package.json file located in the local working directory - everything inside is there for us to use!

import {resolve} from 'path';
import util      from 'util';

// We'll use the first command line argument (argv[2]) as working directory,
// but if there's none we'll just use the directory from which we've executed
// the script
let cwd = process.argv[2] || process.cwd();
let packageJson = require(resolve(cwd, `package.json`));

// Remember that because we use a different format for our dependencies than
// a simple dictionary, we also need to convert it when reading this file
packageJson.dependencies = Object.keys(packageJson.dependencies || {}).map(name => {
    return { name, reference: packageJson.dependencies[name] };
});

getPackageDependencyTree(packageJson).then(tree => {
    console.log(util.inspect(tree, {depth: Infinity}));
});

Now, let’s test this code. Try running it inside a directory that contains the following package.json:

{
    "name": "my-awesome-package",
    "dependencies": {
        "tar-stream": "*"
    }
}

If everything goes According To Plan, here’s what you should obtain (or similar, depending on whether a package has been upgraded since the time this article has been written):

Undefined Reference

You might notice a weird reference on the following snippet: undefined. It’s actually expected! This reference is used on the root package in order to inform the linker (more on that later) that this package is a bit special. In a real-life situation, we would probably want to use a special type of reference (for example root:///path/to/package), but in our case it’s not necessary.

{ name: "my-awesome-package",
  reference: undefined,
  dependencies:
   [ { name: 'tar-stream',
       reference: '1.5.4',
       dependencies:
        [ { name: 'bl',
            reference: '1.2.1',
            dependencies:
             [ { name: 'readable-stream',
                 reference: '2.2.11',
                 dependencies:
                  [ { name: 'core-util-is', reference: '1.0.2', dependencies: [] },
                    { name: 'inherits', reference: '2.0.3', dependencies: [] },
                    { name: 'isarray', reference: '1.0.0', dependencies: [] },
                    { name: 'process-nextick-args',
                      reference: '1.0.7',
                      dependencies: [] },
                    { name: 'safe-buffer', reference: '5.0.1', dependencies: [] },
                    { name: 'string_decoder',
                      reference: '1.0.2',
                      dependencies: [ { name: 'safe-buffer', reference: '5.0.1', dependencies: [] } ] },
                    { name: 'util-deprecate', reference: '1.0.2', dependencies: [] } ] } ] },
          { name: 'end-of-stream',
            reference: '1.4.0',
            dependencies:
             [ { name: 'once',
                 reference: '1.4.0',
                 dependencies: [ { name: 'wrappy', reference: '1.0.2', dependencies: [] } ] } ] },
          { name: 'readable-stream',
            reference: '2.2.11',
            dependencies:
             [ { name: 'core-util-is', reference: '1.0.2', dependencies: [] },
               { name: 'inherits', reference: '2.0.3', dependencies: [] },
               { name: 'isarray', reference: '1.0.0', dependencies: [] },
               { name: 'process-nextick-args',
                 reference: '1.0.7',
                 dependencies: [] },
               { name: 'safe-buffer', reference: '5.0.1', dependencies: [] },
               { name: 'string_decoder',
                 reference: '1.0.2',
                 dependencies: [ { name: 'safe-buffer', reference: '5.0.1', dependencies: [] } ] },
               { name: 'util-deprecate', reference: '1.0.2', dependencies: [] } ] },
          { name: 'xtend', reference: '4.0.1', dependencies: [] } ] } ] }

Perfect. Now, let’s try to run it with larger packages. Let’s try with babel-core! Use the following package.json file :

{
    "dependencies": {
        "babel-core": "*"
    }
}

Don’t worry, I’ll wait.

… Still waiting.

… Still… wait, is this script still running? That’s not good, right?

At this point we can safely assume that there’s something wrong in our code - Babel is not that large, and the execution should have stopped a long time ago. In order to better understand what happened, open the babel-core page on Yarnpkg, and check its dependencies. You should see babel-register. Good. Now, open the babel-register page on Yarnpkg, and check its own dependencies. You should see… Yup. Babel-core. Now can you guess what happened? Because of the circular dependency, we’ve been iterating over babel-core, then babel-register, then babel-core, then… etc. Eventually, our code will end up using too much RAM and will get killed by the OS. That’s really not good.

Fortunately, the fix is fairly easy! Remember that in Node, node_modules directories can be nested. If a package can’t be located inside the current directory node_modules, Node will try looking for it inside the parent directory node_modules, then its grandparent node_modules, etc, until it finds a satisfying match. Let’s take advantage of that:

// Look, we've added an extra optional parameter! ---------------------------------v
async function getPackageDependencyTree({name, reference, dependencies}, available = new Map()) {

    return {name, reference, dependencies: await Promise.all(dependencies.filter(volatileDependency => {

        let availableReference = available.get(volatileDependency.name);

        // If the volatile reference exactly matches the available reference (for
        // example in the case of two URLs, or two file paths), it means that it
        // is already satisfied by the package provided by its parent. In such a
        // case, we can safely ignore this dependency!
        if (volatileDependency.reference === availableReference)
            return false;

        // If the volatile dependency is a semver range, and if the package
        // provided by its parent satisfies it, we can also safely ignore the
        // dependency.
        if (semver.validRange(volatileDependency.reference)
         && semver.satisfies(availableReference, volatileDependency.reference))
            return false;

        return true;

    }).map(async (volatileDependency) => {

        let pinnedDependency = await getPinnedReference(volatileDependency);
        let subDependencies = await getPackageDependencies(pinnedDependency);

        let subAvailable = new Map(available);
        subAvailable.set(pinnedDependency.name, pinnedDependency.reference);

        return await getPackageDependencyTree(Object.assign({}, pinnedDependency, {dependencies: subDependencies}), subAvailable);

    }))};

}

This change adds a filtering pass to our dependencies processing: if any of them happens to be already satisfied by a package made available somewhere in the upstream dependency chain, then we can just skip it, since there isn’t any point in resolving it. Otherwise, we continue as usual, except that we insert them into the registry that contains our dependency chain packages. This way, our own dependencies will be able to skip installing us later on.

If we go back to our babel-core example, it will go like this:

- seeing babel-core@*

  - is it available in a parent module? NO
  - resolve it to babel-core@6.25.0
  - resolve its dependencies

    - seeing babel-register@^6.24.1
    - is it available in a parent module? NO
    - resolve it to babel-register@6.24.1
    - resolve its dependencies

      - seeing babel-core@^6.24.1
      - is it available in a parent module? YES, BECAUSE 6.25.0 MATCHES ^6.24.1
      - skip resolution

Awesome. We now have a working algorithm to compute our full dependency tree. We’re almost done, just two more mandatory steps before we reach the fun and optional parts!

Chapter 5 - Links Awakening

In Chapter 4, we saw how to obtain a complete tree of all of our dependencies. Now, we just have to download their tarballs somewhere, and extract them on the disk. The first part being made trivial by this awesome fetchPackage function we’ve conveniently written not so long ago, our linker will only be a matter of a few lines:

// This function extracts an archive somewhere on the disk
import {extractNpmArchiveTo} from './utilities';

async function linkPackages({name, reference, dependencies}, cwd) {

    let dependencyTree = await getPackageDependencyTree({name, reference, dependencies});

    // As we previously seen, the root package will be the only one containing
    // no reference. We can simply skip its linking, since by definition it already
    // contains the entirety of its own code :)
    if (reference) {
        let packageBuffer = await fetchPackage({name, reference});
        await extractNpmArchiveTo(packageBuffer, cwd);
    }

    await Promise.all(dependencies.map(async (dependency) => {
        await linkPackages(dependency, `${cwd}/node_modules/${dependency.name}`);
    }));

}

And that’s about it. This code will traverse your tree, unpack each package inside its designated directory (check the repository at the end of the article for the extractArchiveTo implementation if you care about it), then iterate over its children and do the same for each of them. Seems good enough, but I feel like we might be forgetting something… oh right! The binaries! See, NPM’s package.json files offers a way for packages to expose utilities to the public (more details here). We’ll need to add a few extra lines to support this use case:

import fs from 'fs-extra';
import path from 'path';

async function linkPackages({name, reference, dependencies}, cwd) {

    // ... same code as before, except for the end:

    await Promise.all(dependencies.map(async ({name, reference, dependencies}) => {

        let target = `${cwd}/node_modules/${name}`;
        let binTarget = `${cwd}/node_modules/.bin`;

        await linkPackages({name, reference, dependencies}, target);

        let dependencyPackageJson = require(`${target}/package.json`);
        let bin = dependencyPackageJson.bin || {};

        if (typeof bin === `string`)
            bin = {[name]: bin};

        for (let binName of Object.keys(bin)) {

            let source = resolve(target, bin[binName]);
            let dest = `${binTarget}/${binName}`;

            await fs.mkdirp(`${cwd}/node_modules/.bin`);
            await fs.symlink(relative(binTarget, source), dest);

        }

    }));

}

Good. But still, I can shake this feeling that… scripts! We’re missing install scripts! Packages can specify commands that should run after a package has been installed (for example, they might want to compile or transpile some code depending on your environment). We don’t execute them yet, but that should be fairly easy:

import cp from 'child_process';
import util from 'util';

const exec = util.promisify(cp.exec);

async function linkPackages({name, reference, dependencies}, cwd) {

    // ... same code as before except the end:

    await Promise.all(dependencies.map(async ({name, reference, dependencies}) => {

        // ... same code as before

        if (dependencyPackageJson.scripts) {
            for (let scriptName of [`preinstall`, `install`, `postinstall`]) {

                let script = dependencyPackageJson.scripts[scriptName];

                if (!script)
                    continue;

                await exec(script, {cwd: target, env: Object.assign({}, process.env, {
                    PATH: `${target}/node_modules/.bin:${process.env.PATH}`
                })});

            }
        }

    }));

}

All your environments are belong to it

Note that we’ve only set the PATH environment variable inside this snippet, but packages usually have access to a whole lot of extra environment variables (more details here). They are rarely used, but if you plan to write a package manager then you’ll have to make sure that you actually define them one way or the other.

Now, calling our linker function will install everything we need on the filesystem! Better yet, all build scripts will be run correctly, meaning you will end up with a working node_modules directory! Good job! Our next chapter will be about performances, things will now start to get really interesting.

Chapter 6 - Lord of the Optimization

Our package manager is working! However, you may notice something … Because we’re not taking advantage of Node’s resolution algorithm, and because we don’t try to remove duplicates from our package tree, we might end up with a really huge node_modules folder! You might think that it’s not that much of a problem, but it has proven to cause issues in the past. For example, on most Windows installations, paths have a hard limit of 260 characters. For packages that are deeply nested, this limit is often exceeded and it breaks things. Fortunately, Node’s resolution algorithm help us by allowing us to move the dependencies lower in the tree, as long as there is no conflicts.

So let’s go! Our job in this chapter will be to decrease the number of packages that get installed on the filesystem, by any means necessary. However, we will also do the best we can to keep our algorithm both simple and encapsulated, so that it can be easily understood by maintainers and contributors alike, and can be switched or disabled in a single line if we need to.

Here’s a possible implementation. It’s not perfect, but it’s a good start! Don’t be scared by its length, most of this is just comments:

function optimizePackageTree({name, reference, dependencies}) {

    // This is a Divide & Conquer algorithm - we split the large problem into
    // subproblems that we solve on their own, then we combine their results
    // to find the final solution.
    //
    // In this particular case, we will say that our optimized tree is the result
    // of optimizing a single depth of already-optimized dependencies (ie we first
    // optimize each one of our dependencies independently, then we aggregate their
    // results and optimize them all a last time).
    dependencies = dependencies.map(dependency => {
        return optimizePackageTree(dependency);
    });

    // Now that our dependencies have been optimized, we can start working on
    // doing the second pass to combine their results together. We'll iterate on
    // each one of those "hard" dependencies (called as such because they are
    // strictly required by the package itself rather than one of its dependencies),
    // and check if they contain any sub-dependency that we could "adopt" as our own.
    for (let hardDependency of dependencies.slice()) {
        for (let subDependency of hardDependency.dependencies.slice()) {

            // First we look for a dependency we own that is called
            // just like the sub-dependency we're iterating on.
            let availableDependency = dependencies.find(dependency => {
                return dependency.name === subDependency.name;
            });

            // If there's none, great! It means that there won't be any collision
            // if we decide to adopt this one, so we can just go ahead.
            if (!availableDependency.length)
                dependencies.push(subDependency);

            // If we've adopted the sub-dependency, or if the already existing
            // dependency has the exact same reference than the sub-dependency,
            // then it becames useless and we can simply delete it.
            if (!availableDependency || availableDependency.reference === subDependency.reference) {
                hardDependency.dependencies.splice(hardDependency.dependencies.findIndex(dependency => {
                    return dependency.name === subDependency.name;
                }));
            }

        }
    }

    return { name, reference, dependencies };

}

And that’s it. We’ll just have to call this function after resolving and before linking, and we’ll get a much simpler tree that will still produce a valid output according to Node’s resolution algorithm!

The devil really was in the details

As we saw in the introduction of this article, a large amount of what makes package managers complex software lies in the details. Our optimizer code suffers from this: despite it working in many cases, it actually has an unfortunate bug related to how binaries are linked. With the code shown above, package binaries will not be installed where they should, because when optimizing we lost the information that would allow the linker to correctly link each binary to the right location. Because of this, they will not be found when running the build scripts. Oops!

Solving this would require adding some fields into our resolution tree nodes that we would then use to track the nodes original locations in the tree. The linker would then be able to link the binaries directly inside its children in a post-processing pass. Unfortunately, it would also make the code much less clear, so we opted not to implement this here. Such is the tough life of package manager writers…

Conclusion - There Really Was a Cake

Finally! After all this time, we have our tiny package manager! You can even see its full code on this repository - you can try it, it really works! It is admittedly pretty basic, kind of slow, and without much features, but we love it nevertheless and that’s all that matters. And because it’s young, there is still room for a lot of evolutions and improvements:

We could implement a powerful CLI that would be similar to Yarn! With progress bars, and emojis, and all those fancy things! In fact, the demo already has progress bars, so that’s a good start!
We could split our functions into modules! Our package manager would then be a simple CLI, and our fetchers / resolvers / linkers would be loaded from a configuration file. Want to link everything using symlinks or hardlinks instead of copying files? Just use another linker than the default one! Want to add support for extra fetchers? Add them to your config files and be done with it! In fact, we even started experimenting with something similar in Yarn.
We could also improve our optimizer so that it would actually work in every case! ;) And assuming a plugin architecture like the one we talked about in the previous bullet point, we could even implement different optimization strategies — from the [--flat](https://yarnpkg.com/lang/en/docs/cli/install/#toc-yarn-install-flat) option to ensure that we wouldn’t use multiple versions of any single package, up to the more esoteric ones that would use more complex algorithms, such as SAT solvers — and all the while without any risk of hurting the package manager core experience!
We could persist our resolution tree to a file on the disk, which we would call yarn.lock, and each time we would need to process a package from within our getPinnedReference and getPackageDependencies functions, we would instead extract that information from the file instead of over the network! (In case you’re wondering, that’s exactly how both Yarn’s yarn.lock and NPM@5’s package-lock.json files work)
We could save the tarballs in some sort of a cache, so that we wouldn’t have to download them from the network multiple times. By doing this we could even install our packages offline, if our cache is sufficiently well furnished!

This is only a short list, far from being exhaustive! Package managers can implement a wide range of features, and all of them can each be improved in a lot of different ways. As you can see, the future looks bright: who can tell what new features and improvements will come during the incoming years? No one can tell for sure, but what I can tell you is to watch this blog for the next Yarn announcement!

I hope you’ve enjoyed this article as much as I’ve taken pleasure in writing it! If you want to discuss it, whether it’s to correct some mistake or to just talk about package managers, ping me on Twitter via @arcanis, or on Yarn’s Discord server where the core team regularly lurks :)

Adding Command Line Aliases for Yarn

Mon, 19 Jun 2017 00:00:00 +0000

One of the core design philosophies of Yarn is to strive for simpleness; a lean CLI without redundant features. That’s why Yarn has resisted adding random built-in shorthands like npm r or an aliases system like the one you can find in Git. We believe that the benefits they could possibly bring to the Yarn experience are not justified by the cost required to build and maintain such a full-fledged subsystem.

We’ve also noticed, however, that it is among one of the most common feature requests we received from the community. People do use aliases for several reasons, for example, to replicate their experiences from the npm command. The good news is, all modern shell environments actually support command aliases in one form or another, and we encourage you to improve your CLI experience using these ways that are baked into your favorite shell already.

Let’s say you check for package distribution tags information pretty often, are a report message addict as well as an emoji hater, and you’d like to have a handy shorthand for this common task. Below we’ve compiled a list of ways to add the command alias of yarn info --verbose --no-emoji <package> dist-tags in a number of popular shells for your convenience:

Bash & Zsh

Bash is the default shell on most Unix-like systems; together with Zsh, they both descended from the earlier Bourne shell and hence syntaxes are largely compatible. To add a simple alias in either Bash or Zsh, simply have the following line added into your .bashrc or .zshrc, respectively:

alias ynf="yarn info --verbose --no-emoji"

Restart your shell and now you’ll be able to do:

ynf react dist-tags

yarn info v0.24.6
verbose 0.261 Checking for configuration file "/Users/gsklee/.npmrc".
verbose 0.262 Checking for configuration file "/Users/gsklee/.npmrc".
verbose 0.262 Checking for configuration file "/Users/gsklee/.nvm/versions/node/v8.1.2/.npmrc".
verbose 0.263 Checking for configuration file "/Users/gsklee/.npmrc".
verbose 0.263 Checking for configuration file "/Users/.npmrc".
verbose 0.265 Checking for configuration file "/Users/gsklee/.yarnrc".
verbose 0.265 Found configuration file "/Users/gsklee/.yarnrc".
verbose 0.267 Checking for configuration file "/Users/gsklee/.yarnrc".
verbose 0.267 Found configuration file "/Users/gsklee/.yarnrc".
verbose 0.268 Checking for configuration file "/Users/gsklee/.nvm/versions/node/v8.1.2/.yarnrc".
verbose 0.268 Checking for configuration file "/Users/gsklee/.yarnrc".
verbose 0.268 Found configuration file "/Users/gsklee/.yarnrc".
verbose 0.27 Checking for configuration file "/Users/.yarnrc".
verbose 0.274 current time: 2017-06-16T09:43:50.256Z
verbose 0.339 Performing "GET" request to "https://registry.yarnpkg.com/react".
verbose 0.488 Request "https://registry.yarnpkg.com/react" finished with status code 200.
{ latest: '15.6.1',
  '0.10.0-rc1': '0.10.0-rc1',
  '0.11.0-rc1': '0.11.0-rc1',
  next: '16.0.0-alpha.13',
  dev: '15.5.0-rc.2',
  '0.14-stable': '0.14.9',
  '15-next': '15.6.0-rc.1' }
Done in 0.28s.

Now, if you’d like to further alias the dist-tags part as well, you’ll need to use a function instead because Bash/Zsh aliases do not accept additional parameters:

function ynftag { yarn info --verbose --no-emoji "$@" dist-tags; }

You’ll then be able to get the same output by simply typing:

ynftag react

Fish

Fish is a newer “exotic shell” that deviates from traditional shell designs. It offers “abbreviations” that expand into full commands live as you type, much like the so-called snippets in modern code editors. To add an abbreviation:

abbr --add ynf yarn info --verbose --no-emoji

When it comes to passing in additional arguments, however, you have to use functions just like in Bash and Zsh:

function ynftag --wraps yarn --description "yarn info --verbose --no-emoji <package> dist-tags"
  yarn info --verbose --no-emoji $argv dist-tags
end

To persist your alias command definition, save it to your autoload directory:

funcsave ynftag

PowerShell

For Windows developers, chances are that you’ve been using PowerShell instead. Unlike Unix shells which are built upon text processing and piping, inputs and outputs in PowerShell are .NET objects; as such, aliases in PowerShell do not work as string substitutions, but rather pointers to existing functions. This means that you’ll need to use functions to define your aliases whether additional parameters are involved or not.

Here is a guidelines-abiding example of the ynftag alias:

function Get-NpmPackageDistributionTags { yarn info --verbose --no-emoji @Args dist-tags }
New-Alias ynftag Get-NpmPackageDistributionTags

Save the code above to one of the many PowerShell profiles that suits you best to persist the definition.

Command Prompt (`cmd`)

If you’re still using the clunky Command Prompt, we believe that it’d be better for you, in the long run, to learn to use PowerShell instead. It’s more capable, modern, and everything is just way more consistent. Nonetheless, here is how you define an alias within the current Command Prompt instance:

doskey ynftag=yarn info --verbose --no-emoji $* dist-tags

Since Command Prompt doesn’t come with a .bashrc equivalent, in order to persist your aliases permanently, you’ll need to create a custom cmdrc.cmd file (could be any name, but we recommend you to stick with the long-standing naming convention) inside your home directory, with the following content:

@echo off
doskey ynftag=yarn info --verbose --no-emoji $* dist-tags

Then modify your Command Prompt shortcut target to:

# Replace `cmdrc.cmd` with the full path that leads to the file.

cmd.exe /k cmdrc.cmd

In Conclusion

Yarn is a powerful JavaScript tool, but it’s also a tool that resides in your shell environment. By leveraging the innate capabilities of your shell, Yarn can do far more for you right now and right away.

Private Registry Support

Fri, 16 Jun 2017 12:00:00 +0000

Today, Yarn already supports a wide variety of different package feeds when fetching and downloading your dependencies. Up until now, there was however a small subset of public and private package feed providers that Yarn could not yet handle very well. One example of these package feed providers that were not yet supported was Visual Studio Team Services (VSTS).

Let’s explain why:

Registry and package location URL differences

Some registries, such as VSTS, use two slightly different URL structures for the location of the package feed and the location of the actual package archive binary itself.
For example, a private feed’s URL would look like this:

// Package feed URL
https://$ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/$FEED_NAME/npm/registry

but the URLs to fetch the actual tar archives would then be returned by the feed in this format:

// Package archive URL
https://$ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/da6e033f-20ad-4ee1-a784-8995dd6836b72/npm/registry/@scope/package-name/-/package-name-0.0.1.tgz

As you can see, the archive’s URL does not contain the actual feed name anymore but rather a random GUID, followed by the package name and version. This differing layout of the path part of the URL lead to Yarn not recognizing that the two URLs actually both do belong to a request to the same registry and would therefore refuse to download the package.

Introducing custom host suffixes

Starting with version 0.26.0, Yarn now understands a new configuration option called custom-host-suffix. This allows you to keep the same strict URL validations for most of your package URLs but also selectively loosen that check for a specific registry provider so that Yarn will now match the URLs where the host part ends with the value from this new option.

Simply add custom-host-suffix to either your global user-level .npmrc or your project’s individual .npmrc and Yarn will be able to download your packages as desired. In the above example of Visual Studio Team Services, the .npmrc should contain an entry like this:

custom-host-suffix='pkgs.visualstudio.com'

We hope you will find this new option useful and are happy that Yarn can now be used in even more projects!

Yarn determinism

Wed, 31 May 2017 09:00:00 +0000

One of the claims that Yarn makes is that it makes your package management “deterministic”. But what exactly does this mean? This blog post highlights how both Yarn and npm 5 are deterministic, but differ in the exact guarantees they provide and the tradeoffs they have chosen.

What is determinism?

Determinism in the context of JavaScript package management is defined as always getting the exact same node_modules folder given a package.json and companion lock file.

What factors does Yarn’s determinism guarantee?

Lockfile

Yarn is fully deterministic as long as all your teammates are using the same Yarn version. In both Yarn and npm 5, the determinism is ensured by lockfiles that contain information about the whole tree. However the lockfile formats are different between these two projects. We haven’t publicly talked about why we chose this format, so we want to walk you through it:

If you’ve ever seen a yarn.lock then you should be pretty familiar with the following structure:

has-flag@^1.0.0:
  version "1.0.0"
  resolved "https://registry.yarnpkg.com/has-flag/-/has-flag-1.0.0.tgz#9d9e793165ce017a00f00418c43f942a7b1d11fa"

supports-color@^3.2.3:
  version "3.2.3"
  resolved "https://registry.yarnpkg.com/supports-color/-/supports-color-3.2.3.tgz#65ac0504b3954171d8a64946b2ae3cbb8a5f54f6"
  dependencies:
    has-flag "^1.0.0"

This is the yarn.lock file generated by running the command yarn add supports-color. This file contains the version of supports-color that our project is using as well as the exact version of all its sub-dependencies.

With this lock file we can ensure that the version of has-flag that supports-color relies on is always the same version.

But there’s one key piece of information that the yarn lockfile doesn’t contain and that’s the hoisting and location of each dependency in the tree. For example, given a yarn.lock it’s impossible to determine what are the top level dependencies unless you have it’s accompanying package.json. Even knowing the top level packages, we still cannot infer what hoisting position packages should be in.

In practice this means that the position of packages in node_modules is computed internally in Yarn, which causes Yarn to be non-deterministic between people using different versions.

The reason that we do this is that this lockfile format is great for diffing. That is, changes to the lockfile can easily be human reviewed. The reason we use a custom format instead of JSON and have everything at the top level is so that it’s easy to read and review. Merge conflicts are usually automatically handled by version control and it reduces thrash.

Hoisting guarantees

Even though Yarn hoisting may differ between versions we still make very strong guarantees around hoisting when the same version of Yarn is used. The most significant of these guarantees is that omitting environmental dependencies like optionalDependencies and devDependencies still influences the position of normal dependencies.

Woah there, that’s a lot of dependency mumbo jumbo. What does that actually mean?

There are several types of dependencies that you can declare in your package.json. Two of these are plain dependencies which are installed all the time, and there are devDependencies which are only installed when you run npm install or yarn within the directory where the package.json file is present.

Due to these features it’s possible to have different layouts of node_modules with omitted dependencies. But Yarn still factors all dependencies into account when determining the position that they should be at in node_modules, so even if they aren’t installed, they still influence the hoisting position of others. This is important as having variance in hoisting position of packages in production and development can cause really weird obscure bugs.

NOTE: This guarantee isn’t unique to Yarn and npm 5 also does this.

How does this compare to npm 5?

npm 5 introduces a rework of the shrinkwrap feature called package-lock. This file includes all the information required to create node_modules as well as all hoisting information. The npm version of the previous yarn.lock would be:

{
  "name": "react-example",
  "version": "1.0.0",
  "lockfileVersion": 1,
  "dependencies": {
    "has-flag": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-1.0.0.tgz",
      "integrity": "sha1-nZ55MWXOAXoA8AQYxD+UKnsdEfo="
    },
    "supports-color": {
      "version": "3.2.3",
      "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-3.2.3.tgz",
      "integrity": "sha1-ZawFBLOVQXHYpklGsq48u4pfVPY="
    }
  }
}

Note that here all the packages listed in the first dependencies object are hoisted. This means that npm can use only the package-lock as the source of truth in order to build the final dependency graph whereas Yarn needs the accompanying package.json to seed it.

This means that npm has better assurances around hoisting position across npm versions at the cost of having a more dense lockfile. There’s currently no plan on how to support package-lock.json in Yarn as the story around lockfile interoperability is unclear. You could however imagine a future where Yarn supports both and updates them in tandem. We’re very interested in community feedback and encourage proposals for how this would work to be submitted as an RFC.

Closing remarks

Each lock file format has different tradeoffs and there doesn’t appear to be a perfect format without disadvantages. It’s important to evaluate what sort of guarantees you’re looking for when deciding what package manager to use.

npm 5 has stronger guarantees across versions and has a stronger deterministic lockfile, but Yarn only has those guarantees when you’re on the same version in favor of a lighter lockfile that is better for review. It’s possible that there’s a lockfile solution that has the best of both worlds, but for now this is current state of the ecosystem and possible convergence could happen in the future.

Hopefully this post has highlighted the determinism guarantees that differ between Yarn and npm and helped you decide what works better for your company or project.

Yarn Create & Yarn 1.0

Fri, 12 May 2017 08:00:00 +0000

Last year was a great time for Javascript newcomers! A lot of starter-kit projects were published, refined, and some of them eventually went on to offer command line tools dedicated to make project creation easier. One such example is create-react-app, but most frameworks have their own tools, with various flavors and syntaxes.

Despite these tools, one problem remains: Users still need to know how to use their package managers before being able to start a new project. They need to know what’s the difference between global packages and local packages, and how to make sure that the binaries are available from the shell, which can sometimes cause subtle issues. Further, because these globally installed tools need to be manually updated, most projects maintain a small cli wrapper that downloads the latest version of the tool itself. Fortunately, we’re in a position where we can help with this to make building new applications more cohesive:

`yarn create <pkg-name>`

With yarn create, you can start building apps with many of the existing projects:

yarn create react-app my-app
yarn create react-native-app my-app
yarn create next-app my-app

When ran, the create command will automatically install or update the requested package, prefixing its name with create-. Running yarn create react-app will start by doing the same thing as yarn global add create-react-app. Then, once the package installed, Yarn will run the executable located in the bin field of the newly installed package’s package.json, forwarding to it any remaining command line argument.

It is important to us to keep the feature small and extensible. Yarn should be a lightweight tool and yarn create is no exception: An immediate implication is that the create command is a completely agnostic tool: we make no assumption regarding what you want to create, and delegate all the behavior to the create-* packages! It is our hope that the community will come up with creative way to use this tool. Creating apps is but only one thing! Feel free to make packages that create tests, readmes, changelogs or anything else you want!

Note: The create- prefix is inserted right before the package name. So, for example, if you run yarn create @ng/app, it will install the @ng/create-app package, then run it.

Other Improvements

yarn create is but one of the many things we have been working on over the last couple of weeks. Thanks to numerous pull requests from many open source contributors, the recent releases also ship with the following features & improvements:

The offline mirror does not require changes to the yarn lockfile any longer (#2970)
Command-line arguments and environment variables can now be set in the yarnrc file (#3033, #3218)
Prepare & prepublish-only lifecycle hooks are now implemented (#3004)
The offline mirror can be pruned if used by a single one of your projects (#2836)
Various improvements for yarn pack (#3175, #3092)

The list of all improvements and bugfixes over the last couple of months can be found in our releases section on GitHub. We would specifically like to thank a team from the Delft University of Technology: Tim van der Lippe, Chris Langhout, Gijs Weterings and Chak Shun Yu. The four of them did a fantastic analysis of the Yarn project and sent pull requests to improve it in many areas. They also pointed out gaps in our test coverage, which our new core contributor Simon Vocella has been working on improving.

Planning for Yarn 1.0

Yarn has made substantial improvements since its initial release 7 months ago and the project recently surpassed 1,000 Pull Requests. Currently, we are planning for the 1.0 release of Yarn which is scheduled for this summer and will come with stability improvements, new features and performance wins. To hear more about Yarn’s present and future, please watch Konstantin’s talk about Building High-Quality JavaScript Tools.

We thank each and every one of you for your help to make this project great. If you’d like to contribute to Yarn, please don’t hesitate to reach out to us on GitHub or on Discord.

Cloudflare security incident and impact on Yarn users

Fri, 24 Feb 2017 14:00:00 +0000

Yarn uses its own proxy to the npm registry in order to allow us to experiment with the way the Yarn client works and allow optimizations in the future around how packages are resolved. This registry is used by all Yarn users by default.

In order to do this we use the popular service, Cloudflare, which is used by thousands of companies and who had offered to work with us to make Yarn installs faster globally.

Recently it was reported that Cloudflare had a serious bug that was leading to requests from other websites being leaked into HTTP responses.

When it comes to registry authentication, the Yarn client differs from the npm client in that when we perform authentication we do not store the resulting token and invalidate it after it’s used.

However, Yarn still allows you to login with your npm account to perform actions such as publishing and downloading private packages. Out of the 70 million requests performed daily we only get 10-30 requests that involve registry authentication. This means that for these requests there was the possibility of user passwords being leaked.

Since the Cloudflare announcement we’ve been in contact and have been assured that Yarn has not been affected and no Yarn users data has been leaked. Even with this assurance we’d recommend that if you’re one of those 30 people a day using Yarn for registry authentication that you reset your password as a precautionary measure.

As a result of this we’re evaluating our security policy and have created a new email address security@yarnpkg.com that can be used to report security vulnerabilities without going through the public issue tracker. We’re also in the process of setting up a HackerOne account and will make an announcement when this is available.

We’d like to apologize for this disruption and want to reaffirm our commitment to security and transparency in cases like these.

Lockfiles should be committed on all projects

Thu, 24 Nov 2016 08:00:00 +0000

Yarn is a new package manager that we built to be consistent and reliable. When installing hundreds or even thousands of third-party packages from the internet you want to be sure that you’re executing the same code across every system.

Yarn maintains consistency across machines in two key ways:

Yarn uses a deterministic algorithm that builds up the entire dependency tree before placing files where they need to be.
Important info from the install process is stored in the yarn.lock lockfile so that it can be shared between every system installing the dependencies.

This lockfile contains information about the exact versions of every single dependency that was installed as well as checksums of the code to make sure the code is identical.

Yarn needs this info about every dependency because packages are constantly changing. New versions of packages are published all the time and since package.json specifies version ranges you need to lock them down to a single version.

These version ranges exist because Yarn follows Semantic Versioning, or “SemVer”. SemVer is a versioning system designed around “breaking” or “non-breaking” changes.

When you have a version such as v1.2.3, it’s broken into three parts:

Major (1.x.x) – Changes that may cause user code to break
Minor (x.2.x) – Changes that add new features (but should not break user code)
Patch (x.x.3) – Changes that are fixing bugs in previous versions (but do not add new features and should not break user code)

When a package publishes a new version, the author bumps major, minor, or patch based on the changes that they have made as a way to communicate them.

Users of packages should typically welcome minor and patch versions but should be wary of major versions as they could break your code.

Version ranges are a way of specifying which types of changes you want to accept and which versions you want to prevent. These version ranges then resolve down to a single version which is either the version you have installed or the latest published version that matches your version range.

If you don’t store which version you ended up installing, someone could be installing the same set of dependencies and end up with different versions depending on when they installed. This can lead to “Works On My Machine” problems and should be avoided.

Also, since package authors are people and they can make mistake, it’s possible for them to publish an accidental breaking change in a minor or patch version. If you install this breaking change when you don’t intend to it could have bad consequences like breaking your app in production.

Lockfiles lock the versions for every single dependency you have installed. This prevents “Works On My Machine” problems, and ensures that you don’t accidentally get a bad dependency.

It also works as a security precaution: If the package author is either malicious or is attacked by someone malicious and a bad version is published, you do not want that code to end up running without you knowing about it.

Libraries vs Applications

There are two primary types of projects that use Yarn:

Libraries – Projects that get published as packages to the registry and installed by users. (i.e. React or Babel)
Applications – Projects that only consume other packages, typically building some kind of product. (i.e. Your company’s app)

For applications, most developers agree that lockfiles are A Good Idea™. But there has been some question about using them when building libraries.

When you publish a package that contains a yarn.lock, any user of that library will not be affected by it. When you install dependencies in your application or library, only your own yarn.lock file is respected. Lockfiles within your dependencies will be ignored.

It is important that Yarn behaves this way for two reasons:

You would never be able to update the versions of sub-dependencies because they would be locked by other yarn.lock files.
Yarn would never be able to fold (de-duplicate) dependencies so that compatible version ranges only install a single version.

Some have wondered why libraries should use lockfiles at all if they do not and should not affect users. Even further, some have said that using lockfiles when developing libraries creates a false sense of security since your users could be installing different versions than you.

This seems to logically makes sense, but let’s dive deeper into the problem.

Development Dependencies

So far we’ve been talking about dependencies as if there were only one type of dependency when in fact there are several different types. These are broken down into two categories:

Runtime – Dependencies that are used by the project’s code and needed when the code is run.
Development – Dependencies that are only needed to work directly on the project

When a library is installed by a user, only the runtime dependencies are installed. The development dependencies are only ever installed when working directly on the project that specifies them.

Each type of dependency creates a whole tree of dependencies which are needed for them to be used.

It turns out that most projects (libraries or applications) have far more dependencies for development than for runtime. The tree of dependencies created by a projects development dependencies is almost always the largest part of the total.

You can blame this on how frustratingly complicated JavaScript development is, but it seems to hold true across every ecosystem. You almost always need more code to develop projects than you need to run them.

When working on a library, it is far more likely that a development dependency breaks just because there are more of them that could break.

The Breaking Change Race

When a package accidentally publishes a breaking change it starts the clock on who will be the first person to catch it. Whoever installs that breaking change first will (most likely) be the first to discover it.

Let’s imagine we have a package called left-pad which takes a string and adds a specified amount of padding in front of it. This small and seemingly harmless package is used by a really big project, which we’ll just call… Babel.

One day, the maintainer of left-pad decides they are going to do something unspeakably evil and make it pad the right side instead. They publish it as a patch version which quickly spreads to everyone using it.

Remember, Babel is a really huge project with tens of thousands of users and dozens of contributors. Let’s try and figure out who will catch the padding fiasco first.

Looking at build history, Babel was installed in CI (with the left-pad dependency) exactly 103 times in the last 30 days.
Looking at statistics from the registry, Babel was installed (with left-pad) by users over 5.5 Million times in the same 30 days.

To put that in a different measurement:

Contributors install Babel on average every 7 hours
Users install Babel about on average every 0.5 seconds

When the left-pad incident happened, users discovered it almost immediately. Notifying the Babel contributors via GitHub issues, tweets, Facebook messages, emails, phone calls, smoke signals, and by carrier pigeon.

Babel contributors couldn’t possibly prevent users from being affected by every error. In order to do so, every 0.5 seconds they would need to be able to run CI, notice the error, find a fix, publish a new version, and get every user to upgrade. There’s just no way to make this happen.

This might seem like a bit too extreme of an example to apply broadly, but the reasoning stays the same just with different numbers. Libraries should always be installed more frequently by users than by contributors. If that isn’t true, it’s because no one uses that library anyways– and we should not be optimizing our workflow around code that no one uses.

User testing

Many library developers work very hard to maintain a test suite that has 100% code coverage. This coverage ensures that every single line of code is run at least once during the tests.

But it still does not catch everything. If a library has even a small number of users, it will be far better tested by the users than it will be by any test suite the contributors can come up with.

Users just write more code, and the code they write is not always what a library author will expect. The more popular the library the more edge cases users will find. Users will do things that you didn’t even think was possible– it will horrify you. It will make you question the goodness of humanity.

Even if a contributor happened to beat users to finding a breaking change, there’s a pretty decent chance they won’t catch it anyways. Untested edge cases are the most likely things to break.

The contributor’s burden

Now let’s talk a bit about the social side of not using lockfiles in libraries.

As mentioned previously, the majority of breaking changes that occur in library dependencies will be development dependencies.

Some percentage of these breaking changes will be caught and (hopefully) fixed by regular contributors. However, the remaining breaking changes will be caught by new or infrequent contributors.

Contributing to a project for the first time is a very intimidating experience for anyone who is not an open source veteran. If the first thing a potential new contributor runs into is a broken build or test suite, they could be so intimidated that they decide to forget about contributing back.

Even as someone who contributes to lots of open source projects, it’s a terrible thing to have to debug a build system for a few hours when you just want to fix a bug or add a small feature to a library.

The user’s burden

Of course not every breaking change is a development dependency, and theoretically contributors could catch breaking changes before users notice them. In that (narrow) scenario, aren’t we shifting the burden from contributors to users?

First, it’s not going to be every user that is affected by this. It’s well agreed upon that applications should be using lockfiles, and if they are then they won’t be affected by sudden breaking changes.

We’re only talking about users that are either installing a package for the first time, or are upgrading the versions of their dependencies.

For users upgrading their dependencies, they should be trained to look out for breaking changes and if they encounter one, to roll back the version to a working state and open an issue in the library.

The only really negative experience that we are adding is for new users. Which is admittedly terrible, you want new users to have the best possible experience.

But remember that they already face this burden in the majority of cases. It’s only when contributors could have caught breaking changes before users experienced them that we are now placing an additional burden on new users.

It’s also important to note that new users make up a minority of the total installations. The majority of the time it will be existing users that are the first to catch breaking changes because they are the ones installing a library the most.

In Closing

There is a simple universal rule that everyone should follow with Yarn: If you are installing a new dependency or upgrading an existing one you should check to make sure the package works as intended and is not breaking your code.

You should follow that rule regardless of what your libraries are doing.

Without lockfiles it gets even more complicated: In applications or libraries, if there is no lockfile, you will have to check the dependencies every time you install or re-install them and make sure that everything still works. Otherwise the build might be broken or the tests might fail. You could break something without even realizing it. You could run into situations where code works on your machine but no one else’s.

The idea that not using lockfiles in libraries somehow saves users from encountering breaking changes is at best extremely rare and at worst is never true.

By not using lockfiles in libraries, the only tangible thing you accomplish is making the project harder to contribute to.

We should work as a community to keep all of our libraries up to date. We should go out and build tooling for automatically upgrading dependencies that makes it painless to do. There has been attempts at such tooling before, but we can do better.

Please commit your yarn.lock files.

Running Yarn offline

Thu, 24 Nov 2016 08:00:00 +0000

Repeatable and reliable builds for large JavaScript projects are vital. If your builds depend on dependencies being downloaded from network, this build system is neither repeatable nor reliable.

One of the main advantages of Yarn is that it can install node_modules from files located in file system. We call it “Offline Mirror” because it mirrors the files downloaded from registry during the first build and stores them locally for future builds.

“Offline mirror” is different from cache that both npm CLI and Yarn have. Caches store already unzipped tarballs downloaded from registry, they also can be implementation specific and may be invalid between multiple versions of CLI tools. The tarballs in “Offline mirror” can be consumed by any version Yarn that will build cache based on them. It is also easier to store files when they are compressed.

Let’s set up “Offline mirror” for a simple JS project

{
  "name": "yarn-offline",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "is-array": "^1.0.1",
    "left-pad": "^1.1.3",
    "mime-types": "^2.1.13"
  }
}

When you run yarn install, the generated yarn.lock file has sections for every dependency:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1

is-array@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/is-array/-/is-array-1.0.1.tgz#e9850cc2cc860c3bc0977e84ccf0dd464584279a"

left-pad@^1.1.3:
  version "1.1.3"
  resolved "https://registry.yarnpkg.com/left-pad/-/left-pad-1.1.3.tgz#612f61c033f3a9e08e939f1caebeea41b6f3199a"

mime-db@~1.25.0:
  version "1.25.0"
  resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.25.0.tgz#c18dbd7c73a5dbf6f44a024dc0d165a1e7b1c392"

mime-types@^2.1.13:
  version "2.1.13"
  resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.13.tgz#e07aaa9c6c6b9a7ca3012c69003ad25a39e92a88"
  dependencies:
    mime-db "~1.25.0"

Each of these dependencies have a resolved field with a remote URL. If you delete your node_modules folder and run yarn install again, Yarn will download the same resolved dependencies specified in this lockfile. It will even guarantee that no one modified the files since your first installs by verifying checksum for each of them.

However, if for some reason these urls are unreachable during your build, it will fail. To solve this, we’ll need an “Offline mirror”.

Set up .yarnrc

First we need to setup a directory to be our “Offline mirror” storage, we can do that with yarn config command:

$ yarn config set yarn-offline-mirror ./npm-packages-offline-cache
yarn config v0.23.2
success Set "yarn-offline-mirror" to "./npm-packages-offline-cache".
✨  Done in 0.06s.

./npm-packages-offline-cache is an example location relative to home folder where all the source .tar.gz files will be downloaded to from the registry.

Offline mirror does not come with removing tarballs. In order to keep the cache folder up to date, you need to add the following to the config file:

This feature is only available in version 0.23.0 and above.

$ yarn config set yarn-offline-mirror-pruning true
yarn config v0.23.2
success Set "yarn-offline-mirror-pruning" to "true".
✨  Done in 0.06s.

This will create a .yarnrc file in your HOME directory. Let’s move this file to the project root so that offline mirror would be used only for this project.

$ mv ~/.yarnrc ./

Initialize the new lockfile

Remove node_modules and yarn.lock that were generated previously and run yarn install again:

$ rm -rf node_modules/ yarn.lock
$ yarn install
yarn install v0.17.8
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 0.57s.

The dependency resolutions in your yarn.lock should look the same as the original:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1

is-array@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/is-array/-/is-array-1.0.1.tgz#e9850cc2cc860c3bc0977e84ccf0dd464584279a"

left-pad@^1.1.3:
  version "1.1.3"
  resolved "https://registry.yarnpkg.com/left-pad/-/left-pad-1.1.3.tgz#612f61c033f3a9e08e939f1caebeea41b6f3199a"

mime-db@~1.25.0:
  version "1.25.0"
  resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.25.0.tgz#c18dbd7c73a5dbf6f44a024dc0d165a1e7b1c392"

mime-types@^2.1.13:
  version "2.1.13"
  resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.13.tgz#e07aaa9c6c6b9a7ca3012c69003ad25a39e92a88"
  dependencies:
    mime-db "~1.25.0"

The offline cache filename is derived from the base name of the resolved URL. It will be stored in the npm-packages-offline-cache folder that was configured earlier. Each resolved dependency also contains a checksum after the file name to ensure that no one mangles with the downloaded files.

And inside the “Offline mirror” folder we have the .tgz files that yarn will use for the following builds without reaching out to network.

$ ls npm-packages-offline-cache/
is-array-1.0.1.tgz    left-pad-1.1.3.tgz    mime-db-1.25.0.tgz    mime-types-2.1.13.tgz

How can you test to make sure it is offline?

Clear your global cache with “yarn cache clean”

Turn off wifi

Run “yarn install –offline”. The offline flag will make sure yarn does not reach out to the network

In a nutshell, to enable “Offline mirror” for your project you need:

add “yarn-offline-mirror” configuration to .yarnrc file

generate a new yarn.lock with “yarn install” command

A few tips and tricks

Updating your package

If you want to make sure you have a clean cached modules, here are few of the steps you can take:

Remove the package first. Make sure you have “yarn-offline-mirror-pruning” set to true in your .yarnrc file
Clear the yarn cache with “yarn cache clean” before adding the updated version of the package
Add your package

The “yarn-offline-mirror-pruning” will help clean up any unlinked dependencies. When you add the updated package, it will check the yarn cache first and pull any missing dependencies from there. This will prevent yarn adding new tarball back with the updated package. You want to make sure the yarn cache is all clean before you do any adding for cache module.

You can check in “Offline mirror” into git or mercurial repository

The “Offline Mirror” can be shared between build servers or development machines in any way that is convenient: a Box / Dropbox folder, stored in source control or on a network drive. At Facebook the offline mirror lives inside of our big Mercurial “monorepo”.

Whether to commit binary files into a repository or not depends on the number and size of your project’s dependencies. For example, out of 849 React Native dependencies totaling 23MB, only 10% are larger than 30KB.

Many Facebook teams, including the React Native team, decided to check in their “Offline mirror”. They all share the same “Offline mirror” which means that most dependencies for new projects are often already checked into that folder, so the cost of storing the packages in source control gets lower the more projects use it.

Let’s compare checking in node_modules to checking in “Offline mirror”

The React Native team used to check in the node_modules folder but they hit several limits:

node_modules contains more than 37,000 files (and more than 100,000 files back when we were using npm2). This had a bad performance impact on our Mercurial repository.
Reviewing Pull Requests that changed a dependency was quite hard as all the files in node_modules that were added and removed created a ton of noise, making code reviews unpleasant

In comparison, updating a third-party dependency with the Offline Mirror adds just a few files that are very easy to review:

$ yarn add shelljs@0.7.0 --dev
yarn add v0.23.2
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 4 new dependencies.
├─ interpret@1.0.1
├─ rechoir@0.6.2
└─ shelljs@0.7.0
│  └─ glob@7.1.1
✨  Done in 8.15s.

$ git diff
diff --git a/package.json b/package.json
index 4619f16..7acb42f 100644
--- a/package.json
+++ b/package.json
@@ -220,7 +220,7 @@
     "mock-fs": "^3.11.0",
     "portfinder": "0.4.0",
     "react": "~15.3.1",
-    "shelljs": "0.6.0",
+    "shelljs": "0.7.0",
     "sinon": "^2.0.0-pre.2"
   }
 }
diff --git a/yarn.lock b/yarn.lock
index 11ce116..f5d81ba 100644
--- a/yarn.lock
+++ b/yarn.lock
...
-shelljs@0.6.0:
-  version "0.6.0"
-  resolved https://registry.yarnpkg.com/shelljs/-/shelljs-0.6.0.tgz#ce1ed837b4b0e55b5ec3dab84251ab9dbdc0c7ec
+shelljs@0.7.0:
+  version "0.7.0"
+  resolved https://registry.yarnpkg.com/shelljs/-/shelljs-0.7.0.tgz#3f6f2e4965cec565f65ff3861d644f879281a576
+  dependencies:
+    glob "^7.0.0"
+    interpret "^1.0.0"
+    rechoir "^0.6.2"

 shellwords@^0.1.0:
   version "0.1.0"

$ git status
On branch testing-yarn
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   package.json
    modified:   yarn.lock

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    yarn-offline-mirror/interpret-1.0.1.tgz
    yarn-offline-mirror/rechoir-0.6.2.tgz
    yarn-offline-mirror/shelljs-0.7.0.tgz

no changes added to commit (use "git add" and/or "git commit -a")

Did you know that Yarn is also distributed as a single bundle JS file in releases that can be used on CI systems without internet access?

**yarn-.js** (for Node 5+) and **yarn-legacy-.js** (for Node 4) can be used stand-alone in CI systems without a need to install it.

Just check it into your project’s repository and use it in the build script:

node ./yarn-0.23.2.js install

This is quite convenient for teams that use multiple operating systems and want to have atomic updates for Yarn.

This is going to get better

The “Offline mirror” was implemented early in Yarn’s development cycle and we are working on improving it in a backwards compatible way:

The resolved field is used both for offline mirror paths and registry URIs. This means that the yarn.lock file that React Native team uses internally can’t be shared with the open source community because the React Native team does not sync the offline mirror with the open source version of React Native. Issue.
There is an improved workflow being considered for future versions of Yarn. It is not drastically different but some settings and lock files may change.