Yarn

Lockfiles should be committed on all projects

Thu, 24 Nov 2016 08:00:00 +0000

Yarn is a new package manager that we built to be consistent and reliable. When installing hundreds or even thousands of third-party packages from the internet you want to be sure that you’re executing the same code across every system.

Yarn maintains consistency across machines in two key ways:

Yarn uses a deterministic algorithm that builds up the entire dependency tree before placing files where they need to be.
Important info from the install process is stored in the yarn.lock lockfile so that it can be shared between every system installing the dependencies.

This lockfile contains information about the exact versions of every single dependency that was installed as well as checksums of the code to make sure the code is identical.

Yarn needs this info about every dependency because packages are constantly changing. New versions of packages are published all the time and since package.json specifies version ranges you need to lock them down to a single version.

These version ranges exist because Yarn follows Semantic Versioning, or “SemVer”. SemVer is a versioning system designed around “breaking” or “non-breaking” changes.

When you have a version such as v1.2.3, it’s broken into three parts:

Major (1.x.x) – Changes that may cause user code to break
Minor (x.2.x) – Changes that add new features (but should not break user code)
Patch (x.x.3) – Changes that are fixing bugs in previous versions (but do not add new features and should not break user code)

When a package publishes a new version, the author bumps major, minor, or patch based on the changes that they have made as a way to communicate them.

Users of packages should typically welcome minor and patch versions but should be wary of major versions as they could break your code.

Version ranges are a way of specifying which types of changes you want to accept and which versions you want to prevent. These version ranges then resolve down to a single version which is either the version you have installed or the latest published version that matches your version range.

If you don’t store which version you ended up installing, someone could be installing the same set of dependencies and end up with different versions depending on when they installed. This can lead to “Works On My Machine” problems and should be avoided.

Also, since package authors are people and they can make mistake, it’s possible for them to publish an accidental breaking change in a minor or patch version. If you install this breaking change when you don’t intend to it could have bad consequences like breaking your app in production.

Lockfiles lock the versions for every single dependency you have installed. This prevents “Works On My Machine” problems, and ensures that you don’t accidentally get a bad dependency.

It also works as a security precaution: If the package author is either malicious or is attacked by someone malicious and a bad version is published, you do not want that code to end up running without you knowing about it.

Libraries vs Applications

There are two primary types of projects that use Yarn:

Libraries – Projects that get published as packages to the registry and installed by users. (i.e. React or Babel)
Applications – Projects that only consume other packages, typically building some kind of product. (i.e. Your company’s app)

For applications, most developers agree that lockfiles are A Good Idea™. But there has been some question about using them when building libraries.

When you publish a package that contains a yarn.lock, any user of that library will not be affected by it. When you install dependencies in your application or library, only your own yarn.lock file is respected. Lockfiles within your dependencies will be ignored.

It is important that Yarn behaves this way for two reasons:

You would never be able to update the versions of sub-dependencies because they would be locked by other yarn.lock files.
Yarn would never be able to fold (de-duplicate) dependencies so that compatible version ranges only install a single version.

Some have wondered why libraries should use lockfiles at all if they do not and should not affect users. Even further, some have said that using lockfiles when developing libraries creates a false sense of security since your users could be installing different versions than you.

This seems to logically makes sense, but let’s dive deeper into the problem.

Development Dependencies

So far we’ve been talking about dependencies as if there were only one type of dependency when in fact there are several different types. These are broken down into two categories:

Runtime – Dependencies that are used by the project’s code and needed when the code is run.
Development – Dependencies that are only needed to work directly on the project

When a library is installed by a user, only the runtime dependencies are installed. The development dependencies are only ever installed when working directly on the project that specifies them.

Each type of dependency creates a whole tree of dependencies which are needed for them to be used.

It turns out that most projects (libraries or applications) have far more dependencies for development than for runtime. The tree of dependencies created by a projects development dependencies is almost always the largest part of the total.

You can blame this on how frustratingly complicated JavaScript development is, but it seems to hold true across every ecosystem. You almost always need more code to develop projects than you need to run them.

When working on a library, it is far more likely that a development dependency breaks just because there are more of them that could break.

The Breaking Change Race

When a package accidentally publishes a breaking change it starts the clock on who will be the first person to catch it. Whoever installs that breaking change first will (most likely) be the first to discover it.

Let’s imagine we have a package called left-pad which takes a string and adds a specified amount of padding in front of it. This small and seemingly harmless package is used by a really big project, which we’ll just call… Babel.

One day, the maintainer of left-pad decides they are going to do something unspeakably evil and make it pad the right side instead. They publish it as a patch version which quickly spreads to everyone using it.

Remember, Babel is a really huge project with tens of thousands of users and dozens of contributors. Let’s try and figure out who will catch the padding fiasco first.

Looking at build history, Babel was installed in CI (with the left-pad dependency) exactly 103 times in the last 30 days.
Looking at statistics from the registry, Babel was installed (with left-pad) by users over 5.5 Million times in the same 30 days.

To put that in a different measurement:

Contributors install Babel on average every 7 hours
Users install Babel about on average every 0.5 seconds

When the left-pad incident happened, users discovered it almost immediately. Notifying the Babel contributors via GitHub issues, tweets, Facebook messages, emails, phone calls, smoke signals, and by carrier pigeon.

Babel contributors couldn’t possibly prevent users from being affected by every error. In order to do so, every 0.5 seconds they would need to be able to run CI, notice the error, find a fix, publish a new version, and get every user to upgrade. There’s just no way to make this happen.

This might seem like a bit too extreme of an example to apply broadly, but the reasoning stays the same just with different numbers. Libraries should always be installed more frequently by users than by contributors. If that isn’t true, it’s because no one uses that library anyways– and we should not be optimizing our workflow around code that no one uses.

User testing

Many library developers work very hard to maintain a test suite that has 100% code coverage. This coverage ensures that every single line of code is run at least once during the tests.

But it still does not catch everything. If a library has even a small number of users, it will be far better tested by the users than it will be by any test suite the contributors can come up with.

Users just write more code, and the code they write is not always what a library author will expect. The more popular the library the more edge cases users will find. Users will do things that you didn’t even think was possible– it will horrify you. It will make you question the goodness of humanity.

Even if a contributor happened to beat users to finding a breaking change, there’s a pretty decent chance they won’t catch it anyways. Untested edge cases are the most likely things to break.

The contributor’s burden

Now let’s talk a bit about the social side of not using lockfiles in libraries.

As mentioned previously, the majority of breaking changes that occur in library dependencies will be development dependencies.

Some percentage of these breaking changes will be caught and (hopefully) fixed by regular contributors. However, the remaining breaking changes will be caught by new or infrequent contributors.

Contributing to a project for the first time is a very intimidating experience for anyone who is not an open source veteran. If the first thing a potential new contributor runs into is a broken build or test suite, they could be so intimidated that they decide to forget about contributing back.

Even as someone who contributes to lots of open source projects, it’s a terrible thing to have to debug a build system for a few hours when you just want to fix a bug or add a small feature to a library.

The user’s burden

Of course not every breaking change is a development dependency, and theoretically contributors could catch breaking changes before users notice them. In that (narrow) scenario, aren’t we shifting the burden from contributors to users?

First, it’s not going to be every user that is affected by this. It’s well agreed upon that applications should be using lockfiles, and if they are then they won’t be affected by sudden breaking changes.

We’re only talking about users that are either installing a package for the first time, or are upgrading the versions of their dependencies.

For users upgrading their dependencies, they should be trained to look out for breaking changes and if they encounter one, to roll back the version to a working state and open an issue in the library.

The only really negative experience that we are adding is for new users. Which is admittedly terrible, you want new users to have the best possible experience.

But remember that they already face this burden in the majority of cases. It’s only when contributors could have caught breaking changes before users experienced them that we are now placing an additional burden on new users.

It’s also important to note that new users make up a minority of the total installations. The majority of the time it will be existing users that are the first to catch breaking changes because they are the ones installing a library the most.

In Closing

There is a simple universal rule that everyone should follow with Yarn: If you are installing a new dependency or upgrading an existing one you should check to make sure the package works as intended and is not breaking your code.

You should follow that rule regardless of what your libraries are doing.

Without lockfiles it gets even more complicated: In applications or libraries, if there is no lockfile, you will have to check the dependencies every time you install or re-install them and make sure that everything still works. Otherwise the build might be broken or the tests might fail. You could break something without even realizing it. You could run into situations where code works on your machine but no one else’s.

The idea that not using lockfiles in libraries somehow saves users from encountering breaking changes is at best extremely rare and at worst is never true.

By not using lockfiles in libraries, the only tangible thing you accomplish is making the project harder to contribute to.

We should work as a community to keep all of our libraries up to date. We should go out and build tooling for automatically upgrading dependencies that makes it painless to do. There has been attempts at such tooling before, but we can do better.

Please commit your yarn.lock files.

Running Yarn offline

Thu, 24 Nov 2016 08:00:00 +0000

Repeatable and reliable builds for large JavaScript projects are vital. If your builds depend on dependencies being downloaded from network, this build system is neither repeatable nor reliable.

One of the main advantages of Yarn is that it can install node_modules from files located in file system. We call it “Offline Mirror” because it mirrors the files downloaded from registry during the first build and stores them locally for future builds.

“Offline mirror” is different from cache that both npm CLI and Yarn have. Caches store already unzipped tarballs downloaded from registry, they also can be implementation specific and may be invalid between multiple versions of CLI tools. The tarballs in “Offline mirror” can be consumed by any version Yarn that will build cache based on them. It is also easier to store files when they are compressed.

Let’s set up “Offline mirror” for a simple JS project

{
  "name": "yarn-offline",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "is-array": "^1.0.1",
    "left-pad": "^1.1.3",
    "mime-types": "^2.1.13"
  }
}

When you run yarn install, the generated yarn.lock file has sections for every dependency:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1

is-array@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/is-array/-/is-array-1.0.1.tgz#e9850cc2cc860c3bc0977e84ccf0dd464584279a"

left-pad@^1.1.3:
  version "1.1.3"
  resolved "https://registry.yarnpkg.com/left-pad/-/left-pad-1.1.3.tgz#612f61c033f3a9e08e939f1caebeea41b6f3199a"

mime-db@~1.25.0:
  version "1.25.0"
  resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.25.0.tgz#c18dbd7c73a5dbf6f44a024dc0d165a1e7b1c392"

mime-types@^2.1.13:
  version "2.1.13"
  resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.13.tgz#e07aaa9c6c6b9a7ca3012c69003ad25a39e92a88"
  dependencies:
    mime-db "~1.25.0"

Each of these dependencies have a resolved field with a remote URL. If you delete your node_modules folder and run yarn install again, Yarn will download the same resolved dependencies specified in this lockfile. It will even guarantee that no one modified the files since your first installs by verifying checksum for each of them.

However, if for some reason these urls are unreachable during your build, it will fail. To solve this, we’ll need an “Offline mirror”.

Set up .yarnrc

First we need to setup a directory to be our “Offline mirror” storage, we can do that with yarn config command:

$ yarn config set yarn-offline-mirror ./npm-packages-offline-cache
yarn config v0.17.8
success Set "yarn-offline-mirror" to "./npm-packages-offline-cache".
✨  Done in 0.06s.

./npm-packages-offline-cache is an example location relative to home folder where all the source .tar.gz files will be downloaded to from the registry.

This will create a .yarnrc file in your HOME directory. Let’s move this file to the project root so that offline mirror would be used only for this project.

$ mv ~/.yarnrc ./

Initialize the new lockfile

Remove node_modules and yarn.lock that were generated previously and run yarn install again:

$ rm -rf node_modules/ yarn.lock
$ yarn install
yarn install v0.17.8
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 0.57s.

The dependency resolutions in your yarn.lock should now look like this:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1

is-array@^1.0.1:
  version "1.0.1"
  resolved is-array-1.0.1.tgz#e9850cc2cc860c3bc0977e84ccf0dd464584279a

left-pad@^1.1.3:
  version "1.1.3"
  resolved left-pad-1.1.3.tgz#612f61c033f3a9e08e939f1caebeea41b6f3199a

mime-db@~1.25.0:
  version "1.25.0"
  resolved mime-db-1.25.0.tgz#c18dbd7c73a5dbf6f44a024dc0d165a1e7b1c392

mime-types@^2.1.13:
  version "2.1.13"
  resolved mime-types-2.1.13.tgz#e07aaa9c6c6b9a7ca3012c69003ad25a39e92a88
  dependencies:
    mime-db "~1.25.0"

The resolved field for each resolution should now contain file names relative to the npm-packages-offline-cache folder that was configured earlier. Each resolved dependency also contains a checksum after the file name to ensure that no one mangles with the downloaded files.

And inside the “Offline mirror” folder we have the .tgz files that yarn will use for the following builds without reaching out to network.

$ ls npm-packages-offline-cache/
is-array-1.0.1.tgz    left-pad-1.1.3.tgz    mime-db-1.25.0.tgz    mime-types-2.1.13.tgz

In a nutshell, to enable “Offline mirror” for your project you need:

add “yarn-offline-mirror” configuration to .yarnrc file

generate a new yarn.lock with “yarn install” command

A few tips and tricks

You can check in “Offline mirror” into git or mercurial repository

The “Offline Mirror” can be shared between build servers or development machines in any way that is convenient: a Dropbox folder, stored in source control or on a network drive. At Facebook the offline mirror lives inside of our big Mercurial “monorepo”.

Whether to commit binary files into a repository or not depends on the number and size of your project’s dependencies. For example, out of 849 React Native dependencies totaling 23MB, only 10% are larger than 30KB.

Many Facebook teams, including the React Native team, decided to check in their “Offline mirror”. They all share the same “Offline mirror” which means that most dependencies for new projects are often already checked into that folder, so the cost of storing the packages in source control gets lower the more projects use it.

Let’s compare checking in node_modules to checking in “Offline mirror”

The React Native team used to check in the node_modules folder but they hit several limits:

node_modules contains more than 37,000 files (and more than 100,000 files back when we were using npm2). This had a bad performance impact on our Mercurial repository.
Reviewing Pull Requests that changed a dependency was quite hard as all the files in node_modules that were added and removed created a ton of noise, making code reviews unpleasant

In comparison, updating a third-party dependency with the Offline Mirror adds just a few files that are very easy to review:

$ yarn add shelljs@0.7.0 --dev
yarn add v0.17.8
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 4 new dependencies.
├─ interpret@1.0.1
├─ rechoir@0.6.2
└─ shelljs@0.7.0
│  └─ glob@7.1.1
✨  Done in 8.15s.

$ git diff
diff --git a/package.json b/package.json
index 4619f16..7acb42f 100644
--- a/package.json
+++ b/package.json
@@ -220,7 +220,7 @@
     "mock-fs": "^3.11.0",
     "portfinder": "0.4.0",
     "react": "~15.3.1",
-    "shelljs": "0.6.0",
+    "shelljs": "0.7.0",
     "sinon": "^2.0.0-pre.2"
   }
 }
diff --git a/yarn.lock b/yarn.lock
index 11ce116..f5d81ba 100644
--- a/yarn.lock
+++ b/yarn.lock
...
-shelljs@0.6.0:
-  version "0.6.0"
-  resolved shelljs-0.6.0.tgz#ce1ed837b4b0e55b5ec3dab84251ab9dbdc0c7ec
+shelljs@0.7.0:
+  version "0.7.0"
+  resolved shelljs-0.7.0.tgz#3f6f2e4965cec565f65ff3861d644f879281a576
+  dependencies:
+    glob "^7.0.0"
+    interpret "^1.0.0"
+    rechoir "^0.6.2"

 shellwords@^0.1.0:
   version "0.1.0"

$ git status
On branch testing-yarn
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   package.json
    modified:   yarn.lock

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    yarn-offline-mirror/interpret-1.0.1.tgz
    yarn-offline-mirror/rechoir-0.6.2.tgz
    yarn-offline-mirror/shelljs-0.7.0.tgz

no changes added to commit (use "git add" and/or "git commit -a")

Did you know that Yarn is also distributed as a single bundle JS file in releases that can be used on CI systems without internet access?

**yarn-.js** (for Node 5+) and **yarn-legacy-.js** (for Node 4) can be used stand-alone in CI systems without a need to install it.

Just check it into your project’s repository and use it in the build script:

node ./yarn-0.17.8.js install

This is quite convenient for teams that use multiple operating systems and want to have atomic updates for Yarn.

This is going to get better

The “Offline mirror” was implemented early in Yarn’s development cycle and we are working on improving it in a backwards compatible way:

The resolved field is used both for offline mirror paths and registry URIs. This means that the yarn.lock file that React Native team uses internally can’t be shared with the open source community because the React Native team does not sync the offline mirror with the open source version of React Native. Issue.
There is an improved workflow being considered for future versions of Yarn. It is not drastically different but some settings and lock files may change.

Yarn: A new package manager for JavaScript

Tue, 11 Oct 2016 08:00:00 +0000

We’re pleased to announce the open source release of Yarn, a collaboration between Facebook, Exponent, Google, and Tilde. With Yarn, engineers still have access to the npm registry, but can install packages more quickly and manage dependencies consistently across machines or in secure offline environments. Yarn enables engineers to move faster and with confidence when using shared code so they can focus on what matters — building new products and features. Read the full announcement on code.facebook.com.