The npm Blog

How we deploy at npm

ceejbot — Wed, 19 Oct 2016 12:33:04 -0700

Here’s how we deploy node services at npm:

cd ~/code/exciting-service
git push origin +master:deploy-production

That’s it: git push and we’ve deployed.

Of course, a lot is triggered by that simple action. This blog post is all about the things that happen after we type a git command and press Return.

Goals

As we worked on our system, we were motivated by a few guiding principles:

deploying should be so easy there’s no reason not to;
rolling back a push should be as easy as the original push was;
our team is small: automate everything!
everything a service needs to start must be on the host with it;
each step is simple & useful on its own;
everything can be run by hand (or by a Slack bot!) if necessary;
every step can be run many times without harm.

Why? We want no barriers to pushing out code once it’s been tested and reviewed, and no barriers to rolling it back if something surprising happens — so any friction in the process should be present before code is merged into master, via a review process, not after we’ve decided it’s good. By separating the steps, we gain finer control over how things happen. Finally, making things repeatable means the system is more robust.

Thousand-foot view

What happens when you do that force-push to the deploy-production branch? It starts at the moment an instance on AWS is configured for its role in life.

We use Terraform and Ansible to manage our deployed infrastructure. At the moment I’m typing, we have around 120 AWS instances of various sizes, in four different AWS regions. We use Packer to pre-bake an AMI based on Ubuntu Trusty with most of npm’s operational requirements, and push it out to each AWS region.

For example, we pre-install a recent LTS release of node as well as our monitoring system onto the AMI. This pre-bake greatly shortens the time it takes to provision a new instance. Terraform reads a configuration file describing the desired instance, creates it, adds it to any security groups needed and so on, then runs an Ansible playbook to configure it.

Ansible sets up which services a host is expected to run. It writes a rules file for the incoming webhooks listener, then populates the deployment scripts. It sets up a webhook on GitHub for each of the services this instance needs to run. Ansible then concludes its work by running all of the deploy scripts for the new instance once, to get its services started. After that, it can be added to the production rotation by pointing our CDN at it, or by pointing other processes to it through a configuration change.

This setup phase happens less often than you might think. We treat microservices instances as disposable, but most of them are quite long-lived.

So our new instance, configured to run its little suite of microservices, is now happily running. Suppose you then do some new development work on one of those microservices. You make a pull request to the repo in the usual way, which gets reviewed by your colleagues and tested on Travis. You’re ready to run it for real!

You do that force-push to deploy-staging, and this is what happens: A reference gets repointed on the GitHub remote. GitHub notifies a web hooks service listening on running instances. This webhooks service compares the incoming hook payload against its configured rules, decides it has a match, & runs a deploy script.

Our deploy scripts are written in bash, and we’ve separated each step of a deploy into a separate script that can be invoked on its own. We don’t just invoke them through GitHub hooks! One of our Slack chatbots is set up to respond to commands to invoke these scripts on specific hosts. Here’s what they do:

pull code with git & run npm install
update configuration from our config db
roll services one process at a time

Each step reports success or failure to our company Slack so we know if a deploy went wrong, and if so at which step. We emit metrics on each step as well, so we can annotate our dashboards with deploy events.

Conventions

We name our deploy branches deploy-foo, so we have, for instance, deploy-staging, deploy-canary, and deploy-production branches for each repo, representing each of our deployment environments. Staging is an internal development environment with a snapshot of production data but very light load and no redundancy. Canary hosts are hosts in the production line that only take a small percentage of production load, enough to shake out load-related problems. And production is, as you expect, the hosts that take production traffic.

Every host runs a haproxy, which does load balancing as well as TLS termination. We use TLS for most internal communication among services, even within a datacenter. Unless there’s a good reason for a microservice to be a singleton, there are N copies of everything running on each host, where N is usually 4.

When we roll services, we take them out of haproxy briefly using its API, restart, then wait until they come back up again. Every service has two monitoring hooks at conventional endpoints: a low-cost ping and a higher-cost status check. The ping is tested for response before we put the service back into haproxy. A failure to come back up before a timeout stops the whole roll on that host.

You’ll notice that we don’t do any cross-host orchestration. If a deploy is plain bad and fails on every host, we’ll lose at most 1 process out of 4, so we’re still serving requests (though at diminished capacity). Our Slack operational incidents channel gets a warning message when this happens, so the person who did the deploy can act immediately. This level of orchestration has been good enough thus far when combined with monitoring and reporting in Slack.

You’ll also notice that we’re not doing any auto-scaling or managing clusters of containers using, e.g., Kubernetes or CoreOS. We haven’t had any problems that needed to be solved with that kind of complexity yet, and in fact my major pushes over the last year have been to simplify the system rather than add more moving parts. Right now, we are more likely to add copies of services for redundancy reasons than for scaling reasons.

Configuration

Configuration is a perennial pain. Our current config situation is best described as “less painful than it used to be.”

We store all service configuration in an etcd cluster. Engineers write to it with a command-line tool, then a second tool pulls from it and writes configuration at deploy time. This means config is frozen at the moment of deploy, in the upstart config. If a process crashes & restarts, it comes up with the same configuration as its peers. We do not have plans to read config on the fly. (Since node processes are so fast to restart, I prefer killing a process & restarting with known state to trying to manage all state in a long-lived process.)

Each service has a configuration template file that requests the config data it requires. This file is in TOML format for human readability. At deploy time the script runs & requests keys from etcd namespace by the config value, the service requesting the config, and the configuration group of the host. This lets us separate hosts by region or by cluster, so we can, for example, point a service at a Redis in the same AWS data center.

Here’s an example:

> furthermore get /slack_token/
slack_token matches:
/slack_token == xoxb-deadbeef
/slack_token.LOUDBOT == XOXB-0DDBA11
/slack_token.hermione == xoxb-5ca1ab1e
/slack_token.korzybski == xoxb-ca11ab1e
/slack_token.slouchbot == xoxb-cafed00d

Each of our chatbots has a different Slack API token stored in the config database, but in their config templates they need only say they require a variable named slack_token[1].

These config variables are converted into environment variable specifications or command-line options in an upstart file, controlled by the configuration template. All config is baked into the upstart file and an inspection of that file tells you everything you need to know.

Here’s LOUDBOT’s config template:

app = "LOUDBOT"
description = "YELL AND THEN YELL SOME MORE"
start = "node REAL_TIME_LOUDIE.js"
processes = 1

[environment]
  SERVICE_NAME = "LOUDBOT"
  SLACK_TOKEN = "{{slack_token}}"

And the generated upstart file:

# LOUDBOT node 0

description "YELL AND THEN YELL SOME MORE"

start on started network-services
stop on stopping network-services
respawn
setuid ubuntu
setgid ubuntu
limit nofile 1000000 1000000

script
    cd /mnt/deploys/LOUDBOT
    SERVICE_NAME="LOUDBOT" \
    SLACK_TOKEN="XOXB-0DDBA11" \
    node REAL_TIME_LOUDIE.js \
    >> logs/LOUDBOT0.log 2>&1
end script

This situation is vulnerable to the usual mess-ups: somebody forgets to override a config option for a cluster, or to add a new config value to the production etcd as well as to the staging etcd. That said, it’s at least easily inspectable, both in the db and via the results of a config run.

Open source

The system I describe above is sui generis, and it’s not clear that any of the components would be useful to anybody else. But our habit as an engineering organization is to open-source all our tools by default, so everything except the bash scripts is available if you’d find it useful. In particular, furthermore is handy if you work with etcd a lot.

jthooks - add and remove webhooks from a Github repo
jthoober - handle incoming Github webhooks to run shell scripts
etcetera - read configuration from etcd & populate an upstart template
furthermore - get & set etcd keys conveniently, with regexp searching

[1] The tokens in this post aren’t real. And, yes, LOUDBOT’s are always all-caps.

Hello, Yarn!

izs — Tue, 11 Oct 2016 08:00:33 -0700

Today, Facebook announced that they have open sourced Yarn, a backwards-compatible client for the npm registry. This joins a list of other third-party registry clients that include ied, pnpm, npm-install and npmd. (Apologies if we missed any.) Yarn’s arrival is great news for npm’s users worldwide and we’re happy to see it.

Like other third-party registry clients, Yarn takes the list of priorities that our official npm client balances, and shifts them around a little. It also solves a number of problems that Facebook was encountering using npm at their unique global scale. Yarn includes another take on npm’s shrinkwrap feature and some clever performance work. We’ve also been working on these specific features, so we’ll be paying close attention.

Does it work with npm?

Mostly! We haven’t had time to run extensive tests on the compatibility of Yarn, but it seems to work great with public packages. It does not authenticate to the registry the way the official client does, so it’s currently unable to work with private packages. The Yarn team is aware of this issue and have said they’ll address it.

Is Facebook forking the community?

Whenever a big company gets involved in an open source project, there’s some understandable anxiety from the community about its intentions.

Yarn publishes to npm’s own registry by default, so Yarn users continue to be part of the existing community and benefit from the same 350,000+ packages as users of the official npm client. Yarn pulls packages from registry.yarnpkg.com, which allows them to run experiments with the Yarn client. This is a proxy that pulls packages from the official npm registry, much like npmjs.cf.

Like so many other companies around the world, Facebook benefits from the united open source JavaScript community on npm.

This is how open source works

As I said at the start, we’re happy to see Yarn join the ranks of open source npm clients. This is how open source software is supposed to work!

The developers behind Yarn — Seb, James, Christoph, and Konstantin — are prolific publishers of npm packages and pillars of the npm community.

Through their efforts, Facebook and others have put a lot of developer time into this project to solve problems they encountered. Sharing the fruits of their labor will allow ideas and bugfixes to flow back and forth between npm’s official client and all the others. Everyone benefits as a result.

Yarn also shows that one of the world’s largest tech companies, which is already behind hugely popular JavaScript projects like React, is invested in and committed to the ongoing health of the npm community. That’s great news for JavaScript devs everywhere.

We’re pleased to see Yarn get off to such a great start, and look forward to seeing where it goes.

out with the newww and in with the www: an update on the npm website

agdubs — Thu, 28 Jul 2016 15:50:35 -0700

From its inception, npm has been keenly focused on open source values. As we’ve grown as a company, however, we’ve learned the important lesson that making source code available under an open license is the bare minimum for open source software. To take it even further, we’ve also learned that “open source” doesn’t necessarily mean community-driven. With these insights in mind, the web team has decided to make some changes to the community interface of npm’s website — with the goal of creating a more efficient and effective experience for everyone involved.

Summary

npm/newww is being retired and made private
npm/www has been created for new issues and release notes

Sunsetting `npm/newww`

As you may (or may not!) have noticed, the repo that used to home npm’s website (npm/newww) isn’t in sync with the production website (http://www.npmjs.com).

A few months back, the team made the executive decision to close source the npm website. There were several reasons for this:

Running the npm website locally was/still is very difficult (even for npm members!) due to a dependency on private API endpoints
Open source tier Travis builds were holding back development pace

This was a super tough call, and there were strong arguments from both sides. In the end, though, the team reached a unified understanding that this was both the best call for the company and for the community. The repo will be officially shutting down tomorrow, Friday, July 29, 2016.

One of the things we’re aware of is that many in the Node community were using the website as an example repo for using the Hapi framework. While we’re completely flattered by this, we honestly don’t believe the codebase is currently in a state to serve that role — it’s a katamari of many practices over many years rolled into one right now!

That being said, we do care about sharing our work with the world, and intend to and are excited to publish many of the website components as packages that will be open sourced and reusable.

Introducing `npm/www`

In place of the npm/newww repo, we’ve created npm/www! The goals of this repo are to give the community a place to:

File issues about display and performance bugs on the website
Request new features of the website
Publish release notes about the website

While the source code for the website will no longer be available, the hope is that this new repo can be a more effective way to organize and respond to the needs the community has. We’re super excited to hear your thoughts, questions, and concerns — head over to npm/www now so we can start collaborating!

npm has a new CTO

seldo — Mon, 18 Jul 2016 11:46:44 -0700

I don’t want to bury the lede, so here it is: npm has a new CTO, and her name is CJ Silverio. My title is changing from CTO to COO, and I will be taking over a range of new responsibilities, including modeling our business, defining and tracking our metrics, and bringing that data to bear on our daily operations, sales, and marketing. This will allow Isaac to concentrate more on defining the product and strategy of npm as we continue to grow.

CJ will be following this post with a post of her own, giving her thoughts about her new role. I could write a long post about how awesome CJ is — and she is awesome — but that wouldn’t achieve much more than make her embarrassed. Instead I thought I’d take this chance to answer a question I get a lot, before I forget the answer:

What does a CTO do, anyway?

The answer is that it depends on the needs of the company. npm has grown from 3 people to 25 in the past 2.5 years, and in that time my job changed radically from quarter to quarter. Every time I got the hang of the job, the needs of the company would shift and I found myself doing something new. So this is my list of some of the things a CTO might do. Not all of them are a good idea, as you’ll see. The chronological order is an over-simplification: I was doing a small piece of all of these tasks all the time, but each quarter definitely had a focus, so I’ve talked about it then.

Q1 2014: just another ops engineer.

Started this quarter: CJ, Raquel.

npm Inc had a bumpy launch: the registry was extremely unstable, because it was running on insufficient hardware and had not been architected for high uptime. Our priority was to get the registry to stay up. I was spinning up hardware by hand, without the benefit of automation. By April we had found the hot spots and mostly met the load, but CJ was the first person to stridently make the case that we had to automate our way out of this. I handed operations to her.

Q2 2014: architect.

Started: Ben, Forrest, Maciej.

Once the fires were out, we could finally think about building products, and we had a choice: do we build a paid product on top of the current (highly technically indebted) architecture, or build a new product and architecture? We decided on a new, modular architecture that we could use to build npm Enterprise first, and then extend later to become “Registry 2.0”. Between recruitment, management, and other duties, I discovered by the end of the quarter that it was already impossible to find time to write code.

Q3 2014: manager

This was the quarter we dug in and built npm Enterprise. My job became primarily that of an engineering manager: keeping everybody informed about what everybody else was up to, assigning tasks, deciding priorities of new work vs. sustaining and operational work, and handling the kind of interpersonal issues which every growing company experiences. I found I was relying on CJ a lot when solving these kinds of problems.

Q4 2014: sales and evangelism

Started: Rebecca

With npm Enterprise delivered to its first customer, we started learning how to sell it. I went to conferences, gave talks, went to meetings and sales calls, wrote documentation and blog posts, and generally tried to make noise. I was never particularly good at any of this, so I was grateful when Ben took over npm Enterprise as a product, which started around this time.

Q1 2015: data scientist

In February 2014 I had written the system that to this day serves our download counts, but we were starting a process of raising our series A, and that data wasn’t good enough. I dredged up my Hadoop knowledge from a previous job and started crunching numbers, getting new numbers we hadn’t seen before, like unique IP counts and other trends. This is one job I’m keeping as I move to COO, since measuring these metrics and optimizing them is a big part of my new role.

Q2 2015: recruiter

Started: Ernie, Ryan

We’d been hiring all the time, of course, but we closed our series A in Q1 2015, so there was a sudden burst of recruitment at this time, most of whom didn’t actually start until the next quarter. By the end of this process we’d hired so many people that I never had to do recruitment again: the teams were now big enough to interview and hire their own people.

Q3 2015: strategic planner

Started: Kat, Stephanie, Emily, Jeff, Chris, Jonathan, Aria, Angela

With so many new people, we had a sudden burst of momentum, and it became necessary for the first time to devote substantial effort to planning “what do we do next?” Until this point the next move had been obvious: put out the fire, all hands on deck. Now we had enough people that some of them could work on longer-term projects, which was good news, but meant we had to pull our heads up and think about the longer term. To accomplish this, I handed management of nearly all the engineering team to CJ, who became VP of engineering.

Q4 2015: project manager

Started: Ashley, Andrea, Andrew (yes, it was confusing)

We had already launched npm Private Modules, a single-user product, but it hadn’t really taken off. We were sure we knew why: npm Organizations, a product for teams, was demanded by nearly everybody. It was a lot more complicated, and with more people there was a lot more coordination to do, so I started doing the kind of time, tasks, and dependency management of a project manager. I will be the first to admit that I was not particularly good at it, and nobody was upset when I mostly gave this task to Nicole the following quarter. We launched Orgs in November, and it was an instant hit, becoming half of npm’s revenue by the end of the year.

Q1 2016: head of product

Started: Nicole, Jerry

Now with two product lines and a bunch of engineers, fully defining what the product should do (or not do), and what the next priority was, became critical. Isaac was too busy CEO’ing to do this, so he gave it to most available person: me. This was not a huge success, partly because I was still stuck in project management mode, which is a very different thing, and partly because I’m just not as creative as Isaac when it comes to product. Everybody learned something, even if it was “Laurie isn’t very good at this”.

Q2 2016: interim CEO

Started: Kiera

Isaac’s baby was born on April 1st (a fact that, combined with his not having mentioned they were even expecting a baby, led many people to assume at first that his announcement of parenthood was a joke). He went on parental leave for most of Q2, so I took over as interim CEO. CJ, already VP of eng, effectively started being CTO at this time.

Today: COO

When Isaac came back from parental leave, we’d learned some things: I had, of necessity, handled the administrative and executive functions of a CEO for a quarter. CJ had handled those of a CTO. We now had two people who could be CTO, and one overloaded CEO with a talent for product. The course of action was obvious: Isaac handed over everything he could that was not-product to me, to focus on product development, while I handed over CTO duties to CJ. We needed a title for “CEO stuff that isn’t product” and picked COO mostly because it’s a title people recognize.

All hail CJ

You’ll notice a common thread, which is that as I moved to new tasks I was mostly handing them to CJ. Honestly, it was pretty clear to me from day 1 that CJ was just as qualified to be CTO as I was, if not more — she has an extra decade’s worth of experience on me and is a better engineer to boot. The only thing she lacked was the belief that she could, and over the last two and a half years it has been a pleasure watching her confidence grow as she’s mastered every new challenge I put in front of her, and more than a little funny watching her repeatedly express surprise at her ability to do all these things. It’s been like slowly persuading an amnesiac Clark Kent that he is, in fact, Superman.

I’ve often referred to CJ as npm’s secret weapon. Well, now the secret is out. npm has the best CTO I could possibly imagine, and I can’t wait to see what she does next.

package tarball read outage today

ceejbot — Wed, 06 Jul 2016 14:45:04 -0700

Earlier today, July 6, 2016, the npm registry experienced a read outage for 0.5% of all package tarballs for all network regions. Not all packages and versions were affected, but the ones that were affected were completely unavailable during the outage for any region of our CDN.

The unavailable tarballs were offline for about 16 hours, from mid-afternoon PDT on July 5 to early morning July 6. All tarballs should now be available for read.

Here’s the outage timeline:

Evening of 2016 July 1: We pushed code that implemented a change in how we generate etags for tarballs for for new publications only.
Midday 2016 July 6: We commenced running a script that updated all existing tarballs on all servers to the new etags scheme. This ran over the course of some hours.
10pm PDT 2016 July 5: We received a report of some 502s being returned by some servers. In this course of investigating this, we observed a known issue with a very slow process leak in an existing service. We cleared up the known issue in the mistaken belief that this was causing the 502s.
6am PDT 2016 July 6: We received more reports of 502s on tarballs, and opened a status incident.
8am PDT: The root cause of the issue was identified as a negative file modification times for the specific affected tarballs. A new script was immediately run to fix mtimes on all tarball files appearing in the logs as producing 502 errors.

Over the next hour 502 rates fell to the normal 0.

What we’re doing to prevent outages like this next time

We’re adding an alert on all 500-class status codes, not just 503s. This alert will catch the category of errors, not simply this specific problem.

We’re also revising our operational playbook to encourage examination of our CDN logs more frequently; we could have caught the problem very soon after introducing it if we had carefully verified that our guess about the source of 502s had resulted in making them vanish from our CDN logging. We can also do better with tools for examining the patterns of errors across POPs, which would have made it clearer to us immediately that the error was not specific to the US East coast and was therefore unlikely to have been caused by an outage in our CDN.

Read on if you would like the details of the bug.

nginx and etags

The root cause for this outage was an interesting interaction of file modification time, nginx’s method of generating etags, and cache headers.

We recently examined our CDN caching strategies and learned that we were not caching as effectively as we might, because of a property of nginx. Nginx’s etags are generated using the file modification time as well as its size, roughly as mtime + '-' + the file size in bytes. This meant that if mtimes for package tarballs varied across our nginx instances, our CDN would treat the files from each server as distinct, and cache them separately. Getting the most from our CDN’s caches and from our users’ local tarball caches is key to good performance on npm installs, so we took steps to make the etags match across all our services.

Our chosen scheme was to set the file modification time to the first 32-bit BE integer from their md5 hash. This was entirely arbitrary but looked sufficient after testing in our staging environment. We produced consistent etags. Unfortunately, the script that applied this change to our production environment failed to clamp the resulting integer, resulting in negative numbers for timestamps. Ordinarily, this would result in a the infamous Dec 31, 1969 date one sees for timestamps before the Unix epoch.

Unfortunately, negative mtimes triggered an nginx bug. Nginx will serve the first request for a file in this state and deliver the negative etag. However, if there is a negative etag in the if-none-match header nginx attempts to serve a 304 but never completes the request. This resulted in the the bad gateway message returned by our CDN to users attempting to fetch a tarball with the bad mtime.

You can observe this behavior yourself with nginx and curl:

The final request never completes even though nginx has correctly given it a 304 status.

Because this only affected a small subset of tarballs, not including the tarball fetched by our smoketest alert, all servers remained in the pool. We have an alert on above-normal 503 error rates served by our CDN, but this error state produced 502s and was not caught.

All the tarballs that were producing a 502 gateway timeout error turned out to have negative timestamps in their file mtimes. The fix was to touch them all so their times were inconsistent across our servers but valid, thus both busting our CDN’s cache and dodging the nginx behavior.

tl;dr

The logs from our CDN are invaluable, because they tell us what quality of service our users are truly experiencing. Sometimes everything looks green on our own monitoring, but it’s not green from our users’ perspective. The logs are how we know.

Introducing add-ons for npm Enterprise

bencoe — Tue, 05 Jul 2016 05:58:57 -0700

Today, npm Enterprise has just grown hugely more extensible and powerful with the release of npm Enterprise add-ons.

It’s now possible to integrate third-parties’ developer tools directly into npm Enterprise. This has the power to combine what were discrete parts of your development workflow into a single user experience, and knock out the barriers that stand in the way of bringing open source development’s many-small-reusable-parts methodology into larger organizations.

What are npm Enterprise add-ons?

npm Enterprise now exposes an API that allows third-party developers to build on top of our npm Enterprise product:

add additional UI components to enterprise customers’ private npm websites;
add hooks to package installation and publishing events;
automate important parts of the software development lifecycle — for example, to automatically apply a security scan when you attempt to install a package for the first time.

With this deceptively simple functionality, developers can offer a huge amount of value to enrich the process of using npm within the enterprise.

Why?

Enterprise developers already want to take advantage of the same code discovery, re-use, and collaboration enjoyed by millions of open source developers, billions of times every month. But this requires accommodating their companies’ op-sec, licensing, and code quality processes, which often predate the modern era.

For example…

Enterprises have strict security restrictions.

In the past, it was possible to manually research the security implications of external code. But with the average npm package relying on over 100 dependencies and subdependencies, this process just doesn’t scale.

Without a way to ensure the security of each package, a company can’t take advantage of open source code.

Enterprises strictly enforce license compliance.

Software that is missing a license, or that’s governed by a license unblessed by a company’s legal department, simply can’t be used at larger companies. Much like security screening, many companies have relied upon manually reviewing the license requirements of each piece of external code. And just like security research, trying to manually confirm the licensing of every dependency (and their dependencies, and their dependencies…) is impossible to scale.

Enterprise developers need a way to understand the license implications of packages they’re considering using, and companies need a way to certify that all of their projects are legally kosher.

Enterprises depend on good software development practices.

Will bug reports be patched quickly? Is the code written well? Do packages rely on stale or abandoned dependencies? These questions demand answers before an enterprise can consider relying on open source code.

Without a way to quantitatively analyze the quality of every code package in a project, many enterprise teams simply don’t adopt open source code or workflows for mission-critical projects.

Our three launch partners, Node Security Platform, FOSSA, and bitHound, address these concerns, respectively.

You can learn about the specifics of each of them here:

By integrating them directly into the tool that enterprise developers use to browse and manage packages, we make it as easy as possible to scratch enterprise development’s specific itches. As more incredible add-ons join the platform, the barriers to open source-style development at big companies get knocked down, one by one.

Now what?

The Node Security Platform, FOSSA, and bitHound add-ons are available to existing npm Enterprise customers today. Simply contact us at support@npmjs.com to get set up.

If you’re looking to bring npm Enterprise and add-ons into your enterprise, let us show you how easy it is with a free 30-day trial.

Interested in building your own add-on? Awesome. Stay tuned: API documentation is on its way.

This is just the beginning

The movement to bring open source code, workflows, and tools into the enterprise is called InnerSource, and it’s the beginning of a revolution.

When companies develop proprietary code the same way communities build open source projects, then the open source community’s methods and tooling become the default way to build software.

Everyone stands to benefit from InnerSource because everyone stands to benefit from building software the right way: open source packages see more adoption and community participation, companies build projects faster and cheaper without re-inventing wheels, and developers are empowered to build amazing things.

Add-ons are an exciting step forward for us. We’re thrilled you’re joining us.

dealing with problematic dependencies in a restricted network environment

nexdrew — Fri, 10 Jun 2016 14:38:30 -0700

When using npm Enterprise, we sometimes encounter public packages in our private registry that need to fetch resources from the public internet when being installed by a client via npm install.

Unfortunately, this poses a problem for developers who work in an environment with limited or no access to the public internet.

Let’s take a look at some of the more common types of problems in this area and talk about ways we can work around them.

Note that these problems are not specific to npm Enterprise — but to using certain public packages in any limited-access environment. That being said, there are some things that npm (as an organization and software vendor) can do to better prevent or handle some of these problems. We’re still working to make these improvements.

Diagnosing the Problem

Typically, developers will discover the problem when installing packages from their private registry. When this happens, we need to determine the type of problem it is and where in the dependency graph the problematic dependency resides.

Problem Types

Here are some common problem types:

Git repo dependency

This is when a package dependency is listed in a package.json file with a reference to a Git repository instead of with a semver version range. Typically these point to a particular branch or revision in a public GitHub or Bitbucket repository. They are mainly used when the package contents have not been published to the public npm registry.

When the npm client encounters these, it attempts to fetch the package from the Git repository directly, which is a problem for folks who do not have network access to the repository.
Shrinkwrapped package

This is when the internal contents of a package contain an npm-shrinkwrap.json file that lists a specific version and URL to use for each mentioned package from the dependency tree.

During a normal npm install, the npm client attempts to fetch the dependencies listed in npm-shrinkwrap.json directly from the URLs contained in the file. This poses a problem when the client installing the shrinkwrapped package does not have access to the URLs that the shrinkwrap author has access to.
Package with install script or node-gyp dependency

This is when a package attempts to defer some setup process until the package is installed, using a script defined in package.json, which typically involves building platform-specific binaries or Node add-ons on the client’s machine.

On a typical install, the npm client will find and run these scripts in order to automatically fetch and build the required resources, targeting the platform that the client is running on. But when limited internet access means the necessary resources cannot be fetched, the install will fail. Most likely the package will be unusable until the end result of running the install script on the client’s machine is achieved.

Direct vs Transitive

To determine the location of the problematic dependency, we can boil it down to two categories:

Direct dependency

A direct dependency is one that is explicitly listed in your own package.json file — a dependency that your project/package uses directly in code or in an npm run script.
Transitive dependency

A transitive dependency is one that is not explicitly listed in your own package.json file — a dependency that comes from anywhere in the tree of your direct dependencies’ dependencies.

Potential Solutions

The same way publishing a package to the public registry requires access to the public internet, most of these solutions require Internet access, at least on a temporary basis. Once the solution is in place, then access to public resources can be restricted.

For starters, remember that it’s generally a good idea to use the latest version of the npm client. To install or upgrade to the latest version, regardless of what version of Node you have installed, run npm i -g npm@latest (and make sure npm -v prints the version that was installed).

Let’s go over the problem types in more detail.

Replacing a Git dependency

Unfortunately, a dependency that references a Git repository (instead of a semver range for a published package) must be replaced with a published package. To do this, you’ll need to first publish the Git repository as a package to your npm Enterprise registry and then fork the project with the Git dependency and replace the dependency with the package you published. Then, publish the forked project, and use that package as a dependency (instead of the original).

It’s usually a good idea to open an issue on the project with the Git dependency, politely asking the maintainers to replace the Git dependency, if possible. Generally, we discourage using Git dependencies in package.json, and it’s typically only used temporarily while a maintainer waits for an upstream fix to be applied and published.

Example: let’s replace the "grunt-mocha-istanbul": "christian-bromann/grunt-mocha-istanbul" Git dependency defined in version 4.0.4 of the webdriverio package, assuming that webdriverio is a direct dependency and grunt-mocha-istanbul is a transitive dependency.

We’ll tackle this in two main steps: forking and publishing the transitive dependency, and forking and publishing the direct dependency.

Step 1: Fork-publish the transitive dependency

Clone the project that is referenced by the Git dependency

Optionally, you can create a remote fork first (e.g., in GitHub or Bitbucket) and then clone your fork locally. Otherwise, you can just clone/download the project directly from the remote repository. It’s a good idea to use source control so you can keep a history of your changes, but you could also probably get away with downloading and extracting the project contents.

Example:
```
git clone https://github.com/christian-bromann/grunt-mocha-istanbul.git
```
Create a new branch to hold your customizations

Again, this is so you can keep a history of your changes. It’s probably a good idea to include the current version of the package in the branch name, in case you need to repeat these steps when a later version is available.

Example:
```
cd grunt-mocha-istanbul
git checkout -b myco-custom-3.0.1
```
Add your scope to the package name in package.json

In our example, change "grunt-mocha-istanbul" to "@myco/grunt-mocha-istanbul".
Commit your changes to your branch and publish the scoped package to your npm Enterprise registry

Assuming you have already configured npm to associate your scope to your private registry, publishing should be as simple as npm publish.

Example:
```
git add package.json
git commit -m 'add @myco scope to package name'
npm publish
```

Step 2: Fork-publish the direct dependency

Clone the project’s source code locally

Either create a remote fork first (e.g., in GitHub or Bitbucket) and clone your fork locally, or just clone/download the project directly from the original remote repository. It’s a good idea to use source control so you can keep a history of your changes.

Example:
```
git clone https://github.com/webdriverio/webdriverio.git
```
Create a new branch to hold your customizations

This is so you can keep a history of your changes. It’s probably a good idea to include the current version of the package in the branch name, in case you need to repeat these steps when a later version is available.

Example:
```
cd webdriverio
git checkout -b myco-custom-4.0.4
```
Add your scope to the package name in package.json

In our example, change "webdriverio" to "@myco/webdriverio".
Replace the Git dependency with the scoped package

This means updating the reference in package.json, and it may mean updating require() or import statements too. You should basically do a find-and-replace, finding the unscoped package name and judiciously replacing it with the scoped package name.

In our example, we only need to update the reference in package.json from "grunt-mocha-istanbul": "christian-bromann/grunt-mocha-istanbul" to "@myco/grunt-mocha-istanbul": "^3.0.1".
Commit your changes to your branch and publish the scoped package to your npm Enterprise registry

Assuming you’ve already configured npm to associate your scope to your private registry, publishing should be as simple as npm publish.

In our example of webdriverio, we next need to deal with the shrinkwrap URLs before we can publish (handled below). In other scenarios, it may be possible to publish now.

Example:
```
git add .
git commit -m 'replace git dep with scoped fork'
npm publish
```
Update your downstream project(s) to use the scoped package as a direct dependency (in package.json and in any require() statements)

In our example, this basically means doing a find-and-replace to find references to webdriverio and judiciously replace them with @myco/webdriverio. However, webdriverio also contains an npm-shrinkwrap.json file. We’ll cover that in the next section.

Working with a shrinkwrapped package

It just so happens that our sample direct dependency above (webdriverio) also uses an npm-shrinkwrap.json file to pin certain dependencies to specific versions. Unfortunately the shrinkwrap file contains hardcoded URLs to the public registry. We need a way to either ignore or fix the URLs.

Shrinkwrap Option 1: Ignore the shrinkwrap

A quick workaround is to install packages using the --no-shrinkwrap flag. This will tell the npm client to ignore any shrinkwrap files it finds in the package dependency tree and, instead, install the dependencies from package.json in the normal fashion.

This is considered a workaround rather than a long-term solution: it’s possible that installing from package.json will install versions of dependencies that don’t exactly match the ones listed in npm-shrinkwrap.json, even though the versions of the package’s direct dependencies are guaranteed to be within the declared semver range.

Example:

npm install webdriverio --no-shrinkwrap

(As noted above, webdriverio@4.0.4 also has a Git dependency, so just ignoring the shrinkwrap isn’t quite enough for this package.)

Shrinkwrap Option 2: Fix the URLs

If you want to use the exact versions from the shrinkwrap file without using the URLs in it, you’ll have to use your own custom fork of the project that contains a modified shrinkwrap file.

Here’s the general idea:

(Note that steps 1-3 are identical to the fork-publish instructions for a direct dependency above. If you’ve already completed them, skip to step 4.)

Clone the project’s source code locally

Either create a remote fork first (e.g., in GitHub or Bitbucket) and clone your fork locally, or just clone/download the project directly from the original remote repository. It’s a good idea to use source control so you can keep a history of your changes.

Example:
```
git clone https://github.com/webdriverio/webdriverio.git
```
Create a new branch to hold your customizations

This is so you can keep a history of your changes. It’s probably a good idea to include the current version of the package in the branch name, in case you need to repeat these steps when a later version is available.

Example:
```
cd webdriverio
git checkout -b myco-custom-4.0.4
```
Add your scope to the package name in package.json

In our example, change "webdriverio" to "@myco/webdriverio".
Use rewrite-shrinkwrap-urls to modify npm-shrinkwrap.json, pointing the URLs to your npm Enterprise registry

Unfortunately this is slightly more complicated than a find-and-replace, since the tarball URL structure of the public registry is different than the one used for an npm Enterprise private registry.

In the example below, replace {your-registry} with the base URL of your private registry, e.g., https://npm-registry.myco.com or http://localhost:8080. The value you use should come from the Full URL of npm Enterprise registry setting in your Enterprise admin UI Settings page.

Example:
```
npm install -g rewrite-shrinkwrap-urls
rewrite-shrinkwrap-urls -r {your-registry}
git diff npm-shrinkwrap.json
```
Commit your changes to your branch and publish the scoped package to your npm Enterprise registry

Assuming you’ve already configured npm to associate your scope to your private registry, publishing should be as simple as npm publish.

Be mindful of any prepublish or publish scripts that may be defined in package.json. You can try skipping those scripts when publishing via npm publish --ignore-scripts, but running the scripts may be necessary to put the package into a usable state, e.g., if source transpilation is required.

Example:
```
git add npm-shrinkwrap.json package.json
git commit -m 'add @myco scope to package name' package.json
git commit -m 'rewrite shrinkwrap urls' npm-shrinkwrap.json
npm publish
```
Note that a prepublish script will probably need to install the package’s dependencies in order to run. In this case, npm install will be executed first. If this happens, it should pull all dependencies in the shrinkwrap file from your registry. If any of those packages don’t yet exist in your registry, you’ll need either to enable the Read Through Cache setting in your Enterprise instance or to manually add the packages to the white-list by running npme add-package webdriverio from your server’s shell and answering Y at the prompt to add dependencies.
Update your downstream project(s) to use the scoped package as a direct dependency (in package.json and in any require() statements)

In our example, this basically means doing a find-and-replace to find references to webdriverio and judiciously replace them with @myco/webdriverio.

This is less than ideal, obviously. We’re currently considering ways to improve handling of shrinkwrapped packages on the server side, but a better solution is not yet available.

Packages with install scripts or node-gyp dependencies

Some packages want or need to run some script(s) on installation in order to build platform-specific dependencies or otherwise put the package into a usable state. This approach means that a package can be distributed as platform-independent source without having to prebundle binaries or provide multiple installation options.

Unfortunately this also means that these packages typically need access to the public internet in order to fetch required resources. In these cases, we can’t really do much to work around this approach, other than attempt to isolate the steps of fetching the package from the registry and set up the platform-specific resources it needs.

Ignore install scripts

As a quick first attempt, you can ignore lifecycle scripts when installing packages via npm install {pkg-name} --ignore-scripts.

Unfortunately, install scripts typically do some sort of platform-specific setup to make the package usable. Thus, you should review the install or postinstall scripts from the package’s package.json file and determine if you need to attempt to run them separately or somehow achieve the same result manually.

Set up node-gyp build toolchain manually

When node-gyp is involved in the setup process, the package requires platform-specific binaries to be built and plugged into the Node runtime on the client’s system. In order to build the binaries, the package will typically need to fetch source header files for the Node API.

The best we can do is attempt to setup the node-gyp build toolchain manually. This requires Python and a C/C++ compiler. You can read more about this at the following locations:

General installation: https://github.com/nodejs/node-gyp#installation

Windows issues: https://github.com/nodejs/node-gyp/issues/629

A good example of a package with a node-gyp dependency is node-sass.

Once the build toolchain is in place, the package’s install script may not need to fetch any external resources.

What’s next?

If you’ve made it all the way to the end, surely you’ll agree that npm could be handling things better to minimize challenges faced by folks with restricted internet access. We feel it’s in the community’s best interest to at least raise awareness of these problems and their potential workarounds until we can get a more robust solution in place.

If you have feedback or questions, as always, please don’t hesitate to let us know.

Introducing hooks: get notifications of npm registry and package changes as they happen

ceejbot — Wed, 01 Jun 2016 08:52:55 -0700

Today, we’re excited to announce a simple, powerful new way to track changes to the npm registry — and build your own amazing new developer tools: hooks.

What?

Hooks are notifications of npm registry events that you’ve subscribed to. Using hooks, you can build integrations that do something useful (or silly) in response to package changes on the registry.

Each time a package is changed, we’ll send an HTTP POST payload to the URI you’ve configured for your hook. You can add hooks to follow specific packages, to follow all the activity of given npm users, or to follow all the packages in an organization or user scope.

For example, you could watch all packages published in the @npm scope by setting up a hook for @npm. If you wanted to watch just lodash, you could set up a hook for lodash.

When?

If you have a paid individual or organizational npm account, you can start using hooks right now.

Each user may configure a total of 100 hooks, and how you use them is up to you: you can put all 100 on a single package, or scatter them across 100 different packages. If you use a hook to watch a scope, this counts as a single hook, regardless of how many packages are in the scope. You can watch any open source package on the npm registry, and any private package that you control (you’ll only receive hooks for packages you have permission to see).

How?

Create your first hook right now using the wombat cli tool.

First, install wombat the usual way: npm install -g wombat. Then, set up some hooks:

Watch the npm package: wombat hook add npm https://example.com/webhooks shared-secret-text

Watch the @slack organization for updates to their API clients: wombat hook add @slack https://example.com/webhooks but-sadly-not-very-secret

Watch the ever-prolific substack: wombat hook add --type=owner substack https://example.com/webhooks this-secret-is-very-shared

Look at all your hooks and when they were last triggered: wombat hook ls

Protip: Wombat has several other interesting commands. wombat --help will tell you all about them.

We’re also making public an API for working with hooks. Read the docs for details on how you can use the API to manage your hooks without using wombat.

Why?

You can use hooks to trigger integration testing, trigger a deploy, make an announcement in a chat channel, or trigger an update of your own packages.

To get you started, here are some of the things we’ve built while developing hooks for you:

npm-hook-receiver: an example receiver that creates a restify server to listen for hook HTTP posts. Source code.
npm-hook-slack: the world’s simplest Slackbot for reporting package events to Slack; built on npm-hook-receiver.
captain-hook: a much more interesting Slackbot that lets you manage your webhooks as well as receive the posts.
wombat: a CLI tool for inspecting and editing your hooks. This client exercises the full hooks API. Source code.
ifttt-hook-translator: Code to receive a webhook and translate it to an IFTTT event, which you can then use to trigger anything else you can do on IFTTT.
citgm-harness: This is a proof-of-concept of how node.js’s Canary in the Gold Mine suite might use hooks to drive its package tests. Specific package publications trigger continuous integration testing of a different project, which is one way to test that you haven’t broken your downstream dependents.

What do you think?

We’re releasing hooks as a beta, and where we take it from here is up to you. What do you think about it? Are there other events you’d like to watch? Is 100 hooks just right, too many, or not enough?

We’re really (really) (really) interested to see what you come up with. If you build something useful (or silly) using hooks, don’t be shy to drop us a line or poke us on Twitter.

What’s next?

This is the tip of an excitement iceberg — exciteberg, if you will — of cool new ways to use npm. Watch this space!

npm ♥ you!

Announcing npm for Atlassian Bitbucket Pipelines and an improved Bitbucket Connect add-on

bencoe — Tue, 24 May 2016 06:00:45 -0700

The 283,000 (!) packages in the npm Registry are only useful if it’s easy for developers to integrate them into their projects and deploy their code, so we’re excited by any chance to streamline your workflow.

If you work with Bitbucket, starting today, it’s easier than ever to install and publish npm private packages — to either your npm account or your self-hosted npm Enterprise installation.

Bitbucket Pipelines is a new continuous integration service built into Bitbucket Cloud for end-to-end visibility, from coding to deployment. We’re excited that they’re launching with npm support.

Why?

When it’s easier to work with packages, developers do. This integration helps split monolithic projects into multiple npm modules. More, smaller packages means code discovery and re-use, which saves you from reinventing the wheel.
It’s all in one place. See pipeline information and work with npm right in Bitbucket commits, branches, and PRs, without context switching and waiting for feedback to see if you broke a build.
You can automate tedious parts of the development workflow. (Did you write a test for that?…)
It’s easy. To get started, just add a build configuration file into a Bitbucket repo.

How?…

Working with private packages in the npm registry

Use the bitbucket-pipelines.yml supplied in this repository.
Set the NPM_TOKEN environment variable.
- This token can be found in your local ~/.npmrc, after you log in to the registry.
Enable pipelines

Working with private packages in npm Enterprise

Use the bitbucket-pipelines.yml supplied in this repository.
Set the NPM_TOKEN environment variable:
- This token can be found in your local ~/.npmrc, after you log in to the registry.
Set NPM_REGISTRY_URL to the full URL of your private registry (with scheme).
Enable pipelines.

New in the Bitbucket Connect add-on: private packages

Alongside the new pipelines integration, the npm for Bitbucket add-on has been updated to support private modules.

This helps complete an elegant CI/CD workflow:

Bitbucket Pipelines installs your private and public npm modules, ensuring that your build continues passing;
npm for Bitbucket Cloud provides constant feedback, ensuring that an up-to-date version of your module has been published.

Get started by installing the add-on now.

Now what?

We have more exciting integrations and improvements in the … pipeline (sorry), but it helps to know what matters to you. Don’t be shy to share feedback in the comments or hit us up on Twitter.

running npm Enterprise on Google Compute Engine

bencoe — Tue, 03 May 2016 07:01:02 -0700

Last month, we released a “one-click” installer for npm Enterprise on AWS. Fresh on its heels, we’re excited to announce support for Google Compute Engine

Getting npm Enterprise up and running on GCE is easy:

Install and configure the Google Cloud SDK

Download the Google Cloud SDK.
Authenticate: gcloud auth login.
Configure your project and default zone:
- gcloud config set project my-project
- gcloud config set compute/zone us-east1-d

Run the npm Enterprise deploy template

Fetch npm’s configuration template:

curl -XGET https://raw.githubusercontent.com/npm/npme-installer/master/npme-gce.jinja > /tmp/npme-gce.jinja
Run the npm Enterprise deploy template:

gcloud deployment-manager deployments create npme-deployment --config /tmp/npme-gce.jinja --properties="zone=us-east1-d"

Note: You can replace us-east1-d with whatever zone you’d like to deploy to.

Go grab a coffee

When you get back, if you visit your cloud console you’ll see a running server called npm-enterprise. That’s all there is to it!

Configure your instance

You can configure your instance by visiting port 8800. Our npm Enterprise will point you in the right direction with information on the various settings that you can configure.

What’s next?

It’s our continued goal to make npm Enterprise painless to run, regardless of your infrastructure. Are we missing a platform that you’d love for us to support? Let us know in the comments.

how many npm users are there?

seldo — Tue, 26 Apr 2016 16:07:46 -0700

A story in 6 graphs

How many npm users are there? It’s a surprisingly tricky question.

There are a little over 211,000 registered npm users, of whom about 73,000 have published packages. But far more people than that use npm: most things npm does do not require you to login or register. So how many are there, and what are they up to? Our first and most obvious clue is the number of packages they are downloading:

How many packages are downloaded?

That’s over a billion downloads per week, and that only counts package installs that weren’t already in cache – about 66% of npm package installs don’t require downloading any packages, because they fetch from the cache. Still, the growth is truly, without exaggeration, exponential.

So what’s driving all the downloads? Are the same people building ever-more-complicated applications, downloading more packages to do it? Are the same people running build servers over and over? Or are there actually more people? Our next clue comes from the number of unique IPs hitting the registry:

How many IPs hit the registry?

Here’s a ton of growth again, close to 100% growth year-on-year, but much more linear than the downloads: 3.1 million IPs hit the registry in March. Of course, IP addresses are not people. Some of these IPs are build servers and other robots. Other IP addresses are companies or educational institions that serve thousands or even tens of thousands of people. So while it doesn’t correlate perfectly, generally speaking, more IPs means more people are using npm.

How many times does npm get run?

Every time npm runs it generates a unique ID that it sends as a header for every request it makes during that run. This ID is random and not tied to you in any way, and once npm finishes running it is deleted. We use this ID for debugging the registry: it lets us see that these 5, 10 or 50 requests were all part of the same operation, which makes it easier to see what’s going on when somebody has a problem. It also makes it possible to say roughly how many times npm is run – or at least, run in a way that makes a request to the registry. There were 84 million npm sessions in March: this number is growing faster than IPs, but less quickly than downloads.

We can take these last two and combine them:

How many packages per npm run?

This number is interesting because it’s not going anywhere. The ratio of packages downloaded to npm sessions is essentially constant (this is not the same as the number of packages downloaded per install, because many npm sessions don’t result in downloads). But this is a clear signal: the number of packages per install isn’t rising. Applications aren’t getting notably more complicated; people are installing packages more often because they are writing more applications.

Here’s a final clue:

How many packages per IP?

The number of packages downloaded by an IP is also rising linearly. So, not only are more people using npm, but the people who are already using npm are using it more and more frequently. And then of course there’s this number:

Web users

Another way of counting npm users is counting people who visit npm’s website. This also grew enormously; 400% since we started the company. In the last 90 days, npm saw just over 4 million unique users visit our site. Ordinarily, you take web user numbers with a grain of salt – there’s lots of ways they can be wrong. But combined with the IPs, the sessions, and the download numbers, we think that number is probably accurate, maybe even a little conservative.

Invisible users

There are so many sources of error! There are robots who crawl the registry. There are lots of companies who host their own internal registry caches, or run npm Enterprise, and so have their own npm website and registry and never hit ours. There’s the entire nation of China, which finds it difficult to access npm through the great firewall, and is served by our hard-working friends at cnpmjs. There’s errors that inflate our numbers, and errors that deflate them. If you think we’re way off, let us know. But we think we have enough for a good guess.

4 million npm users

We think there are four million npm users, and we think that number is doubling every year. Over at the node.js foundation, they see similar growth numbers. Not only are there more of them, but they’re more engaged than ever before. This is awesome! The 25 very hard working people of npm Inc. thank you for your participation and your contributions to the community, and we hope to see even more of you.

run npm Enterprise on AWS with just a few clicks

nexdrew — Thu, 07 Apr 2016 09:02:16 -0700

In a previous blog post we showed you how easy it is to run npm Enterprise on Amazon Web Services. Today, we’re happy to announce the public availability of the npm Enterprise Amazon Machine Image (AMI). Now, it’s even easier to run your own private npm registry and website on AWS!

Using our AMI, there is nothing to install. Just launch an instance, configure it using the npm Enterprise admin web UI, and you’re done: it’s a true point-and-click solution for sharing and managing private JavaScript packages within your company.

Let’s take a quick look at the details.

1. Find the AMI in your preferred AWS region

We have AMIs for several AWS regions. When you launch a new instance in the AWS EC2 Console, find the right one by searching for the relevant AMI ID under the Community AMIs tab. Note that new AMI versions are published about every month and include the date of publication in the AMI name.

Here’s a list of the AMI IDs by region:

us-east-1 (N. Virginia): ami-edd65bfa
us-west-1 (N. California): ami-61db9d01
us-west-2 (Oregon): ami-dc34f6bc
eu-central-1 (Frankfurt): ami-5ec13431
ap-southeast-2 (Sydney): ami-7d32181e

Ensure the AMI comes from owner 666882590071.

If you don’t see your preferred region in the list above, contact our support team, and we’ll get one created for you!

2. Launch it

When you launch an instance of the AMI, you’ll need to:

Choose an instance type: use m3.large or better
Enter a storage size: must be at least 16 GB; we recommend 50-100 GB for typical installs
Select or create a security group: open ports 22 (ssh), 8080 (registry), 8081 (website), and 8800 (npm Enterprise admin UI)
Select or create a .pem key pair: this allows you to ssh into your server instance

It’s not necessary, but if you’d prefer to attach an EBS volume for registry data that is separate from the root volume, you can. However, the root EBS volume cannot be smaller than 16 GB.

For more information (or screenshots) on any of the above, see our docs for Running npm Enterprise in AWS.

3. Configure and start the appliance

You don’t have to, but you can ssh into your EC2 instance to make sure it’s up and running. If you do, you should see a welcome message like the following:

Open your favorite web browser, access your server on port 8800, and follow the prompts to configure and start your appliance.

You’ll need a license key. If you haven’t already purchased one, you can get a free trial key here.

For more information on configuring npm Enterprise, visit our docs.

That’s it! Once you’ve configured and started the appliance, your private npm registry and website are ready for use. See this document for configuring your npm CLI to use your new private registry.

We’re continually striving to provide you the best solutions for distributing, discovering, and reusing your JavaScript code and packages. We hope this AMI makes it just that much easier to leverage the same tools within your organization that work so well in open source communities around the world - a concept we refer to as InnerSource.

As always, if you have questions or feedback, please reach out.

npm registry is now fully HTTPS!

agdubs — Fri, 01 Apr 2016 10:42:29 -0700

Effective yesterday morning, all requests to the npm registry are made via HTTPS.

What’s different

npm has accepted HTTPS or HTTP requests for some time. (In fact, only the initial hop from your client to Fastly, our CDN, would use HTTP.) We returned most data to you via HTTPS, but we would serve JSON containing package metadata over HTTP if you requested it via HTTP.
Starting today, requests you make to the registry over HTTP still work, but we will return all data via HTTPS.
In a few weeks, Fastly will redirect all HTTP requests to HTTPS. This won’t break your HTTP requests, but they’ll be a bit slower because of the redirect. You will be able to avoid that delay by using HTTPS in your first request.

Practically this means:

If you request http://registry.npmjs.org/pkgname you get a JSON response
That JSON used to include http://registry.npmjs.org/pkgname/-/pkgname-1.2.3.tgz
Now it’s https for the dist.tarball url, https://registry.npmjs.org/pkgname/-/pkgname-1.2.3.tgz.
Soon, a request to http://registry.npmjs.org/pkgname will 301 (redirect) over to https://registry.npmjs.org/pkgname

Does this mean that package data is/was insecure?

No! The CLI client checks a shashum to verify the package and that check always has been over HTTPS.

How it affects you

It probably doesn’t. However, a small number of users currently replicate registry data using third-party tools such as Artifactory with configurations to only communicate insecurely.
This is easily fixed by reconfiguring your tool, or by replicating the npm registry using another method.

How to weather these changes

We’ve developed an ecosystem of tools that you can use to replicate the registry in a way that is resilient to these changes:

_changes feed: https://skimdb.npmjs.com/registry/_changes?descending=true&limit=10

For every change in a package in the registry, the whole package object (with changes) gets emitted as data on the _changes feed of CouchDB.
follower: https://github.com/npm/concurrent-couch-follower

Users wishing to follow the changes feed can use our CouchDB follower wrapper, which will ensure you don’t miss any documents even if you process them asynchronously.
normalizer: https://github.com/npm/normalize-registry-metadata

Finally, we also provide a normalizer, so that you can clean up the data you receive, and implement the changes from the changes feed.

We will never stop making replicating public packages utterly trivial. If anything, we’ll keep making it easier.

We believe these tools should minimize any disruption from our transition to HTTPS — but of course there are edge cases! If you experience difficulty, we want to hear about it and help you out. As always, don’t be shy to reach out: support@npmjs.com.

Happy replicating!

fixing a bearer token vulnerability

agdubs — Thu, 31 Mar 2016 15:54:32 -0700

Last week, npm@2.15.1 (npm LTS) and npm@3.8.3 were released to latest. Among other improvements, these fix a vulnerability that could cause the unintentional leakage of bearer tokens.

Here are details on this vulnerability and how it affects you.

How to update npm

An up to date npm is the most secure npm. Update npm to get this patch, as well as other patches:

npm install npm@latest -g

Think you’re at risk? Generate a new bearer token

If you believe that your bearer token may have been leaked, invalidate your current npm bearer tokens and rerun npm login to generate new tokens. Keep in mind that this may cause continuous integration builds in services like Travis to break, in which case you’ll need to update the tokens in your CI server’s configuration.

Details of the vulnerability

Since 2014, npm’s registry has used HTTP bearer tokens to authenticate requests from the npm’s command-line interface. A design flaw meant that the CLI was sending these bearer tokens with every request made by logged-in users, regardless of the destination of their request. (The bearers only should have been included for requests made against a registry or registries used for the current install.)

An attacker could exploit this flaw by setting up an HTTP server that could collect authentication information, then use this authentication information to impersonate the users whose tokens they collected. This impersonation would allow them to do anything the compromised users could do, including publishing new versions of packages.

With the fixes we’ve released, the CLI will only send bearer tokens with requests made against a registry.

Will this break my current setup?

Maybe.

npm’s CLI team believes that the fix won’t break any existing registry setups. Due to the large number of registry software suites out in the wild, though, it’s possible our change will be breaking in some cases.

If so, please file an issue describing the software you’re using and how it broke. Our team will work with you to mitigate the breakage.

Thanks to the community

Thanks to Mitar, Will White & the team at Mapbox, Max Motovilov, and James Taylor for reporting this vulnerability to npm. You can learn more about npm’s security policy on our security page.

-/-

Node.js has also posted about this disclosure. You can read that here.

on dependency squatter packages

agdubs — Wed, 30 Mar 2016 18:06:06 -0700

This week, after we announced changes to our unpublish policy, some community members created a series of packages that depend on every package in the registry. Functionally, these packages exist to ensure that every package has at least one dependent package. A full list of these packages is here.

Other members of the community quickly notified us of these packages. In response, we have contacted the authors of these packages and we are removing all of these packages from the registry, effective 6:00pm PST today (March 30, 2016).

All of the authors we contacted were cooperative and acted in good faith to our requests :) This is not a fight- we’re communicating this situation for the sake of transparency, not drama.

Here’s why we’ve taken this step:

These packages violate npm’s Terms of Use. Much like a squatter package, packages like these have no individual functionality beyond depending on other packages.
These packages perpetuate a misunderstanding of our updated unpublish policy. Some community members are concerned that packages like these make unpublishing packages completely impossible after the new 24 hour window, but this is not the case. These packages violate npm’s terms of use, so they aren’t valid package dependents. Having them alone wouldn’t make it impossible to unpublish a package — meaning the exercise doesn’t even meet its intended goal.
These packages are harmful. If an npm user were to try to install one of them, the installation most likely would fail. Worse, there’s the small chance that installing would succeed … in filling up their hard drive with a lot of junk, and spinning their CPU for a very long time. We can’t guarantee the safety or security of all packages in the registry, but when we learn about packages with harmful effects like these, we have a responsibility to act to protect the community.

We utterly depend on our community — for making the npm registry useful and safe for everyone, and for helping us make policies that balance everyone’s needs and keep everyone safe. We’re grateful to everyone who reached out to alert us to these packages — honestly, thank you! — and we’re also open to continue discussing it with you to understand your concerns. If you have concerns or questions, please contact community@npmjs.com.

Here is the email we sent to the authors of these packages.

changes to npm’s unpublish policy

agdubs — Tue, 29 Mar 2016 09:57:35 -0700

One of Node.js’ core strengths is the community’s trust in npm’s registry. As it’s grown, the registry has filled with packages that are more and more interconnected.

A byproduct of being so interdependent is that a single actor can wreak significant havoc across the ecosystem. If a publisher unpublishes a package that others depend upon, this breaks every downstream project that depends upon it, possibly thousands of projects.

Last Tuesday’s events revealed that this danger isn’t just hypothetical, and it’s one for which we already should have been prepared. It’s our mission to help the community succeed, and by failing to protect the community, we didn’t uphold that mission.

We’re sorry.

This week, we’ve seen a lot of discussion about why unpublish exists at all. Similar discussions happen within npm, Inc. There are important and legitmate reasons for the feature, so we have no intention of removing it, but now we’re significantly changing how unpublish behaves and the policies that surround it.

These changes, which incorporate helpful feedback from a lot of community members, are intended to ensure that events like Tuesday’s don’t happen again.

Our new policy

Going forward, if you try to unpublish a given package@version:

If the version is less than 24 hours old, you can unpublish it. The package will be completely removed from the registry. No new packages can be published using the same name and version.
If the version is older than 24 hours, then the unpublish will fail, with a message to contact support@npmjs.com.
If you contact support, they will check to see if removing that version of your package would break any other installs. If so, we will not remove it. You’ll either have to transfer ownership of the package or reach out to the owners of dependent packages to change their dependency.
If every version of a package is removed, it will be replaced with a security placeholder package, so that the formerly used name will not be susceptible to malicious squatting.
If another member of the community wishes to publish a package with the same name as a security placeholder, they’ll need to contact support@npmjs.com. npm will determine whether to grant this request. (Generally, we will.)

Examples

This can be a bit difficult to understand in the abstract. Let’s walk through some examples.

Brenna is a maintainer of a popular package named “supertools”. Supertools has 3 published versions: 0.0.1, 0.3.0, and 0.3.1. Many packages depend on all the versions of supertools, and, across all versions, supertools gets around 2 million downloads a month.

Brenna does a huge refactor and publishes 1.0.0. An hour later, she realizes that there is a huge vulnerability in the project and needs to unpublish. Version 1.0.0 is less than 24 hours old. Brenna is able to unpublish version 1.0.0.

Embarrassed, Brenna wants to unpublish the whole package. However, because the other versions of supertools are older than 24 hours Brenna has to contact support@npmjs.com to continue to unpublish. After discussing the matter, Brenna opts instead to transfer ownership of the package to Sarah.
Supreet is the maintainer of a package named “fab-framework-plugin”, which has 2 published versions: 0.0.1 and 1.0.0. fab-framework-plugin gets around 5,000 downloads monthly across both versions, but most packages depend on it via ^1.0.0.

Supreet realizes that there are several serious bugs in 1.0.0 and would like to completely unpublish the version. He attempts to unpublish and is prompted to talk to support@npmjs.com because the 1.0.0 version of his package is older than 24 hours. Instead, Supreet publishes a new version with bug fixes, 1.0.1.

Because all dependents are satisfied by 1.0.1, support agrees to grant Supreet’s request to delete 1.0.0.
Tef works for Super Private Company, which has several private packages it use to implement static analysis on Node.js packages.

Working late one night, Tef accidentally publicly publishes a private package called “@super-private-company/secrets”. Immediately noting his mistake, Tef unpublishes secrets. Because secrets was only up for a few minutes — well within the 24 window for unrestricted unpublishes — Tef is able to successfully unpublish.

Because Tef is a responsible developer aware of security best-practices, Tef realizes that the contents of secrets have been effectively disclosed, and spends the rest of the evening resetting passwords and apologizing to his coworkers.
Charlotte is the maintainer of a package called “superfoo”. superfoo is a framework on which no packages depend. However, the consultancy Cool Kids Club has been using it to develop their applications for years. These applications are private, and not published to the registry, so they don’t count as packages that depend on superfoo.

Charlotte burns out on open source and decides to unpublish all of their packages, including superfoo. Even though there are no published dependents on superfoo, superfoo is older than 24 hours, and therefore Charlotte must contact support@npmjs.com to unpublish it.

After Charlotte contacts support, insisting on the removal of superfoo, npm deprecates superfoo with a message that it is no longer supported. Whenever it is installed, a notice is displayed to the installer.

Cool Kids Club sees this notice and republishes superfoo as “coolfoo”. Cool Kids Club software now depends on “coolfoo” and therefore does not break.

Changes to come

This policy is a first step towards balancing the rights of individual publishers with npm’s responsibility to maintain the social cohesion of the open source community.

The policy still relies on human beings making human decisions with their human brains. It’s a fairly clear policy, but there is “meat in the machine”, and that means it will eventually reach scaling problems as our community continues to grow.

In the future, we may extend this policy (including both the human and automated portions) to take into account such metrics as download activity, dependency checking, and other measures of how essential a package is to the community.

Moving forward

In balancing individual and community needs, we’re extremely cognizant that developers feel a sense of ownership over their code. Being able to remove it is a part of that ownership.

However, npm exists to facilitate a productive community. That means we must balance individual ownership with collective benefit.

That tension is at the very core of open source. No package ecosystem can survive without the ability to share and distribute code. That’s why, when you publish a package to the registry, you agree to our Terms of Service. The key lines are:

Your Content belongs to you. You decide whether and how to license it. But at a minimum, you license npm to provide Your Content to users of npm Services when you share Your Content. That special license allows npm to copy, publish, and analyze Your Content, and to share its analyses with others. npm may run computer code in Your Content to analyze it, but npm’s special license alone does not give npm the right to run code for its functionality in npm products or services.

When Your Content is removed from the Website or the Public Registry, whether by you or npm, npm’s special license ends when the last copy disappears from npm’s backups, caches, and other systems. Other licenses, such as open source licenses, may continue after Your Content is removed. Those licenses may give others, or npm itself, the right to share Your Content with npm Services again.

These lines are the result of a clarification that we asked our lawyer to make for the purposes of making this policy as understandable as possible. You can see that in this PR.

We don’t try to hide our policies; in fact, we encourage you to review their full list of changes and updates, linked from every policy page.

We acknowledge that there are cases where you are justified in wanting to remove your code, and also that removing packages can cause harm to other users. That’s exactly why we are working so hard on this issue.

This new policy is just the first of many steps we’ll be taking. We’ll be depending on you to help us consider edge cases, make tough choices, and continue building a robust ecosystem where we can all build amazing things.

You probably have questions about this policy change, and maybe you have a perspective you’d like to share, too.

We appreciate your feedback, even when we can’t respond to all of it. Your participation in this ecosystem is the core of its greatness. Please keep commenting and contributing: you are an important part of this community!

Please post comments and questions here. We’ve moved to a Github issue for improved moderation.

Package install scripts vulnerability

seldo — Fri, 25 Mar 2016 22:16:44 -0700

Disclaimer: we had been told this vulnerability would be disclosed on Monday, not Friday, so this post is a little rushed and may be edited later.

As disclosed to us in January and formally discussed in CERT vulnerability note VU#319816, it is possible for a maliciously-written npm package, when installed, to execute a script that includes itself into a new package that it then publishes to the registry, and to other packages owned by that user.

npm cannot guarantee that packages available on the registry are safe. If you see malicious code on the registry, report it to support@npmjs.com and it will be taken down.

How to protect yourself

If you are installing a package that you do not trust, you can avoid this vulnerability by running

npm install --ignore-scripts

If you wish to never run scripts at install time, you can instead run

npm config set ignore-scripts true

Either or both of these steps will prevent you from spreading a worm at install time.

If you install a package that contains malicious code and then execute it (e.g. by require()ing it into your code) it could still perform malicious actions. You should not execute any software downloaded from the Internet if you do not trust it, including software downloaded from npm.

Why does this vulnerability exist?

Installation and other lifecycle scripts are a useful tool that allows package authors to set up configuration, compile binary dependencies, and perform other actions that make using npm packages convenient.

On balance, it’s npm’s belief that the utility of having installation scripts is greater than the risk of worms. This is a tradeoff that we will continue to evaluate.

Is this new?

Package scripts have been a feature of npm since the very beginning. The implications of this feature were clear from the start, but not everyone in the ever-expanding npm community is fully aware of them. Disclosures of this kind are helpful for that reason.

What happens if a worm is published?

You should report malicious packages to support@npmjs.com. Per our terms of service, they will be taken down. Authors publishing malicious code to the registry may be banned from the registry.

npm monitors publish frequency. A spreading worm would set off alarms within npm, and if a spreading worm were identified we could halt all publishing while infected packages were identified and taken down.

Can npm scan packages for worms?

npm is working with security vendors to introduce enhanced security vulnerability scanning and mitigation services. This work is underway but not yet ready.

At root, it is impossible to guarantee that any new piece of software is benign short of manually inspecting it, as mobile app stores do. The work required to do this would be prohibitively expensive. Instead, we rely on users to flag suspicious packages and act quickly to remove them from the registry.

What else can be done?

Other potential steps can be taken to make publishing without an author’s knowledge harder, including implementing 2-factor authentication on publishing. This functionality is already available via integrations in npm On-Site, and npm is working to make various 2-factor solutions available to the public registry. This work is also not yet complete.

What if I…

Ultimately, if a large number of users make a concerted effort to publish malicious packages to npm, malicious packages will be available on npm. npm is largely a community of benevolent, helpful people, and so the overwhelming majority of software in the registry is safe and often useful. We hope the npm community continues to help us to keep things that way, and we will do our best to continuously improve the reliability and security of the registry.

kik, left-pad, and npm

izs — Wed, 23 Mar 2016 18:21:47 -0700

Earlier this week, many npm users suffered a disruption when a package that many projects depend on — directly or indirectly — was unpublished by its author, as part of a dispute over a package name. The event generated a lot of attention and raised many concerns, because of the scale of disruption, the circumstances that led to this dispute, and the actions npm, Inc. took in response.

Here’s an explanation of what happened.

Timeline

In recent weeks, Azer Koçulu and Kik exchanged correspondence over the use of the module name kik. They weren’t able to come to an agreement. Last week, a representative of Kik contacted us to ask for help resolving the disagreement.

This hasn’t been the first time that members of the community have disagreed over a name. In a global namespace for unscoped modules, collisions are inevitable. npm has a package name dispute resolution policy for this reason. That policy encourages parties to attempt an amicable solution, and when one is impossible, articulates how we resolve the dispute.

The policy’s overarching goal is this: provide npm users with the package they expect. This covers spam, typo-squatting, misleading package names, and also more complicated cases such as this one. Entirely on this basis, we concluded that the package name “kik” ought to be maintained by Kik, and informed both parties.

So far, this followed a process that is routine, though rare. What happened next, though, was unprecedented.

Under our dispute policy, an existing package with a disputed name typically remains on the npm registry; the new owner of the name publishes their package with a breaking version number. Anyone using Azer’s existing kik package would have continued to find it.

In this case, though, without warning to developers of dependent projects, Azer unpublished his kik package and 272 other packages. One of those was left-pad. This impacted many thousands of projects. Shortly after 2:30 PM (Pacific Time) on Tuesday, March 22, we began observing hundreds of failures per minute, as dependent projects — and their dependents, and their dependents… — all failed when requesting the now-unpublished package.

Within ten minutes, Cameron Westland stepped in and published a functionally identical version of left-pad. This was possible because left-pad is open source, and we allow anyone to use an abandoned package name as long as they don’t use the same version numbers.

Cameron’s left-pad was published as version 1.0.0, but we continued to observe many errors. This happened because a number of dependency chains, including babel and atom, were bringing it in via line-numbers, which explicitly requested 0.0.3.

We conferred with Cameron and took the unprecedented step of re-publishing the original 0.0.3. This required relying on a backup, since re-publishing isn’t otherwise possible. We announced this plan at 4:05 PM and completed the operation by 4:55 PM.

The duration of the disruption was 2.5 hours.

What worked

We stand by our package name dispute resolution policy, and the decision to which it led us.

Given two packages vying for the name kik, we believe that a substantial number of users who type npm install kik would be confused to receive code unrelated to the messaging app with over 200 million users.

The dispute resolution policy minimizes disruption.

Transferring ownership of a package’s name doesn’t remove current versions of the package. Dependents can still retrieve and install it. Nothing breaks.

Had Azer taken no action, Kik would have published a new version of kik and everyone depending upon Azer’s package could have continued to find it.

It was abrupt unpublishing, not our resolution policy, that led to yesterday’s disruptions.

The community stepped in.

It’s pretty remarkable that Cameron stepped in to replace left-pad within ten minutes. The other 272 affected modules were adopted by others in the community in a similar time. They either re-published forks of the original modules or created “dummy” packages to prevent malicious publishing of modules under their names.

We’re grateful to everyone who stepped in. With their explicit permission, we are working with them to transfer these to npm’s direct control.

What didn’t work

Unrestricted un-publishing caused a lot of pain.

There are historical reasons for why it’s possible to un-publish a package from the npm registry. However, we’ve hit an inflection point in the size of the community and how critical npm has become to the Node and front-end development communities.

Abruptly removing a package disrupted many thousands of developers and threatened everyone’s trust in the foundation of open source software: that developers can rely and build upon one another’s work.

npm needs safeguards to keep anyone from causing so much disruption. If these had been in place yesterday, this post-mortem wouldn’t be necessary.

Poor communication made matters worse.

In the immediate wake of yesterday’s disruption, and continuing even now on blogs and Twitter, a lot of impassioned debate was based on falsehoods.

npm did not “steal” Azer’s code.

left-pad was open-source code, and explicitly allows republishing by any other author. That’s what happened in this case.
This incident did not arise because of intellectual property law.

We’re aware that Kik and Azer discussed the legal issues surrounding the “Kik” trademark, but that wasn’t pertinent. Our decision relied on our dispute resolution policy. It was solely an editorial choice, made in the best interests of the vast majority of npm’s users.
npm won’t suddenly take your package name.

Our guiding principle is to prevent confusion among npm users. In the rare event that another member of the community requests our help resolving a conflict, we work out a resolution by communicating with both sides. In the overwhelming majority of cases, these resolutions are amicable.

It took us too long to get you this update. If this were a purely technical operations outage, our internal processes would have been much more up to the challenge.

What happens next

There are technical and social aspects to this problem. Any reasonable course of action must address both of these.

We will make it harder to un-publish a version of a package if doing so would break other packages.

We are still fleshing out the technical details of how this will work. Like any registry change, we will of course take our time to consider and implement it with care.
We will make it harder to maliciously adopt an abandoned package name.

If a package with known dependents is completely unpublished, we’ll replace that package with a placeholder package that prevents immediate adoption of that name. It will still be possible to get the name of an abandoned package by contacting npm support.
We are updating our internal policies to help our team stay in sync and address community conflict more effectively.

To Recap (tl;dr)

We dropped the ball in not protecting you from a disruption caused by unrestricted unpublishing. We’re addressing this with technical and policy changes.
npm’s well-established and documented dispute resolution policy was followed to the letter. This is not a legal dispute.
We’ll continue to do everything we can to reduce friction in the lives of JavaScript developers.

In a community of millions of developers, some conflict is inevitable. We can’t head off every disagreement, but we can earn your trust that our policies and actions are biased to supporting as many developers as possible.

Introducing Greenkeeper for npm On-Site

bencoe — Tue, 15 Mar 2016 09:30:40 -0700

Keeping dependencies up to date in your modules is a tedious chore, but it’s very important; the further you drift away from the latest release of a dependency, the more difficult and risky an upgrade becomes. I’ve found that, despite developers’ best intentions, dependencies tend to gradually drift behind:

Enter Greenkeeper. Greenkeeper solves the problem of keeping your dependencies up to date:

automatically creating a branch when dependent modules have changed;
kicking off CI, so that you can see if the changes break your build;

and

providing detailed context about relevant updates in a pull request:

Open source developers (like me!) who have integrated Greenkeeper into their continuous integration workflow have found it indispensible. But what if you’re bringing open source’s best practices to the enterprise?

Good news!

Announcing Greenkeeper for npm On-Site

We’re excited today to announce that Greenkeeper now integrates with npm On-Site. The integration allows you to configure Greenkeeper’s automated dependency management workflow for the modules that you develop privately within your company.

npm and Inner Source

This integration fits wonderfully into our mission of Inner Source: empowering developers within companies of all sizes to benefit from the awesome tooling and best practices of the open source community.

Greenkeeper solves a very real problem for developers — keeping dependencies up to date — and it’s already seen wide adoption in the open source world. We’re excited to partner with Greenkeeper to bring this same slick workflow to companies’ private module development.

If you’d like to give Greenkeeper a shot, they’d love to hear from you: contact Greenkeeper about starting an enterprise trial.

And, as always, if you’d like to talk about how to bring npm into your large organization — or just see a demo of how easy it is to spin up an npm instance behind your firewall — give it a whirl: discover npm On-Site

Practices Perfected In OSS Can Reshape Enterprise Development

bencoe — Mon, 15 Feb 2016 11:48:59 -0800

npm, Inc. helps more than 3 million open source developers build communities, learn, collaborate, and ultimately publish software that’s used by many of the largest companies in the world.

I believe that this open source development process works. Some great studies back up this belief: In their 2015 article InnerSource: Internal Open Source at PayPal, PayPal describes introducing open source practices into their engineering culture, and the results are exciting:

The results were visible after 6 months. The Checkout Platform team spends 0% of its time rewriting code and just 10% reviewing submissions. The team was able to do a major refactoring and a 4x increase in performance without planning for it. The mindset moved from blocking change to mentoring and coaching.

In this post I focus on tooling, one important aspect of the open source process:

how developers interact with a community to discover best practices, and in turn find the best tools;
how this ad-hoc standardization process puts developers in a good place to automate all the things.

I’ll conclude by discussing the steps that npm On-Site and other industry leaders are taking to bring these open source tools and practices into the enterprise.

Best Practices are Community Driven

It’s amazing to watch best practices spread across OSS communities. Just a few years ago, projects rarely had unit-tests; now unit-tests are one of the first things folks look for in a healthy project. Developers learned running large open source projects that writing descriptive unit-tests was a great habit: It helped people familiarize themselves with codebases, and prevented regressions as new features were added.

How do good habits like unit-testing spread? There’s a wonderful feedback loop created by the social sites where people share and discover code:

npm’s website maintains a list of most-depended-upon modules, getting a module on this list is a great motivator for module builders. But how do developers get on this list? What works best is studying the projects of other maintainers already on this list, interacting with the developers, learning from them, and emulating their methodologies. This process means habits and best practices spread quickly across the whole community.

Badges are a great example of this process:

Among the first things developers look for in a healthy OSS project is a set of green badges along the top of its README documentation. These badges emerged organically from the developer community to indicate qualities including passing unit-tests, healthy code coverage percentages, and up-to-date project dependencies. The emergence of badges has, in turn, lead to better software development practices in the community. So cool!

Best Practices Center Around Great Tools

As effective development habits spread across the community, developers start to converge on the tools that best facilitate these practices. Let’s look at the top 10 modules listed on the npm registry:

10/10 are published to npm.
10/10 host their code on GitHub.
9/10 run their unit tests on Travis CI.
3/10 use coveralls.io for tracking test coverage.

As the developer community arrives at a standard set of tools, incentives grow for these tools to work together elegantly.

Consider one average workflow of an npm module developer:

a user pushes code to GitHub and creates a pull request;
a build kicks off on Travis CI, installing modules from npm, and running tests;
the GitHub pull request is automatically updated with status messages from Travis CI and Coveralls;
upon build completion, coveralls.io maintains a history of coverage information, and will fail a build if this coverage decreases;
once the pull request is merged, the package is published to npm.

Various checks and balances automatically ensure code quality along the way — and at the end of the day, all of our badges remain green.

With the right set of tools, tasks that were once frustrating and manual — e.g., making sure that a pull request doesn’t break your project — become fun.

InnerSource: Don’t Leave Your Best Tools at Home

npm is dedicated to the concept of InnerSource. Originally coined by Tim O'Reilly in 2000, the term refers to applying the best open source practices and tools to building enterprise software.

To help make InnerSource a reality, npm On-Site has teamed up with Travis CI Enterprise, Coveralls Enterprise, and GitHub Enterprise to bring OSS developers’ elegant workflow to the enterprise.

What does this look like exactly?

Travis CI, Coveralls.io, and npm On-Site are working to build a similar installation experience. If you can get one product up and running, you can run the full suite. We’re also working together to keep our roadmaps in alignment; features will be built that compliment the full suite of products.

I’m excited about this direction for npm On-Site, and it’s wonderful to collaborate with other Open Source leaders. With hard work and evangelism, InnerSource can help make writing code at work as much fun as writing open source software on weekends. At the same time, open source practices can help make the enterprise software development cycle faster, while producing higher quality software.

Do you think that Open Source practices apply well in the Enterprise? Are there other useful tools that make your development workflow better? I would love to hear from you.

The npm Blog

How we deploy at npm

Goals

Thousand-foot view

Conventions

Configuration

Open source

Hello, Yarn!

Does it work with npm?

Is Facebook forking the community?

This is how open source works

out with the newww and in with the www: an update on the npm website

Summary

Sunsetting npm/newww

Introducing npm/www

npm has a new CTO

What does a CTO do, anyway?

Q1 2014: just another ops engineer.

Q2 2014: architect.

Q3 2014: manager

Q4 2014: sales and evangelism

Q1 2015: data scientist

Q2 2015: recruiter

Q3 2015: strategic planner

Q4 2015: project manager

Q1 2016: head of product

Q2 2016: interim CEO

Today: COO

All hail CJ

package tarball read outage today

What we’re doing to prevent outages like this next time

nginx and etags

tl;dr

Introducing add-ons for npm Enterprise

What are npm Enterprise add-ons?

Why?

Enterprises have strict security restrictions.

Enterprises strictly enforce license compliance.

Enterprises depend on good software development practices.

Now what?

This is just the beginning

dealing with problematic dependencies in a restricted network environment

Diagnosing the Problem

Problem Types

Direct vs Transitive

Potential Solutions

Replacing a Git dependency

Step 1: Fork-publish the transitive dependency

Step 2: Fork-publish the direct dependency

Working with a shrinkwrapped package

Shrinkwrap Option 1: Ignore the shrinkwrap

Shrinkwrap Option 2: Fix the URLs

Packages with install scripts or node-gyp dependencies

Ignore install scripts

Set up node-gyp build toolchain manually

What’s next?

Introducing hooks: get notifications of npm registry and package changes as they happen

What?

When?

How?

Why?

What do you think?

What’s next?

Announcing npm for Atlassian Bitbucket Pipelines and an improved Bitbucket Connect add-on

Working with private packages in the npm registry

Working with private packages in npm Enterprise

New in the Bitbucket Connect add-on: private packages

Now what?

running npm Enterprise on Google Compute Engine

Install and configure the Google Cloud SDK

Run the npm Enterprise deploy template

Go grab a coffee

Configure your instance

What’s next?

how many npm users are there?

A story in 6 graphs

How many packages are downloaded?

How many IPs hit the registry?

How many times does npm get run?

How many packages per npm run?

Sunsetting `npm/newww`

Introducing `npm/www`