inconshreveable

Launching Equinox and the importance of software packaging

Tue, 26 Apr 2016 00:00:00 -0700

I was proud to launch Equinox last week: a service that makes it easy to package and distribute production Go applications.

In some ways, I’ve finally come full circle. My first job in the tech industry was packaging software. The first company I worked for had offices in a converted warehouse - a squat, colorless building with offices ringing a great, empty space. Perched against the back wall was a lone workstation on a folding table with a mess of wires underneath. And that’s where I first began packaging software. I would sit at that desk and burn copies of our compiler, one-by-one at 8x speed on to blank CD-Rs.

Of course, packaged software needs good identifying metadata. So while each disc went through its 5 minute burn period, I printed CD labels - four at a time - onto special adhesive paper and then carefully applied the label to a disc with a smoothing motion to avoid air bubbles. This was also my earliest use of containers; I assembled pre-printed boxes, folding each of the four sides upright, slotting hooked corner tabs together to lock the cardboard into holding its form.

Boxes were packed with a pink flyer advertising upcoming product launches, a yellow flyer encouraging you to visit a website (or mail in the form on the back) to register your software, and the CD slipped into a simple paper and plastic case. End-to-end integrity of the transported bits was enforced by shrinkwrap.

Years later, everything is different and yet still the same. We still ship bits to customers, but it’s via fiber and cable and wireless in our homes and offices. Containers are digital things with their own subindustry and millions of VC dollars. Instead of shrinkwrap, we use cryptographic algorithms to sign and verify code. Maybe the biggest change though is how little most software shops even think about these concerns.

Developers don’t package their own software much anymore, even digitally. In the golden age of web services, we ship software by adding a new resource endpoint. The bits are packed once for our rigidly controlled datacenter environment. Even the web browser in 2016, for all its faults, is an impressively consistent application delivery platform that makes it easy to safely deliver the most up-to-date version of your app to customers. When we deploy to app stores, we run a vendor-provided tool that manages the entire process, wrapping up our bits appropriately for the walled gardens before shipping it off to an opaque vendor-specific cloud service.

Three years ago I launched ngrok.com - a tunneling reverse proxy service. As part of that, I distribute a command-line tool available on all major platforms (and many minor ones, too). It has given me the privilege to watch users fail to install software in every possible way. Your users will download your software for the wrong operating system, for the wrong computer architecture and they will install older versions from third-party distributors. They will double-click command line apps, paste terminal commands into web browsers and try to run unzip on tar.gz archives. They do not know how to change their PATH environment variable or what it even is. They will try to run your software on machines built 20 years ago without the SSE2 instruction set.

Software packaging and installation is about first impressions. It is the first step of the conversion funnel after a customer agrees you have a solution to their problems. The UX of that installation experience is crucially important. Installation sets the tone and expectations of your user before they’ve even begun using your product. And if it fails, the rest of your product experience is moot because it will never run. A smooth installation experience establishes the initial trust of your user in the quality of your software.

For indie developers who aren’t shipping to the browser or an App Store, it is exceedingly difficult to build simple installation experiences. Instead of the tedium of assembling cardboard boxes, packaging modern software is a frustrating exercise in learning poorly-designed, vendor-specific toolchains, XML manifest files and installation behaviors. You must buy code-signing certificates with week-long turnarounds after apply for official business registrations. The tools for each platform often only run on that OS itself, adding additional burdens in resources and ops. Updating software to the latest version is part of this experience as well and requires an entirely new set of tools and knowledge of platform-specific behaviors and security mindfulness.

And that’s why we built Equinox: to solve these problems and make creating first-class installations experiences easier so that you can spend more time shipping your product and delighting your customers. For any Go program, Equinox automatically builds Windows .msi, OS X .pkg, Linux .deb and .rpm installers, .zip and .tgz archives in addition to publishing new releases to your own Homebrew tap. Equinox adds a small package to your app that enables self-updating functionality with just a few lines of code.

We’re starting with an admittedly small, but growing niche. Equinox only supports Go programs, but we’ve got bigger plans down the road to widen the scope of what we can distribute and how.

Ready to get started? Sign up at equinox.io or drop us a line at contact@equinox.

Your live coding demo is boring

Fri, 13 Nov 2015 00:00:00 -0800

I’ve been to a lot of conferences and meetups over the past year and one thing I can say categorically about the experience is that if you write code live on stage, then you are boring me.

For better or worse, what we do as software engineers is not a good spectator sport. I can think faster than you can type. Writing code doesn’t mix well with oration. You will fumble your speech and settle into long, ennui-inducing pauses. The demo will likely fail because of syntax errors in code, configuration, or command lines. And while you eat up time in an impromptu debugging session, I will reach for my phone. Live coding demos fail so often that I’m more surprised when they work than when they fail.

That does not mean you shouldn’t show code on stage. Please, by all means, put code up there on your slide and talk about it. Just don’t write it in real time. Live coding is lazy; it is a shortcut for creating quality presentation material. On well-prepared slides, your code will be magnified to ensure that I can read it. It will be highlighted so that you can direct my attention appropriately.

If you want to show something interactive do that with a video. You can talk over the video and guide my interest of what is happening on screen. You can edit a video to add highlights, insert pauses, speed it up 1.5x, or whatever else is needed to suit your speaking style and flow well with the presentation. Oh right, and it’s guaranteed to work.

This applies with equal force to demos of CLI tools or anything heavy with keyboard input.

Less live demos, more high quality presentations.

The Neomonolith

Wed, 07 Oct 2015 00:00:00 -0700

Monolith or Microservices?

In the modern silicon-valley startup zeitgeist, the CTO of a formative software company faces a pivotal decision. Will the fledgeling software service be a monolith or microservices? The stakes are high: rewrites are measured in man-years.

There is no industry consensus on whether either approach is strictly-better. Despite that ambivalence, there is general agreement of two points:

Monoliths that exceed some threshold will be broken apart eventually. At a large enough scale, only a service-oriented design is workable.
There are only two options. It’s either service-oriented or a monolith, and you must choose.

The Problem

Much has been written contrasting the pros and cons of monolith and microservice designs in a general context. [1] This territory is well-covered so I omit discussion of it here. Instead we will take a more narrow focus on the initial stage of the business. The neomonolith is most suited to this formative period: when scale is small and code changes land at a breakneck pace.

You can begin with a monolith. Sometimes this is called a sacrificial architecture because of the inevitable transition to services at large scale. It will enable immediate, rapid progress but inevitably ossify your codebase in a Kafkaesque entanglement of broken interfaces. That may be a reasonable tradeoff: technical debt is irrelevant if the business fails.

Alternatively, you can construct a microservices architecture from the get-go. Such a design comes with a steep premium. You must stand up and operate new pieces infrastructure, most of which is non-standardized, alpha-quality and changing rapidly. In addition, with a monolith, the work of monitoring, alerting, configuration, and a local development is paid once. But with a microservice design, that cost must be paid for every service. Starting with microservices is such a risky proposition for a startup that many proponents of the design recommend building monolith-first.

A False Architectural Dichotomy

This architectural question is a false dichotomy: there is an intermediate, hybrid option. I call it the The Neomonolith. The neomonolith is a superior design for the nascent systems built by startups. It combines strengths of both approaches with a less onerous set of shortcomings. I’ve had great success applying this pattern building both ngrok.com and equinox.io.

The ideas described herein are not new. Even this repackaging is not genuinely novel. Other efforts have promoted similar architectural designs, but most are couched in the context of a particular language or framework. [2] The neomonolith pattern is not limited to a particular language or framework – indeed, one of its strengths is the tolerance of heterogeneity in implementation languages!

What is the Neomonolith?

A neomonolith can be conceptualized as a special-case deployment of a microservice architecture.

Like a microservices architecture, a neomonolith is composed of small, independent services communicating with well-defined RPC interfaces. But like a monolith, each machine runs the same set of code. Each server is an identical image that runs all of the individual services. All services start when a host boots and stop when it shuts down.

Services communicate with each other over ports bound on the loopback interface. A service listens on the same port regardless of which host it runs on, obviating the need for any service discovery system.

Containers are optional. They can be helpful if you are struggling with resource contention between misbehaving services, but they are easy to add later if you are not prepared to pay their operational cost up front.

The Best of Both Worlds

The neomonolith provides the best of both worlds, combining benefits from both monolith and microservice styles:

Microservice benefits

Independence of development - services can be developed independently, even in different code repositories
Independence of deployment - services can be deployed without affecting others
Independence of failure - e.g. a file descriptor leak in one process will not affect others on the same host
Enforced interfaces and the single responsibility principle - services are small and must communicate over well-defined interfaces
Heterogeneity of language/frameworks - services communicate over APIs, allowing teams to “use the right tool for the job” if that would be more productive

Monolith Benefits

No orchestration complexity - all boxes run an identical set of code and processes
No dynamic service scheduling - services start on boot, halt at shutdown
No service discovery - services listen on the same port on all hosts
Easy scaling - capacity can be added by deploying additional machines with the same image behind a load balancer
Defer distributed systems problems - use of loopback enables you to defer robust handling of uncommon failures caused by unreliable networks [3]

Three of the monolith benefits listed begin with the word ‘No’. Anyone who has operated large-scale systems understands why. Every additional piece of infrastructure you run is an operational liability; each adds another point of failure. Institutional knowledge and software must be built to configure, monitor and alert on these new pieces.

But beyond that, the infrastructure of a typical microservice PaaS deployment bears yet another oft-ignored cost. Complicating the production infrastructure makes a software system harder to reason about. It becomes more difficult to build a mental model of how the entire system operates. This makes production incidents harder to debug: “Is it our code, or a problem with the PaaS?” It also increases the knowledge and tools required for development, ratcheting up the barrier to entry for new programmers.

A neomonolith dispenses with all of that complexity, greatly simplifying the software system. Problems that still plague microservice PaaS installations are often non-issues. Deployment requires only tried-and-tested tools familiar to anyone who has done ops in the last 10 years.

Drawbacks

No scaling independence - you must scale out all services to meet the demand of the most resource-intensive one
Requires investment in operational tooling - monitoring, alerting, and local dev must be repeated or reusable for each service
Refactoring across RPC interfaces - it’s still more difficult than refactoring over a monolith’s functions
Limited failure isolation - if a service dies, its consumers can’t talk to another one running on a different host

A neomonolith is not a panacea. It shares faults of both monoliths and service-oriented systems. It is more effort than starting monolith-first and it will not scale out forever like service-oriented systems in place at Amazon or Google. On the other hand, most of us don’t run at that scale. It’s important to remember that not running tens of thousands of machines is an advantage that you should leverage while it lasts.

Transitioning to a complete microservice design

Once the tools improve and mature, that there will be no question about whether you begin with a sacrificial monolith first. But right now, it is not the future yet.

The single most important benefit of a neomonolith is the clear path to a full microservices PaaS deployment. The up-front investment in defining RPC interfaces and scoping out services means that when you’re ready to deploy on a Mesos/CoreOS/Kubernetes cluster, it will require no major architectural changes. Switching to service discovery or cluster proxies instead of using loopback is a task measured in weeks, not years. Moreover, PaaS elements can be added piecemeal for a smooth transition. For example, sketching the first two steps of a hypothetical switch:

In short, not only does starting with a neomonolith today provide you with a whole host of development benefits, but it additionally positions your engineering team for a painless transition in the future if you must scale up to a more traditional service-oriented design.

Video

I gave a lightning talk at GopherCon 2015 that is the spiritual precursor to this piece in which I called this design ‘The Lego Monolith’.

Footnotes

Microservice/Monolith tradeoffs
- http://martinfowler.com/articles/microservices.html
- http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
Prior Art
Yes, loopback can fail when you overflow the kernel buffers, although I’ve yet to observe this ever happen in production. Whether you choose to defer error handling and retries for network problems depends of course on your failure tolerance for that particular operation.

Sweat the small stuff

Tue, 09 Sep 2014 00:00:00 -0700

ngrok is a tunneling, reverse proxy that establishes secure tunnels from a public endpoint to a locally running network service while capturing all traffic for inspection and replay. It is an open-source project on GitHub.

When you double-click a command-line application on any modern Windows OS, a new cmd.exe window opens running the application that was launched. When the program terminates, Windows closes the whole cmd.exe window. When executed with no arguments, almost all command line apps print their help text and then exit. From the perspective of a user who just double-clicked a command line app, they will see a command prompt open, display a large amount of text that they don’t have time to read and then close almost instantly. Anyone who didn’t initially learn on a command line will probably remember this initial growing pain.

To say the least, this is a tremendously poor user experience. Not only did the program fail to work, but it did so without displaying any error message or diagnostic text that explains why it did not work. There isn’t even anything to search Google or Stack Overflow for!

Authors of most CLI tools simply choose to ignore that they completely fail for this cohort of users. It’s easy to rationalize that it’s not your problem, that it’s an error in how the user is interacting with their operating system. “All of my users are technical enough to understand that!” you cry. And I’ll tell you that they will have to be, if that’s your attitude.

I don’t have that luxury. Until about six months ago, I received a support email almost every week from an ngrok user who was running into this exact issue. ngrok’s target userbase spans everyone from network engineers to designers to hobbyist gamers who want to host their own minecraft server. In short, ‘non-technical’ describes a large percentage of the userbase. Here are a few examples of support emails:

i downloaded it and install it a terminal window poped out and nothing happened.

hi i have download for window when i try command prompt i suddent open and hide can you help me?

When I try to open the .exe file command prompt opens very briefly and then nothing happens. Please help?

Windows version for ngrok doesn’t work. I downloaded ngrok for windows 7 , 64 bit . When I click ngrok.exe , console flashes for half second and then vanishes.

I don’t get these support emails anymore. But before we dive into why that’s the case, I want to explain how I support ngrok.

The ngrok support strategy

As of this writing, there are over 17,000 registered ngrok users, and undoubtedly more who use it without an account. I support ngrok entirely by myself. There is no IRC channel for ngrok. There is no mailing list. Instead, I have made it extraordinarily easy to reach me, personally. Every page of ngrok.com not only has an email address that goes directly to me, but it also has an Olark chat window that you can use to communicate with me directly and synchronously at any hour of the day when I’m in front of my computer. The next version of ngrok will include this direct chat on the local introspection interface of every ngrok client. I’ve even idly toyed with including a phone number.

I have a secret, though: I hope you never use that chat window. I don’t want to talk to you. You’re all really lovely people, but I just don’t have the time for those interruptions. I’m being a bit hyperbolic here, of course. But think of it this way: when a new user has to talk to a human in order to fix a problem they’re having, the time-to-value of your product has just ballooned from seconds to minutes or possibly even hours or days. If a user contacts me to solve an issue, ngrok is no longer self-service and it has failed them as a product. Sometimes this is an error in the core functionality or an omission in the documentation. Those are the low hanging fruit of your support problem because they’re easy to fix. The more common cause is that the UX of the tool is not good enough, i.e. ngrok behaved in some way which did not match with the user’s expectations and it failed to communicate what change could be made to correct this expectation.

The goal of this support system I’ve constructed is to attempt to visit upon myself every pain that using ngrok inflicts on my customers. When you really boil it down to its essence, ngrok’s support strategy is to make it as easy as possible to contact me and then to optimize that conversion rate to zero. This is how you make a better product, this is how you really listen to your customers.

It works. I was sufficiently frustrated receiving these emails every week and typing the same explanation over and over again, that I decided to implement a fix.

Pursuing a technical solution

The easy way to solve this problem is to create an FAQ or Knowledge Base item (or maybe in a larger organization to hire extra support staff). This is suboptimal. Users may not know to look in the FAQ. They may not know that it exists at all, or if it does what keywords they should search for. They may not even speak English! Even supposing neither of those apply and the user figures out what they were doing wrong, they will still experience the initial frustration of ngrok inexplicably failing. Don’t forget that this is part of ngrok’s first impression. A user just downloaded my application and is trying to use it for the first time. Failing in any way will decrease the perceived value of the product and make them that much less likely to pay for it and recommend it to their network.

With this in mind, I opted to pursue a technical fix. I initially didn’t know if this would work, but here’s the idea I experimented with:

If the parent process is named “explorer.exe” then display a message instructing users to open cmd.exe instead of running it from the command line, then sleep for a few seconds so they can read it.

So at that point, I went into the Go standard library, and found os.Getppid(). It returns the parent process’ id which I could check against the id of explorer.exe. I tried it out and . . . nothing; it didn’t work. I discovered that, strangely os.Getppid() was always returning -1 until I read the source code of syscall_windows.go to find:

// TODO(brainman): fix all needed for os
func Getppid() (ppid int) { return -1 }

Oops! So I read some of the Win32 API documentation, and a bunch more examples from the standard library about how to call those functions from Go. Finally I had a working implementation of os.Getppid(), and when I wired it all into the stdlib, the whole thing worked!

Figuring that others could benefit from the work I’d done, I did two things. First, I submitted a changeset to the Go standard library to implement Getppid() for Windows. This was merged into Go 1.4. I then created a simple Go package which used os.Getppid() (and vendored the implementation for pre-1.4 versions) plus the other necessary Windows APIs to check if the parent process was explorer.exe. You can find it on github at inconshreveable/mousetrap.

I released ngrok 1.7 three months ago with mousetrap integrated, and I’ve only had a single support email about this issue since then from someone who didn’t understand how to open cmd.exe and use a command prompt. Victory!

Sweat the small stuff

In the process of writing up this blog post, I wondered if I could do even better. Is there a way I could eliminate that last support email? After searching around for a bit, I stumbled on cmd.exe’s /K option which instructs cmd.exe to run the program and remain open after it terminates. The soon-to-released ngrok 2.0 contains the final iteration of this feature. When you double click it from explorer, it will relaunch itself using cmd /K so that it executes as if you had opened the prompt yourself and run the command. Windows even launches the prompt in the directory of the double-clicked executable so there aren’t even problems with executable location. From there, a user can be instructed to type a proper incantation to invoke ngrok with the correct arguments. I’ve also wired it up to print out some additional help text that explains that you shouldn’t try double-clicking it from explorer again.

ngrok lives in a crowded marketplace. There are any number of similar programs and services. In the end, what makes ngrok the best in class tool is the attention to detail. It is the uncompromising focus on user experience. Sweat the small stuff, it matters more than you think.

Principles of designing Go APIs with channels

Tue, 08 Jul 2014 00:00:00 -0700

Channels are concurrent-safe queues that are used to safely pass messages between Go’s lightweight processes (goroutines). Together, these primitives are some of the most popularly touted features of the Go programming language. The message-passing style they encourage permits the programmer to safely coordinate multiple concurrent tasks with easy-to-reason-about semantics and control flow that often trumps the use of callbacks or shared memory.

Despite their power, channels are rare to find in public APIs. I combed through the Go standard library for examples. As of Go 1.3, there are more than 6,000 public APIs across 145 packages. Among those thousands, there are only 5 unique uses of channels.

There is little guidance on the tradeoffs and decisions to make when using channels in a public API. By “public API” I mean “any programmatic interface whose implementer and user are two different humans”. This article will go in depth to provide a set of principles and rationale on how to use channels appropriately in public APIs. Some cases that break the rules are discussed at the end.

Principle #1

An API should declare the directionality of its channels.

Examples

#### time.After

func After(d Duration) <-chan Time

signal.Notify

func Notify(c chan<- os.Signal, sig ...os.Signal)

Although not commonly used, Go allows you to specify the directionality of a channel. The language spec says:

The optional <- operator specifies the channel direction, send or receive. If no direction is given, the channel is bidirectional.

The important part is that a directional operator in your API signature will be enforced by the compiler.

t := time.After(time.Second)
t <- time.Now()  // won't compile (send to receive-only type <-chan Time)

In addition to the safety granted by compiler enforcement, these directional operators help consumers of your API understand the direction of data flow just by looking at the type signature.

Principle #2

An API that sends an unbounded stream of values into a channel must document how it behaves for slow consumers.

Examples

#### time.NewTicker

// NewTicker returns a new Ticker containing a channel that will send the
// time with a period specified by the duration argument.
// It adjusts the intervals or drops ticks to make up for slow receivers.
// ...
func NewTicker(d Duration) *Ticker {
    ...
}

signal.Notify

// Notify causes package signal to relay incoming signals to c.
// ...
// Package signal will not block sending to c
// ...
func Notify(c chan<- os.Signal, sig ...os.Signal) {

ssh.Conn.OpenChannel

// OpenChannel tries to open an channel. 
// ...
// On success it returns the SSH Channel and a Go channel for
// incoming, out-of-band requests. The Go channel must be serviced, or
// the connection will hang.
OpenChannel(name string, data []byte) (Channel, <-chan *Request, error)

Whenever an API sends an unbounded stream of values into a channel, the implementation will be faced with the decision about what to do if sending a value into the channel would block. This can occur either because the channel is full, or because it is unbuffered and no goroutine is ready to receive a new value. Choosing the appropriate behavior depends on the API, but an implementation must make a decision. For example, the ssh package chooses to block, and documents that if you don’t receive values that your connection will hang. signal.Notify and time.Tick choose not to block and instead drop values silently.

Unfortunately, the language does not provide a way to specify the intended behavior as part of a type or function signature. As a designer of an API, you must specify the behavior in your documentation, otherwise it is undefined. Since we are more often consumers of APIs than designers of them, it can be helpful to remember the converse rule, which is a warning:

You can never determine the behavior of an API that sends an unbounded stream of values over a channel to slow consumers without reading the documentation or the implementation.

Principle #3

An API that sends a bounded set of values into a channel it accepted as an argument must document how it behaves for slow consumers. ### BAD Example #### rpc.Client.Go

func (client *Client) Go(serviceMethod string,
                         args interface{}, 
                         reply interface{}, 
                         done chan *Call
                         ) *Call

This is similar to the second principle, except it’s for APIs sending a bounded series of values. Unfortunately, there’s not a good example of this in the standard library. The only API in the standard lib which does this is rpc.Client.Go and it violates this principle. The documentation says:

Go invokes the function asynchronously. It returns the Call structure representing the invocation. The done channel will signal when the call is complete by returning the same Call object. If done is nil, Go will allocate a new channel. If non-nil, done must be buffered or Go will deliberately crash.

Go sends a bounded number of values (just 1, when the remote call completes). But notice that because the channel is passed into the function, that it still suffers from the slow consumer problem. Even though you must pass a buffered channel to this API, sending on that channel could still block if the channel is full. The documentation does not define the behavior under this circumstance. Time to read the source code:

src/pkg/net/rpc/client.go

171 func (call *Call) done() {
172      select {
173      case call.Done <- call:
174          // ok
175      default:
176          // We don't want to block here.  It is the caller's responsibility to make
177          // sure the channel has enough buffer space. See comment in Go().
178          if debugLog {
179              log.Println("rpc: discarding Call reply due to insufficient Done chan capacity")
180          }
181      }
182  }

Uh oh! If the done channel isn’t appropriately buffered, your RPC replies may just disappear into the ether!

Principle #4

An API that sends an unbounded stream of values into a channel should accept the channel as an argument instead of returning a new channel.

Examples

#### signal.Notify

func Notify(c chan<- os.Signal, sig ...os.Signal)

ssh.NewClient

func NewClient(c Conn, chans <-chan NewChannel, reqs <-chan *Request) *Client

When I first saw the API for signal.Notify, I was confused. “Why does it take a channel as an input instead of returning a channel for my use?” I wondered. “Using the API requires the caller to allocate a channel, shouldn’t the API just do that for you, like this?”

func Notify(sig ...os.Signal) <-chan os.Signal

The documentation helps us understand why this is not a good choice:

Package signal will not block sending to c: the caller must ensure that c has sufficient buffer space to keep up with the expected signal rate.

signal.Notify takes the channel as an argument because it gives the caller control over the amount of buffer space. This allows the caller to choose how many signals it can safely miss while responding to a previous one by trading away memory to buffer those signals.

This control of buffer size also matters for high-throughput systems. Imagine this interface to a high-throughput publish/subscribe system:

func Subscribe(topic string, msgs chan<- Msg)

The more messages pushed through that channel, the greater the chance that the channel synchronization could become a performance bottleneck. Because the API allows the caller to create the channel, it delegates the decision about buffering, and thus performance tuning to the caller. This is a more flexible design.

If it’s just about controlling the size of the buffer, one could argue that an API like the following would suffice:

func Notify(sig ...os.Signal, bufSize int) <-chan os.Signal

A channel argument is still preferrable to this design because it allows the caller to wait for multiple types of signals dynamically with a single channel. This provides more flexibility to your callers both for the structure of the program and performance characteristics. As a thought experiment, let’s work with our Subscribe API to build code for the requirements, “subscribe to the ‘newcustomer’ channel, and for each message, subscribe to the topic for that customer.” If the API allows us to pass the receiving channel as an argument, we might write:

msgs := make(chan Msg, 128)
Subscribe("newcustomer", msgs)
for m := range msgs {
    switch m.Topic {
    case "newcustomer":
        Subscribe(msg.Payload, msgs)
    default:
        handleCustomerMessage(m)
}

But if the channel is returned, the caller is forced into a design with a separate goroutine for every subscription. This can cost additional memory and synchronization time in whatever piece is responsible for the demultiplexing:

for m := range Subscribe("newcustomer") {
    go subCustomer(m.Payload)
}

func subCustomer(topic string) {
    for m := range Subscribe(topic) {
        handleCustomerMessage(m)
    }
}

Principle #5

An API which sends a bounded number of values may do so safely by returning an appropriately buffered channel. ### Examples: #### http.CloseNotifier

type CloseNotifier interface {
        // CloseNotify returns a channel that receives a single value
        // when the client connection has gone away.
        CloseNotify() <-chan bool
}

time.After

func After(d Duration) <-chan Time

When an API sends a bounded number of values into a channel, it can return a buffered channel that has enough room for all the values it will send. The directionality indicator on the returned channel guarantees that the caller cannot break this contract. Channels returned by CloseNotify and After take advantage of this.

On the other hand, be aware that these calls could be more flexible by allowing the caller to pass in a channel to receive values, but then they would be forced to cope with cases where the channel is full (Principle #3). For example, this an alternative, more flexible CloseNotifier:

type CloseNotifier interface {
        // CloseNotify sends a single value with the ResponseWriter whose
        // underlying connection has gone away.
        CloseNotify(chan<- http.ResponseWriter)
}

But the cost of the extra flexibility doesn’t seem worth paying since it is unlikely that a single caller would ever want to wait on multiple close notifications. After all, close notifications only make sense within the context of a single connection, and connections are typically largely independent.

P-P-P-Principle Breakers

Some of the APIs we’ve examined break some of the principles. They warrant a closer look.

Breaking Principle #1

An API should declare the directionality of its channels. ### Example #### rpc.Client.Go

There’s no directionality indicator on the done channel you pass in:

func (client *Client) Go(serviceMethod string,
                         args interface{}, 
                         reply interface{}, 
                         done chan *Call
                         ) *Call

Without diving in too deeply, this happens because the done channel is returned to you as part of the Call struct.

type Call struct {
        // ...
        Done          chan *Call  // Strobes when call is complete.
}

This flexibility is required so that a done channel can be allocated for you if you pass nil. Fixing this would require removing Done from the Call struct and creating two functions:

func (c *Client) Go(method string, 
                    args interface{},
                    reply interface{}
                    ) (*Call, <-chan *Call)

func (c *Client) GoEx(method string,
                      args interface{},
                      reply interface{},
                      done chan<- *Call
                      ) *Call

Breaking Principle #4

An API that sends an unbounded stream of values into a channel should accept the channel as an argument instead of returning a new channel.

Examples

#### go.crypto/ssh

func NewClientConn(c net.Conn, addr string, config *ClientConfig)
    (Conn, <-chan NewChannel, <-chan *Request, error)

time.Tick

func Tick(d Duration) <-chan Time

The go.crypto/ssh package returns channels of unbounded streams nearly everywhere. ssh.NewClientConn is just one of those APIs. A better API that gives the callers more control and flexibility would instead be:

func NewClientConn(c net.Conn,
                   addr string,
                   config *ClientConfig,
                   channels chan<- NewChannel,
                   reqs chan<- *Request
                   ) (Conn, error)

time.Tick violates this principle as well, but it’s easy to forgive. It’s rare that you’ll ever be creating that many tickers, and you typically want to handle them independently anyways. Buffering doesn’t make much sense in this case either.

More and revisions

This material was eventually turned into a talk with some updates and changes, those start about half-way through.

Principles of designing Go APIs with channels talk from GopherCon India 2015

Thanks to Kyle Conroy, Jeff Lindsay, shazow, and Blake Mizerany for feedback on drafts.

Automate away your problems: combatting illegal abuses of ngrok

Thu, 29 May 2014 00:00:00 -0700

At its core, ngrok is an open reverse proxy. That shudder you heard was the sound of millions of network administrators crying out in horror. Open proxies, sometimes called open relays or open resolvers are excellent resources for spammers, cybercrime, DDOS attacks and more. “Open” in these cases means that the service is available without any authentication or payment.

Open reverse proxies like ngrok are traffic sinks, and thus not prone to those types of abuses. They won’t relay any requests out to the public internet. Instead, ngrok.com suffers from the opposite problem that most hosting providers and Platforms-as-a-Service like Heroku, Azure and App Engine deal with: people like to host illegal content on us.

Unlike most platform providers, ngrok doesn’t even require you to create an account before you can begin hosting content over a tunnel. I don’t require accounts because of my fanatical devotion to first-time user experience. Minimizing the time to value of the service is very important to me. Unfortunately, this lower barrier to entry does put me at a disadvantage when it comes to dealing with illegal content.

Recently, some miscreants have taken to using ngrok for hosting phishing websites which attempt to collect login credentials for popular, high-value websites like Google Accounts. They’ve stuck to hosting these on the only types of tunnels you can create without signing up on ngrok.com which are randomly-chosen hexadecimal subdomains that look like 39ac91f.ngrok.com. And while most of us would immediately recognize a bank login page hosted on that domain as suspicious, there are plenty of humans without the internet-savvy to recognize the illegitimacy of such a site.

It’s hard to blame them, honestly. URLs are often opaque, magic looking things with weird base64 encoded state passed around by the ColdFusion framework or ASP.NET web forms or something. Some mobile browsers have also taken to hiding them under certain circumstances. On top of all that, ngrok.com provides free TLS-secured tunnels, which means that an ngrok-hosted phishing site renders with that “secure lock” symbol we trained an entire cohort of internet users to believe meant a website was trustworthy.

Inevitably, when illegal content gets hosted on ngrok.com, it is brought to the attention of my hosting provider and, in turn, to me. Hosting a phishing site (or any illegal content) is against the terms of service of pretty much every hosting provider out there and mine takes it pretty seriously.

A motivating outage

During the first couple weeks encountering this phenomenon, I banned the sites manually as I was notified of them. But on Sunday, May 18th, ngrok suffered its first major outage (approximately five hours) in nearly nine months because I wasn’t fast enough. My hosting provider notified me of a phishing site and then subsequently powered down the ngrok servers because I failed to deal with it before the given deadline. The cause was as pedestrian and dull as they come: it was a Sunday morning and I was still asleep. After I got ngrok back online, I considered my next steps. What changes could I make to combat illegal content more effectively or reduce the incidence rate? I came up with a number of possible solutions.

Possible Solutions

Only provide TLS tunnels to paid accounts

Phishing sites are less attractive without TLS/HTTPS because our CA system has unfortunately conflated the fundamentally separate concepts of encryption and trust. In the end, other illegal content would be just as useful without https, and I’m a big proponent of encryption everywhere, so I decided against this.

This would severely compromise ngrok’s first-time user experience and ease of use, both of which are hugely important in ngrok’s popularity. Moreover, creating an ngrok account does not require anything besides an email address, which it doesn’t bother to verify since obtaining a valid one is so easy anyways. This wouldn’t help either really.

Inspect tunnel traffic on the wire for illegal content patterns

One of the core tenets of the ngrok.com service is that it does not inspect your traffic at all beyond reading the header field necessary to perform the multiplexing. All traffic inspection and replay is done client side. While I’m sure that this would likely be effective, it’s hard to get right and it’s not a route I want to go down unless I have no other options. This is a last resort.

Make ngrok a completely paid service

Even if I did this and there was no free tier (which would fundamentally change ngrok’s audience and global appeal), it’s likely that I would still need some sort of free trial. I could keep upping the barrier to entry, (like requiring a credit card number) but it only hurts ease of use.

Automating the banhammer

In the end, I decided that what I really want to start with is just a more efficient way to respond to these illegal sites. Ideally, I’d like to give my hosting provider an administrative tool that they could use to shut down illegal tunnels without putting me in the critical path. Of course, it’s a little bit scary to hand a giant banhammer for your entire service over to someone else, but it’s a reasonable compromise so long as I get notifications and can asynchronously verify that it’s not being abused.

These types of administrative interfaces aren’t all that uncommon, and I tried suggesting one to my provider, but they weren’t responsive to any deviation from their procedure of opening tickets against me. But I really wanted a completely automated workflow, so in the end, I got out my proverbial programmatic roll of duct-tape and automated around them. Let’s talk about how it works.

The anatomy of a hack

The goal was to automate the entire process so that as soon as I get a notification that an illegal site is hosted on an ngrok.com subdomain, the following will all happen, without a human involved:

Forward the email notification through to my personal email address
Hellban the offending subdomain and the IP address of the tunneling client. If someone was silly enough to do this with an account, ban their user id as well
Notify my hosting provider that the site was blocked through their ticketing system
Notify my phone so that I can verify that the site was indeed illegal

All total, I ended up writing a small custom extension to the ngrok server and about a hundred or so lines of Python to handle all of the blacklisting.

Automated handling of email

I get notifications of new tickets against me in email. So far as I can tell, this is the only mechanism by which I can automate the handling of them. I run my email through postfix, and it turns out that automatically handling email is super easy with postfix (sidenote: not exactly, it took me a while to figure out that you can only do this with aliases, not with virtual aliases). Here’s how you set up your /etc/aliases file to invoke a script for every email to an address:

1 banhammer:        |/var/services/ngrok.blacklist/venv/bin/ngrok.ban

Now emails to banhammer on the local machine will invoke the given script. If you’re using virtual aliases you’ll need to properly point an entry in your virtual aliases file as well:

1 banhammer@ngrok.com      banhammer@localhost

The script

This is a slightly pared-down version of the script that I wrote up to automate the whole process. It walks through each of the important steps in the process: reading out the email, parsing it, forwarding it to my personal address, communicating with the blacklist service, and then posting an update back to my provider. It is a testament to the power of the Python language, standard library and its ecosystem of third-party modules that allow me to accomplish so much in ~60 lines of code:

 1 import sys, email, smtplib, re, requests, bs4
 2 
 3 # read in the message from stdin and parse it
 4 rawmsg = sys.stdin.read()
 5 msg = email.message_from_string(rawmsg)
 6 
 7 # send the mail through to my personal address
 8 smtp = smtplib.SMTP("localhost", 25)
 9 smtp.sendmail(get_from(msg.get("From")), config.my_email, msg.as_string())
10 smtp.quit()
11 
12 # only process messages from the provider
13 if not verify_authenticity(msg):
14     return
15 
16 ticket_id = None
17 domains = set()
18 # look in each part of the email
19 for part in msg.walk():
20     try:
21         # find all the ngrok.com subdomains to ban
22         domains.update(re.findall(r"[\w]+\.ngrok\.com", part.get_payload()))
23 
24         # look for the ticket id in the email
25         m = re.search(TICKET_ID_REGEX, part.get_payload())
26         if m:
27             ticket_id = m.group(1)
28     except:
29         # skip message parts without a payload
30         pass
31 
32 # blacklist each domain requested
33 for d in domains:
34     resp = requests.post(config.blacklist_service_url, data={"hostname": d})
35     resp.raise_for_status()
36 
37 if len(domains) > 0 and ticket_id != None:
38     s = requests.Session()
39     resp = s.post("https://provider.com/login", data={
40         "username": config.provider_username,
41         "password": config.provider_password
42     })
43     resp.raise_for_status()
44 
45     # load the ticket page to pull out the CSRF token
46     resp = s.get("https://provider.com/ticket/{}".format(ticket_id))
47     resp.raise_for_status()
48 
49     # find the csrf token
50     soup = bs4.BeautifulSoup(resp.text)
51     csrf_token = soup.select("#ticket input[name=csrf_token]")[0]["value"]
52 
53     # construct and post the response
54     msg = "The site {} has been blocked"
55     if len(domains) > 1:
56         msg = "The sites {} have been blocked"
57     resp = s.post("https://provider.com/ticket/{}".format(ticket_id), data={
58         "message": msg.format(", ".join(domains)),
59         "csrf_token": csrf_token
60     })
61     resp.raise_for_status()

The blacklist service

The blacklist service is fairly simple. It just manages a database table of blacklist entries and pushes updates over HTTP into the ngrokd server to immediately update its blacklist maps. It’s also responsible for notifying my phone and doing some metrics collection. I’m not going to walk over this part of the code since it’s basically just a little CRUD service that talks to some database tables and external HTTP services.

It’s just the beginning

It’s my expectation that this problem will never go away so long as ngrok is a successful service and that I’m in for a long arms race and game of whack-a-mole. Automation will help, but to what extent it acts as a deterrent for those using ngrok to accomplish dishonorable ends remains to be seen. At the very least, I experience a brief moment of magical bliss whenever my phone lights up to notify me that my automation has done in a few seconds something which used to be a manual process that could have hours of latency before I could respond.

For those of you who work at hosting providers and services platforms: how do you deal with this problem? What automation do you have in place?

gonative: Cross compiling Golang programs with native libraries

Wed, 30 Apr 2014 00:00:00 -0700

Go has execellent support for cross compilation. There are scores of tutorials and helper tools on the internet with instructions on how to do it. This, in and of itself, is rather astounding because of how simple it is:

1 # set up cross compilation to windows_amd64
2 # you only need to do this once
3 cd $GOROOT/src
4 GOOS=windows GOARCH=amd64 ./make.bash --no-clean
5 
6 # whenever you want to cross compile
7 cd $YOUR_APP
8 GOOS=windows GOARCH=amd64 go build

There is a well-known limitation to this simplicty. If you want to use any C libraries, you’ll need to compile and link them with Cgo and from there you’re back to the mess that is cross-compiling C. Upon hearing this limitation you might say something totally reasonable like:

That’s not really a problem, there are tons of programs that don’t use Cgo.

And you’d be right, until I told you:

Some critical parts of the standard library use Cgo.

What exactly are those parts exactly? It turns out that if you cross-compile Go to Darwin or Linux your programs won’t use the system DNS resolver. They also can’t use the native host certificate store. They also can’t look up the user’s home directory, either. For some applications, that doesn’t matter so much. For ngrok, though, all three could matter, and two have already caused bugs.

A better solution

When I spoke at GopherCon, I told everyone that if they want to distribute production applications they should be compiling natively because of this limitation. I was wrong. It turns out that there’s a clever workaround for this that has been mentioned occasionally on the mailing list and which minux just reminded me of when he posted:

If the only cgo-dependent packages are in the standard library, then you do have workaround without setting up a full cross compilation environment.

Download the binary distribution for the target platform, extract the pkg/$GOOS_$GOARCH directory into your $GOROOT/pkg …

Which is quite clever and I want the simplicity of that build and deployment story. But instead of doing this as a one off, I decided to automate the entire process since I want you all to use it as well and because I’m going to need to do it again at every new release of Go.

gonative

gonative is a simple tool which creates a build of Go that can cross compile to all platforms while still using the Cgo-enabled versions of the stdlib packages. It’s a simple Go program of a few hundred lines and is available on github! If you’re the author of a Go program doing multi-platform distribution, please try it out and contribute:

github.com/inconshreveable/gonative

It’s really easy to install and run:

1     go get github.com/inconshreveable/gonative
2     gonative

I’ve tested targeting Windows/Linux/Darwin and they all work correctly!

About that portabililty thing . . .

One of the oft-touted advantages of Go programs is that they’re “dependency-free”. They don’t require a runtime or link against dynamic libraries which your users might not have installed on their systems.

So when I compiled my native-stdlib linux_386 test code and tried to run it on an amd64 machine, I was rather confused when I was confronted with this error message:

1     alan@inconshreveable:~$ ./test 
2     -bash: ./test: No such file or directory

minux’s help on the mailing list helped me understand what was going on. Let’s read the ELF header of this binary.

1     alan@inconshreveable:~$ readelf -l test

Elf file type is EXEC (Executable file)
Entry point 0x8066a50
There are 9 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x08048034 0x08048034 0x00120 0x00120 R E 0x1000
  INTERP         0x000bed 0x08048bed 0x08048bed 0x00013 0x00013 R   0x1
      [Requesting program interpreter: /lib/ld-linux.so.2]
  LOAD           0x000000 0x08048000 0x08048000 0x68e70 0x68e70 R E 0x1000
  LOAD           0x069000 0x080b1000 0x080b1000 0xc9403 0xc9403 R   0x1000
  LOAD           0x133000 0x0817b000 0x0817b000 0x10580 0x23554 RW  0x1000
  DYNAMIC        0x133080 0x0817b080 0x0817b080 0x00098 0x00098 RW  0x4
  TLS            0x000000 0x00000000 0x00000000 0x00000 0x00008 R   0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x4
  LOOS+5041580   0x000000 0x00000000 0x00000000 0x00000 0x00000     0x4

See that INTERP section? The one that says:

[Requesting program interpreter: /lib/ld-linux.so.2]

/lib/ld-linux.so.2 is the 32-bit ELF interpreter that is required for programs that need to dynamically link libraries. My linux box didn’t have it installed, so the program was failing.

But wait, you said that Go programs doesn’t require a runtime or dynamically linked libraries!

And that’s totally true when you cross compile without native libraries. Those executables don’t call anything via C APIs. But when you link against native libraries, there are a few libraries you need to load dynamically. Let’s take a look at them:

alan@inconshreveable:~$ ldd test
        not a dynamic executable

What!? This hung me up for a while too, until I remembered seeing that DYNAMIC section in the ELF headers. It turns out that if you don’t have the 32-bit libraries on your system, ldd can’t analyze the binary properly. minux explains:

actually if you take a look at ldd’s source code you will find that it’s a script and just set some magic environment variable and then execute the program, essentially it’s asking the elf interpreter to dump needed shared library.

so if you don’t have the elf interpreter, the program will fail to execute and ldd mistake that as the program not being dynamically linked.

If we use readelf instead, we can see what’s going on:

alan@inconshreveable:~$ readelf -d test
Dynamic section at offset 0x133080 contains 19 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
more …

So now we can finally see that our natively compiled program now depends on the 32-bit elf interpreter so that it can dynamically load libpthread and libc.

A fun detour

Understanding executable formats and the guts of how Linux sets up your program to support dynamic linking is not an area I need to wander into frequently, so it was rather enjoyable to push my knowledge deeper into this area. For those of you who don’t wade there often either, I hope this was useful in understanding more about exactly how your programs are loaded and run and what you need to be aware of when you compile with native vs. non-native Go libraries. And for those of you who need to distribute Go programs that use native libs, try out gonative! I hope it makes things a little less painful.

Relevant mailing list thread

https://groups.google.com/forum/#!msg/golang-nuts/2XoGUvBalcw/ErSWiTlO17kJ

Code at 30,000 feet

Thu, 17 Oct 2013 00:00:00 -0700

A week ago, the post “I wrote FAT on an airplane, for heaven’s sake”, on Raymond Chen’s popular blog The Old New Thing was making the rounds on the interwebs. In the dramatized story, a frustrated Bill Gates wants his engineers to stop wasting time on some trivial performance tuning, don’t ya know he wrote a file system on an airplane, for heaven’s sake.

And while Gates’ claim is obviously hyperbolic, it’s certainly not the first time I’ve heard “on a plane” used to exemplify programming virtuosity. A former colleague liked to say of another, “He doesn’t code much anymore, but he wrote the whole counters system on the plane flight back from a pitch meeting.” The boastful or reverent nature with which we say these things betrays the interesting assumption that airplanes are not conducive to effective work.

So that got me thinking about when I’ve worked on airplanes (or buses, most recently). And for me, I think the exact opposite is true. Planes (and trains and automobiles) are a perfect incubator for productivity because they compose four major factors of a productive environment which rarely align in an office setting or at home:

No interruptions

On an airplane, your interruptions are limited. You won’t get any text messages or phone calls. No one will come over to your desk for a quick question. Most of your fellow travelers are uninterested in conversation. They want to sleep, read or watch their in-flight entertainment. If you’re lucky enough to be on a flight without wifi, you won’t even get those pesky emails or chat messages.

Few distractions

There’s just not that much to do on an airplane. You can’t get another cup of coffee down the street or step away to bang out a round of Foosball. Until recently, the lack of pervasive wifi on flights made it even better by curbing that impulsive news-feed, Twitter update scrolling habit you can’t seem to shake. Unless you brought along a book or really enjoy watching last year’s romantic comedy, there are just very few ways you can get yourself off-task. For breaks, I just close my computer and my eyes for five to ten minutes. I find it lends itself well to sub-conscious problem solving.

Guaranteed time

I might have put this first because it’s perhaps the most important. On the ground, even if you manage to isolate yourself from interruptions and distractions, how long will it last? I have a serious mental aversion to entering a highly focused state when I know I won’t be able to maintain it. “I have that meeting at 11:30, so I don’t want to get started on anything big” is a variation on a thought that is all too common for me. On a plane flight though, you’ve got a printed receipt for a very real guarantee of uninterrupted time.

A brief period for inspiration

Excepting those with aviophobia, that meditative, relaxed, sub-conscious problem-solving mental state that’s credited with making the shower a place of great inspiration reminds me a lot of takeoff. It’s a 10-15 minute lull where no electronic devices are permitted and your mind can drift around solutions to problems you’re intending on solving when you hit cruising altitude.

Of course, there are undesirable side-effects. An isolating flight might prevent me from accessing useful documentation and Stack Overflow. I find I end up reading the source code for third-party libs more frequently up in the air. And sometimes I really do want to take a break outside and unwind in the sun. But on balance, my creativity and productivity spikes frequently during my reclusive retreats in the sky. And judging by the follow up post and comments on The Old New Thing, I’m not the only one.

The next time you want to impress someone with your coding chops, tell them about that time your hacked up an OS scheduler while on a conference call with a tab open to Reddit.

ngrok tunnels: better, faster, stronger

Wed, 25 Sep 2013 00:00:00 -0700

This week, I released a new version of ngrok that includes a number of significant improvements. This effort involved a serious refactoring of the ngrok codebase and a redesign of the ngrok protocol. The result is a cleaner, more extensible design as well as an improved set of core functionality that includes the following features and improvements.

Tunnels behind port-restricted networks and firewalls

ngrok’s raison d’etre is to selectively expose services behind firewalls and NATs to the public internet while capturing the traffic for inspection and replay. While most users have no problems getting ngrok to work, there was a small segment of users who reported that ngrok failed to connect on some restricted networks. The root cause of this issue is the implementation of network policies which block traffic on all ports except those responsible for the most commonly used internet protocols (HTTP, DNS, SMTP, etc). In an effort to make ngrok work absolutely everywhere, ngrok.com now combats these draconian filters by running all of its traffic over port 443 by default. Since all ngrok connections are encrypted with TLS, ngrok traffic looks sufficiently similar to regular HTTPS traffic to defeat all port/traffic restricting networks that I’ve been able to test.

Tunnels behind HTTP proxies

Unfortunately, the change to port 443 alone doesn’t allow ngrok to run on all restricted networks. Many corporate networks reject all traffic to the public internet and provide HTTP proxy servers as the only public gateway. Supporting ngrok tunnels in this environment was another common support request. The latest version of ngrok now supports establishing its tunnels through an HTTP proxy by honoring the traditional *nix environment variable “http_proxy” like other tools. You may also specify a value explicitly by setting the http_proxy parameter in the configuration file. ngrok uses the same CONNECT method used by browsers to run HTTPS traffic over an HTTP proxy.

Multiple simultaneous tunnels

One of the most major changes in this version of ngrok was adding the support to run multiple ngrok tunnels simultaneously. That means it’s now possible to run a single ngrok client which creates multiple public endpoints that route to multiple local services. Here’s a new ngrok client running:

ngrok

Tunnel Status                 online
Version                       1.3/1.3
Forwarding                    http://blog.ngrok.com -> 127.0.0.1:4000
Forwarding                    https://blog.ngrok.com -> 127.0.0.1:4000
Forwarding                    https://api.ngrok.com -> 127.0.0.1:8080
Forwarding                    http://hacks.inconshreveable.com -> 127.0.0.1:9090
Forwarding                    tcp://ngrok.com:44764 -> 127.0.0.1:22
Web Interface                 127.0.0.1:4040
...

To enable you to easily create multiple tunnels without complicating the command-line interface, ngrok now supports reading from a simple YAML configuration file for starting multiple tunnels. Here’s the configuration file:

authtoken: "..."
tunnels:
  blog:
    proto:
      http: 4000
      https: 4000

  api:
    auth: "user:password"
    proto:
      https: 8080

  personal:
    hostname: "hacks.inconshreveable.com"
    proto:
      http: 9090

  ssh:
    proto:
      tcp: 22

Then you can start any number of your pre-configured tunnels from the command line, like so:

ngrok start blog api personal

Here’s another example to just start remote access to my machine:

ngrok start ssh

I expect this capability to be especially valuable to contractors and consultants working on multiple projects simultaneously as well as for developers who are working on distributed systems that involve running multiple public components. I already use it myself to switch between sharing different projects without rewriting command-line switches all of the time.

Faster tunneled connections

Some users of ngrok in countries outside of the United States reported that requests routed through ngrok were very slow to complete. To understand why this was the case, I will briefly explain how ngrok tunnels a new connection to a public ngrok endpoint through to your local service.

The old procedure ngrok used for tunneling a new connection was as follows:

ngrokd server receives a new public connection to your-app.ngrok.com
ngrokd server sends a Request Proxy Connection Message to the ngrok client
ngrok client establishes new connection to the ngrokd server
ngrokd server joins the public connection and the proxy connection to service the request

Now that you understand the ngrok protocol for tunneling a connection, it’s easy to understand why it was so slow. The setup time for tunneling a new connection involved a significant amount of network latency, including: - Sending a message from the server to the client (.5 network round trips) - A new TCP connection (1 network round trip) - A TLS negotiation (2 network round trips)

In cases where the ngrok client was across the world or on slow connections, 3.5 network round trips could easily add over a full second to connection setup times!

The latest version of ngrok now pre-fetches connections so it can immediately service new requests. The ngrokd implementation now executes the following logic: 1. ngrokd server receives a new public connection to your-app.ngrok.com 1. ngrokd server retrieves a proxy connection from a pool and joins it with the public connection to service the request 1. ngrokd server sends a new Request Proxy Connection Message to the ngrok client to replace the connection removed from the pool

Now, in the common case when a connection is available in the pool, zero network round trips are required before the connection begins to tunnel. Avoiding the time to establish these connections makes tunneling connections feel much, much faster.

Better Crash Behavior

When older versions of ngrok crashed, they destroyed your terminal. In order to display the terminal interface, ngrok uses termbox-go which changes the mode of your terminal and needs to be cleanly shut down to return your terminal to a proper state. ngrok now wraps the execution of most of its goroutines with a function that will execute a controlled shutdown of the program if the goroutine panics. In the unfortunate but inevitable case that ngrok crashes, a user will no longer be confronted with a wrecked terminal. Instead, they will be presented with a clean stack trace and instructions for reporting the bug on Github.

Simplify and improve self-hosting

All of the code for ngrok and ngrokd (the client and server) has always been open-source, but running your own ngrokd server wasn’t entirely trivial. I’ve made a great effort to make running your own ngrokd server easier than ever. I’ve posted a guide on how to self-host your own service that’s only six steps.

Self-hosting: How to run your own ngrokd server

ngrok on linux/arm

Another common inquiry I received was whether ngrok ran on linux/arm devices and if so where the download was available. ngrok certainly runs on Linux/ARM (in fact, I developed this new version entirely on a Chromebook), and as of today, ngrok.com now has download links available for the builds. Now you can try it out on your Chromebook or RaspberryPi without compiling!

HTTPS for everyone, by default

Finally, I’d like to mention one last change to ngrok that is unrelated to the new version. Previously, ngrok only allowed you to use HTTPS tunnels after you signed up on ngrok.com. In response to the news regarding the surveillance of internet traffic and possible weaknesses in internet encryption implementations, I’m making my own small effort to improve the state of internet security. I’ve made the following changes to the ngrok.com service:

ngrok.com now issues https tunnels without any need to signup for an account on all versions of ngrok
ngrok.com now supports TLS1.2 for all capable clients
ngrok.com now supports TLS forward secrecy for all capable clients

Get it

The latest version of ngrok is available for download on ngrok.com.

Download ngrok

Support ngrok

If you love ngrok and would like to see more improvements, please show your support in any of the following ways:

Tweet about ngrok

File a bug or feature request

You can file bugs or feature requests on the ngrok project on github.

Contribute code

ngrok is an open source project! Submit a pull request to fix a bug or add a new feature on the ngrok project on github.

Pay for ngrok

ngrok is a pay-what-you-want service. You can pay for ngrok after you sign-up for an account on ngrok.com

inconshreveable

Launching Equinox and the importance of software packaging

Your live coding demo is boring

The Neomonolith

Monolith or Microservices?

The Problem

A False Architectural Dichotomy

What is the Neomonolith?

The Best of Both Worlds

Microservice benefits

Monolith Benefits

Drawbacks

Transitioning to a complete microservice design

Video

Footnotes

Sweat the small stuff

The ngrok support strategy

Pursuing a technical solution

Sweat the small stuff

Principles of designing Go APIs with channels

Principle #1

Examples

signal.Notify

Principle #2

Examples

signal.Notify

ssh.Conn.OpenChannel

Principle #3

src/pkg/net/rpc/client.go

Principle #4

Examples

ssh.NewClient

Principle #5

time.After

P-P-P-Principle Breakers

Breaking Principle #1

Breaking Principle #4

Examples

time.Tick

More and revisions

Automate away your problems: combatting illegal abuses of ngrok

A motivating outage

Possible Solutions

Only provide TLS tunnels to paid accounts

Require signup before you can create any tunnels

Inspect tunnel traffic on the wire for illegal content patterns

Make ngrok a completely paid service

Automating the banhammer

The anatomy of a hack

Automated handling of email

The script

The blacklist service

It’s just the beginning

gonative: Cross compiling Golang programs with native libraries

A better solution

gonative

About that portabililty thing . . .

A fun detour

Relevant mailing list thread

Code at 30,000 feet

No interruptions

Few distractions

Guaranteed time

A brief period for inspiration

ngrok tunnels: better, faster, stronger

Tunnels behind port-restricted networks and firewalls

Tunnels behind HTTP proxies

Multiple simultaneous tunnels

Faster tunneled connections

Better Crash Behavior

Simplify and improve self-hosting

ngrok on linux/arm

HTTPS for everyone, by default

Get it

Support ngrok

Tweet about ngrok

File a bug or feature request

Contribute code

Pay for ngrok