Evert Pot's blog

Creating a fake download counter with Web Components

2024-07-02T16:07:33+00:00

Over the years I’ve written several open source libraries. They’re mostly unglamorous and utilitarian, but a bunch of them obtained got a decent download count, so I thought it would be fun to try and get a grand total and show a ‘live’ download counter on my blog.

This is how that looks like:

My open source packages were downloaded roughly 138945563 times.

Like most live counters, this number isn’t tracked in real-time. Instead it just uses a start number and updates the number based on an average number of downloads.

One day it might be nice to make it live, but this is a static blog and this would need proper hosting and a database.

Why is this number so high?

This data comes from NPM and Packagist, and they both count any download. So this number doesn’t represent necessarily 138 million users, but simply this many downloads by any means including bots, CI environments and so on. I think for both package managers the goal was probably not to have a realistic representation of users, but rather a number that makes developers feel good. And I like that. It’s nice to see a number go up and it’s still a nice proxy for relative popularity.

The Web Component

This seemed like a good use-case for a web component. Always wanted to build one! I was surprised how easy it was.

This is how this looks in the HTML:

<p>
  My open source packages were downloaded roughly
  <strong>
    <download-counter inc-per-day="157444" date="2024-06-29T15:34:00Z" >
      138945563
    </download-counter>
  </strong> times.
</p>

What’s nice is that if Javascript is not enabled, or Web Components are not supported, this will just fall back on showing the static number.

I included 3 parameters:

The last recorded download count (in the element value)
When that number was recorded (date)
Average number of downloads per day.

I wanted to include the date because I only intend to update these numbers rarely, so we need to know how many downloads have elapsed since the last time.

Writing the web component was surprisingly straightforward too. Here is it in its fully glory:

class DownloadCounter extends HTMLElement {

  connectedCallback() {
    this.count = +this.textContent;
    this.date = new Date(this.getAttribute('date'));
    this.inc = (+this.getAttribute('inc-per-day')) / (3600 * 24 * 1000)
    this.calculateCurrentDownloads();

  }

  calculateCurrentDownloads() {

    const currentDownloads =
      Math.floor(
      this.count +
      (Date.now()-this.date) * this.inc
      );

    // Intl.NumberFormat adds thousands separators
    this.textContent = Intl.NumberFormat().format(currentDownloads);

    setTimeout(
      () => this.calculateCurrentDownloads(),
      // Add some randomnes
      Math.floor(Math.random() * 150)+50,
    );

  }

}

customElements.define(
  'download-counter',
  DownloadCounter,
);

Now just include the file in the HTML and you’re good to go:

<script type="module" src="/assets/js/downloadcounter.mjs"></script>

Obtaining the data

I’ve published 20 PHP libraries on packagist and 30 Javascript libraries on NPM. This is too much to count, so instead I wrote some scripts that pull in the data for their respective APIs.

You cannot easily ask these APIs what the download count or download rate at a given date was, and for both the numbers might be a bit delayed. So I’ve opted to simply:

Get a grand total of all downloads up until this date.
Wait at least 24 hours or more.
Get another grand total and use the difference to caclulate average download rate.

Below is my approach for getting the numbers from NPM and Packagist. It’s a bit messy and imperative.

NPM

The NPM api gave me weird, incomplete results for getting the whole list of packages I’ve authored, so I worked around this by hardcoding some package names, and then augmenting this with the result of 3 searches.

This gives us the full list of packages I’m interested in:

// Searches

const npmSearches = [
  // by author
  'author:evrt',

  // by org name
  'scope:badgateway',
  'scope:curveball',
];

// Hardcoded extra packages that for some reason didn't get returned with
// the searches
const npmPackages = {
  'davclient.js': 0,
  'structured-headers': 0,
  'bigint-money': 0,
  'fetch-mw-oauth2': 0,
  'hal-types': 0,
  'react-ketting': 0,
  'ketting': 0,
  'html-form-enhancer': 0,
  'changelog-tool': 0,
};


async function fetchNpmPackageList() {

  for(const search of npmSearches) {

    const res = await fetch('https://registry.npmjs.org/-/v1/search?text=' + search);
    const body = await res.json();

    for(const object of body.objects) {
      npmPackages[object.package.name] = 0;
    }
  }

}

NPM doesn’t return absolute total download counts but instead we can get a count for a specific time range with a maximum range of 18 months. I’ve opted to instead to get download counts per year, counting backwards for each package until get a result of 0.

async function fetchNpmDownloadCounts(packageName) {

  let year = new Date().getFullYear();
  let count = 0;
  let yearCount;

  do {
    yearCount = await fetchNpmDownloadCountsByYear(packageName, year);
    count += yearCount;
    year--;
  } while (yearCount > 0);

  console.log('%s: %i', packageName, count);
  return count;
}

async function fetchNpmDownloadCountsByYear(packageName, year) {

  const res = await fetch(`https://api.npmjs.org/downloads/point/${year}-01-01:${year}-12-31/${packageName}`);
  const body = await res.json();
  return body.downloads;

}

Packagist

Getting totals from packagist was considerably easier:

const packagistOrgs = [
  'sabre',
  'evert',
];

const packagistPackages = {};

async function fetchPackagistPackages() {

  for(const vendor of packagistOrgs) {
    const res = await fetch(`https://packagist.org/packages/list.json?vendor=${vendor}`);
    const body = await res.json();
    for(const pkg of body.packageNames) {
      packagistPackages[pkg] = 0;
    }
  }

}

async function fetchPackagistDownloadCounts(packageName) {

  const res = await fetch(`https://packagist.org/packages/${packageName}/stats.json`);
  const json = await res.json();
  const count = json.downloads.total;

  console.log('%s: %i', packageName, count);

  return count;

}

Putting it together

async function main() {

  await fetchNpmPackageList();
  await fetchPackagistPackages();

  for(const pkg of Object.keys(npmPackages)) {
    npmPackages[pkg] = await fetchNpmDownloadCounts(pkg);
  }

  for(const pkg of Object.keys(packagistPackages)) {
    packagistPackages[pkg] = await fetchPackagistDownloadCounts(pkg);
  }

  const packagesCombined = [];
  for(const [packageName, downloads] of Object.entries(npmPackages)) {
    packagesCombined.push({
      ecosystem: 'npm',
      packageName,
      downloads
    });
  }
  for(const [packageName, downloads] of Object.entries(packagistPackages)) {
    packagesCombined.push({
      ecosystem: 'packagist',
      packageName,
      downloads
    });
  }
  console.log(packagesCombined);
  console.log('Total: %i', packagesCombined.reduce((acc, cur) => acc+cur.downloads, 0));

}
await main();

Conclusion

Whenever I need small bits of Javascript to enhance a web application (and not a full-blown framework) I tend to write code that looks for an element with a particular class, and then start my logic and hook up events.

Web Components seems like a really great replacement for that.

Got comments? Liked this aticle? You can reply to this Mastodon post to make the comments show up here.

Moving on from Mocha, Chai and nyc.

2024-04-21T03:00:00+00:00

I’m a maintainer of several small open-source libraries. It’s a fun activity. If the scope of the library is small enough, the maintenance burden is typically fairly low. They’re usually mostly ‘done’, and I occasionally just need to answer a few questions per year, and do the occasional release to bring it back up to the current ‘meta’ of the ecosystem.

Also even though it’s ‘done’, in use by a bunch of people and well tested, it’s also good to do a release from time to time to not give the impression of abandonment.

This weekend I released a 2.0 version of my bigint-money library, which is a fast library for currency math.

I originally wrote this in 2018, so the big BC break was switching everything over to ESM. For a while I tried to support both CommonJS and ESM builds for my packages, but only a year after all that effort it frankly no longer feels needed. I was worried the ecosystem was going to split, but people stuck on (unsupported) versions of Node that don’t support ESM aren’t going to proactively keep their other dependencies updated, so CommonJS is for (and many others) in the past now. (yay!)

Probably the single best way to keep maintenance burden for packages low is to have few dependencies. Many of my packages have 0 dependencies.

Reducing devDependencies also helps. If you didn’t know, node now has a built-in testrunner. I’ve been using Mocha + Chai for many many years. They were awesome and want to thank the maintainers, but node --test is pretty good now and has pretty output.

It also:

Is much faster (about twice as fast with Typescript and code coverage reporting, but I suspect the difference will grow with larger code bases).
Easier to configure (especially when you’re also using Typescript. Just use tsx --test).
It can output test coverage with (--experimental-test-coverage).

Furthermore, while node:assert doesn’t have all features of Chai, it has the important ones (deep compare) and adds better Promise support.

All in all this reduced my node_modules directory from a surprising 159M to 97M, most of which is now Typescript and ESLint, and my total dependency count from 335 to 141 (almost all of which is ESLint).

Make sure that Node’s test library, coverage and assertion library is right for you. It may not have all the features you expect, but I keep my testing setup relatively simple, so the switch was easy.

OAuth2 client updates

2024-02-05T15:00:00+00:00

I just released v2.3.0 of @badgateway/oauth2-client, which I wrote because there weren’t any lean, 0-dependency oauth2 clients with modern features such as PKCE.

This new version includes support for:

Resource Indicators for OAuth 2.0 (RFC8707).
OAuth2 Token Revocation (RFC7009).

Hope you like it!

Using JSX on the server as a template engine

2023-11-14T13:14:02+00:00

The React/Next.js ecosystem is spinning out of control in terms of magic and complexity. The stack has failed to stay focused and simple, and it’s my belief that software stacks that are too complex and magical must eventually fail, because as sensibilities around software design change they will be unable to adapt to those changes without cannibalizing their existing userbase.

So while React/Next.js may be relegated to the enterprise and legacy systems in a few years, they completely transformed front-end development and created ripple effects in many other technologies. One of many great ideas stemming from this stack is JSX. I think JSX has a chance to stay relevant and useful beyond React/Next.

One of it’s use-cases is for server-side templating. I’ve been using JSX as a template engine to replace template engines like EJS and Handlebars, and more than once people were surprised this was possible without bulky frameworks such as Next.js.

So in this article I’m digging into what JSX is, where it comes from and how one might go about using it as a simple server-side HTML template engine.

What is JSX?

JSX is an extension to the Javascript language, and was introduced with React. It usually has a .jsx extension and it needs to be compiled to Javascript. Most build tools people already use, like ESbuild, Babel, Vite, etc. all support this natively or through a plugin. Typescript also natively supports it, so if you use Typescript you can just start using it without adding another tool.

It looks like this:

const foo = <div>
  <h1>Hello world!</h1>
  <p>Sup</p>
</div>;

As you can see here, some HTML is directly embedded into Javascript, without quotes. It’s all syntax. It lets you use the full power of Javascript, such as variables and loops:

const user = 'Evert';
const todos = [
  'Wash clothes',
  'Do dishes',
];

const foo = <div>
  <h1>Hello {evert}</div>
  <ul>
    {todos.map( todo => <li>todo</li>)}
  </ul>
</div>;

It has a convention to treat tags that start with a lowercase character such as <h1> as output, but if the tag starts with an uppercase character, it’s a component, which usually is represented by a function:

function HelloWorldComponent(props) {

  return <h1>Hello <span>{props.name}</span></h1>

}

const foo = <section>
  <HelloWorldComponent name="Evert" />
</section>;

Anyway, if you’re reading this you likely knew most of this but it’s important to state that this are all JSX features, not React.

#Inspo

JSX probably has it’s roots in E4X (and more directly XHP). E4X was a way to embed XML in Javascript. E4X has been supported in Firefox for years, and was a part of ActionScript 3 onwards, but there’s a major conceptual difference between E4X and JSX.

With E4X you embed XML documents as data, similar to defining a JSON object in a Javascript file. After defining the XML document you can manipulate it. A fictional transpiler for E4X could (as far as I can tell) could just take the XML structure and turn it into a string and pass it to DOMParser.parseFromString(). In E4X there are no variables, functions, components, or logic. Similar to how a Regular expression is part of the JS language.

JSX is quite different to this. It’s fully integrated in the language and it effectively compiles down to nested function definitions. These functions are don’t get turned into HTML until they are called by some render function. Before this they are defined but not evaluated.

So while E4X and JSX share that they both make XML/HTML first-class citizens in the language, the goals and features are wildly different. JSX really is a DSL.

JSX Transpiled

Lets turn the previous example into Typescript, and see what Typescript does with it:

function HelloWorldComponent(props: { name: string }) {

  return <h1>Hello <span>{props.name}</span></h1>

}

const foo = <section>
  <HelloWorldComponent name="Evert" />
</section>;

By default, Typescript will turn it into this:

function HelloWorldComponent(props) {
    return React.createElement("h1", null,
        "Hello ",
        React.createElement("span", null, props.name));
}
const foo = React.createElement("section", null,
    React.createElement(HelloWorldComponent, { name: "Evert" }));

Now, there’s definitely a react dependency here, but in recent versions of Typescript, we can configure it to use a different ‘factory’ for JSX. By setting the jsx and jsxFactory settings in tsconfig.json, we can get Typescript to output more generic code:

function HelloWorldComponent(props) {
    return _jsxs("h1", { children: ["Hello ", _jsx("span", { children: props.name })] });
}
const foo = _jsx("section", { children: _jsx(HelloWorldComponent, { name: "Evert" }) });

So what’s _jsx? This is a ‘jsx factory’ function that can be provided by React, but also a number of other libraries such as Preact or Solid.js. It’s also relatively easy to create your own.

What’s interesting then about JSX is that while it’s syntax, it’s not always clear what this yields:

const foo = <h1>Sup!</h1>

Because what’s in foo depends on what JSX Factory was used during transpiling. This is frustrating because it requires out of band information and external configuration, but also a benefit because it opens the doors to alternative implementations.

Using JSX as a template engine

When generating HTML on the server with Node, it’s pretty common to use template engines such as EJS or Handlebars. When building a new (multi-page) web application, it occurred to me that neither are quit as good as JSX.

Some of the advantages of JSX are:

Deep IDE/Language server/intellisense integration, because it’s all syntax.
Static type checking with Typescript.
It also enforced correctly structured HTML. No way to forget to close an element.
By default dynamic data is escaped. They made it challenging to get unescaped HTML!

This had me wondering, can we use server-side microframeworks such as Koa, Fastify or Express but instead of string-based template parsers and get all the benefits of JSX?

Turns out the answer is, yes! It’s not only pretty simple to implement, it works exceedingly well.

I’ve implemented this pattern 3 times now for different frameworks, here’s some sample code that works:

Koa

Here’s an example of a small controller:

import Router from 'koa-router';

const router = new Router();

router.get('/foo.html', ctx => {

  ctx.response.body = <h1>Hello world with JSX in Koa!</h1>;

});

As you can see here we can set JSX straight on the body property. A middleware ensures this gets transformed into HTML.

Fastify

Similar to Koa, we can use JSX instead where otherwise HTML strings would be used:

import { FastifyInstance } from 'fastify'

export function routes(fastify: FastifyInstance) {

  fastify.get('/hello-world', request => <h1>Sup Fastify!</h1>);

}

How does this magic work?

For both of these I used the React library. Despite it’s ecosystem being rather bulky, standalone React is still pretty lean and highly optimized.

For both of these frameworks, I simply created a middleware that checks if the body is a React node, and if so use React’s renderToStaticMarkup to do the transformation before the response is sent.

Koa middleware

import { renderToStaticMarkup } from 'react-dom/server';
import { isValidElement } from 'react';
import { Context, Middleware, Next } from 'koa';

export const jsx: Middleware = async(ctx: Context, next: Next) => {

  await next();

  if (isValidElement(ctx.response.body)) {
    ctx.response.body = '<!DOCTYPE html>\n' + renderToStaticMarkup(
      ctx.response.body
    );
    ctx.response.type = 'text/html; charset=utf-8';
  }

};

This is how it’s used:

const application = new Koa();
application.use(jsx);

Fastify hooks

This one needed a few more hacks, but the result is worth it:

import { onSendHookHandler, preSerializationHookHandler } from 'fastify';
import { isValidElement } from 'react';
import { renderToStaticMarkup } from 'react-dom/server';

/**
 * The preserialization hook lets us transform the response body
 * before it's json-encoded.
 *
 * We use this to turn React components into an object with a ___jsx key
 * that has the serialized HTML.
 */
export const preSerialization: preSerializationHookHandler<unknown> = async (_request, reply, payload: unknown) => {

  if (isValidElement(payload)) {
    reply.header('Content-Type', 'text/html');
    return {
      ___jsx: '<!DOCTYPE html>\n' + renderToStaticMarkup(payload as any)
    };
  } else {
    return payload;
  }

};

/**
 * The onSendHookHandler lets us transform the response body (as a string)
 * We detect the ___jsx key and unwrap the HTML.
 */
export const onSend: onSendHookHandler<unknown> = async (_request, _reply, payload: unknown) => {

  if (typeof payload==='string' && payload.startsWith('{"___jsx":"')) {
    return JSON.parse(payload).___jsx;
  }
  return payload;

};

To use the hook:

const fastify = Fastify();

// These 2 hooks allow us to render React/jsx tags straight to HTML
fastify.addHook('preSerialization', jsxRender.preSerialization);
fastify.addHook('onSend', jsxRender.onSend);

Using Preact or your own factory should also totally work here.

Limitations to this approach

Users of React and other frameworks may be used to pulling in data asynchronously and updating their document. This approach only allows for a single pass, so this means that all dynamic data to your JSX templates needs to be fetched in advance and passed down as props.

This means no hooks or state.

To me this is an advantage because the paradigm it replaces is regular templates, and with those the data is typically also passed in ‘at the top’.

It would certainly be possible to implement Suspend and allow for asynchronous data fetching or wait for other signals, but I haven’t yet needed this.

Frequently asked questions

How do you handle routing?

The short answer is: you don’t. Routing is already handled by your framework, and each route/endpoint/controller is responsible for rendering it’s entire page.

To reuse things like the global layout, you use components. A slightly fictionalized example of a page for us looks like this:

ctx.body = <PublicLayout>

  <div>
    <h1>Welcome back!</h1>
  </div>
  <form method="post" action="/login">

    <label>
      <span>Email</span>
      <input type="email" name="email" required minLength={10} />
    </label>
    <label>
      <span>Password</span>
      <input type="password" name="password" required minLength={4} />
    </label>
    <button type="submit">Log In</button>

  </form>
</PublicLayout>;

This in effect ‘inherits’ from PublicLayout, which looks like this:

type Props = {
  children: JSX.Element[]|JSX.Element;
  className: string;
}

/**
 * This is the main application wrapper, shared by all HTML pages.
 */
export function PublicLayout(props: Props) {

  return <html>
    <head>
      <title>Sprout Family</title>
      <link href="/css/main.css" rel="stylesheet" type="text/css" />
      <meta name="viewport" content="width=device-width, initial-scale=1" />
      <script src="/js/utils.js"></script>
    </head>
    <body className={props.className}>
      { props.children }
    </body>

  </html>;

}

Is there still a reason to use Handlebars or EJS?

I think one benefit of these template languages are they they are their own isolated format with limited capabilities.

This is helpful when for example you build an application and let your own users edit templates. Perhaps you for example have a ‘welcome email’ and want to let your users/tenants modify it.

Giving them the full power of a Javascript + a DSL that needs to be transpiled might be a negative here, because limiting features makes it easier to evolve systems and there’s also a major security component.

My template engine of choice for these is probably handlebars or even the more limited variant mustache.

Anyway

Hope this was interesting. If there’s interest in turning my Koa and Fastify code into real NPM packages, let me know! Happy to do it if there’s a non-0 number of users.

In the future I might even be interested in developing my own JSX Factory that allows awaiting for async components.

If any of this sounds interesting, you have a scathing critique or found punctuation in the wrong place, you can leave a comment by replying to this Mastodon post.

Why aren't there more 80% jobs?

2023-10-12T20:29:12+00:00

There’s been a bit of a trend recently for some companies to move to 4-day workweeks. This is making a decent amount of noise, but the actual number of companies offering this still seems pretty few and far between. It’s not hard to imagine why CEOs might feel this is risky, considering that many don’t even trust people to work from home.

What I don’t see much are 80% jobs, which are 32 hour jobs, at 80% salary. This should be a low-risk proposition that I think a lot of people in the tech industry would take, if it were an option.

My background

I grew up in the Netherlands, but I moved to Canada for my first programmer job at age 20. I stayed for a few years, moved back and started a job there, and eventually moved back to Canada again and pretty much settled here.

I got strong roots and professional experience in both places, but there was one thing that surprised me most in the Netherlands that I haven’t really seen in North America.

Apparently Netherlands has a very high rate of part-time careers. Statistics suggest that they are the highest in the world by a large margin. My personal experience matches this too. I know a number of people in the Netherlands that don’t work full-time, and I think there are more opportunities for part-time career jobs. I don’t see them much in North America.

OECD Chart: Part-time employment rate, Total, % of employment, Annual, 2022

My first-hand experience is when I was interviewing a few years into my career at a company named Ibuildings in Utrecht, Netherlands. After the interview I was told I pretty much had the job, and then the interviewer asked me if I wanted to work 4 or 5 days per week.

I only had Canadian work experience, so I was pretty shocked being asked this. Picking 4 days seemed like a no-brainer to me. I would get paid for 32 hours per week, instead of 40 and got to pick a day of the week I wanted off (I picked Friday). At this company I believe most people took the 4-day option. So yes, I got paid 20% less, but with the tax bracket I was in this was closer to 10%.

For me it was great. 5 days actually feels like a lot, and 2 days in a weekend never feels quite enough. A sentiment so common it’s boring talk about it. I used the extra time to work on open source, errands, and picked a few small freelance gigs here and there.

I’m leaning pretty socialist, but even with a capitalist hat on, I’m surprised this doesn’t happen more. Why not give employees the option to do this? Not everyone will take the option, but for those that do here’s some advantages:

Advantages of offering 80% jobs

For the salary cost of 4 employees at 40 hours, you get 5 employees at 32 hours.
Those 5 employees are more rested, and may be generally happier.
There’s a lower risk of burn-out and attrition.
You get the combined experience of 5 people instead of 4.
Also I don’t believe for the 80% workers, you only get 80% output. Most people just aren’t productive all the time.
Advertising this as a benefit in your company may also be attractive to candidates for hiring.

There are also some drawbacks. Every employee will have some overhead such as a laptop, benefits and general admin. Probably a larger expense is the cost to recruit, although in the long run a lower attrition rate might make up for that a bit.

But my intuition tells me that this overhead is probably well worth the benefit of a smarter, larger, happier workforce.

Some notes:

I want to stress that working 4 days should be presented as a choice. Some people prefer to work more for more money, and you don’t want to scare them.
Another, even more lightweight option for irriationally risk-averse traditionalists is to offer 90% jobs, resulting in an extra day every 2 weeks.

Does OAuth2 have a usability problem? (yes!)

2023-04-27T02:18:00+00:00

I read an interesting thread on Hackernews in response to a post: “Why is OAuth still hard in 2023”. The post and comments bring up a lot of real issues with OAuth. The article ends with a pitch to use the author’s product Nango that advertises support for supporting OAuth2 flows for 90+ APIs and justifying the existence of the product.

We don’t need 90 browsers to open 90 websites, so why is this the case with OAuth2? In a similar vain, the popular passport.js project has 538(!) modules for authenticating with various services, most of which likely use OAuth2. All of these are NPM packages.

Anyway, I’ve been wanting to write this article for a while. It’s not a direct response to the Nango article, but it’s a similar take with a different solution.

My perspective

I’ve been working on an OAuth2 server for a few years now, and last year I released an open source OAuth2 client.

Since I released the client, I’ve gotten several new features and requests that were all contributed by users of the library, a few of note are:

Allowing client_id and client_secret to be sent in request bodies instead of the Authorization header.
Allow ‘extra parameters’ to be sent with some OAuth2 flows. Many servers, including Auth0 require these.
Allow users to add their own HTTP headers, for the same reason.

What these have in common is that there’s a lot of different OAuth2 servers that want things in a slightly different/specific way.

I kind of expected this. It wasn’t going to be enough to just implement OAuth2. This library will only work once people start trying it with different servers and run into mild incompatibilities that this library will have to add workarounds for.

Although I think OAuth2 is pretty well defined, the full breadth of specs and implementations makes it so that it’s not enough to (as an API developer) to just tell your users: “We use OAuth2”.

For the typical case, you might have to tell them something like this:

We use OAuth2.
We use the authorization_code flow.
Your client_id is X.
Our ‘token endpoint’ is Y.
Our ‘authorization endpoint’ is Z.
We require PKCE.
Requests to the “token” endpoint require credentials to be sent in a body.
Any custom non-standard extensions.

To some extent this is by design. The OAuth2 spec calls itself: “The OAuth 2.0 Authorization Framework”. It’s not saying it is the protocol, but rather it’s a set of really good building blocks to implement your own authentication.

But for users that want to use generic OAuth2 tooling this is not ideal. Not only because of the amount of information that needs to be shared, but also it requires users of your API to be familiar with all these terms.

A side-effect of this is that API vendors that use OAuth2 will be more likely roll their own SDKs, so they can insulate users from these implementation details. It also creates a market for products like Nango and Passport.js.

Another result is that I see many people invent their own authentication flows with JWT and refresh tokens from scratch, even though OAuth2 would be good fit. Most people only need a small part of OAuth2, but to understand which small part you need you’ll need to wade through and understand a dozen IETF RFC documents, some of wich are still drafts.

Sidenote: OpenID Connect is another dimension on top of this. OpenID Connect builds on OAuth2 and adds many features and another set of dense technical specs that are (in my opinion) even harder to read.

OAuth2 as a framework is really good and very successful. But it’s not as good at being a generic protocol that people can write generic code for.

Solving the setup issue

There’s a nice OAuth2 feature called “OAuth 2.0 Authorization Server Metadata”, defined in RFC8414. This is a JSON document sitting at a predictable URL: https://your-server/.well-known/oauth-authorization-server, and can tell clients:

Which flows and features are supported
How to pass credentials
URLs to every endpoint.

Here’s an example from my server:

{

    "issuer": "http://localhost:8531",
    "authorization_endpoint": "/authorize",
    "token_endpoint": "/token",
    "token_endpoint_auth_methods_supported": [
        "client_secret_basic"
    ],
    "token_endpoint_auth_signing_alg_values_supported": [
        "RS256"
    ],
    "jwks_uri": "http://localhost:8531/.well-known/jwks.json",
    "scopes_supported": [
        "openid"
    ],
    "response_types_supported": [
        "token",
        "code",
        "code id_token"
    ],
    "grant_types_supported": [
        "client_credentials",
        "authorization_code",
        "refresh_token"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "service_documentation": "http://localhost:8531",
    "ui_locales_supported": [
        "en"
    ],
    "introspection_endpoint": "/introspect",
    "revocation_endpoint": "/revoke",
    "revocation_endpoint_auth_methods_supported": [
        "client_secret_basic"
    ]

}

If your server and client supports this, it can simplify the setup a great deal. Here’s an example using my client:

import { OAuth2Client } from '@badgateway/oauth2-client';

const client = new OAuth2Client({
  server: 'https://my-auth-server/',
  clientId: 'my-client-id'
});

The problem is majority of OAuth2 servers and clients don’t support this, so even if your server did, your setup instructions would still need to include all the ‘setup info’ for clients that don’t support the discovery document.

And for the clients that do support it, you would need to call out that your users’ client needs support for RFC8414.

So while I think the discovery spec is great and solves real problems; it alone cannot solve the OAuth2 usability problem.

My proposal

Currently work is underway to define OAuth 2.1. OAuth 2.1 will remove features that are considered insecure or bad practices (such as implicit and password flows) and PKCE is brought in as a core feature.

If you’re a OAuth 2 client or server maintainer and kept up with the specs, you likely are already compatible with OAuth 2.1.

I don’t think OAuth 2.1 goes far enough.

I think that this proposal should require support for the discovery document and make it a required step to find features and endpoints. I also think it should have an opinion on how clients should support custom extensions and how they might work. (or forbid them).

This version of OAuth should also provide a way to discover the discovery endpoint (hear me out).

If a client makes a HTTP request to an API, and the API replies with 401, it should tell the client where to find the discovery document and which flow(s) to use.

Lastly, I think that it should be renamed to OAuth 3, so API vendors no longer have to state:

We use OAuth 2.
We use the authorization_code flow.
Your client_id is X.
Our ‘token endpoint’ is Y.
Our ‘authorization endpoint’ is Z.
We require PKCE.
Requests to the “token” endpoint require credentials to be sent in a body.
Any custom non-standard extensions.

But instead:

We use OAuth 3.
Your client_id is X.

A nice aspect of this proposal is that OAuth 2 clients can still talk to OAuth 3 servers, it also doesn’t obsolete the OAuth 2 framework.

Perhaps you could name this “OAuth 2.1: the good parts”, but I think increasing the major version number sends a clear signal to users they should be looking for a OAuth 3 library.

Then maybe, 10 years from now we no longer need 538 Passport.js modules for 538 APIs. Perhaps browsers could even facilitate authentication.

Final notes

A future OAuth version should also explicitly allow http://localhost as a redirect_url. We need to be able to test.
I’m aware that there was a OAuth 3 proposal, which is now called XYZ (or maybe GNAP?). I’m not very familiar with this proposed protocol.
XKCD 927 is funny, but ultimately a conversation stopper and a bit cynical.

Switching to Fedora from Ubuntu

2023-03-30T16:00:00+00:00

It seems like every 7-8 years I’m switching operating systems. In 2006 I first started using Apple, because it was just so damn cool to have a well working Unix-like system. (I’ll never forget you Snow Leopard).

In 2015 I switched to Ubuntu. Apple’s Software seemed to hit rock bottom at this point from a quality perspective, and hardware seemed to go obsolete at a rate I hadn’t seen before. Fed up paying a premium price for a sub-par product, it was time for a change.

Ubuntu’s fall

Ubuntu was the obvious choice. I want something that just works, and Dell’s XPS 13 Developer Edition ships with Ubuntu which means hardware support from Dell itself. Breath of fresh air and fast as hell. The experience was similar to what people have been saying about the new M1 chips. But it’s fast because of software, not hardware.

But something changed with Ubuntu in recent years. I think linux users have a thicker than usual skin when it comes to software issues and are willing to look past more things. But Ubuntu’s quality has been consistently falling. I was being worn down, and it seems to be a common sentiment in my bubble.

The best example is Ubuntu’s package manager Snap. A pretty good idea, I like the branding too but the execution hasn’t been there. Ubuntu users have been effectively beta-testing this for years. Just to give you an idea I made a giant list of bugs that I ran into when Ubuntu switched Firefox from apt to Snap.

To be honest I feel a bit bad ragging on Ubuntu, because without knowing anything about how the project and Canonical is run, my intuition is that the steam is just kind of running out for them. Ubuntu feels ‘depressed’, but maybe it’s all in my head.

Onto Fedora

So I pledged to try something new in 2022, and it took me another 14 months to find the energy and motivation to actually follow through.

I went for Fedora. Named after a fashion faux pas, it seems to be on the #2 spot for desktop linux. It’s funded by Red Hat, and seems to have a focus on keeping it’s packages very up to date, which I like and why I originally switched from Debian to Ubuntu!

I’m a week and a bit in, and here are my first impressions.

Installation

Installation was super smooth. I always forget to make a separate /home mount, so it took a while to move everything to an external disk and back.

The one thing I always forget to move is my MySQL databases, and today was no exception.

Non-free stuff

Fedora does not ship with things that aren’t open source. Nothing against that philosophy (awesome in fact), but personally I don’t mind adding some binaries for a better experience.

I miss Ubuntu’s Additional Drivers tool, because it told me what to install. I’m sure the drivers I need are available for Fedora, but I don’t know what to look for which makes me slightly worried my computer is not running optimally. Battery feels worse but that could also be my imagination.

Video in Firefox didn’t work at all in stock Fedora. I had to install ffmpeg to get it to barely function, but then I discovered RPM Fusion, where I got an even better ffmpeg, plus gstreamer and Intel drivers and I can now watch beautiful smooth 4K video, and confirmed with intel_gpu_top that I’m using hardware acceleration.

Gnome

Ubuntu used to have their own desktop environment called Unity. In 2018 they switched to Gnome, but they modified Gnome to keep their Unity look.

This felt like a good move, because it let them kept their look while taking advantage of all the Gnome plumbing.

One drawback is that Ubuntu was usually a bit behind with Gnome features.

Fedora uses stock Gnome. As a result there’s more consistency overall, and it’s nice to have the latest features. I miss the strong visual identity Ubuntu has though. It sets itself apart from Mac and Windows.

I’m sure I can customize Fedora to be more fun, but if I’m being real with myself I haven’t changed my desktop background in 10 years, and so this will never happen.

Flatpak

The only thing that Flatpak had to do to be better than Snap was to not remind me of its existence unless I’m installing something, and so far it’s done that. Flathub is nice, and I love the idea of developers not having to repackage for every distro under the sun.

Bugginess

Fedora does generally feel more stable. Fewer background processes pegged at 100%. 3rd party application support doesn’t seem as good on first sight. I had to find workarounds for KeeppassXC and Element to not crash all the time.

This is no fault of the Fedora team, but it does tell you something about how much Fedora is on other people’s radars. If developers test 1 linux distro, it will be Ubuntu.

Software Center app

I falsely assumed that the Software Center application was buggy due to Ubuntu’s Snap changes, but it also hangs all the time in Fedora.

The solution is to kill the gnome-software process if you want to use ‘Software’ more than once per session. Apparently this has been an issue for at least 12 months

Things are never perfect, but even with some of the issues I ran into I’m very happy I switched. It’s hard to describe, but things feel more solid.

To get to this state I did have to put in some work and research, but not at a level of recompiling kernels.

If you are comfortable googling, using the terminal for some commands and interpreting an occasional error message I would now recommend Fedora over Ubuntu.

If you are not a technical user or programmer-adjacent, I think Ubuntu is still going to be the choice with the least amount of friction. In large I think this is simply because it is the biggest, has the most eyes and the most support.

I’m excited though. Fresh start!

Supporting CommonJS and ESM with Typescript and Node

2023-03-21T17:11:00+00:00

I maintain a few dozen Javascript libraries, and recently updated many of them to support CommonJS and Ecmascript modules at the same time.

The first half of this article describes why and how I did it, and then all the hoops I had to jump through to make things work.

Hopefully it’s a helpful document for the next challenger.

A quick refresher, CommonJS code typically looks like this:

const MyApp = require('./my-app');
module.exports = {foo: 5};

And ESM like this:

import MyApp from './my-app.js';
export default {foo: 5};

Except if you use Typescript! Most Typescript uses ESM syntax, but actually builds to CommonJS. So if your Typescript code looks like the second and think ‘I’m good’, make sure you also take a look at the built Javascript code to see what you actually use.

Why support ESM

The general vibe is that ESM is going to be the future of Javascript code. Even though I don’t think the developer experience is quite there yet, but more people will start trying ESM.

If you decided to plunge in with ESM, I want my libraries to feel first-class.

For example, I’d want you to be able to default-import:

import Controller from '@curveball/controller';

At the same time most people are still on CommonJS, and I want to continue to support this without breaking backwards compatibility for the forseeable future.

The general approach

My goal is for my packages to ‘default’ to ESM. From the perspective of a library maintainer you will be dealing with ESM.

During the build phase steps are being made to generate a secondary CommonJS build. As a maintainer you might run into CommonJS-specific issues, but I suspect people will only see those if the CI system reports an issue.

Our published NPM packages will have a directory structure roughly like this:

- package.json
- tsconfig.json

- src/     # Typescript source
- cjs/     # CommonJS
- esm/     # ESM
- test/

We include the original typescript sources to make step-through debugging work well, and have a seperate directory for the ESM and CommonJS builds.

If you just want to skip to the end and see an example of a package that has all this work done, check out my @curveball/core package:

Github source: https://github.com/curveball/core
Published NPM package contents: https://www.npmjs.com/package/@curveball/core?activeTab=code

Typescript features we don’t use

esModuleInterop

We don’t use the esModuleInterop setting in tsconfig.json. This flag lets you default-import non-ESM packages like this:

import path from 'node:path';

instead of the more awkward:

import * as path from 'node:path';

On the surface esModuleInterop seems like it would be helpful, but it has a major problem: if our libraries use esModuleInterop, anyone who uses that library is forced to also turn it on.

So turning by turning esModuleInterop off we don’t force anyone downstream into a specific setting. I generally recommend anyone writing libraries to turn this off to reduce friction and ironically increase interopability.

Path mapping

Typescript lets you map arbitrary paths to specific directories in your source. This is popular because it lets you do things like:

import MyController from '@controllers/my';

Instead of being forced to always do relative import

import MyController from '../../../controllers/my';

This is quite a popular feature for larger code-bases because people find relative paths annoying.

I’m also not using this feature. A big issue is that only Typescript is aware of this configuration, so any other tooling that uses your files may need to be configured separately to also understand this, which doesn’t always work.

So while I don’t have an exact list of exact reasons or configurations where this fails, it’s a common enough issue that I’ve decided to just avoid this feature altogether, for the sake of reducing magic. If you do insist on using path mapping, you’re on your own.

Configuring package.json

Modern node versions now have a syntax in package.json that lets you specify both the default CommonJS and ESM files. These are the relevant lines of one of our packages:

{
  "name": "@curveball/controller",
  "version": "0.5.0",
  "description": "A simple controller pattern for Curveball.js",
  "type": "module",
  "exports": {
    "require": "./cjs/index.js",
    "import": "./esm/index.js"
  },
  "main": "cjs/index.js"
}

When specifying your package.json this way, Node will automatically load the correct (cjs/esm) directory depending on what the user needs. This all happens automatically. Neat!

main is no longer used, but I’m keeping it in case of older tooling.

Building with Typescript

Before we made this change, we just had a src/ and dist/ directory for Typescript and Javascript files respectively. To do the build, we could just run npx tsc.

This still works, except npx tsc now builds to esm for the regular developer flow.

But when we build for creating the NPM package, we need to create both. We use a Makefile for this, but these are roughly the commands to run:

npx tsc --module commonjs --outDir cjs/
echo '{"type": "commonjs"}' > cjs/package.json

npx tsc --module es2022 --outDir esm/
echo '{"type": "module"}' > esm/package.json

As you can also see that both our cjs and esm packages get a 1-line package.json file with a type property.

This is required because Node needs to know wether files with a .js extensions should be interpreted as CommonJS or ESM. It can’t figure it out after opening it.

Node will look at the nearest package.json to see if the type property was specified.

It’s also possible to use .cjs or .mjs file extensions, but this doesn’t play well with Typescript as Typescript can’t automatically adjust import paths for you.

Changes we had to make in our code

I think it’s reasonable to say that that vanilla javascript and javascript modules are slightly different programming languages

They are interopable, but not the same. They have slightly different syntax and behavior. This is why you also need to tell HTML in advance what kind of flavour of Javascript you’re going to be using:

<script type="module" src="my-module.mjs"></script>

So naturally I expected to have to make some changes to write code in a way that works identical in both targets.

An obvious example is top-level await which only works in ESM. Here are all the things I ran into:

Extensions in imports

All our (local) typescript imports had to change from:

import Foo from './foo';

to:

import Foo from './foo.js';

ESM requires extensions, whereas in CommonJS it’s not needed. Here’s a sed command I used to change all these in bulk (probably not perfect, so you’ll really want to be able to roll this back):

find . -name "*.ts" | xargs -n1 sed -i "s/from '\(.*\)';/from '\\1.js'/g"

So the strange thing with all this is that your code will have statements like:

import Foo from './foo.js';

But actually the file in that directory is called ./foo.ts (.ts not .js) yes, this is confusing and annoying.

I don’t really want a future contributor of my library to have to understand the build chain, so it’s a leaky abstraction and a limitation of Typescript.

I hope this is rectified in the future. I understand the philosophy of wanting to avoid magic as much as possible, but just give me a setting to globally rewrite a .ts to .js when generating the .js files.

I secretly suspect that despite Typescript explicitly not supporting this, this might still happen later anyway. Maybe they don’t want to commit to building this before they have a good plan how 🤞.

Directory imports

In CommonJS in node you can import a directory, and if a index.js exists in that directory, it will open that. ESM requires exact paths, so your

import * from './controllers'

Now has to be:

import * from './controllers/index.js'

`dirname` and `filename`

Node defines 2 useful very variables in every CommonJS file, probably borrowed from PHP.

__dirname is a reference to the path the current file is in, and __filename is the whole path + filename.

I used these to load in non-javascript assets from relative paths. For example, I had a snippet like this to get the version of the current package:

import * as fs from 'node:fs';

const pkg = JSON.parse(
  fs.readFileSync(
     __dirname + '../package.json'
  )
);

console.log(pkg.version);

Unfortunately, these variables no longer exist in ESM. Instead we have import.meta. In Node, this is an object looking like this:

{
  url: 'file:///home/evert/src/curveball/controller/test.mjs'
}

So instead of a filesystem path, we get a file:// url with the location. So to write the equivalent in ESM we get something like this:

import * as fs from 'node:fs';
import * as url from 'node:url';

const pkg = JSON.parse(
  await fs.readFile(
    url.fileURLToPath(url.resolve(import.meta.url, './package.json')),
    'utf-8',
  )
);

console.log(pkg.version);

The challenge is writing this code in a way that will work with both. The ‘crazy’ way to solve this is to take an Error object, and get the path from the string that appears in the stack trace. This means parsing the stacktrace :(

import * as url from 'node:url';
import * as path from 'node:path';

function getDirName() {

  const err = new Error()
  console.log(err.stack?.split('\n')[1]);
  const fileUrl = err.stack?.split('\n')[1].match(/at .* \((file:.*):\d+:\d+\)$/);

  if (!fileUrl) throw new Error('This misrable idea failed');

  return path.dirname(url.fileURLToPath(fileUrl[1]));

}

console.log(getDirName());

I don’t believe that the exact format of the stack trace is stable or even the same across different interpreters, so this seems like a bad idea.

The problem is that the more obvious approach doesn’t work:

import * as path from 'node:path';
import * as url from 'node:url';

/** @ts-ignore CommonJS/ESM fun */
const dirname = 
  typeof __dirname !== 'undefined' ?
  __dirname :
  /** @ts-ignore CommonJS/ESM fun */
  path.dirname(url.fileURLToPath(import.meta.url));

The above code creates a new dirname variable. While this works for ESM, it breaks for CommonJS on Node. The reason is that import is not a regular object, it’s syntax and even if the branch with import.meta never gets run, Node will error while parsing:

/home/evert/src/curveball/controller/cjs/dirname.js:9
    path.dirname(url.fileURLToPath(import.meta.url));
                                          ^^^^

SyntaxError: Cannot use 'import.meta' outside a module

Sadness!

I don’t have a satisfying solution for this yet. I’ve worked around this in some of my packages by avoiding this altogether, or rewriting certain files after running them. eval('import.meta.url') causes the ESM build to fail because technically things running in eval is no longer in a module scope.

Importing CommonJS modules with ‘default exports’

Some of our packages rely on 3rd party dependencies that are written in plain Javascript and use CommonJS.

In CommonJS dependencies you can ‘default’ export things like:

module.exports = function foo(a, b) { a+b; };

If we were to use this library in Typescript (CJS build) we could use it this way:

import * as WebSocket from 'ws';
const ws = new WebSocket('ws://localhost:8000');

But if we are in ESM land in Typescript, this doens’t work. We actually get an object with a property named .default. For these modules, this is the approach I’ve come up with:

import * as WebSocketImp from 'ws';

// ESM shenanigans
const WebSocket = 'default' in WebSocketImp ? (WebSocketImp.default as any) : WebSocketImp;

Yep! It’s a pain. I’ve had to do this for websocket, chai-as-promised, ajv, raw-body, accepts and http-link-header. In case you haven’t figured it out, I’m building a framework.

Testing

To have confidence in both builds, I want to write tests that run against both. My tests are written using Mocha. My tests are also written in Typescript. By default, they only run against the ESM build, and I expect myself and other developers to be in this mode during the main ‘development loop.

To make Mocha understand Typescript, we need to install the ts-node package, and add the following properties to the root package.json:

"mocha": {
  "loader": [
    "ts-node/esm"
  ],
  "recursive": true,
  "extension": [
    "ts",
    "js",
    "tsx"
  ]
}

This all works perfectly well, but what about CommonJS? For this I had 2 approaches: Either reconfigure mocha to use the CommonJS Typescript loader instead of the ESM typescript loader, but this was too much of a pain to get right with writing configuration files on the fly.

Instead, I decided to just ask Typescript to build the entire project including tests in a separate directory and then just run Mocha on the javascript files.

mkdir -p cjs-test
cd test; npx tsc --module commonjs --outdir ../cjs-test
echo '{"type": "commonjs"}' > cjs-test/package.json
cd cjs-test; npx mocha --no-package

This required a separate tsconfig.json in our test directory.

Conclusion

So this is a fairly large amount of annoyances to work through, some without a solution. Would I recommend this?

I think if you’re in my position where:

You’re doing this for a library.
You consider ESM the future, but want to continue to support CommonJS users.
The experience of users of your library is far more important than your own experience.

Then, I think this is a good idea. If you don’t want this hassle and want just one target, I think CommonJS will still give you the best experience.

ESM will probably catch up at some point, but right now I think we’re still in the awkward phase.

Winding down Bad Gateway

2023-02-20T19:18:15+00:00

In 2019 I started Bad Gateway as a software development agency. Last year we grew all the way to 7 people. It was crazy challenging, especially with Covid in the mix; but ultimately could not get the company into a good financial state to be able to carry on.

Big thanks to my co-workers and partners Ju, Becky, Phil, Michael, Siep, Richard and Syed. I’m incredibly grateful you came on this journey with me. We shared some tears, exchanged some words but mostly had lots of laughs. Despite the challenges I feel my relationship with you has only strengthened and I wish you well in the next steps of your career.

Also thank you to our customers and especially Underknown who’ve stuck with us since the start.

Despite this outcome, I have a hard time seeing the last few years as a failure. It’s been hella fun, and I feel we stuck to our values even in times of crisis. Off to the next adventure.

Building a simple CLI tool with modern Node.js

2023-02-13T19:05:14+00:00

I’m a maintainer of several dozen open source libraries. One thing I’ve always done is maintain a hand-written changelog.

Here’s an example from a12n-server

0.22.0 (2022-09-27)
-------------------

Warning note for upgraders. This release has a database migration on the
`oauth2_tokens` table. For most users this is the largest table, some
downtime may be expected while the server runs its migrations.

* #425: Using a `client_secret` is now supported with `authorization_code`,
  and it's read from either the request body or HTTP Basic Authorization
  header.
* The service now keeps track when issuing access tokens, whether those tokens
  have used a `client_secret` or not, which `grant_type` was used to issue them
  and what scopes were requested. This work is done to better support OAuth2
  scopes in the future, and eventually OpenID Connect.
* Fixed broken 'principal uri' in introspection endpoint response.
* OAuth2 service is almost entirely rewritten.
* The number of tokens issued is now displayed on the home page.
* Large numbers are now abbreviated with `K` and `M`.
* #426: Updated to Curveball 0.20.
* #427: Typescript types for the database schema are now auto-generated with
  `mysql-types-generator`.

These are all written in Markdown. You might think: isn’t Git also a log? Why bother hand-writing these?

The reason is that the audience for these is a bit different. I want to bring attention to the things that are the most important for the end-user, and focus on the impact of the change to the user.

I thought it would be handy to write a CLI tool that makes it a bit easier to maintain these. So, I did!. If you are curious what kind of technology choices went into this, read on.

Goals and features

The tool should be able to do the following:

Reformat changelogs (a bit like prettify) (changelog format)
Add an entry via the command line (changelog add --minor -m "New feature").
Automatically set the release date (changelog release)
Pipe a log of a specific version to STDOUT, so it can be used by other tools (like integrating with github releases).

I also had a bunch of a non-functional requirements:

Use the latest Node features.
Use up to date Javascript standards and features (ESM).
Avoid dependendencies unless it’s unreasonable to do so.
Make it low maintanance.

Want to find the finished tool right now? It’s open source so just go to Github.

The implementation

ESM & Typescript

Ecmascripts modules worked really well here. It’s a small change of habits, but the general recommendation I would have is to just save your files as .mjs and start using it.

Here’s the first few lines of parse.mjs:

// @ts-check
import { Changelog, VersionLog } from "./changelog.mjs";
import { readFile } from 'node:fs/promises';

/**
 * @param {string} filename
 * @returns {Promise<Changelog>}
 */
export async function parseFile(filename) {

  return parse(
    await readFile(filename, 'utf-8')
  );

}

The CommonJS -> ESM transition is not without pain, but for an new project like this it’s really the ideal choice. (top level await 🎉)

I’ve also made the choice to not write my code in Typescript, but use JSDoc annotations instead (These are the @param and @returns comments).

Not everyone knows that you don’t need to write .ts files to benefit from Typescript. Typescript can also check your Javascript files quite strictly, and there’s a lot of work still being done in adding new features to this syntax.

This has the benefit of not needing a build phase. You don’t even need Typescript during development which reduces the barrier to entry.

Here’s my minimal tsconfig.json:

{
  "compilerOptions": {
    "target": "es2022",
    "module": "esnext",
    "rootDir": "./",
    "allowJs": true,
    "checkJs": true,

    "moduleResolution": "node",
     
    "noEmit": true,
    "strict": true,
    "useUnknownInCatchVariables": false,
  
  }
}

The Typescript documentation has a page detailing the JSDoc annotations they support, if you’d like to learn more.

Command line parsing

CLI tools need to be able to parse command line options. Since Node 18.3 (backported to Node 16.17) Node comes with a built-in options parser.

Here’s a sample of the code:

import { parseArgs } from 'node:util';

const { positionals, values } = parseArgs({
  options: {
    help: {
      type: 'boolean',
      short: 'h',
      default: false,
    },
    all: {
      type: 'boolean',
      default: false,
    },
    message: {
      type: 'string',
      short: 'm'
    },
    patch: { type: 'boolean' },
    minor: { type: 'boolean' },
    major: { type: 'boolean' },
  },
  allowPositionals: true,
});

This confguration adds flags like --major, lets you specify a message with --message "hello!" or use a short-hand alternative like -m "Hi".

Does it do everything? No! There’s packages out there that are more sophisticated, use colors, automatically create help screens but they also ship with large dependency trees.

In my case, this was good enough.

Check out the Node docs for more info.

Testing

Most people probably use Jest or Mocha as their test framework, but since Node 18 (also backported to 16) Node has a built-in test-runner.

It has an API similar to Mocha and Jest with keywords like it, test, describe, before, etc.

Here’s a sample of one of my tests:

// @ts-check
import { test } from 'node:test';
import { parse } from '../parse.mjs';
import * as assert from 'node:assert';

test('Parsing changelog metadata', async () => {

  const input = `Time for a change
=========

0.2.0 (????-??-??)
------------------

* Implemented the 'list' command.
* Added testing framework.

0.1.0 (2023-02-08)
------------------

* Implemented the 'help' and 'init' commands.
*
`;

  const result = await parse(input);

  assert.equal('Time for a change', result.title);
  assert.equal(2, result.versions.length);
  
  assert.equal(null, result.versions[0].date);
  assert.equal('0.2.0', result.versions[0].version);
  assert.equal('2023-02-08', result.versions[1].date);
  assert.equal('0.1.0', result.versions[1].version);

});

To run the tests, just run node --test. No configuration needed, it will discover tests following directory and filename conventions.

The Node 18 test output is a bit rough, it’s the TAP format and looks like this:

TAP version 13
# Subtest: /home/evert/src/changelog-tool/test/parse.mjs
    # Subtest: Parsing changelog metadata
    ok 1 - Parsing changelog metadata
      ---
      duration_ms: 1.713409
      ...
    # Subtest: Parsing changelog entries
    ok 2 - Parsing changelog entries
      ---
      duration_ms: 0.2595
      ...
    # Subtest: Preface and postface
    ok 3 - Preface and postface
      ---
      duration_ms: 0.193591
      ...
    1..3
ok 1 - /home/evert/src/changelog-tool/test/parse.mjs
  ---
  duration_ms: 70.901055
  ...
1..1
# tests 1
# pass 1
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 81.481441

In Node 19 new test reporters will be shipped, but I haven’t tested this yet.

Frankly, after using this I don’t know if would use Mocha anymore. I’ve been using probably over a decade, and it has some nice features and benefits, for the kinds of tests I write (and I write a lot), I think there’s anything I need beyond what Node is offering here.

Some links:

package.json, annotated

I wanted to end this article with how I’ve set up my package.json, so you can see how it all ties together. (If only npm supported JSON5 so I could keep my comments right in the package 😭).

{
  // The name of the package, and how it's published on NPM
  "name": "changelog-tool",

  // Package version!
  "version": "0.5.0",

  // This will show up in NPM searches
  "description": "A CLI tool for manipulating changelogs",

  // This tells Node to assume this is an ESM package. Not strictly needed
  // if we use the .mjs extension everywhere though.
  "type": "module",

  // If you use this package programmatically (not on CLI), this file is
  // where all 'exports' are for this package.
  "main": "index.mjs",

  "scripts": {
    // Test runner!
    "test": "node --test",

    // I like to keep Typescript running in the terminal to insta-warn me of
    // of any issues.
    "watch": "tsc --watch"
  },

  // This helps people discover the package on npmjs.org
  "keywords": [
    "changelog",
    "markdown"
  ],

  // It's me!
  "author": "Evert Pot (https://evertpot.com/)",

  // Do (almost) anything you want
  "license": "MIT",

  "engine": {
    // Warn people if they haven't upgraded yet
    "node": ">16"
  },

  "bin": {
    // When people install this package, this makes `npx changelog` work. If
    // you installed this package globally, you now have a `changelog` command
    // at your disposal.
    "changelog": "./cli.mjs"
  },
  "devDependencies": {
    // The only 2 dependencies. You don't even really need these if you just want
    // use or hack the package. Github actions will run Typescript as well.
    "@types/node": "^18.11.19",
    "typescript": "^4.9.5"
  }
}

Conclusion

I love building new things and being deliberate about every choice.

The result is that I’m more likely to end up with something minimal, low-maintance and I gain a deeper understanding of the tools I use.

In the near future I would probably make all these choices again. The Node test runner is fast and low-cruft, ESM is great when it works and not needing a build step feels right for a project this size.

I hope this inspires someone in the future to start their next project from an empty directory instead of copying a large boilerplate project.

changelog-tool project on Github.

Knex (with MySQL) had a very scary SQL injection

2023-01-12T21:31:43+00:00

Knex recently released a new version this week (2.4.0). Before this version, Knex had a pretty scary SQL injection. Knex currently has 1.3 million weekly downloads and is quite popular.

The security bug is probably one of the worst SQL injections I’ve seen in recent memory, especially considering the scope and popularity.

If you want to get straight to the details:

Check out the Github issue, which was opened 7 years ago(!)
An article from Ghostccamm explaining the vulnerability.
CVE-2016-20018.

My understanding of this bug

If I understand the vulnerability correctly, I feel this can impact a very large number of sites using Knex. Even more so if you use Express.

I’ll try to explain through a simple example. Say, you have MySQL table structured like this:

CREATE TABLE `users` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`id`)
)

And you have a query that does a SELECT using Knex:

const lookupId = 2;

const result = await knex('users')
  .select(['id', 'name'])
  .where({
    id: lookupId
  });

You’d expect the query to end up roughly like this

SELECT `id`, `name` FROM `users` WHERE `id` = 2

The issue is when the user controls the value of lookupId. If somehow they can turn this into an object like this:

const lookupId = {
  name: 'foo'
}

You might expect an error from Knex, but instead it generates the following query:

SELECT `id`, `name` FROM `users` WHERE `id` = `name` = 'foo'

This query is not invalid. I don’t fully understand fully understand MySQL’s behavior, but it causes the WHERE clause to be ignored and the result is equivalent to:

SELECT `id`, `name` FROM `users`

I think it has something to do with how MySQL casts things. In MySQL this yields true (or 1):

SELECT 1 = 'foo' = 'bar'

Query strings and Express

One place where this is especially scary, is if lookupId was provided by a user, via a JSON body or query string.

Consider this example from an Express route:

app.get('/', async (req, res) => {
  const result = await knex('users')
    .select(['id', 'name'])
     .where({id: req.query.id});

  res.send(result)
})

Here the id in the WHERE clause is provided by the URL query string:

If a the server is opened with something like:

http://localhost:3000/?id=2

It will return a single user, but if it was opened using:

http://localhost:3000/?id%5Bname%5D=foo

It will return every user. And this issue is not limited to SELECT, it could also trigger in a WHERE clause in DELETE.

The reason this works is that by crafting URL query parameters in a special way, a user can have things in req.query show up as objects or arrays.

Express calls this the ‘extended’ syntax and turns it on by default. In my opinion this is a bad default because it’s not really what users expect will happen. PHP does this as well, and I believe this may have been where Express (or specifically the qs library) got the syntax from.

This is why I think Express users are expecially likely to be vulnerable. I think that most developers in professional settings are decent at validating JSON request bodies with a variety of tools, but I’ve noticed this is often not the case for query parameters. I believe a big reason for this is that the ‘extended’ syntax is not well known and ‘surprising’ behavior. Many people assume these are just strings. This is not intended as a plug, but this is also why the default behavior in our Curveball framework is to not support this. In most cases it’s not needed, and if you do want it you can use qs and explicitly opt-in.

Other examples

But of course, this issue is not limited to query strings. If you’re not validating input somewhere and this ends up in a .where() statement there’s risk.

A specific example is a situation where part of the contents of your WHERE clause is unguessable.

For instance, say you had a database table with records that users can only see if they know a secret code, and it’s selected with:

SELECT * FROM coupons WHERE product_id = 5 AND coupon_code = 'BIG_OOF2023'

A user presumably would have to know the coupon_code to see the coupons, but if there’s no code to validate coupon_code is a string, a user would have the power to change the query to this:

SELECT * FROM coupons WHERE product_id = 5 AND coupon_code = product_id = 'bla'

Which is equivalent to:

SELECT * FROM coupons WHERE product_id = 5

And thus no longer requiring coupon codes to get ALL the discounts.

Lastly, if you can rewrite a query that expects 1 result to return every record in a database is also an easy way to cause a Denial of service attack.

This does demonstrate again how is critical validating is and throw errors whenever you get data you don’t expect. Even if under normal circumstances nothing weird can happen, a library you use might do the wrong thing with unexpected data instead of just rejecting it.

On Knex

It’s quite unfortunate to see that this went unpatched for so long. I’d invite anyone reading this to try to not pile on on the (likely volunteer) authors but think about the larger ecosystem.

This bug was hidden in plain sight. Lots of people must have randomly ran into it and a ticket was opened. Probably the most responsible thing to do would have been what @rgmz did: do their best to contact the authors, failing that contact Github Security Team. After the Github Security Team also weren’t able to connect to the authors, make a CVE which puts this on every everyone’s radar. This ultimately led to a random bystander make their first contribution and submit a fix.

Knex feels high risk now though, and I can’t help wondering what else might be unpatched. I’ve only recently made the jump from just writing my own queries for like 20 years to Knex, but I’m probably reversing that decision for my next project.

I wish JSON5 was more popular

2023-01-09T21:29:36+00:00

As developers we write a lot of code, but we also deal with a lot of configuration files.

The three major formats I tend to use day to day are:

JSON
YAML
.env

And, they all kinda suck. JSON feels like it should never have become a format that people hand-write. So many quotes, and and configuration files need comments to tell users why certain decisions were made. .env has a specific purpose (and it’s ok at that), but it’s not a great universal format, and YAML has always been difficult to read and write to me. I can somehow never retain the syntax and end up copy-pasting things from examples.

Why YAML is difficult for me

A small example from Github workflows/actions:

steps:
  - uses: actions/checkout@v2
  - uses: actions/setup-node@v2
    with:
      node-version: 14
      registry-url: https://registry.npmjs.org/
  - run: npm ci
  - run: npm publish

I couldn’t tell you why uses has a dash in front, and node-version does not. If there’s a difference in how a YAML reader outputs them, I’m not sure how I would be able to retain this while writing YAML.

I also use/love home assistant, which lets you write some pretty cool automations using YAML. I wanted to play with this but it’s been a barrier I’ve not been able to overcome. I don’t know if it’s me. I’m been working as a programmer for 22 years. I’m decent at it, but when when I chat with some of my peers (hi mhum!) they did not share my sentiment.

YAML can also have very surprising behavior, with casting types:

From the linked article, this:

- country1: ca
- country2: no

Becomes:

- country1: ca
- country2: false

It’s a bit cherry picked, and I’m sure there’s YAML linters out there that help avoid the pitfalls, but in my mind configuration files should be simple.

There’s some configuration formats I like, such as TOML and JSON5. They strike the right balance to me with being easy to read and write, unambigious, supporting comments, strictness and not being incredibly hard to write a parser for.

TOML is like ini files on steroids, and JSON5 is JSON but with fewer quotes, comments and multi-line strings.

I could write my NPM configuration file as package.json5 and automatically convert it to package.json but that feels too surprising. My projects are already kind of eclectic, so I want the ‘plumbing’ to be unsurprising. Plus there’s the whole chicken and egg thing with needing a JSON5 parser before we have dependencies.

I’d love the NPM project to adopt JSON5. It seems like a great fit. JSON and YAML can’t be the final word for human-maintained data formats. It’s so obviously sub-optimal.

If NPM adopted JSON5, I would annotate so much in my package.json. I’d document why a dependency is needed, why we are stuck using a previous major version of a dependency and what the purpose is of each script.

I wouldn’t know what format would be ideal for Github Actions. Maybe the answer is ‘nothing’ and they need a good DSL.

And while we’re at it, stop polluting my projects root directory! Can’t we all agree on a .meta directory for finding configuration files?

Neko - A brief history and porting to Javascript

2022-11-06T21:50:00+00:00

In the early 90’s, being a frisian kid obsessed with computers there weren’t a ton of ways to get access to new software or learn more about computers.

The two main ways were exchanging 3.5” diskettes with friends, or go to the library. One book I remember more than others was “Windows for Kinderen” (“Windows for Kids”) by Addo Stuur.

I must have been around 10 years old and was obsessed by this book. It covered Windows 3.0, and when you got the book from the library, it came with a diskette filled to the brim with shareware. Mostly games and useless toys, but it still baffles me thinking they were able to cram it all on a 1.44 megabyte disk. Using floppys from the libraries was even back then a risky business given that they’re writable! Luckily this mostly went ok.

One that I remembered particularly well was ‘Neko’, an application that renders a cat in a window that follows your mouse. This must have been a popular thing to make at the time, because the diskette somehow had space for 3(!) different ports of this same application.

I don’t know what reminded me of Neko last week, but I started doing some more research, and found out that the first version was written all the way back in the 1980’s by Naoshi Watanabe for the NEC PC 9801.

Neko for the NEC PC 9801 (1980's)

After that, it was ported to the Macintosh in 1989 by Kenji Gotoh, and this art style seems to be the basis of almost every future port:

Neko on Macintosh (1989)

In 1991, IBM included it in OS/2! Apparently they paid 300,000 YEN, which is about $3000 CAD in todays money. At this point it also became public domain.

Neko on OS/2 (1991)

Since then there’s been countless ports for every platform. If you’re running linux you might be able to install one by just running:

apt install oneko

I also decided to make a version. Neko is now close to 40, so my Neko is no longer interested in following the mouse, and prefers to just sleep all day unless you wake it.

If you want to know more, the Eliot Akira wrote a great article with way more information about Neko and all its ports, this is also where I took the screenshots from this page.

You can also check out the source of my port ‘jneko’. It uses the assets of oneko, which are public domain. It was a fun exercise to create a little state machine.

Creating the assets

To create the assets for jneko, I had to convert them from the .xbm format from oneko.

I did this with imagemagick on the command line using a command like this to loop through all the files:

for f in ../oneko/bitmaps/neko/*.xbm
  convert "$f" `basename "${f%.xbm}.png"`

Sadly I still had to open each one of them with Gimp and add the alpha layer.

I hadn’t really heard of the .xbm format, but when I accidentally opened one with an editor I was surprised to see that they’re actually a text format:

#define awake_width 32
#define awake_height 32
static char awake_bits[] = {
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x04,
   0x40, 0x10, 0x10, 0x02, 0x80, 0x28, 0x28, 0x01, 0x00, 0x49, 0x24, 0x00,
   0x06, 0x44, 0x44, 0x60, 0x18, 0x84, 0x42, 0x18, 0x60, 0x82, 0x83, 0x06,
   0x00, 0x02, 0x80, 0x00, 0x00, 0x22, 0x88, 0x00, 0x0f, 0x22, 0x88, 0x78,
   0x00, 0x22, 0x88, 0x00, 0x00, 0x02, 0x80, 0x00, 0x00, 0x3a, 0xb9, 0x00,
   0x00, 0x04, 0x40, 0x00, 0x00, 0x08, 0x20, 0x00, 0x00, 0x70, 0x1c, 0x02,
   0x00, 0x40, 0x04, 0x05, 0x00, 0x20, 0x88, 0x04, 0x00, 0x10, 0x50, 0x02,
   0x00, 0x08, 0x20, 0x01, 0x00, 0x0b, 0xa0, 0x01, 0x80, 0x0c, 0x61, 0x02,
   0x40, 0x18, 0x31, 0x04, 0x40, 0x10, 0x11, 0x04, 0xc0, 0x11, 0x11, 0x07,
   0x60, 0x90, 0x13, 0x0c, 0xe0, 0xff, 0xfe, 0x0f, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};

Turns out this is the X BitMap format, which has the interesting property that it aside from an image format, they’re also valid C source files. This has the advantage that they can easily be statically compiled into C files without requiring a parser.

This is really similar to the history of the JSON format, which was designed to be a subset of Javascript, allowing Javascript programs to read the format without a new parser (using eval). Ironically the current JSON format is no longer a subset of Javascript, and Javascript engines now have a JSON.parse() function built in.

A deep dive into the source

jneko uses a state machine. All its possible states and transitions are expressed with this configuration object:

const stateMachine = {

  // Name of the state
  sleep: {

    // Which images are used for the state
    image: ['sleep1', 'sleep2'],

    // How quickly we loop through these images
    imageInterval: 1,

    // What state does this go to when clicked
    click: 'awake'
  },
  awake: {
    image: 'awake',

    // We automaticall transition to this state
    nextState: 'normal',

    // How long it takes to transition
    nextStateDelay: 2.5,
  },
  normal: {
    image: 'mati2',

    // If there's multiple nextState values, a random one is picked.
    // You make make 1 state more likely by addding it multiple times
    // to the array
    nextState: ['normal', 'normal', 'normal', 'tilt', 'scratch', 'yawn'],
    nextStateDelay: 1.5,
  },
  tilt: {
    image: 'jare2',
    nextState: 'normal',
    nextStateDelay: 1,
  },
  yawn: {
    image: 'mati3',
    nextState: ['normal', 'normal', 'sleep'],
    nextStateDelay: 1,
  },
  scratch: {
    image: ['kaki1', 'kaki2'],
    imageInterval: 0.1,
    nextState: 'normal',
    nextStateDelay: 3,
  }
};

Before Neko starts, we need to preload the images. This ensures that the animations happen instantly and not after a delay.

/**
 * The filenames are a bit funny because they were taken straight from
 * the oneko project.
 */
const imageNames = [
  'awake',
  'jare2',
  'kaki1',
  'kaki2',
  'mati2',
  'mati3',
  'sleep1',
  'sleep2',
];
/**
 * Images are actually 32x32 but it's too small for modern screens.
 */
const nekoSize = 64;
const images = Object.fromEntries(imageNames.map(name => {
  const image = new Image(nekoSize, nekoSize);
  image.src = '/assets/posts/neko/bitmaps/' + name + '.png';
  return [name, image]
}));

Finally, this class does all the heavy lifting:

class Neko {

  constructor(elem) {

    // The HTML element that hosts neko
    this.elem = elem;
    this.stateMachine = stateMachine;

    // Creating a new <img> element
    this.imgElem =  new Image(nekoSize, nekoSize);

    elem.appendChild(this.imgElem);
    this.imgElem.addEventListener('click', () => this.onClick());

    this.setState('sleep');

  }

  /**
   * This property is used to keep track of states with multiple frames
   */
  #animationIndex = 0;

  renderImage() {

    let name = this.stateMachine[this.#state].image;

    this.#animationIndex++;
    if (Array.isArray(name)) {
      name = name[this.#animationIndex % name.length];
    }
    this.imgElem.src = images[name].src;
  }

  #state = null;
  #nextStateTimeout = null;
  #imageCycleInterval = null;

  setState(stateName) {

    clearTimeout(this.nextStateTimeout);
    clearInterval(this.imageCycleInterval);

    if (Array.isArray(stateName)) {
      // If stateName was supplied as an array of strings, we'll randomly
      // pick a new state.
      stateName = stateName[Math.floor(Math.random()*(stateName.length))];
    }
    if (!this.stateMachine[stateName]) {
      throw new Error('Uknown state: ' + stateName);
    }
    this.#state = stateName;
    const stateData = this.stateMachine[this.#state];

    // If there was a nextState, we automatically transition there after
    // a delay.
    if (stateData.nextState) {
      this.nextStateTimeout = setTimeout(
        () => this.setState(stateData.nextState),
        stateData.nextStateDelay * 1000
      );
    }

    // This block is responsible for cycling through multiple images
    // of the current state.
    if (stateData.imageInterval) {

      this.imageCycleInterval = setInterval(
        () => this.renderImage(),
        stateData.imageInterval*1000
      );

    }
    this.renderImage();
  }
  onClick() {
    const stateData = this.stateMachine[this.#state];
    if (stateData.click) {
      // If the current state had a 'click' property, we'll transition
      // to that state, otherwise it's ignored.
      this.setState(stateData.click);
    }

  }

}

Now if you’d like to call Neko, make a <div id="neko"></div> somewhere. and hook up Neko with:

new Neko(document.getElementById('neko'));

To make Neko not look ugly, tell the browser to not use anti-aliasing when scaling:

#neko img {
  /* Both are needed to support all browsers currently. */
  image-rendering: pixelated;
  image-rendering: crisp-edges;
}

Thanks for reading! If you have any comments you can reply to this tweet or this mastadon post to make it show up below this article automatically.

Taking a look at Mastodon

2022-11-01T19:35:00+00:00

I’ve been a Twitter user and fan since 2007. With Twitter’s future looking a bit grim, I started looking around if there’s another place to go.

Twitter can’t really be replaced with anything else, because everyone’s Twitter experience is unique to them and their community. For me, it’s the main way I stay in touch with my Open Source / HTTP / API / Hypermedia bubble and some friends. Losing that would suck! Unfortunately, there’s no way that group would all flock to another platform.

But for the ones that do try something else, the talk of the town seems to be Mastodon.

Mastodon is interesting. On the surface it might just seem like a Twitter clone, but it’s based on a federated protocol called ‘ActivityPub’. What this means in practice is that there’s no central server. There’s many instances. Each of these instances is managed by different people, and many of them focus on specific interests.

With email, it doesn’t matter which provider you go with. Thanks to universal SMTP standards that every server uses, you can exchange messages with everyone else. This is the same with Mastodon. You’re not siloed into a single instance, and you can follow people from any other instance. Unlike email, it appears that with Mastodon you can actually migrate to different instances if you don’t like your current one.

This has some interesting side effects too. I joined the IndieWeb instance, which is a community I already loved. And even though I’m not siloed in, I get access to a local feed of like-minded people from that community. Everything feels new and more intimate.

Also, instead of one central authority that you have to trust to make the right moderation decisions, you can join one of many that aligns with your values, and you can block entire instances that don’t.

So should you join? If you use Twitter to stay on top of news and follow high profile people then probably not. If you’re like me, you might be able to find a community that fits your interest.

Will I stick to this? Who knows… but Twitter, like everything before, will fall out of favor one day and I’m enjoying Mastodon’s ad-free, open source, developer-friendly experience. Reminds me of early Twitter or mid-2000’s blogging culture.

Links

If you’re just getting started, check out https://joinmastodon.org/ and find a server.
Follow me!
To find your Twitter friends on Mastodon try Twitodon and Fedifinder.
To automatically cross-post everything from Twitter to Mastodon and vice versa, use Moa.

Lastly, one of the interesting results of Mastodon building on open protocols, is that it allows alternative implementations.

The Microblog.pub project lets you self-host a single-user instance. Instead of joining some large instance, you deploy an instance on your own domain that’s just for you. Can’t get more control than that, and this might be something I’ll consider in the future.

I don’t see why this blog couldn’t one day also be a ‘microblog’ and part of the fediverse.

Porting Curveball to Bun

2022-09-13T16:30:00+00:00

Bun is the hot new server-side Javascript runtime, in the same category as Node and Deno. Bun uses the JavascriptCore engine from Webkit, unlike Node and Deno which use V8. A big selling point is that it’s coming out faster in a many benchmarks, however the things I’m personally excited about is some of it’s quality of life features:

It parses Typescript and JSX by default (but doesn’t type check), which means there’s no need for a separate ‘dist’ directory, or a separate tool like ts-node.
It loads .env files by default.
It’s compatible with NPM, package.json, and many built-in Node modules.

I also like that it’s ‘Hello world HTTP server’ is as simple as writing this file:

// http.ts
export default {
  port: 3000,
  fetch(request: Request): Promise<Response> {
    return new Response("Hello world!");
  },
};

And then running it with:

bun run http.ts

Bun will recognize that an object with a fetch function was default-exported, and start a server on port 3000. As you can see here, this uses the standard Request and Response objects you use in a browser, and can use async/await.

These are all things that didn’t exist when Node and Express were first created, but seem like pretty good ideas for something built today. I don’t think using Request and Response are good for more complex use-cases (streaming responses, 1xx responses, trailers, upgrading to other protocols, getting tcp connection metadata like remoteAddr are some that come to mind), because these objects are designed for clients first.

But in many cases people are just building simple endpoints, and for that it’s great.

Bun supports a ton of the standard Node modules, but it’s also missing some such as support for server-side websockets and the node http/https/https packages, which for now makes it incompatible with popular frameworks like Express.

Porting Curveball

Curveball is a Typescript micro-framework we’ve been developing since mid-2018 as a modern replacement for Express and Koa. A key difference between Curveball and these two frameworks is that it fully abstracts and encapsulates the core ‘Request’ and ‘Response’ objects Node provides.

This made it very easy to create a lambda integration in the past; instead of mapping to Node’s Request and Response types, All I needed was simple mapping function for Lambdas idea of what a request and response looks like.

To get Express to run on AWS Lambda the Node http stack needs to be emulated, or a full-blown HTTP/TCP server needs to be started and proxied to. Each of these workarounds require a ton of code from libraries like serverless-express.

So with Bun up and coming, either the same work would need to be done to emulate Node’s APIs, or Bun would would need to add full compability for the Node http module (which is eventually coming).

But because of Curveball’s message abstractions it was relatively easy to get up and running. Most of the work was moving the Node-specific code into a new package.

Here’s the Curveball ‘hello world’ on Bun:

import { Application } from '@curveball/kernel';

const app = new Application();

// Add all your middlewares here! This should look familiar to Koa users.
app.use( ctx => {
  ctx.response.body = {msg: 'hello world!'}; 
});

export default {
  port: 3000,
  fetch: app.fetch.bind(app)
};

It’s still a bit experimental, but the following middlewares are tested:

And because JSX also just works in Bun, it’s relatively easy to use it to generate HTML:

import { Application } from '@curveball/kernel';
import reactMw from '@curveball/react';
import React from 'react';

const app = new Application();

app.use(reactMw());

app.use( ctx => {

  ctx.response.type = 'text/html; charset=utf-8';
  ctx.response.body = <html>
    <body>
      <div>
        <h1>Hello world!</h1>
      </div>
    </body>
  </html>;

});

export default {
  port: 3000,
  fetch: app.fetch.bind(app)
};

This is not a full-blown hydration/server-rendering solution, but JSX has replaced template engines like EJS and Handlebars for us at Bad Gateway. JSX lets you do the same things, but you get the advantage of type checking, it’s harder to create XSS bugs and full Javascript access.

Final thoughts on Bun

Compared to Deno, It was considerably easier to port Curveball to Bun. Deno was a much greater challenge, due to it having such a radically different idea about packages and module loading. This was over a year ago, so it’s worth giving a shot again.

So, I’m curious where all of this goes! Perhaps Bun is the future, or perhaps we’ll see Node get parity with Bun and making it obsolete. Either way competition is good.

I feel Bun has a better chance than Deno, because it delivers many of its advantages and features while mostly staying inside with the Node ecosystem.

Although as it turns out Deno also has changed their tune and also made it a new goal to be compatible. I can’t help wondering if this was inspired by Bun’s recent success as well.

Ubuntu bungled the Firefox Snap package transition

2022-09-01T09:39:29+00:00

I’m not a Snap hater. On paper it’s a good idea, but as a user I shouldn’t really be aware that ‘snaps’ even exist. In Ubuntu 21.10, Firefox became a snap package.

Arguably the browser is the most important application in an operating system. Here’s a non-exhaustive list of issues I’ve personally ran into. I should note that some of these issues are now fixed, but I wanted to illustrate what Ubuntu launched with:

Printing is completely broken, and I can only print to PDF.
KeePassXC, an open source password managers’ browser extension no longer works.
Firefox thinks that when opening ‘localhost:8080’ should open the URI scheme ‘localhost’ and tries to find an application that supports this scheme (now fixed!)
Gnome shell integration extension, the primary way to install gnome add-ons is now broken.
‘Set image as desktop background’ is broken.
Opening applications via a custom URI scheme no longer asks for confirmation, this makes it possible to (for example) launch a bittorrent client like Transmission via a magnet: uri without asking a user.
The Mozilla VPN product has a neat feature that lets you have specific containers always use a VPN. This doesn’t work.
Firefox creates a ‘firefox.tmp’ directory in the Downloads folder (fixed!)
When there’s an update to the Firefox package, the following notification appears, once per day. Restarting Firefox does not make this go away. The official answer is to run snap refresh firefox to make it go away.
GSConnect Firefox Add-on is broken.

This is just the stuff I ran into myself, (and I have reported most of these). I imagine the total list of bugs must be way higher. I don’t usually go out and complain on the internet like this, especially when it’s about open source projects. I’m a Linux user, so I’ve kind of come to expect things to not be quite as polished as some of its commercial counterparts. They’re small trade-offs to support Open Source.

However, I’m so surprised by the lack of quality control for arguably the #1 application on the #1 linux distro I’m frankly flabbergasted, and for the first time since switching from Debian to Ubuntu 15ish years ago I’m considering jumping ship again. What happened here?

Comments? Reply to this tweet

On syntactic sugar

2022-08-13T18:12:29+00:00

Ever so often the term ‘syntactic sugar’ comes when people discuss language features, and it’s not uncommon to see the word ‘just’ right in front of it; some examples:

The ‘just’ has a lot of meaning here. To me it suggests that language features that are ‘just’ syntactic sugar, aren’t quite as important as features that aren’t. Maybe it even suggests to me that the language would be fine without.

So while the above two examples both argue that both Javascript classes and async/await aren’t syntactic sugar, they also kind of come to the defence of those features and justify their existence. In other cases when people call something syntactic sugar, it’s often in a context that’s somewhat dismissal of the feature.

I think this is a bit odd and also confuses people, especially since any actual definition I’ve found is generally positive, such as this one from Wikipedia.

In computer science, syntactic sugar is syntax within a programming language that is designed to make things easier to read or to express. It makes the language “sweeter” for human use: things can be expressed more clearly, more concisely, or in an alternative style that some may prefer.

The thing is, isn’t every language feature beyond the bare minimum of what makes a language turing-complete syntax sugar?

You don’t need classes because you can use functions and structs.
You don’t really need types because everything can fit in a string.
Functions can be implemented with goto.
Multiply can be implemented with addition.
or and and, xor can be implemented with nand.
async/await can be implemented with generators.

These are all incredibly useful features that make it easier to read and write code and express ideas. Almost every language feature could be considered syntactic sugar. That’s not a bad thing, it’s just uninteresting to point out.

A new OAuth2 client for Javascript

2022-06-20T19:40:00+00:00

Frustrated with the lack of well maintained, minimal OAuth2 libraries, I wrote my own. This new OAuth2 library is only 3KB gzipped, mainly because it has 0 dependencies and relies on modern APIs like fetch() and Web Crypto which are built in Node 18 (but it works with Polyfills on Node 14 and 16).

It has support for key features such as:

authorization_code with PKCE support.
password and client_credentials grants.
a fetch() wrapper that automatically adds Bearer tokens and refreshes them.
OAuth2 endpoint discovery via the Server metadata document (RFC8414).
OAuth2 Token Introspection (RFC7662).

If your server does support the meta-data document, here’s how simple the process can be:

client_credentials example

import { OAuth2Client } from '@badgateway/oauth2-client';

const client = new Client({
  clientId: '..',
  clientSecret: '..',
  server: 'https://my-auth-server.example'
});

const tokens = await client.clientCredentials();

Without the meta-data document, you will need to specify settings such as the tokenEndpoint and possibly the authorizationEndpoint depending on which flow you are using.

authorization_code example

The authorization_code flow is a multi-step process, so a bit more involved. The library gives you direct access to the primitives, allowing you to integrate in your own frameworks and applications.

import { OAuth2Client, generateCodeVerifier } from '@badgateway/oauth2-client';

const client = new OAuth2Client({
  server: 'https://authserver.example/',
  clientId: '...',
});

// Part of PCKE
const codeVerifier = await generateCodeVerifier();

// In a browser this might work as follows:
document.location = await client.authorizationCode.getAuthorizeUri({
  redirectUri: 'https://my-app.example/',
  state: 'some-string',
  codeVerifier,
  scope: ['scope1', 'scope2'],
});

Handling the redirect back

const oauth2Token = await client.authorizationCode.getTokenFromCodeRedirect(
  document.location,
  {
    redirectUri: 'https://my-app.example/',
    state: 'some-string',
    codeVerifier,
  }
);

Docs and download

Github
npmjs

Reasons why abolishing DST in the US will be worse for users and developers

2022-03-18T08:51:00+00:00

Daylight savings time is hated by many, and twice per year a discussion reignites to get rid of it. Lot of folks feel this is a great idea. This year this decision seems especially close in the US. If this law passes, it will probably also change where I live 🇨🇦.

No doubt there’s lots of benefits and advantages to this, I don’t want to dispute that. This is also not an endorsement against that change, but I do however want to bring light to at least one disadvantage:

The timezone change was for a lot of developers a twice-per-year reminder that we need to use timezone databases and libraries.

This is useful, because every year many countries change their timezone rules. This means that if you schedule something in the future, say.. 2pm 6 months from, you don’t know yet with absolute certainty what UTC timestamp this will be. This is especially important when scheduling between people in multiple timezones.

The right way to handle this is to store the intended local time + a location such as America/Toronto. EST is not enough, because it’s EDT half the year, and obviously neither is -0500. I grew up in the netherlands, and it was only when I got into programming I realized that our timezone is not +0100 all year round, unlike what we learned in school.

Timezones are relatively stable in North America, but the US also made a change in 2007, and it sounds like we may have another one in the future!

So this bi-annual time change was a great reminder to many developers that timezones are a thing, and you can’t just naively assume a UTC time + an offset is enough. Even more so for teams that are spread cross-continent because the DST change doesn’t fall on the same day. Currently I’m in the 3 weeks per year the time difference between me and my parents is 5 instead of 6 hours.

A lot of programming is (seems?) anglo-centric. A similar situation is that before Emoji became wide-spread it was way more common to see a lot more issues around encoding non-ascii characters 🤷 (billpg). Especially in languages that don’t have good native unicode support (looking at you PHP).

So if DST goes away in North America, I predict we’ll see more people assuming using the offset is enough, resulting in bugs related to:

Times in countries that have not yet abolished DST.
Countries that ever change timezone rules. (This happens more often than you think!)
Applications that deal with historical data.

It doesn’t help that one of the most common date formats (ISO 8601) uses an offset! (2022-03-18T17:05:30.996-0400). This is OKish for things that have already happened, but not good for anything in the future.

So when you hear developers excited about the US abolishing DST because it will make their (work) life simpler, remind them this is only true if you never intend your software to be used outside of North America, or when the entire rest of the world makes the same change and also freezes all timezone rules forever!

Log4j, Faker and Black Swan Events

2022-03-04T22:20:00+00:00

In December log4j, a library that’s used by a massive amount of projects had a major vulnerability, and in January the author of ‘Faker’ and ‘Color’ went nuclear, released a intentionally buggy version that broke a lot of projects (temporarily).

A lot has been said about both of these, but I wanted to add my own (very late) take to this.

I think the general theme of what people are writing about this is is: this is a major problem with open source. Critical projects aren’t getting enough funding.

In the case of Faker.js/Color.js, a lot of people in the Node.js community suggested to no longer specify ‘version ranges’ for dependencies but lock everything to a specific version in package.json.

I however found myself disagreeing with both of those conclusions.

For both of these major issues, I looked at the events unfolding and I have a hard time reaching the conclusion that there’s a big problem that needs to be solved. It was quite the opposite actually.

It’s hard to find exact statistics, but the sources I could find suggests that there’s at least 100,000s of open source developer as many open source packages. If we have couple of major issues like Log4j and Faker.js every year, that feels like a number that’s not really worth optimizing for.

Proprietary or well-funded software can also have major security bugs, and in each of these cases workarounds popped up fast and the issues were fully resolved within 2 weeks for most users.

Yes, the issues themselves were problematic and I have sympathy for the developers involved and how they were treated, but the conclusion I got from these events is that things are actually pretty robust and correct themselves in the face of major issues.

A counter example of the open source world, a report recently came out that Linux developers generally fix bugs faster compared to Apple, Google or Microsoft by a wide margin.

I don’t want to deny that that open source needs more funding. It’s a complex issue that I don’t know enough about, but If there are indeed major structural issues related to how open source is developed, we should look at widespread problems and not ‘black swan events’.

I’ve often seen organizations trying to find and fix root causes for any issue that pops up, but all these process changes can lead to a situation where it’s hard to get anything done. An important part of any post-mortem should be ‘how can we prevent this in the future’, but also ‘is this issue common enough to prevent it from happening again’.

Request bodies in GET requests

2022-01-29T19:14:00+00:00

12 years ago I asked on Stack Overflow: Are HTTP GET requests allowed to have request bodies?. This got a 2626 upvotes and a whopping 1.6 million views, so clearly it’s something lots of people are still curious about, and in some cases disagree with the accepted answer.

Because it keeps popping up in my Stack Overflow notifications (and I compulsively visit the site), the question has lived in my head rent-free. I keep adding context in my head, and I’ve been meaning to write some of this down for a few years now and hopefully evict it.

Anyway, if you’re just looking for a quick answer, it’s ‘No, you shouldn’t do this.’, you should probably use QUERY.

Undefined behavior

A number of people (most famously ElasticSearch) have gotten this wrong, but why? I think it’s because of this sentence in the HTTP Spec:

A payload within a GET request message has no defined semantics

That sentence could easily suggest that there’s no specific behavior associated to request bodies with GET requests, and that the behavior is left up to the implementor.

The reality is that this is more like Undefined behavior from languages like C/C++. My understanding is that leaving certain aspects of the C language undefined (instead of for example requiring an error to be thrown) leaves room for compiler implementations to make certain optimizations. Some compilers also have fun with this; GCC hilariously starts a video game in a specific case of undefined behavior which really brings home this point.

If you were to write a C program that relies on how a compiler dealt with specific undefined behavior, it means your program is no longer a portable C program, but it’s written in variant of C that only works on some compilers.

The same applies for HTTP as well. It’s true that undefined behavior means that you as a server developer can define it, but you are not an island!

When working with HTTP, there’s servers but also load balancers, proxies, browsers and other clients that all need to work together. The behavior isn’t just undefined server-side, a load balancer might choose to silently drop bodies or throw errors. There’s many real-world examples of this. fetch() for example will throw an error.

This hasn’t stopped people from doing this anyway. OpenAPI removed support for describing GET request bodies in version 3.0 (and DELETE, which has the same issue!), but was quitely added back in 3.1 to not prevent people from documenting their arguably broken APIs.

Why it’s not defined

The best source I have is this quote from Roy Fielding in 2007. Roy Fielding coined REST is and is one of the main authors of the HTTP/1.1 RFCs.

Yes. In other words, any HTTP request message is allowed to contain a message body, and thus must parse messages with that in mind. Server semantics for GET, however, are restricted such that a body, if any, has no semantic meaning to the request. The requirements on parsing are separate from the requirements on method semantics. So, yes, you can send a body with GET, and no, it is never useful to do so. This is part of the layered design of HTTP/1.1 that will become clear again once the spec is partitioned (work in progress).

….Roy

(His message was originally sent to the now-dead rest-discuss group on Yahoo Groups, but I found an archive in JSON format)

However, I always found this answer unsatisfying. I understand that you might want a low-level protocol design that just passes messages containing headers, bodies, urls, methods and statuses and not concern itself with method-specific behavior.

That design goal doesn’t preclude GET from specifically rejecting request bodies though, and it also doesn’t preclude the spec from mentioning that request bodies should not be emitted.

So, unless there’s a different reason for this I’m going to have to assume that this was well-intentioned but just not written clearly enough.

Despite this explanation from one of the authors of the HTTP spec itself, I noticed some people will still claim a different interpretation, making a sort of a ‘death of the author’ argument. It’s a bit funny to think about ‘death of the author’ in technical specs, but I think they have a point. I believe that the definition of a standard such as HTTP is not the written RFC, it’s how everyone actually implements it.

The goal of the spec is to accurately describe the standard, not to define it. If what everyone is doing is different from the author’s intention, the spec has failed to accurately describe the standard, not vice versa.

The team behind the HTTP specs believes this too, which is why we’ve seen new releases of HTTP/1.1 without increasing the version. In each iteration major steps are taken to capture real-world usage, sometimes even in conflict with the original RFC2616.

However, in this case it’s irrelevant. Many popular HTTP implementations will throw errors when seeing GET bodies, such as fetch(). Using GET bodies will give you such poor interopability that it’s worth continuing to not do this.

In 2019 I’ve opened a ticket to request to fix this, and they listened. The upcoming HTTP/1.1 is going to include the following text:

Although request message framing is independent of the method used, content received in a GET request has no generally defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of HTTP/1.1).

A client SHOULD NOT generate content in a GET request unless it is made directly to an origin server that has previously indicated, in or out of band, that such a request has a purpose and will be adequately supported.>An origin server SHOULD NOT rely on private agreements to receive content, since participants in HTTP communication are often unaware of intermediaries along the request chain.

Source.

Why should `GET` have a body?

I think we feel this way, because we’ve been told over and over again that it’s a good idea to use protocols in a ‘semantically correct’ way.

But why should be care about semantics? 2 reasons:

If you do things in a semantically correct way, other systems can make assumptions about how it should behave. For example: A PUT request may be automatically repeated if it failed the first time due to a network failure or 5xx error. A GET request may be cached if it contained a Cache-Control header.
It’s self-documenting behavior. If you see a GET request it tells a developer something is being retrieved.

So while #1 doesn’t really apply here (there’s no technical advantages, because it’s undefined behavior), but #2 makes sense.

So if we decided to use GET for our complex search, what can we do? Body is obviously not allowed; our only options are headers and the URL. There’s a number of issues with this: Encoding UTF-8 is unclear, no real support for documents/mine-types, length limitations. It’s a bit of a mess!

Why was it designed not to support bodies?

Most developers working with HTTP and learning about it is in the context of APIs, but this is not the original use-case.

It all starts with HTML and browsers and URLS. The world wide web was designed as a distributed hypertext document system, where every document has a global unique identifier called a URL.

To retrieve a hypertext document, you would do a GET on this url, to replace or create the document a PUT and to remove the document DELETE. (Note that it took longer to get a writable web, but I wanted to illustrate what I think the intended design is without being encumbered by facts).

Taken from this perspective, the idea of ‘parameters’ makes less sense. GET doesn’t just mean “Do a safe request and read something from the server”, it means: ‘Give me the document with this name’.

This specific feature is in part why the web was so successful. Given that the URL comprehensively describes to a client exactly how to connect to a server and retrieve a document, it made the <a> HTML work. It also let people make bookmarks, share links via email/chat and paint it on store-fronts.

The URL is the web’s superstar and killer feature. HTTP just let you find documents associated with the URL. HTTP/0.9 didn’t even have headers and can be read during a bathroom break.

Our needs have evolved, has HTTP?

I still maintain that even if you’re building a REST api, it’s good to think of your endpoints as documents, and the GET, PUT and DELETE as the most important HTTP methods to manipulate these, using the URL as the primary key.

But, sometimes we need to do something more complicated and it would be nice if HTTP had a prescribed way to handle cases where we want to describe a search query as a document.

The answer traditionally was to just use POST. POST is kind of your no-guarantees catch all method. Often also used to tunnel other protocols that don’t really speak HTTP well, such as SOAP, XMLRPC and more recently GraphQL.

But there’s a new contender. Now I think your best option is QUERY. This is a new method, that’s not quite standard yet but any compliant HTTP client and server should already support it. The goal of QUERY is specifically to solve this use-case.

QUERY solves the following problems:

It’s basically a ‘safe’ POST. This means the implication is that it’s for reading and safe to repeat.
It also is self-descriptive. It tells a developer that this is a API that’s focused on reading.
The spec also makes it cachable.

I think you should still continue the use GET for the majority of cases, but there’s a clear answer now for what to do when that doesn’t fit.

Anyway…

That’s a lot of writing for what seemingly is a simple question. It’s just one of those things that’s been bouncing around my head for a bit too long.

I hope it’s out of my system!

Big thank you to everyone who originally answered my SO question.

Wanna discuss?

Hit me up on Twitter, Github Discussions, or read the Hacker News Thread

Hello 2022!

2022-01-06T04:08:28+00:00

Yesterday I received an email from a reader asking ‘Are you ok?’.

It’s been nearly 8 months since the last time I wrote here. In that last post I celebrated blogging on this website for 15 years with some consistency, so perhaps it’s a bit ironic for that to be immediately followed by complete silence.

The last big gap in blogging for me was in 2017, the year I joined Yelp. This experience was so depressing, every day I was done work I had no creative energy left for anything else.

2021 was a bit different though. 2 years back I started a software development agency, which grew from 2 to 5 people in the last year. The stakes have increased quite a bit, and it’s taken up a lot of my emotional reserves.

I’ve also made the mistake of not taking any vacation all year. There was just not much gas left in the tank. This is so stupid. Less time off doesn’t result in more productivity. I know this, but the last 2 years there’s been little travel or activities due to lockdowns and restrictions. Every day looks the same and it kind of just flew by.

Over the last holidays I’ve taken an actual break though, and have since started several new projects and buzzing with new ideas. I’m still motivated to work on Curveball and Ketting (we use it every day for almost every customer!), and I’ve also started a series of live streams in which I build a Time Tracking application with Hypermedia on twitch.tv/evrt3.

If this sounds interesting, the first few episodes are up on my youtube channel, but I’ll share more on this blog later.

I’m also preparing for a tech talk on January 19th for Toronto JS. It’s online and free!

So am I ok? I think I am? This year is off to a good start. I just have to make sure I don’t forget to take it easy.

Happy stupid new year! I hope it sucks less!

15 years of blogging

2021-05-29T20:56:54+00:00

I’m a few days late, but as of May 25, 2021 I’ve had this blog for 15 years! I guess it also makes this my longest running project.

The blog went through quite a few iterations and platforms, but the current Github Pages iteration is about 8 years old already.

This is how it looked between 2006 and 2010:

This was definitely the most elaborate design, but eventually I got sick of a hand-written PHP and switched to a more plain Habari-based blog.

I’ve had plans for years to move away from Github pages again, and do a custom build. Github pages is so easy to use, but the lack of a server-side language stifles my creativity a bit. I want to add a regular-old comment system again and add websockets for bits of creativity.

Some stats

Since I started measuring in 2010, Google Analytics has reported 2 million pageviews!
I’ve published 426 posts. The number of posts is trending downwards year-over-year. I find it harder to just write about a small interesting thing I came across, and posts tend to be more long-form now.
I wrote articles in 15 countries, in 5 continents!
I got 101 email subscribers.

Live map of every post on this blog. (see how I did this)

The most popular post ever is about following redirects in PHP, which probably has to do more with good search keywords than anything. 144,565 pageviews.

I also wrote 68 posts for every HTTP status code. Combined, they got 202,656 page views as of today.

My proudest post is about HTTP/2, Server Push and compound documents, because I added a bunch of interactivity that I think a lot of people loved. I’d like to do more like this, but it was extremely time consuming. It must have taken at least 100 hours!

Thanks!

Thanks everyone for reading and supporting! It’s all the great responses that give me the energy and motivation to keep going =)

JWT should not be your default for sessions

2021-05-10T20:37:00+00:00

Cookies

When designing web applications, (especially the traditional HTML kind), you will at one point have to figure out how to log a user in and keep them logged in between requests.

The core mechanism we use for this are cookies. Cookies are small strings sent by a server to a client. After a client receives this string, it will repeat this in subsequent requests. We could store a ‘user id’ in a cookie, and for any future requests we’ll know what user_id the client was.

Cookie: USER_ID=123

But this is very insecure. The information lives in the browser, which means users can change USER_ID and be identified as a different user.

Sessions

The traditional way to solve this is what’s known as a ‘session’. I don’t know what the earliest usage of sessions is, but it’s in every web framework, and has been since web frameworks are a thing.

Often, sessions and cookies are described as 2 different things, but they’re really not. A session needs a cookie to work.

Cookie: MY_SESSION_ID=WW91IGdvdCBtZS4gRE0gbWUgb24gdHdpdHRlciBmb3IgYSBmcmVlIGNvb2tpZQ

Instead of a predictable user id, we’re sending the client a completely random session id that is impossibly hard to guess. The ID has no further meaning, and doesn’t decode to anything. This is sometimes called an opaque token.

When a client repeats this session id back to the server, the server will look up the id in (for example) a database, which links it back to the user id. When a user wants to log out, the session id is removed from the data storage, which means the cookie is no longer associated with a user.

Where is the session data stored?

Languages like PHP have a storage system for this built in, and will by default store data in the local filesystem. In the Node.js ecosystem, by default this data will be in ‘memory’ and disappear after the server restarts.

These approaches make sense on developer machines, or when sites were hosted on long-lived bare-metal servers, but these days a deploy typically means a completely fresh ‘system’, so this information needs to be stored in a place that outlives the server. An easy choice is a database, but it’s common for sites to use systems like Redis and Memcached, which works for tiny sites, but still works at massive scales.

Encrypted token

Over 10 years ago, I started working a bit more with OAuth v1 and similar authentication systems, and I wondered if we could just store all the information in the cookie and cryptographically sign it:

Despite getting some good answers, I didn’t go through with it as I didn’t feel confident enough in making this secure, and I felt it required a better understanding in crypto than I did.

A few years later, we got JWT, and it’s hot shit! JWT itself is a standard for encrypting/signing JSON objects and it’s used a LOT for authentication. Instead of an opaque token in a cookie, we actually embed the user_id again, but we include a signature. The signature can only be generated by the server, and it’s calculated using a ‘secret’ and the actual data in the cookie.

This means that if the data is tampered with (the user_id was changed), the signature no longer matches.

So why is this useful? The best answer I have for this, is that it’s not needed to have a system for session data, like Redis or a database. All the information is contained in the JWT, it means your infrastructure is in theory simpler. You’re potentially making fewer calls to a data-store on a per-request basis.

Drawbacks

There are major drawbacks to using JWT.

First, it’s a complicated standard and users are prone to get the settings wrong. If the settings are wrong, in the worst case it could mean that anyone can generate valid JWTs and impersonate anyone else. This is not a beginners-level problem either, last year Auth0 had a bug in one of their products that had this very problem.

Auth0 is(or was? they just got acquired) a major vendor for security products, and ironically sponsor the jwt.io website. If they’re not safe, what chance does the general (developer) public have?

However, this issue is part of a larger reason why many security experts dislike JWT: it has a ton of features and a very large scope, which gives it a large surface area for potential mistakes, by either library authors or users of those libraries. (alternative stateless tokens to JWT exists, and some of them do solve this.)

A second issue is ‘logging out’. With traditional sessions, you can just remove the session token from your session storage, which is effectively enough to ‘invalidate’ the session.

With JWT and other stateless token this is not possible. We can’t remove the token, because it’s self-contained and there’s no central authority that can invalidate them.

This is typically solved in three ways:

The tokens are made very short lived. For example, 5 minutes. Before the 5 minutes are over, we generate a new one. (often using a separate refresh token).
Maintain a system that has a list of recently expired tokens.
There is no server-driven log out, and the assumption is that the client can delete their own tokens.

Good systems will typically use the first two. An important thing to point out is, in order to support logout, you’ll likely still need a centralized storage mechanism (for refresh tokens, revocation lists or both), which is the very thing that JWT were supposed to ‘solve’.

Sidenote: some people like JWT because it’s fewer systems to hit per request, but that contradicts with being able to revoke tokens before they expire.

My favourite solution to this is keep a global list of JWTs that have been revoked before they expired (and remove the tokens after expiry). Instead of letting webservers hit a server to get this list, push the list to each server using a pub/sub mechanism.

Revoking tokens is important for security, but rare. Realistically this list is small and easily fits into memory. This largely solves the logout issue.

A last issue with JWT is that they are relatively big, and when used in cookies it adds a lot of per-request overhead.

All in all, that’s a lot of drawbacks just to avoid a central session store. It’s not my opinion that JWT are universally a bad idea or without benefits, but there is a lot to consider.

Why are they popular?

One thing that’s surprised me when reading tech blogs, is that there is a lot of chatter around JWT. Especially on Medium and subreddits like /r/node I see intros to JWT on an extremely regular basis.

I realize that that doesn’t mean that ‘JWTs are more popular than session tokens’, for the same reason that GraphQL isn’t more popular than REST, or NoSQL than relational databases: it’s just not that interesting to write about the technology that’s been tried and tested for a decade or more (See: Appeal to novelty). In addition, subject experts that write about new solutions are likely going to have different problems & scale than the majority of their own readers.

However, these new technologies create a lot more buzz than their simpler counterparts, and if enough people keep talking about the hot thing, eventually this can translate to actual adoption, despite it being a sub-optimal choice for the majority of simple use-cases.

This is similar to many newer developers learning how to build SPAs with React before server-rendered HTML. Experienced devs would likely feel that server-rendered HTML should probably be your default choice, and building an SPA when needed, but this is not what new developers are typically taught.

Adopting complex systems before the simple option is considered is something I see happening more, but it surprised me for JWT.

As an exercise, I looked up the top most popular (by votes) posts on /r/node that mention JWT. I was going to go over the first 100, but got bored after the top 12.

From these 12 articles and Github repos:

1 mentions using a revocation list, 3 mention refesh tokens. The remaining articles and github repositories simply have no means of logging out.
1 article mentions that it might be better to use a standard session storage instead.
1 article uses both a standard session storage and JWT, making JWT unneeded.
1 github repository ships with pre-generated private keys. (yup)
Most of posts use expiry times of weeks or months and 3 posts never expire their JWTs.

Except 1, the quality of these highly upvoted posts was extremely low, the authors were likely not qualified to write about this and can potentially cause real-world harm.

All this at least confirmed my bias that JWT for security tokens is hard to get right.

On JWT and scale

Going through tons of Reddit posts and comments also got me a more refined idea of why people think JWTs are better. The top reason everywhere is: “It’s more scalable”, but it’s not obvious at what scale people think they’ll start to have issues. I believe the point where issues start to appear is likely much higher than is assumed.

Most of us aren’t Facebook, but even at ‘millions of active sessions’, a distributed key->value store is unlikely going to buckle.

Statistically, most of us are building applications that won’t make a Raspberry Pi break a sweat.

Conclusion

Using JWTs for tokens add some neat properties and make it possible in some cases for your services to be stateless, which can be desirable property in some architectures.

Adopting them comes with drawbacks. You either forego revocation, or you need to have infrastructure in place that be way more complex than simply adopting a session store and opaque tokens.

My point in all this is not to discourage the use of JWT in general, but be deliberate and careful when you do. Be aware of both the security and functionality trade-offs and pitfalls. Keep it out of your ‘boilerplates’ and templates, and don’t make it the default choice.

Acknowledgements

Thanks to Nick Chang-Fong and Dominik Zogg for providing feedback and suggestions to this article!

Evert Pot's blog

Creating a fake download counter with Web Components

Why is this number so high?

The Web Component

Obtaining the data

NPM

Packagist

Putting it together

Conclusion

Moving on from Mocha, Chai and nyc.

OAuth2 client updates

Using JSX on the server as a template engine

What is JSX?

#Inspo

JSX Transpiled

Using JSX as a template engine

Koa

Fastify

How does this magic work?

Koa middleware

Fastify hooks

Limitations to this approach

Frequently asked questions

How do you handle routing?

Is there still a reason to use Handlebars or EJS?

Anyway

Why aren't there more 80% jobs?

My background

Advantages of offering 80% jobs

Does OAuth2 have a usability problem? (yes!)

My perspective

Solving the setup issue

My proposal

Final notes

Switching to Fedora from Ubuntu

Ubuntu’s fall

Onto Fedora

Installation

Non-free stuff

Gnome

Flatpak

Bugginess

Software Center app

Would I recommend Fedora?

Supporting CommonJS and ESM with Typescript and Node

Why support ESM

The general approach

Typescript features we don’t use

esModuleInterop

Path mapping

Configuring package.json

Building with Typescript

Changes we had to make in our code

Extensions in imports

Directory imports

__dirname and __filename

Importing CommonJS modules with ‘default exports’

Testing

Conclusion

Winding down Bad Gateway

Building a simple CLI tool with modern Node.js

Goals and features

The implementation

ESM & Typescript

Command line parsing

Testing

package.json, annotated

Conclusion

Knex (with MySQL) had a very scary SQL injection

My understanding of this bug

Query strings and Express

Other examples

On Knex

I wish JSON5 was more popular

Why YAML is difficult for me

Neko - A brief history and porting to Javascript

Creating the assets

A deep dive into the source

Taking a look at Mastodon

Links

`dirname` and `filename`

Why should `GET` have a body?