Aral Balkan

End-to-end encrypted Kitten Chat

mail@ar.al (Aral Balkan) — Mon, 20 Feb 2023 09:59:56 +0000

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download Small Is Beautiful #27 directly, and watch it with your favourite video player.

Small Is Beautiful (Feb, 2023): End-to-end encrypted Kitten Chat (an example peer-to-peer Small Web app using Kitten. Follow the tutorial to build it yourself from scratch or browse the source code).

In this hour-and-a-half long Small is Beautiful live stream recording, I show you how WebSockets, project-specific secrets, and authenticated routes work in Kitten and migrate a centralised WebSocket chat application to an end-to-end-encrypted peer-to-peer Small Web chat application in Kitten.

The example also makes use of the native support for htmx and Alpine.js in Kitten.

You can follow Small is Beautiful from the fediverse (we stream it using our own Owncast instance) to be notified of future streams.

(Hint: you can install Owncast using Site.js.)

Transcript

Auto-generated transcript from the captions.

Links to other things mentioned or shown during the stream:

Streamed using our own Owncast instance. (Hint: you can install Owncast using Site.js.)

If you like this livestream, please help support out work at Small Technology Foundation with a patronage or donation, or share our videos with your friends!

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Installing Helix Editor Language Servers

mail@ar.al (Aral Balkan) — Mon, 14 Nov 2022 11:18:37 +0000

Helix Editor using the Bash Language Server to show the symbols in the script included in this post.

Helix Editor

I’ve been using Helix Editor as my daily driver for web development for most of this year and – while it has some outstanding issues¹ (what doesn’t?) – I’m really enjoying it.²

One of the issues that might trip new folks is that while it has Language Server Protocol (LSP) support, and while there are default Language Servers configured, installing Helix Editor doesn’t actually install those language servers.³

Another issue is that since many of the language servers are written in Node.js and installed as global modules, if you update your version of Node (e.g., using nvm.fish on fish shell), you will also have to reinstall your language servers.

Needless to say, this can get tedious so here’s a little script I hacked together today for myself to easily install (and reinstall) the Language Servers I use for web development (mostly HTML, JS, CSS, and Node.js with Kitten these days) in Helix Editor.

I’m sharing it below in case it helps anyone else. Please feel free to adapt and use it for yourself.

Usage

Copy the code into a file named, e.g., install-helix-language-servers

Make that file executable. e.g.:

chmod +x install-helix-language-servers

Adapt it for your own needs (see notes) and enjoy!

Notes

Only tested on Linux.
Requires Node.js, Rust, Bash, wget, gunzip, and tar.

If you’re running this on Fedora Silverblue or other immutable Linux distribution, make sure you do so from a mutable container (e.g., via Toolbox or Distrobox) that has the Rust toolchain installed so that the Language Servers that are installed by Cargo (e.g., TOML) can be compiled properly.

Note: TOML Language Server requires latest Rust to compile. Tested to work on 1.65.0 (does not work on 1.60.0). See https://github.com/tamasfe/taplo/issues/349

If you’re on a non-Linux platform, please modify the script accordingly.

You can find installation instructions for all Language Servers supported by Helix Editor at: https://github.com/helix-editor/helix/wiki/How-to-install-the-default-language-servers

Code

#!/usr/bin/env bash

BINARY_HOME="${HOME}/.local/bin"
DATA_HOME="${XDG_DATA_HOME:-$HOME/.local/share}"

echo "Installing Language Servers for Helix Editor:"

# Work in a throwaway temporary directory so as not to pollute the file system.
temporaryDirectory="/tmp/helix-editor-language-server-installer"
mkdir -p "${temporaryDirectory}"
pushd "${temporaryDirectory}"

# Bash
echo "  • Bash (bash-language-server)"
npm i -g bash-language-server

# HTML, JSON, JSON schema
echo "  • HTML, JSON, and JSON schema (vscode-langservers-extracted)"
npm i -g vscode-langservers-extracted

# JavaScript (via TypeScript)
echo "  • JavaScript (typescript, typescript-language-server)"
npm install -g typescript typescript-language-server

# Markdown (via ltex-ls. Note: this has excellent features like
# spelling and grammar check but is a ~269MB download).
echo "  • Markdown (ltex-ls)"
ltexLsVersion=15.2.0
ltexLsBinaryPath="${BINARY_HOME}/ltex-ls"
ltexLsBaseFileName="ltex-ls-${ltexLsVersion}"
ltexLsFileNameWithPlatform="${ltexLsBaseFileName}-linux-x64"
ltexLsAppDirectory="${DATA_HOME}/${ltexLsBaseFileName}"
rm "${ltexLsBinaryPath}"
rm -rf "${ltexLsAppDirectory}"
wget "https://github.com/valentjn/ltex-ls/releases/download/${ltexLsVersion}/${ltexLsFileNameWithPlatform}.tar.gz"
gunzip "${ltexLsFileNameWithPlatform}.tar.gz"
tar xf "${ltexLsFileNameWithPlatform}.tar"
mv "${ltexLsBaseFileName}" "${DATA_HOME}"
ln -s "${ltexLsAppDirectory}/bin/ltex-ls" "${ltexLsBinaryPath}" 

# TOML
cargo install taplo-cli --locked --features lsp

# Clean up.
popd
rm -rf "${temporaryDirectory}"

echo "Done."

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Soft wrap doesn’t exist yet, for example, which makes it nigh on impossible to edit Markdown in it. For that, I’m currently using MarkText. ↩︎
This is not something I thought I’d ever say for a modal editor but Helix gets a lot of things right. While there is definitely a learning curve, I feel like I now think about my code as code instead of as lines of text and characters as I’m working and that’s a good feeling. ↩︎
This is understandable as not everyone needs or wants to install every language server. Especially considering that language servers can be very large. For example, the ltex-ls language server for Markdown is roughtly a 265MB download. ↩︎

Is the fediverse about to get Fryed? (Or, “Why every toot is also a potential denial of service attack”)

mail@ar.al (Aral Balkan) — Wed, 09 Nov 2022 19:00:00 +0000

Stephen is a big fish to fry. (I’m here all week.)

Warning: the fediverse is about to get Fryed.

Stephen Fryed, that is.

Following the recent takeover of Twitter by a proto-fascist billionaire man-baby, people have been fleeing¹ to the fediverse². Among them are folks who, on Twitter, at least, had millions of followers like Greta Thunberg and, more recently, Stephen Fry.³

“Well, surely that’s a good thing? It’ll get everyone talking about the fediverse, decentralisation, and maybe even that Small Web thing you keep harping on about all the time, Aral, no?”

Well, yes and no… you see, there is such a thing as too much of a good thing. And, on the fediverse today, that appears to be “engagement when you’re popular.” In fact, it could be deadly (to Mastodon instances, that is).

Read on and I’ll try to explain what I mean by using my own account as an example.

How to kill a Mastodon (hint: by being chatty when you’re popular)

Needless to say, I’m not a celebrity.

And yet, on the fediverse, I find myself in a somewhat unique situation where:

I have my own personal Mastodon instance, just for me.⁴
I’m followed by quite a number of people. Over 22,000, to be exact.⁵
I follow a lot of people and I genuinely enjoy having conversations with them. (I believe this is what the cool kids call “engagement”.)

Unfortunately, the combination of these three factors creates a perfect storm⁶ which means that now, every time I post something that gets lots of engagement, I essentially end up carrying out a denial of service attack on myself.

Mastodon: denial-of-service as a service?

Yesterday was my birthday.

So, of course, I posted about it on my Mastodon instance.

It got quite a few replies. And, because it’s only polite, I started replying to everyone with thank-you messages.

Oh, no, you poor, naïve man, you. What were you thinking?!…

I’ll let my friend Hugo Gameiro, who runs masto.host and hosts my instance, explain what happened next:⁷

You just get a lot of engagement and that requires a ton of Sidekiq power to process.

For example, let’s look at your birthday post … besides requiring thousands of Sidekiq jobs to spread your post through all their servers (you have 23K followers, let’s assume 3K different servers⁸), as soon as you create the post 3K Sidekiq jobs are created. At your current plan you have 12 Sidekiq threads, so to process 3K jobs it will take a while because it can only deal with 12 at a time.

Then, for each reply you receive to that post, 3K jobs are created, so your followers can see that reply without leaving their server or looking at your profile. Then you reply to the reply you got, another 3K jobs are created and so on.

If you replied to the 100 replies you got on that post in 10 minutes (and assuming my 3K servers math is right). You created 300K jobs in Sidekiq. That’s why you get those queues.

So what does that mean if you’re not into the technical mumbo-jumbo?

It means I was too chatty while being somewhat popular.

What a traffic jam looks like in Mastodon.

So, what’s the solution?

Well, there’s only one thing you can do when you find yourself in such a pickle: scale up your Mastodon instance.⁹ The problem with that? It starts getting expensive.

Prior to the latest Twitter migration¹⁰, I was paying around €280/year (or a little over €20/month) for my Mastodon instance on a custom plan I had with Hugo from the early days. This week, I upped that to a roughly €50/month plan. And that’s still not enough as my birthday post just showed so Hugo, kindly, has suggested he might have to come up with a custom plan for me.

And yet, the problem is not one that will go away. We can only kick the ball down the road, as it were.

(Unless I piss everyone off with this post, that is.)

Thankfully, by running my own instance, the only person I’m burdening with this additional expense is me. But what if I’d been on a public instance run by someone else instead?

Musk you?

If Elon Musk wanted to destroy mastodon.social, the flagship Mastodon instance, all he’d have to do is join it.¹¹

Thank goodness Elon isn’t that smart.

I jest, of course… Eugen would likely ban his account the moment he saw it. But it does illustrate a problem: Elon’s easy to ban. Stephen, not so much. He’s a national treasure for goodness’ sake. One does not simply ban Stephen Fry.

And yet Stephen can similarly (yet unwittingly) cause untold expense to the folks running Mastodon instances just by joining one.¹²

The solution, for Stephen at least, is simple: he should run his own personal instance.

(Or get someone else to run it for him, like I do.)¹³

Running his own instance would also give Stephen one additional benefit: he’d automatically get verified.

After all, if you’re talking to, say, @stephen@social.stephenfry.com, you can be sure it’s really him because you know he owns the domain.

Personal instances to the rescue

My speech at the European Parliament on the problem with Big Tech and the different approaches provided by Mastodon, the fediverse, and Small Web.

Wait, I’m confused… didn’t you say that personal instances were part of the problem?

Yes and no: they are and they shouldn’t be.

If ActivityPub (the protocol) and Mastodon (a server that adheres to that protocol) were designed to incentivise decentralisation, having more instances in the network would not be a problem. In fact, it would be the sign of a healthy, decentralised network.

However, ActivityPub and Mastodon are designed the same way Big Tech/Big Web is: to encourage services that host as many “users”¹⁴ as they can.

This design is both complex (which makes it difficult and expensive to self-host) and works beautifully for Big Tech (where things are centralised and scale vertically and where the goal is to get/own/control/exploit as many users as possible).

In Big Tech, the initial cost of obtaining such scale is subsidised by vast amounts of venture capital (rich people investing in exploitative and extractive new businesses – which Silicon Valley calls Startups™ – in an effort to get even richer) and it leads to the amassing of the centres¹⁵ we know today as the Googles, Facebooks, and Twitters of the world.

However, unlike Big Tech, the stated goal of the fediverse is to decentralise things, not centralise them. Yet how likely is it we can achieve the opposite of Big Tech’s goals while adopting its same fundamental design?

When you adopt the design of a thing, you also inherit the success criteria that led to the evolution of that design. If that success criteria does not align with your own goals, you have a problem on your hands.

What I’m trying to say is:

Do not adopt the success criteria of Big Tech lest you should become Big Tech.

Bigger is not better

Today, we equate the size of mastodon.social (the instance run by Eugen) with how successful Mastodon (the software created by Eugen) is. This is very dangerous. The larger mastodon.social gets, the more it will become like Twitter.

I can almost hear you shout, “But Aral, it’s federated! At least there’s no lock-in to mastodon.social!”

This is true.

You know what else is federated? Email.

Have you ever heard of a little old email instance called Gmail? (Or perhaps the term “embrace, extend, extinguish?”)

Do you know what happens to your email if Google says (rightly or wrongly) that you’re spam? No one sees your email.

You know what happens if mastodon.social blocks your instance? Hundreds of thousands of people (soon, millions?) do not get a choice in whether they see your posts or not.

What happens when your instance of one blocks mastodon.social? Nothing, really.

That’s quite a power imbalance.

Decentralisation begins at decentring yourself

Mastodon is a not-for-profit, and I have no reason to believe that Eugen has anything but the best of intentions.

However, decentralisation begins at decentring yourself.

It’s in the interests of the fediverse that mastodon.social sets a good example by limiting its size voluntarily.

In fact, this should be built right into the software. Mastodon instances should be limited from growing beyond a certain size. Instances that are already too large should have ways of encouraging people to migrate to smaller ones.

As a community we should approach large instances as tumours: how do we break them up so they are no longer a threat to the organism?

If you take this approach to its logical conclusion, you will arrive at the concept of the Small Web; a web where we each own and control our own place (or places).

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download Small Is Beautiful #23 directly, and watch it with your favourite video player.

Small Is Beautiful (Oct, 2022): What is the Small Web and why do we need it?

Tweet, tweet?

I’m not saying that the current fediverse protocols and apps can, will, or even necessarily should evolve into the Small Web.¹⁶ In the here and now, the fediverse is an invaluable stopgap that provides a safer haven than the centralised cesspits of Silicon Valley.

How long the stopgap lasts will depend on how successful we are at resisting centralisation. Protocol and server designs that incentivise vertical scale will not necessarily make this easy. However, there are social pressures we can use to counter their effects.

The last thing you want is a handful of mini Zuckerbergs running the fediverse. Or worse, to find yourself having become one of those mini Zuckerbergs.

I love that the fediverse exists. And I have the utmost respect for the gargantuan effort that’s going into it.

And yet, I am also very concerned¹⁷ that the design decisions that have been made incentivise centralisation, not decentralisation. I implore us to acknowledge this, to mitigate the risks as best we can, to strive to learn from our mistakes, and to do even better going forward.

So to the ActivityPub and Mastodon folks, I say:

Consider me your canary in the coal mine…

«Chirp! Chirp! Chirp!»

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

After 16 years on Twitter, even I finally deactivated my account and asked for all by data to be deleted last week. It was easy to do as I’ve been on the fediverse for over five years and I’d basically stopped using the hell site almost entirely for over a year. I was just keeping my account active so as not to break over a decade-and-a-half of web links. (Let this be a lesson to you: if you care about not breaking links/content on the web, make sure that you own them instead of Startup Of The Week, Inc.) ↩︎
If you’re wondering what the fediverse is, it’s likely what you call Mastodon. I could write a whole other blog post about why Mastodon is not the fediverse but, thankfully, others have done a great job of explaining the basic concepts already. ↩︎
Greta joined the mastodon.nu instance and already has over 44,000 followers (and is following 50) and Stephen has joined mastodonapp.uk and amassed 27,000 followers in a day or so (and isn‘t following anyone in return at the moment). ↩︎
It’s what I’d call a “personal instance”, an “instance of one”, or “a single-tenant instance.” It’s basically a server where only you have an account. Because Mastodon is federated using the ActivityPub protocol, you can communicate with anyone on any other instance but being on an instance of one is very different to being on an instance of hundreds of thousands, like mastodon.social, for example. ↩︎
On Twitter, my follower count was at roughly 42,000 people before I deactivated my account. Keep in mind that that was after being on the site for 16 years. Similarly, I’ve been on the fediverse for over five years, basically since the beginning, and Stephen Fry has amassed more followers in a single day. This is important to remember as we use my experience to forecast what the instance he joined – mastodonapp.uk – will begin to (or has already begun to) experience. ↩︎
Given the design decisions underpinning the ActivityPub protocol and Mastodon server. ↩︎
When I asked Hugo if I could quote him for this post, he said yes but with the following caveat, which I’m including here so you don’t go off complaining to him (if you have any issues with this post, you can complain to me instead): “You can quote my explanation but as an illustration because there are several aspects that are not 100% accurate and others that I probably don’t even know. For example, if a post/reply includes an image, are two Sidekiq jobs created or only one? If it includes multiple images? If there is a custom emoji that has not federated yet in those posts/replies, etc. But the explanation is probably not very far from the reality in terms of the general functionality of the protocol.” ↩︎
After posting this article from my instance, I watched the Sidekiq queue during the original posting and subsequent replies and it would appear that, currently, the actual number of unique instances my followers are on is ~1,377. ↩︎
Or start blocking followers, or unfollowing people, or staying quiet. None of which are viable options. (Especially the last one… I mean, do you even know me?) ↩︎
Somewhat ironically, I’m apparently responsible for one of the first Twitter migrations (although on a much smaller scale than today’s), back when Mastodon was just starting out. I guess you could say all this is just karma. ↩︎
He’d then get his followers on Twitter to join as many different Mastodon instances as they could and follow him. Finally, with his bootlicker army of incels in place, he’d just have to get chatty with them and watch Eugen’s instance burn to the ground under the weight of it all. ↩︎
On an instance run by donations, for example, this would mean those donations would go to subsidise his account far more than the other accounts on the instance. That is, if they even managed to cover the cost of it to begin with. In Big Tech, like Twitter, the burden placed on the system by celebrities and other exceedingly popular accounts (journalists, etc.) is just a cost of doing business. The cost is subsidised by the corporation because these accounts are the bait that attracts the true assets of the business: you. In Big Tech, you are the livestock being farmed. They just have to keep you distracted enough with enough sparkly things that you don’t realise you’re being farmed.Also, there are economies of scale present in centralised Big Tech systems that simply do not (and should not) exist in decentralised systems. ↩︎
Unlike me, I don’t think Stephen would have to sweat the cost of the server although it will be considerably more than the heavily-subsidised $8/month that Elon would have been charging him. ↩︎
The term “user” is an othering. In Small Tech, we call people “people.” ↩︎
Some would say “tumours”. Hello, I am Some. ↩︎
If anything, the design decisions behind servers like Mastodon show that we need radically different approaches and first-principles design optimised for single-tenant servers and peer-to-peer connections if we want to nurture a Small Web. ↩︎
And I have been from the start. ↩︎

NLnet Grant Application for Domain Rejected

mail@ar.al (Aral Balkan) — Thu, 20 Oct 2022 11:08:59 +0100

On October 12th, 2022, we recevied the following form letter, informing us that our NLnet Grant Application (original application, follow-up questions and answers) for Domain has been rejected.

Dear Aral,

I'm sorry to have to inform you that your project "Domain" (2022-08-099) unfortunately was not selected for a grant.

This in no way means that we do not see the value of the work you proposed. We get many more excellent proposals than we can grant with our limited means, so competition is extremely tough. There are unfortunately many talented independent researchers, developers and community leaders that need funding. In addition to that, the specific scope and rules of play of each call pose (sometimes artificial) limits in our selection that mean excellent projects leave empty-handed - because they just do not fit well enough within a certain fund.

Again, we are very sorry that we cannot offer you support for your good efforts. We hope you are not discouraged, and are able to secure funding elsewhere for the project, for instance from any of these organisations:

 https://NLnet.nl/foundation/network.html

or through other calls from the Next Generation Internet:

 https://www.ngi.eu/opencalls

And do trust that we have your funding need and the outline of your project in the back of our head from now on, and so we might come back to you if an opportunity arises (unless you asked us to destroy your contact details in the application form, in which case we will do so).

If you should have any questions, please let us know. And in case you have another good project in mind in the future, do not hesitate to submit again!

Kind regards,
on behalf of NLnet foundation,

Michiel Leenaars
Strategy Director

This means that, to date, we have received and continue to receive no European Union funding for our work at Small Technology Foundation.

I’m also done wasting time writing grant proposals to organisations that clearly do not care about supporting the work we do.

Instead, there is code to write and we will continue to work for the common good – as we have been doing for almost the past decade – even though we are not funded from the common purse.

If you’d like to help us continue to exist, please feel free to become a patron or make a donation.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Nlnet Grant Application for Domain – First Update: Questions and Answers

mail@ar.al (Aral Balkan) — Wed, 28 Sep 2022 09:31:40 +0100

This is the first update to our NLnet Grant Application for Domain.

You applied to the 2022-08 open call from NLnet. We have some questions regarding your project proposal Domain.

Thank you for getting back to us. Please find the answers, inline, below.

You requested 50000 euro, equivalent of one year of effort. Can you provide some more detail on how you arrived at this time estimate? Could you provide a breakdown of the main tasks, and the associated effort? What rates did you use?

As a funding body, I can see why the question of funding amount is the primary question. However, as someone who has eschewed mainstream income and other benefits to work for the common good, it is the least important one for me so I’ll leave this to the end and tackle your other questions first.

Can you compare Domain to chatons.org and Libre.sh (note the former community also already uses the word Kitten…)?

Let me tackle them separately:

Domain vs. Chatons

CHATONS is an excellent initiative by the lovely folks at Framasoft – who have supported our work in the past with translations of our articles into French – to create “a collective of small structures offering online services (e.g. email, web hosting, collaborative tools, communication tools, etc.)”. They call these structures “hosters.”

Within the CHATONS, model, Domain is a tool to enable the creation of more hosters. Everyday people can then use the Domain instances of those hosters to set up their own small web places.

Once Domain is ready for use, CHATONS is a collective that I can see us considering becoming a part of with Domain and Kitten (after all, as you point out, they’re already called Kittens so even that fits).

Domain vs. Libre.sh

Libre.sh, from what I can tell based on their web site, is a tool for people with technical knowledge to set up web sites using Kubernites or Docker Compose. These are enterprise technologies used by Big Tech and involve a high level of complexity. In the Getting Started guide for the Kubernites version of libre.sh, you are shown how to deploy a cluster with 9 machines: “3 masters, 3 ingresses, 3 compute”. Needless to say, this is a completely different use case than Domain.

The goal of Domain is to enable organisations to become small web hosts, where they can provide a simple interface for everyday people without technical knowledge to set up their own small web places (sites/apps). As such, you can think of it as “Digital Ocean in a box” for anyone who wants to run their own small web host.

Domain provides a holistic solution that integrates the necessary components of provisioning a VPS, managing subdomains (DNS), installing web applications, and (optionally) charging for the service (or otherwise controlling access to resources).

The problem libre.sh solves is how do we “host free software at scale?” while the problem Domain solves is “how do we enable people to host free software that doesn’t scale?” (in other words small web places). And, crucially, how do we make it possible for any organisation to become such a host? And easy for people without technical knowledge to set up small web places using it?

if one would use Domain to install e.g. Yunohost or Sandstorm, what would be the added advantage compared to deploy a preconfigured image of these directly at a VPS hosting company (which is already on offer with some hosters).

You wouldn’t use Domain to install Yunohost or Sandstorm as they are different approaches to solving similar problems. Both Yunohost and Sandstorm offer dashboards for installing existing web applications. These are applications that, for the most part, have been created in the traditional multi-user, Big Tech design, with out-of-process databases, etc.

Domain, on the other hand, is for hosting single-tenant Small Web applications. The kind that you can create, for example, with Kitten.

Domain does not aim to support every type of web application but a very specific type (single tenant, small web application). This is its greatest strength as this focus and control means that we can simplify the experience and reduce the system requirements throughout the stack and lifecycle (e.g., when it comes to automatic updates and maintenance).

Yunohost, Sandstorm and similar projects are great in that they allow more people to install and use existing free/open source “multi-user” web apps that have almost entirely been designed with Big Web architectures.

Domain aims to do the same thing for Small Web apps while reducing complexity and the need for technical knowledge and improving the overall experience.

The Domain approach could be interpreted as poor-mans hosting and/or entry level shared hosting (which is already a very competitively priced economic area). In order to cut prices ,there are definite significant tradeoffs which may bite the user in the longer term.

I wouldn’t call Domain’s approach to democratising hosting by enabling independent organisations to become small web hosts “poor man’s hosting” any more than I would call the platform cooperative movement “poor man’s corporations”. The goal is to democratise not just the ownership of small web places but to do so without centralising them at a single host (e.g., us).

While price is an important factor for commercial providers (in that there is likely a psychological number beyond which it would be deemed too expensive for a small web place), it is definitely not the primary focus. As mentioned in the original funding application, Domain can be run as a private instance (e.g., internally for organisations, neighbourhoods, families, etc.) or using a token-based system (where, for example, a municipality can issue tokens that citizens can exchange for the own small web places instead of using legal tender).

Finally, the type of hosting integrated into Domain is Virtual Private Server (VPS) not shared hosting.

Borrowing a subdomain from someone doesn’t offer much legal standing, whether it is on the public suffix list or not.

Borrowing a subdomain from an organisation offers the exact same legal standing as borrowing a sudomain from a commercial top-level domain (like .com, etc.) provider in that they are both subject to contract law. So whatever terms are in the contract are the legal standing that is offered.

The difference between a commercial top-level domain provider and organisations that will be running domain (like our not-for-profit Small Technology Foundation) is the reason why the domains are being offered. In the former, it is to make a profit. For us, it’s to provide people with a location that they can be easily reached on the Web with.

When a host folds the entire reputation of its user base is immediately destroyed (unlike the case of a TLD, which has an escrow arrangement enforced by their ICANN contract). When someone forgets to backup and things crash, users have little resort. And small payments are expensive to process, eating further into the “Domain” hosters margins. What countermeasures do you propose for such scenario’s to give users peace of mind? Is there some form of service portability?

Indeed, if a host shuts down, this would take down everyone they host. This is a problem inherent with decentralisation and we see it when a fediverse server goes down too. This is not to say that the solution is then to ensure that only a handful of mutli-billion-dollar commercial hosts should exist. Just as the solution to fediverse servers being shut down is not to say that everyone should use Twitter instead because we can trust it’ll be around in one form or other.

Instead, possible mitigations for this fact of life include:

Supporting small web hosts from the commons
Encouraging as many small web hosts as possible so that if one goes away the damage is limited
Ensuring that it is as easy as possible to move between small web hosts (portability)

While the first one would involve political will, the second and third are issues that Domain exists to tackle.

On the topic of payments, Domain initially supports Stripe which, to the best of my knowledge, does not impose higher fees for the sorts of amounts we’re talking about. Microtransactions (which is not what this is) might be a different matter but do not concern our use case for Domain.

Finally, while the initial setup is on a subdomain, it will be easy to configure the site to use any domain (on a regular TLD) after the fact. The initial setup using a subdomain is a design decision to both enable people to have a small web place without paying separately for a commercial domain name and to keep the setup time as quick as possible.

If people are expected to pay 10 euro’s a month, that puts them in the realm of CPanel and Plesk/managed services which are already common with hosting companies for decades - and at prices going down to below 1 euro per month for e.g. Wordpress hosting for which tens of thousands of tutorials and extensions to make everything possible. What planned features are supposed to lure people over from that competition (and the marketing power of players like Strato)?

First off, to clarify: the €10/month number is an initial estimate. It’s based on our own research into what could make Small Technology Foundation sustainable by hosting a Domain instance at small-web.org. It’s also based on my belief that it’s likely the maximum amount you could charge that still feels “small.”

And, again, Domain instances can be private and token-based. The commercial payment aspect is a sad necessity for organisations like ours that need to survive under capitalism today while hopefully creating systems that will mean that eventually we will have kinder and fairer systems tomorrow where we better understand the value of the commons and support it accordingly.

If cheap is the main target, would it not be more convenient to point people to dot.tk et al, which gives them a ‘real’ domain name (discussion about their security aside) at no cost, conveniently exposed by an API that allows for automation?

Cheap really isn’t the main target. If anything, affordable is. And, ideally, in time, we’d love to see small web hosts supported from the common purse for the common good. But until that time, we aim to prove, even within the success criteria of the current system, that we can be sustainable.

If someone wants to use a dot.tk name (not linking to them as their site doesn’t use TLS which doesn’t give me a lot of confidence about them), they will be able to.

You state that a domain on the public suffix list would “provide all the privacy and security guarantees that any other top-level domain can”. Is that actually true? i

Yes, like the other statements in our funding application and that we make in general, it is true :)

(If we wanted to get into lying, we’d be working in the mainstream. You can make a lot of money there doing that. It’s actually quite a sought-after skill.)

To reiterate, from the official description:

It allows browsers to, for example:

Avoid privacy-damaging “supercookies” being set for high-level domain name suffixes

Highlight the most important part of a domain name in the user interface

Accurately sort history entries by site

When a domain is on the Public Suffix List, it is, for all intents and purposes, a top-level domain and is treated as such by browsers. This is a very useful hack for creating domains that are not part of the commercial top-level domain business and is one of the fundamental design decisions in Domain that sets it apart from other initiatives in the area.

A lot of information can already be learned just from the DNS requests the second level nameservers would see. How do you intend to deal with e.g. TLSA records, DNSSEC, SPF, etc? What kind of alternative services will be delivered (for instance e-mail? ) How will e.g. spam and blacklisting impact sibling Domain-hosted efforts?

As a project from a not-for-profit that exists to protect privacy, Domain includes privacy policies for hosts based on data minimisation. While the initial service providers for DNS, VPS, TLS (DNSimple, Hetzner, Let’s Encrpypt), etc., do have the means to gather data, this is true in general (not specific to Domain) and they are bound by the rules of GDPR.

Because of ACME, meanwhile there has been a lot of work on automation of DNS challenges (https://github.com/acmesh-official/acme.sh/tree/master/dnsapi). Would it not make sense to seek a similar approach for the DNS management, where one does not depend on hardcoding a single US based domain name company (DNSimple) as exclusive provider?

DNSimple is not used for TLS certificates. Kitten uses Auto Encrypt (another of our free and open source libraries) to automatically provision TLS certificates using Let’s Encrypt’s HTTP-01 challenges.

DNSimple is used for setting up the domain records for subdomains.

And, again, DNSimple is simply the first provider we are building support for to get Domain up and running. The goal is to support others that have APIs and to thus abstract out the APIs, hopefully commoditising these services to some degree in the process.

How would applications be packaged and maintained/security hardened? Is this something you have experience with?

Applications are “packaged” (insofar as we can use that term for it) in Domain using cloud-init. Currently, this is on standard Ubuntu 22.04 instances with unattended security updates enabled. However, we are keeping our eyes on immutable operating systems like CoreOS to aid in OS updates in the future.

Much of the security of Kitten and Small Web sites comes from their simplicity and small attack surface. There are fewer moving parts, less code, and basic standards being used.

Can you elaborate on deployment and maintenance requirements after initial deployment - which tend to forms a continuous drain on resources at the project level and the user level?

Small Web sites/apps built on Kitten and deployed with Domain will be entirely self-maintaining.

Kitten’s installation process is based on git and it also clones git repositories to deploy apps.

There is also work underway to implement:

Automatic updates for the deployed app (via git)
Automatic updates for Kitten itself (again via git)

Eventually, when immutable server operating systems become available, the goal is to have automatic major version operating system updates as well for completely hands-off maintenance. (In the meanwhile we will be deploying to LTS versions of Ubuntu which will give us quite a few years of headroom on this.)

You mention that setup can be done in under one minute, but surely beyond that someone will have to do the hard work?

No, that’s the whole idea: there isn’t. When you install a small web app and, say, 45 seconds later, it’s ready, it’s actually ready. You hit your domain. You start using it. That’s all there is to it.

Is there a threat analysis that you’ve considered during the design phase - if one of the fellow hosted community members doesn’t update their version of X, how do you prevent compromise of the rest? Is configuration and user management separate from delivering bits? Would Domain packages offer reproducibility, like Nix/Nixpkgs (which has a vast collection of software, see repology.org)?

The security model of Domain follows, in the words of Joanna Rutkowska (of Qubes, etc.) “security through distrust”.

This results in some core design decisions:

Domains are on the Public Suffix List (so no one can set supercookies, etc.)
Every person gets their own virtual private server (we don’t use shared hosting)
Every site is automatically protected by TLS

And, to reiterate, some related properties even if they may not seem to immediately be security-related:

You can use your own domain.
You can easily move away from a host.
All code is free and open.

Regarding reproducibility, since small web apps are installed via git, there isn’t necessarily a build process involved. Where one is (e.g., via npm install), it would be up to the apps themselves to ensure that dependencies are locked and loaded from trusted sources and that commits and tags are signed.

You requested 50000 euro, equivalent of one year of effort. Can you provide some more detail on how you arrived at this time estimate? Could you provide a breakdown of the main tasks, and the associated effort? What rates did you use?

Finally, to return to the first question: I work full time at Small Technology Foundation and my work these days is full-time on Kitten and Domain. So, let’s say I work 40 hours a week on average (it can vary and I often work on the weekends too):

€50K/yr comes down to about ~€26/hour.

Let’s put this into perspective: over a decade ago, when I doing regular development work as a contractor, I was charging €100/hr. My partner, who is contracting with Stately (a startup), makes more than double this amount a year and this is the main source of income that is currently sustaining Small Technology Foundation (the two of us and our dog). Previously, I’ve sold two family homes in Turkey and we’ve relied on a combination of sales of our tracker blocker (Better, now retired) and fees from conference speaking to scrape by.

All this to say that I don’t actually care how much funding we get. (I wonder if this is why they don’t usually let me write the funding applications?) Whether it’s €50K or €0 or something in between, we will keep working on Kitten and Domain and we will keep working on our vision for a Small Web owned and controlled by people, not corporations. We’ve always found a way and we will continue to do so.

What would be nice, however, is to feel like we are supported. We have been working for the common good for almost a decade now and it would be nice to have it funded from the common purse to some degree at least. And it would also be nice to have some stability so we can keep working on this without worrying about how we’re going to exist in X months time.

While I understand that the funding from ngi/NLnet is mostly project (and maybe even feature)-based, I do believe we need to think longer term and support folks like ourselves who are essentially carrying out research and development.

Silicon Valley understands the value of funding teams and then allowing them to pivot, etc., as they explore a problem domain and learn more about it. Sadly, it does so with the worst possible success criteria (how to best farm you for your data and monetise it). As I mentioned during one of my talks at the European Parliament, I hope that we can do the same thing for folks working on technology for the common good.

So I’m not going to give you an hour-by-hour breakdown of tasks because I don’t know what those are going to be beyond a few days’ time. You can keep track of the current ones on the issue trackers of the Kitten and Domain projects.

Apart from that, I get up in the morning and work on Kitten and Domain and that’s what I’ll be doing for the foreseeable future.

Please support us financially if you feel that what we’re working on should exist and you’d like us to exist long enough to make sure that it does.

If you have any other questions, please don’t hesitate to get in touch.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Small Is Beautiful Episode 22 Small Web Kitten Dweb Camp Braid

mail@ar.al (Aral Balkan) — Fri, 16 Sep 2022 10:49:35 +0100

Dear Linux, Privileged Ports Must Die

mail@ar.al (Aral Balkan) — Tue, 30 Aug 2022 14:30:53 +0100

Privileged ports, toffs of the Linux world.

Kitten is a small web server that runs as a user-level service and would never need elevated privileges if it wasn’t for one archaic anti-security feature in Linux that dates back to the mainframe era: privileged ports.

Back to the future

As it was in Unix in the 1980s, so it is now, that any process that wants to bind to a port less than 1024 must have elevated privileges. These ports are known as “privileged ports.” While this was a security feature in the days of dumb terminals, in the age of the World Wide Web, it is a security vulnerability.

Privileged ports lead to dangerous security practices, like server processes forgetting to drop privileges and being run as root.

They’ve been obsolete for quite a while, macOS removed them as of Mojave¹, and there’s no comparable concept on Windows either. Heck, they apparently even cause climate change.

Not to mention, they’re a world of unnecessary hurt to work around.

In fact, even their original implementation in BSD was a hack:

if (lport) {
  u_short aport = htons(lport);
  int wild = 0;

  /* GROSS */
  if (aport < IPPORT_RESERVED && u.u_uid != 0)
    return (EACCES);
  …

The BSD people knew this was a hack; they just did it anyway, probably because it was a very handy hack in their trusted local network environment. Unix has quietly inherited it ever since.

— The BSD r* commands and the history of privileged TCP ports

Listen to what some other folks have to say on the subject:

“So we have web servers, and all other servers whose standard ports happen to fall below 1024, expecting to be started as root, bind(2) to their service address, and then “drop privileges”. At this point, you must think I’m shitting you, but I’m not. This is for real. Failure to drop privileges after binding to listen port is a whole category of vulnerability. It’s like throwing egg on the stairs every morning, and several times later each day carefully classifying various accidents in the accident log as “slipped on eggy stairs”. Stop throwing egg on the stairs!”

— The Persistent Idiocy of “Privileged Ports” on Unix

This port 1024 limit is a security measure. But it is based on an obsolete security model and today it only gives a false sense of security and contributes to security holes.

— Why can only root listen to ports below 1024?

These are the reasons why I suspect the 1024 limit was imposed:

Don’t let users run system-level services on the mainframe.

Don’t let the users hog ports for important services on the mainframe.

Don’t let the users run a bogus service to steal logins on the mainframe.

…and here’s why I think these aren’t relevant anymore:

- The mainframe is now the desktop.

– …I really don’t get it

The workaround

On modern Linux systems, you can configure privileged ports using sysctl:

sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80

However, that setting does not survive a reboot.

You can also configure the setting in a persistent way using a configuration file. e.g., by creating a file called /etc/sysctl.d/99-reduce-unprivileged-port-start-to-80.conf with the following content:

net.ipv4.ip_unprivileged_port_start=80

To remove all privileged ports, you can set that value to 0. For our needs, 80 will do as it means our web server can bind to ports 80 and 443 without requiring superuser privileges.

In fact, the Kitten installer does just this.

But even that has issues.

For one thing, it doesn’t work in rootless containers (e.g., if you’re running on an “immutable” Linux distribution like Fedora Silverblue²) because the configuration file gets added to the container, which doesn’t have systemd running sysctl.d so the configuration setting doesn’t get applied at boot.

So we’re back to having to gain temporary privileges and drop them just to alter this configuration setting every time the server is run.

What a pain in the ass.

The fix

The fix is easy: ship Linux distributions so that privileged ports start from 80 to begin with.³

net.ipv4.ip_unprivileged_port_start=80

Sure, this could also be set to zero but setting it to 80 would fix the number one use case today, which is to allow web servers to bind to ports 80 and 443 without requiring superuser privileges. So I’d be happy with either solution.

And for the three folks in Finland who administer multi-user Linux instances and rely on privileged ports for their mainframe-era security properties, they can always run sysctl and set their port limit to 1024 as it was before.

Yay, everyone’s happy!

This is such an easy fix and one that would improve security across the board that I hope Linux distributions will start implementing it as soon as possible.

All it takes is one major distribution to start the trend and the rest will follow. Please feel free to open issues in your distribution of choice and talk to folks you know to get them to do this.

#Linux #PrivilegedPortsMustDie is what I’m saying.

It’s time to move Linux out of the mainframe era… kicking and screaming, if need be.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

I’m just waiting for someone to tell me the folks at Apple don’t understand security and that macOS is less secure now because they removed privileged ports. ↩︎
And these so-called “immutable” distributions are the future of Linux, not just on the desktop but on servers also (see, for example, Fedora CoreOS, etc.) So the sooner we remove the archaic security anti-feature that is privileged ports, the better. ↩︎
I mean, ideally, the kernel could implement this fix and be done with it for every Linux distribution but I have no idea how to get the kernel folks to implement something. I have a feeling it involves a lot of text-only emails and being told how dumb I am in no uncertain terms by multiple people. So, yeah, if anyone else wants to take that one, please be my guest ;) ↩︎

Lipstick on a Pig: learning the most important lesson in design

mail@ar.al (Aral Balkan) — Wed, 17 Aug 2022 16:15:00 +0100

I just released a little tool called Lipstick on a Pig that helps keep the visual appearance of supported command-line applications in sync with the current light/dark mode setting (colour scheme) of your system in GNOME.

But why is this tool even necessary to begin with?

Let’s start at the beginning…

Getting to GNOME you

The GNOME display environment¹, since version 42, implements support for light and dark appearance styles (aka colour schemes).²

With a GNOME extension like Night Theme Switcher, you can even automate the switch from light mode to dark mode based on time of day (or toggle it manually using a button in the system bar).³

Furthermore, newer terminal applications like Black Box follow GNOME’s colour scheme and automatically switch between their light and dark themes for the console and its content.

This is all great and it means that you can, for example, have your computer switch automatically to dark mode at sunset and have your terminal adapt to it automatically.

The immediately-obvious problem is that very few command-line applications adhere to the terminal’s theme, choosing to override it with their own theme settings, which, rarely (if ever) take the system’s colour scheme setting (light/dark mode) into consideration.

But is this the right problem to solve?

And is the command-line application layer the right place to solve it?

(Surely, I must think so because that’s the problem Lipstick on a Pig solves, right? Not really, no, read on…)

Solving the wrong problem

Think about it: The only reason command-line apps implement theme support is because they cannot rely on terminal apps to do so consistently.

If every terminal app in the world had consistent support for themes, command-line apps could just render their output, happy in the knowledge that it would get rendered in the person’s chosen theme.

So is the terminal app layer the correct place to define our problem?

No.

The fact that every terminal app would have to implement theme support should be a hint that it’s not. That’s a lot of duplicated effort. (And last I checked there wasn’t a standards body for terminal apps, nor do I think we’ll see one anytime soon given they’re not exactly – unlike the web, say – a huge industry.)

So let’s look for the problem one layer higher in the stack, in the display environment/operating system layer.

Is the core problem that the display environment/operating system does not have support for text themes?

Yes.

How do we know that this is the right layer to define our problem at? We know because if we define it at this layer, solving it will also solve it at the terminal app and command-line app layers.

Now that we know where to solve it, articulating the design problem becomes much easier:

“How do we ensure consistent syntax highlighting of text on an operating system that adheres to system colour scheme changes.”

So the ideal solution to this design problem would be for the theme settings that currently reside in a terminal app such as Black Box to instead exist in the Appearance settings of GNOME itself and for every app running on the system, including terminal apps, to use them when rendering syntax-highlighted text.

Were this to be implemented, it would mean terminal apps and command-line apps would no longer have to implement custom theme support themselves.

Which brings us to…

The most important lesson in design

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download Superheroes & Villains in Design directly, and watch it with your favourite video player.

The most important lesson in design is that design does not begin at solving a problem. Design begins at understanding what the right problem to solve is. Which means we must uncover where the problem actually is.

Most times, bad design (which results in tools that are unusable, overly complicated, etc.) doesn’t arise from solving a problem badly (although this can, of course, happen) but from solving the wrong problem.

This was the central thesis of my talk, Superheroes and Villains in Design, that I gave at Thinking Digital Conference in Newcastle almost a decade ago. I’d highly recommend you watch it, especially if you’re working in free technology. It’s more important than ever that the free and open alternatives we build today are not only ethically-sound but accessible, usable, and maybe even – heck, why not? – delightful. That’s the only way to ensure that people will actually use them. The Ethical Design Manifesto and Small Tech Principles can guide you in achieving this.

In the talk, I recount my less-than-delightful experiences trying to buy a ticket on the San Francisco Bay Area Rapid Transit (BART) system. While you can watch the video for all the hilarious details, suffice to say the folks who designed the BART ticketing system solved the wrong problem. They solved the problem of “How do we sell a ticket?” not the problem of “How do we get someone from point A to point B as effortlessly as possible.” If they had solved the latter problem, the confusing ticket machines would either look very different or perhaps not even exist at all.

Another example I give in the talk is that of a whole class of everyday devices that solve the wrong problem: washing machines. Nearly all washing machines today solve the problem of “washing clothes.” This is a problem nearly no one has (unless you wash clothes for a living or have a particular enthusiasm for the craft). However, nearly all of us have the problem of having dirty clothes we want to be clean. While they may at first appear to be the same problem, they’re not. These are two fundamentally different problems. Conflating them is the reason we have washing machines today with a plethora of settings that almost no one really understands instead of machines with almost no controls that you dump your clothes into and get clean clothes out of. (Hands up if you set your machine to the one or two settings you know work well enough and leave it at that.)

I won’t go into it again here as, in the talk, I sketch out what a washing machine would look like if it solved the right problem.

Spoiler alert: it comes out looking nothing like any washing machine you’ve seen.

Solving the right problem

So, to get back to the case in hand, the correct design problem we should be solving is how to ensure that any app on a system that renders syntax highlighted text does so consistently with every other app and in accordance with the system’s colour scheme settings.

Understanding the problem and understanding what level a problem should be solved at are inextricably linked.

We now understand that this problem cannot be solved at the level that Lipstick on a Pig solves it. (Now do you understand why I chose the name?) Therefore, we know that Lipstick on a Pig is not a design solution. Lipstick on a Pig is a hack; a pragmatic short-term workaround; a stopgap.

It’s very important that we don’t conflate hacks and pragmatic workarounds with actual solutions. The former are not without value. In fact, they might keep things ticking along for quite a while. But the moment we forget that they’re just temporary band-aids, we run the risk of incorporating them into our stacks and thereby weakening the foundations of those stacks. Where Murphie’s Hierarchy of Hacks goes, Murphy’s Law is usually not far behind. We eventually end up with brittle structures that can no longer be evolved or iterated upon without other parts of them collapsing, leading to complete rewrites or failure and abandonment of the systems for better alternatives.

Lipstick on a Pig

The problem Lipstick on a Pig solves is “how can we synchronise the visual appearance of command-line apps that implement their own theme systems with system colour scheme changes to provide people with a consistent/legible/readable experience.”

This complexity of the problem statement itself is the first smell that we’re solving the wrong problem at the wrong place.

And I’ve already baked in so many incorrect assumptions based on my belief (right or wrong, as it might be) on the difficulty of effecting change in the other layers in the stack.

I know that command-line apps should not be implementing their own theme systems (the operating system/desktop environment should). And I know that terminal apps should not be doing this either (because, again, the operating system/desktop environment should). I also know that by solving the right problem at the right layer in the stack⁴, it would create less work for – i.e., solve the problem for – all lower layers in the stack.⁵

And yet I’m releasing Lipstick on a Pig today instead of doing any of those other things.

Why?

Because getting people to care about consistency is hard.

And getting people to care about consistency in free/open source is even harder.

Cultural challenges

We still have an uphill battle even to get a desktop environment like GNOME to adhere to its own colour scheme settings. Why? Mostly because some folks like how something looks, consistency be damned.

GNOME has even formalised this inconsistency by not just having a light and dark mode but a “mixed mode.”

What’s mixed mode?

Quite bluntly, it’s an example of bad design (complexity and inconsistency) born out of a culture where design is still, to some extent, not practised from first principles but is subject to the aesthetic whims and fancies of individual developers and the compromises they (often times reluctantly) reach between themselves.⁶

It’s important to understand that these issues do not merely concern aesthetics and that they must not be driven by vanity. If someone has their system set to light or dark mode, there may well be important accessibility-and-usability-related reasons for doing so. So, beyond consistency (which itself lowers the cognitive load of the entire system), not respecting a system setting like light or dark mode is a perfect example of not respecting the people who use the tools we make. We must constantly remind ourselves that our preferences as developers do not override those of the people using the things we make.

On pragmatism and landfills

While I hope that the solution to the correct design problem in this case can be implemented at the correct layer in the stack, we must also be pragmatic. Sometimes all we can do is to implement a sub-optimal workaround (like Lipstick on a Pig) at the wrong layer in the stack.

Which layer?

One that we, ourselves, have control over.

I believe folks call this “pragmatism.”

(This is a concept I am trying more heartily to embrace these days in lieu of exotic expeditions to explore the spiralling depths of confrontation and bike-shedding and regular visits to the chasms of utter hopelessness and despair.)

And yet, even as I write this, I’m painfully aware that such “pragmatism”, alone, explains so much of how bad design comes about and why sometimes it‘s easier to start from scratch rather than to continue building on a system that is comprised of a landfill of hacks and workarounds.

X11, I’m looking at you, kid.

It also should hold a candle to what you can do in your own organisations to ensure that you don’t end up with a landfill on your hands: ensure designers have the responsibility and authority to affect change at any layer of the organisational stack.

And create a culture where design conversations take place from first principles, not personal preferences.

A map of solutions

So Lipstick on a Pig tackles the wrong problem at the most pragmatic and most sub-optimal level possible.

(Yippee! Don’t we all feel better now?)

It requires hacks to be implemented for individual command-line applications and thus has a potentially infinite problem space. This is literally the worst possible place to fix this problem and yet it is also the only place where I can fix it without engaging in (the often times frustrating process of) getting large open source projects to implement design changes.

And yet, if I didn’t believe that these things can change, I wouldn’t be writing this (admittedly rather lengthy, at this point, apologies…) treatise to accompany the launch of a simple little tool.

I do hope that in writing this I might influence those who have control over the more optimal layers of the stack to implement better solutions there. But I don’t have the time or wherewithal to fight those battles myself. I have my own projects – like Domain and Kitten – that I’m working on as a solo developer and they take up almost all my time. Even writing this feels like a luxury I can scantly afford right now while bigger problems remain unsolved.

With that goal in mind, here is a map of solutions, organised by whom it would require buy-in from.

1. Terminal app developers

One step up from Lipstick on a Pig would be to fix the problem at the terminal app layer while still essentially implementing the same technique Lipstick on a Pig uses with light and dark mode configurations for individual command-line apps.

A terminal app like Black Box, for example, could listen for system colour scheme changes and call the corresponding light or dark script. This would remove the need for Lipstick on a Pig and reduce the problem to one of configuration (you would still have to keep your light and dark scripts updated for the command-line apps you use).

Still suboptimal, but better.

2. Terminal app developers and command-line app developers

If we’re willing to open up the social effort required to implement a solution, we can stay at the terminal app layer and implement something better.

The terminal app could set an environment variable (maybe one that’s even eventually standardised by freedesktop.org) like $XDG_TERMINAL_HANDLES_THEME. Command-line apps could, then, easily implement support for light/dark mode by checking for this environment variable and, if it exists, disabling their own theme support to use the system theme.

This solution is much better in that it could, eventually, lead to command-line apps not having to implement themes themselves if enough terminal apps adhered to the standard. This is the second-best solution, and likely the best compromise between purity and pragmatism, given how difficult it would probably be to get GNOME to implement the optimal solution. (If past performance is any indication of future behaviour, I’m likely going to get chided just for writing this. Mais, c’est la vie, non, mes amis?)

3. GNOME developers

Finally, to reiterate, the ideal solution would be for the operating system/desktop environment to implement a solution that can be used not just by terminal apps but by any app that display text with syntax highlighting (e.g., code editors with graphical interfaces).

Until then – or until one of the more optimal solutions outlined above are implemented by others – I hope you’ll find Lipstick on a Pig useful as a stopgap.

And, before you rush off to solve a design problem, remember to always ask yourself the most important question in design:

Am I solving the right problem at the right place?

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Used by operating systems like Ubuntu, Fedora, etc. (Linux is confusing like that.) ↩︎
GNOME’s implementation of colour schemes is currently not holistic. Which is a nice way of saying it’s inconsistent and not very well supported by interfaces within GNOME shell itself. For example, if you want to create your own custom wallpapers for light and dark mode, you have to use GNOME Tweaks. ↩︎
The ability to schedule colour scheme changes based on time of day, having colour scheme changes animate smoothly, and having an easy to reach control for changing the colour scheme manually – in other words, basically everything the Night Theme Switcher extension does – are features that should really be added to GNOME itself and presented with intelligent defaults (e.g., switching to dark mode at sunset). ↩︎
Note that “the stack” does not necessarily just mean a technology stack. It includes socio-economic layers. While we’re mostly looking up and down the technological stack here, it’s quite possible that a wider analysis of a problem will reveal that the correct design solution must be implement in a non-technological layer. It may require, for example, changing the business model of an organisation. If you reach the top of the technology stack and you still cannot find an optimal solution to the problem, that’s a pretty good tell that the design problem (and therefore the solution) is at some higher, non-technological layer. In many ways, these are the hardest design problems to solve. Mostly because, in most organisations, problems at these levels are not seen as design problems and designers do not have the necessary authority or responsibility to tackle them. ↩︎
Were GNOME to implement system themes for syntax-highlighted text for light and dark modes, all apps – including terminal apps and all command-line applications – could consistently adhere to them without doing anything at all. Think of all the code (and thus complexity) removed from the equation were that to happen. ↩︎
Although this has gotten immeasurably better in recent years with the effect of folks like Tobias Bernard. ↩︎

Make Helix Editor follow the system colour scheme

mail@ar.al (Aral Balkan) — Mon, 01 Aug 2022 18:26:49 +0100

Black Box adheres to the system colour scheme in GNOME.

This is a niche post for those of you using a Terminal application that adheres to system light/dark mode settings¹ (like the excellent Black Box by Paulo Queiroz) and Helix Editor.

The problem is that Helix editor currently does not support separate light and dark themes or switching between them automatically when your system does. This is, however, something we can fix with a little bit of scripting.

Prerequisites

Make sure your Helix configuration is at ~/.config/helix/config.toml (or change the paths in the scripts accordingly) and that you have a line in it that sets the theme:

theme = "something"

(What it’s set to is not important as long the line is there so it can be changed by the update script we’re going to write next.)

The scripts

First, create a folder to hold the scripts:

mkdir -p ~/.config/helix/scripts/synchronise-with-system-colour-scheme

Then create the following scripts and ensure they’re executable (chmod +x <name-of-file>):

monitor.sh

#!/usr/bin/env bash

gsettings monitor org.gnome.desktop.interface color-scheme \
  | xargs -L1 bash -c "source ${HOME}/.config/helix/scripts/synchronise-with-system-colour-scheme/update.sh"

update.sh

#!/usr/bin/env bash

# Fail early, fail often.
set -eu -o pipefail

if [[ "$1" == "default" ]]; then
  # Update Helix Editor config to use light theme.
  sed -i 's/theme = ".*"/theme = "onelight"/' ${HOME}/.config/helix/config.toml
else
  # Update Helix Editor config to use dark theme.
  sed -i 's/theme = ".*"/theme = "dracula"/' ${HOME}/.config/helix/config.toml
fi

Edit the script to change the light and dark themes to the ones you want. You can see the ones available on your system using the :theme command in Helix.

You need to keep the monitor script running in the background for this to work. You can do this by simply running ./monitor.sh &, or (recommended), you can run it as a systemd user service.

Before creating the systemd service, create two more helper scripts to enable and disable it:

enable.sh

#!/usr/bin/env bash

# Enables the service to start on boot and also starts it up immediately.
systemctl --user enable helix-system-colour-scheme-synchronisation.service
systemctl --user start helix-system-colour-scheme-synchronisation.service

disable.sh

#!/usr/bin/env bash

# Stop the service immediately and disables it from starting on boot.
systemctl --user stop helix-system-colour-scheme-synchronisation.service
systemctl --user disable helix-system-colour-scheme-synchronisation.service

And, finally, create the systemd user service (in _~/.config/systemd/user and make sure it is executable.)

helix-system-colour-scheme-synchronisation.service

[Unit]
Description=Helix System Colour Scheme Synchronisation Service
Documentation=https://github.com/helix-editor/helix/issues/2158
After=gnome-session.target

[Service]
Type=simple
RestartSec=5
Restart=always
ExecStart=%h/.config/helix/scripts/synchronise-with-system-colour-scheme/monitor.sh

[Install]
WantedBy=gnome-session.target

That’s it.

Run: ./enable in your scripts folder to enable the service and ./disable to disable it.

Helix currently doesn’t automatically reload its configuration when it changes so you’ll have to restart existing instances manually.

I look forward to the day when we can see a smooth fade from a light theme to a dark theme and back when the system theme changes. 🤓👍

Until then, I hope you find this useful.

If you’re using GNOME Console or GNOME Terminal, this post won’t help you as those apps do not adhere to the system colour scheme. ↩︎

NLnet Grant Application for Domain

mail@ar.al (Aral Balkan) — Fri, 29 Jul 2022 11:04:33 +0100

This is my application to get NLnet funding to work on Domain as part of the User-Operated Internet Fund¹

I feel it’s important that such grant applications are made public so everyone has visibility into the process. This will allow us to collectively learn from the experience and perhaps even to improve the process itself.

As such, I’ll be making my end of the process as public as possible by not only sharing my original grant application but any subsequent communication I receive during the process. And I urge you to do the same on your own site when applying for similar grants.

Public money, public process, public code.

Timeline

Saturday, July 30, 2022: Proposal submitted to NLnet.

Tuesday, September 27, 2022: Questions received from NLnet.

Wednesday, September 28, 2022: Answers to questions sent to NLnet.

Wednesday, October 12, 2022: Our proposal is rejected.

Call topic

Thematic call: User-Operated Internet Fund

Contact information

Your name: Aral Balkan
Email address: aral@small-tech.org
Phone numbers: (not including it here for privacy reasons)
Organisation: Small Technology Foundation
Country: Ireland

General project information

Project name: Domain
Website/wiki: https://codeberg.org/domain/app

Abstract: Can you explain the whole project and its expected outcome(s). (1200 characters)

Domain creating an Owncast server in under a minute (live stream demo).

Domain aims to democratise the ownership of web sites and commoditise the gatekeepers.

“The best thing about the internet is that … [f]rom the edges inwards, users themselves can collectively own, operate and rewrite every aspect of the technology and infrastructure they depend on.” – User-Operated Internet Fund

Yes, we can. But it is technically difficult to do so.

This is the problem Domain aims to solve.

Think about what you have to do today to have your own web site (or self-hosted web application) running on your own server:

Buy a domain name from a commercial domain name registrar.
Sign up with a commercial web host (e.g., DigitalOcean, Hetzner, etc.)
Provision a server (e.g., virtual private server) with your web host.
Configure your Domain Name System (DNS) records to point to your server.
Install or configure a web server, the site (or app) you want to host, as well as any other required software (database, etc.).
Ensure your server is sufficiently secure.
Maintain your server by installing updates to your operating system and other related software on a regular basis.

While none of this is terribly difficult for seasoned developers or system administrators to tackle, it is not a process you would expect someone without technical skills to undertake without a considerable investment of time and effort.

And we want everyone to have their own place on the Web, not just those of us who have the relevant technical knowledge.

Furthermore, it can easily take at least half an hour even for technically-savvy folks to go through all those steps to set up a simple site or app (and longer still if you have to configure databases, message queues, etc.) I once watched an experienced software engineer struggle to set up his own Mastodon instance for an afternoon before giving up. ’Nuff said.

The goal of Domain is to reduce this process to under a minute.

(So we’re talking about an order of magnitude difference or more.)

Here’s a video of Domain creating and running a new Owncast server in under a minute.

How does it work?

Domain is being built for use by two different audiences:

Hosts: Organisations/technically-savvy individuals who run domain as a service for their communities.
Everyday people: Who use the Domain instances set up by hosts to commission, own, and control their own web sites and apps.

For hosts

Domain enables organisations and individuals to provide web hosting services for small (personal) web sites and apps to the general public.

As a prerequisite, hosts need to sign up for the following constituent services:

A web hosting provider (initially only Hetzner will be supported²).
A DNS provider (initially only DNSimple will be supported).
A domain name that they’ve added to the Public Suffix List³.
Optionally, an account with a payment provider if they plan to charge for hosting (initially only Stripe will be supported).

The Domain Administration Panel.

Once a host has acquired the required accounts and their domain is on the Public Suffix List, they will use the simple online administration interface provided by Domain to enter the relevant API keys for each of these services.

The VPS Host Settings screen.

Domain combines these fundamental services into a simple and seamless flow to enable everyday people to sign up for their own sites and apps at their own domain.

The App Settings screen.

Hosts can decide which apps and sites people can install. These apps are defined in the online interface by their installation and configuration instructions in a standard format known as cloud-init. It is currently possible in the Domain prototype to install Owncast, for example, as well as a generic Site.js web site.

App Settings: Owncast cloud-init configuration.

Domain will democratise the process of being a web host and enable small organisations and even individuals to host sites and apps for their local communities.

The Small Web design of Domain sets natural limits on how large a host can grow. Our goal is to encourage a community of lots of small, independent, and ideally not-for-profit hosts around the world. Each of which, of course, can customise Domain for their own needs and culture, given that it is released under AGPL version 3. This license also means that any improvements to Domain must be shared back into the commons, thereby strengthening it for everyone. (And it prevents, for example, a venture-capital-funded startup in the future from taking the software, spending a few million dollars to improve it, and then keeping those improvements to themselves to create fragmentation and lock-in.)

Domain is being built with an integrated web server and web development platform called Kitten (previously known as NodeKit).

Kitten is being developed alongside Domain, with Domain’s requirements determining its features.

Kitten will also be used to create and serve small web sites that can be installed with Domain and will be a useful tool itself in lowering the barrier of entry to developing and hosting your own web site or app. While Kitten is not the focus of this grant application, it is important to understand that it is impossible to separate the two as they are being developed in parallel and any support for our development work will benefit both of these intertwined projects.

For everyday people

For everyday people, Domain presents a very simple interface:

Choose a domain and select an app to install. Optionally, provide payment information and, within a minute, you’re up and running with your own site at your own domain.

The Domain interface for everyday people.

As far as I’m aware, there is currently nothing that approaches this in terms of ease of use.

The speed and ease with which everyday people can set up their own sites and apps at their own servers is at least an order of magnitude faster than what’s available today. We believe that this will have a substantial impact in democratising the Web by enabling more people to own and control their own place on it.

Have you been involved with projects or organisations relevant to this project before? And if so, can you tell us a bit about your contributions?

“Technology should be a commons for everyone to enjoy and contribute to, without gatekeepers or persistent threats.” – User-Operated Internet Fund

We formed Ind.ie in 2013 and have been working towards this very ideal ever since. In the intervening years, as our understanding of the problem grew, we honed our focus and limited resources on effecting change were we feel it will be most impactful and least likely to be threatened by proprietary systems and corporate control: the open web.

The Indienet project as covered in New Scientist

In 2017-2018 we worked with the City of Ghent, in Belgium, on prototyping the Indienet project. The goal was to demonstrate how citizens could have their own web sites (at a .gent domain) and own and control their own data when interacting among themselves and with the municipality. Sadly, a conservative local government was elected and our funding for the project was cut.

The Indienet project was the spiritual predecessor to Domain.

The Indienet project taught me several things: first, not to rely on public funding which might be taken away at any moment when the winds of politics change and, secondly, that we must build our own tools and frameworks at all levels of the stack to radically simplify the development and deployment of personal web sites and apps.

As part of our research and development efforts, I’ve been iterating on a personal web server and web development platform since 2019, when I first launched HTTPS Server which became Indie Web Server, and finally Site.js. The latter currently powers all our sites, including https://small-tech.org, my blog at https://ar.al, and, of course, https://sitejs.org itself.

Kitten, the server/framework I am building in parallel to Domain, is the next iteration of these efforts. It makes modern web development about as simple as it can get by eschewing a complicated build process and heavyweight JavaScript frameworks and enabling people to create web sites using plain HTML, CSS, and JavaScript and, optionally, enhancing them with HTML-over-the-wire via htmx and simple interactivity using hyperscript.

Domain is being written in and will be powered by Kitten.

Kitten (and thereby Domain) benefits from components that I created for Site.js, including the auto-encrypt library for simplifying secure sites with automatic Let’s Encrypt certificates and JavaScript Database (JSDB), a simple in-memory, streaming write-on-update JavaScript database for Small Web use that persists to a JavaScript transaction log.

(An earlier iteration of Kitten was called NodeKit and used Svelte instead of htmx. The current prototype of Domain in written in NodeKit and is being ported over to Kitten. The current prototype is able to deploy sites using Site.js. In fact, it’s how we set up our streaming server for Small Technology Foundation at https://owncast.small-web.org.)

Requested support

Requested amount: €50,000

Explain what the requested budget will be used for. Does the project have other funding sources, both past and present? (If you want, you can in addition attach a budget at the bottom of the form):

The requested budget will be used, after taxes, to pay me, a software developer with over two decades of professional development experience, to work on the project for the next 12 months (at a rate that’s less than half of what I was earning working in the mainstream two decades ago).

Our funding sources for Small Technology Foundation are and have been:

(Past) An initial crowdfunding round in 2013.
(Past) My selling two family homes in Turkey.
(Past and present) Patrons and donors.
(Present) Laura’s contracting work with Stately.

Compare your own project with existing or historical efforts.

The closest existing tools and projects to Domain are:

Hosting providers like DigitalOcean, Hetzner, etc. when combined with commercial domain name registration services.

These suffer from the complexity problem mentioned above and are profit-oriented corporate initiatives that do not have any sort of social mission to democratise the Web or technology.

Free and open source self-hosting platforms like Yunohost.

These systems provide web site/app installation functionality but they must themselves be hosted (which requires all the steps that Domain automates and streamlines). To illustrate, you could use Domain to create a server running Yunohost.

What are significant technical challenges you expect to solve during the project, if any?)

In terms of feasibility, I don’t believe there are any major questions left unexplored (as I have been working on the underlying infrastructure and design for several years now and I already have a functional prototype of Domain).

In the coming days, I will be porting Domain from NodeKit to Kitten and evolving Kitten to handle any unmet requirements this process might uncover in it. I feel very good about this as the replacement of Svelte with htmx has removed a large chunk of complexity from (the already lightweight) framework and developing with it feels excellent so far.

Now that the architectural decisions have been locked down, I expect a rather straightforward and sustained development effort with ongoing releases of both Kitten and Domain in the coming year.

Describe the ecosystem of the project, and how you will engage with relevant actors and promote the outcomes?

Initially, my goal is to launch our own Domain instance at https://small-web.org to demonstrate how it works and so it can start bringing in income for Small Technology Foundation. (Our goal is to become sustainable so we can keep working on the Small Web.) I envision that the initial people who play with it to set up their own sites and apps will be people who hear about it because they follow me on the fediverse.

I hope that once other developers see that it works, they might want to start becoming hosts themselves (either individually or with their organisations). My goal is to eventually see a healthy ecosystem of independent (and ideally not-for-profit hosts) around the world using Domain to empower their communities to own and control their own places on the World Wide Web.

I would love to see community groups, cooperatives, municipalities, and other organisations experimenting with alternative means of sustaining such efforts. The token ‘payment’ system in Domain, for example, could be used by a city to mail out codes to all its citizens that they can then use in lieu of payment to set up their own web sites and apps. (This is the model we were exploring with the City of Ghent.)

Finally, although Kitten is not the focus of this grant proposal, since it is being developed in parallel with Domain (and given that Domain is being written in it and will be powered by it), I believe that Kitten will also open up web development to a wider group of people by removing the complexities inherent in current corporate stacks. I foresee it being used to teach web development (at the very least, I foresee myself using it to teach web development).

Attachments

None.

(Please see links in the body of the proposal.)

How may we handle your information

(If you haven’t, please check our privacy policy).

When your project gets selected, we will legally need to retain your information for compliance purposes for at least seven years. Also, in case you are submitting to one of the NGI related funds, at that point we will need to share some information with our (not-for-profit) partner organisations in order so they may assist you with mentoring as well as complementary services e.g. accessibility, documentation, localisation, packaging, etc. By submitting a proposal you grant us permission to handle the data in the proposal in the manner described.

What should we do in the other case, e.g. when your project is not immediately selected?

I allow NLnet Foundation to keep the information I submit on record, should future funding opportunities arise
Send me a copy of this application.

I do wish they had called it the Person-Operated or Human-Operated Internet Fund (or even just the Personal Internal Fund) instead of using the Silicon Valley word ‘user’, which is inherently colonial/ethnographic in nature. This is why we refer to people as ‘people’ in small technology. ↩︎
While initially only a single web host, DNS provider, and payment provider will be supported, our hope is to add other providers so that, eventually, Domain can commoditise these services. The greatest challenge there is that commercial providers usually have custom Application Programming Interfaces (APIs) as part of their business model of locking people into their own services. So our goal will be to work towards better interoperability between service providers as well as abstracting away the differences on our end. ↩︎
Domain makes use of the Public Suffix List in order to bypass the commercial domain name system and enable fully-functional sites to be set up at their own domains in under a minute.

A domain like small-web.org – where Small Technology Foundation will be running our own instance of Domain – which is on the Public Suffix List is, for all intents and purposes, equivalent to a top-level domain and can provide all the privacy and security guarantees that any other top-level domain can (e.g., .com, .org, etc.) ↩︎

Using bound functions to unit test EcmaScript Modules

mail@ar.al (Aral Balkan) — Wed, 23 Mar 2022 15:47:53 +0000

Imagine you have the following EcmaScript module you want to unit test:

// untestable.js
const configurationValue = 'something'

export function functionToTest () {
  return configurationValue
}

The problem is you can’t test it for different values of configurationValue (which, in a real example, might be based on a runtime quality of your app). You can’t because configuration value is private to the module.

However, if you rewrite your function like this:

// testable.js
export const context = {
  configurationValue: 'something'
}

export const functionToTest = (function () {
  return this.configurationValue
}).bind(context)

You can test it by providing different values for configurationValue like this¹:

// tests/index.js
import test from '@small-tech/tape-with-promises'

import { context, functionToTest } from '../testable.js'

const something = 'something'
const somethingElse = 'something else'

test('index-testable', t => {
  t.equals(functionToTest(), something)

  context.configurationValue = somethingElse

  t.equals(functionToTest(), somethingElse)
  t.end()
})

This works (both tests pass) because exports are live bindings.

Why not just use a class? Because sometimes you can’t. Case in point: I came up with this method to test my ES Module Loader in NodeKit.

ES Module Loaders are meant to export resolve() and load() functions and I was configuring them using module-level variables.

I didn’t want to change the function signatures to inject values or introduce conditional logic to support unit tests.

This method allows me to inject the values I need during tests. The only change to the code is exporting a bound function instead of an unbound function and referring to any context variables using this.

Mocking whole modules

You can also use this method to mock imported modules.

For example, if the module you want to test looks like this:

// also-testable.js
import importedFunction from './other-module.js'

export const context = {
  importedFunction
}

export const functionToTest = (function () {
  return this.importedFunction()
}).bind(context)

And this is the module we want to mock:

// other-module.js
export default function () {
  return 'actual value'
}

We can mock it like this in our test:

import test from '@small-tech/tape-with-promises'

import { context, functionToTest } from '../also-testable.js'

const mockFunction = function () {
  return 'mock value'
}

test('index-testable', t => {
  t.equals(functionToTest(), 'actual value')

  context.importedFunction = mockFunction

  t.equals(functionToTest(), 'mock value')
  t.end()
})

Hope this adds another tool to your belt should you need it when unit testing EcmaScript Modules.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

If you’re going to try this out, first npm i --save-dev @small-tech/tape-with-promises. You can run it with npx tape tests/index.js. ↩︎

Decentralisation begins at decentring yourself

mail@ar.al (Aral Balkan) — Wed, 16 Feb 2022 13:34:21 +0000

My talk at the Department of Art & Media Technology, University of Southampton, on Wednesday, February 16th, 2022.

This morning, I was invited to gave a talk at the Department of Art & Media Technology, University of Southampton.

I was invited by Adam Procter, a long-time ally who heads up the department, as part of a new series of talks to help shape department policy on digital and networked technologies.

The talk covered:

Corporate vs human futures and topologies.
Big Web vs Small Web (and web3 vs web0).
Decentralised and decolonial approaches to technology.

Links mentioned during the talk:

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

NodeKit extension for Codium

mail@ar.al (Aral Balkan) — Tue, 15 Feb 2022 11:03:27 +0000

NodeKit for Codium: a reluctant fork of Svelte Language Tools.

This past week, I cobbled together¹ a very basic NodeKit extension for Codium.

Download

nodekit-for-codium-0.5.0.vsix

What’s NodeKit? Watch the NodeKit video introduction to find out.

Install

Either drag the downloaded package to the Extensions panel of Codium or, in your terminal, run the following command from the folder you downloaded the package to:

code --install-extension ./nodekit-for-codium-0.5.0.vsix

If you have Svelte Language Tools installed, this extension will likely clash with it so please disable the other extension while playing with NodeKit.

The NodeKit code in the example keeps a simple ephemeral server-side count and displays it in the browser when the page loads.

About

The NodeKit for Codium extension is a fork of Svelte Language Tools that:

Hides the <data> section of a .page from Svelte so it doesn’t complain about it.
Provides basic JavaScript syntax highlighting for <data> blocks.
Provides very basic JavaScript completions (not Node-specific) for <data> blocks.
Provides full JavaScript syntax highlighting and language intelligence for .data, .get, .post, and .socket files.

Good, perfect and all that

I’m not happy with the current state of this extension but you know what they say about perfect being the enemy of the good.

Basically, without this extension, Codium gets in your way when you add the NodeKit <data> block to your code. With it, it doesn’t and you get proper Svelte language intelligence for your page as well as basic syntax highlighting for your <data> block.

Initially, I did try creating a separate tiny extension that uses request forwarding to forward requests for the <data> block and the rest of the document separately but that didn’t work. And, even if it had worked, being a client-side solution, it would have been Codium-specific.

A forking shame

So, like it or not, if I want other editors to support language intelligence for NodeKit in the future, I currently find myself in the unenviable position of having to fork the Svelte language server.

It is sad that there isn’t a cleaner way to basically say, at the language server level, “hey, see this … region? Treat it as JavaScript.”

I mean this is essentially what the Svelte language server itself does by forwarding different regions of Svelte files to different language services and using source maps to translate the results back but it is not trivial to implement (²).

I’ll likely eventually end up modifying the Svelte language server far more extensively to add proper support for <data> and other NodeScript blocks (like <post>, <socket>, etc.)

Right now, however, given the extension is currently good enough – if not anywhere near perfect – I’m going to get back to developing the core of NodeKit as I build Domain using it.

If you’re playing with NodeKit at this very early stage, I hope the existence of this extension makes your life a little easier. I’m going to keep thinking about tooling around NodeKit to try and balance not recreating the wheel with having a first-rate developer experience.

Or, “Bled for a week to try and get something better together within the limitations of the Language Server Protocol while wading through The Incredible Machine that is the Svelte Language Tools monorepo but finally settled on something that works for now even if it’s not perfect.” ↩︎
At one point it even transpiles the Svelte source to TSX to make use of the TypeScript language service, which is a pragmatic move but it does end up creating the aforementioned Incredible Machine to decipher, especially when you’re first diving through the code. There is, however, a helpful overview of the language-tools and how things work together document that will make your life easier. ↩︎

Everyone Hates Facebook (but this is more than just about Facebook)

mail@ar.al (Aral Balkan) — Mon, 07 Feb 2022 10:15:01 +0000

“And my timelines lit up in a rare unity, the left and right, the rich and the poor, the healthy and the sick, all pleading and nodding and saying, yes, yes please, please do that.” - Adam Dalliance

Well, it’s official, everybody hates Facebook.

But why we hate it matters. As does what we intend to do about it.

For more good reasons to hate Facebook and other people farmers, watch my talk, The Camera Panopticon.

Good reasons to hate Facebook:

Because it is bad for democracy.
Because it is bad for personhood.

Bad reasons to hate Facebook:

Because it doesn’t censor the things your government wants it to.
Because it did censor your favourite neo-Nazi.
Because you want to create the next Facebook and be just as evil as they are but they’re standing in your way. (I’m looking at you VCs and startups, you know who you are.)

So, we all agree that Facebook is a problem.

Some for the right reasons, some for the wrong ones…

But this is not just about Facebook: it’s about any corporation that has the same business model as Facebook. The business model I call “people farming.”

So it’s about Google too. And Snapchat. And TikTok. And, and, and, and… (this is the business model of mainstream technology today.)

So we have a bigger – systemic – problem on our hands. (Ooh, fun!) And it seems everyone has some idea or other about how we should do things differently moving forward.

Bad ways forward

Recreate Facebook but in Europe.
Recreate Facebook but in fucking web3.
Make Facebook share its data with other people farmers so more people farmers can farm you for your data (try saying that quicky five times).

And yes, this really is exactly the EU Commission’s current hair-brained strategy because they can’t think beyond markets and antitrust.

Good ways forward

Watch my speech at the European Parliament in which I summarise the problem and suggest a solution.

Support current non-commercial federated alternatives (the “fediverse”), where there already are viable alternatives to Twitter, YouTube, and Instagram.
Support current non-commercial and single-tenant alternatives for individuals like Owncast for online video streaming.
Support the research and development of the Small Web – a non-commercial, human-scale web made up of spaces owned and controlled by individuals, not corporations.

Watch the recordings of Small is Beautiful, our monthly live stream at Small Technology Foundation, to find out more about the current state of my work on the Small Web.

Hating Facebook is all well and good but please let’s not forget that this is not just about Facebook. It’s about people farming in general.

If Facebook goes away tomorrow but another Facebook takes its place, we won’t have won anything.

So please let’s make sure we understand the differences between the various alternatives and pick the ones that will result in meaningful progress in protecting our personhood and democracy.

(Hint: look at the intent behind an organisation/project. Is it to make a billion dollars or to protect human rights and democracy? And yes, no matter what the capitalists tell you, the two goals are diametrically opposed and mutually exclusive.)

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

NodeKit Update

mail@ar.al (Aral Balkan) — Wed, 26 Jan 2022 11:40:33 +0000

Since the initial demo of NodeKit at last week’s Small Is Beautiful, we now actually have a nodekit command, the server now does naïve restarts on route changes, and routes are now lazily loaded the first time they’re hit.

I wanted to record this before jumping down the rabbit hole of implementing some form of hot module reloading/replacement using WebSockets and MorphDOM.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

The web0 manifesto

mail@ar.al (Aral Balkan) — Thu, 06 Jan 2022 10:55:19 +0000

The web0 manifesto: a short and sweet middle finger to web3.

Is it Covid or bullshit allergy?

Towards the end of 2021, I started noticing more and more “articles” on “web3” cropping up, each one more grating than the last.

It was as if ~~journalists~~ public relations hacks¹ were in a race to the death to see who could best parrot the latest right-libertarian techbro bullshit from bullshit press releases ejected out the fetid bowels of bullshit startups in Silicon Valley, the world’s capital of bullshit.

So, anyway…

I felt we could all use a little antihistamine – not to mention, a different term we could use to differentiate ourselves from the bullshit – so, over the holidays, I created the web0 manifesto.

As I write this, it has been online for a little under six days and signed by 639 people, projects, and organisations.

The site is entirely hand-rolled, doesn’t use any third-party APIs, and runs on Site.js. If you’re interested in some technical trivia, read the footer, glance at the privacy policy, and view the source.

You can also, of course, sign it.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

To these fine individuals I say, you can stop worrying: you do not have imposter syndrome; you are imposters. ↩︎

How to send an email

mail@ar.al (Aral Balkan) — Mon, 20 Dec 2021 18:07:03 +0000

How to send an email by speaking SMTP interactively to an email server. (This is how email works under the hood and it’s simpler than you might think. Play the video and follow along.)

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

The Three Laws of Personal Devices

mail@ar.al (Aral Balkan) — Sat, 18 Dec 2021 13:21:15 +0000

The Universal Declaration of Cyborg Rights states that we extend our selves using digital and networked technologies and that this extended self must be protected under human rights law.

As the primary means by which we extend ourselves today are through our everyday personal devices – computers, mobile phones, the so-called “Internet of Things” and “smart” homes, cars, etc. – we must enshrine the human rights that pertain to our extended selves within concrete laws that protect personhood in the digital network age.

I propose we need no more or less than the following three:

Law 1

Your devices must work in your interests and your interests alone.

Law 2

When a feature can be built so that algorithms and data are kept exclusively on the person’s own device, it must be built that way.

If a feature cannot be built in this manner, all data must be end-to-end encrypted and the owner of the device must be the exclusive holder of the private key.

Law 3

The hardware, software, and services must be free and open.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Saying goodbye to an old friend

mail@ar.al (Aral Balkan) — Thu, 16 Dec 2021 12:18:14 +0000

How do you like them apples? We didn’t and poor Better paid the price.

Trillion-dollar corporations are not your friend.

This isn’t a revelation to me. I’ve been saying it for some time now. And yet, over the years, I’ve taken my share of flack for seeing Apple as a viable mainstream stopgap based on its business model.

Well, that position has become untenable ever since Apple announced, got slammed for, doubled-down on, delayed, and finally, with iOS 15.2, started drip feeding us the changes in an attempt to slowly boil frogs as it continues with its plans to implement client-side scanning on iPhones, iPads, and Macs.

Apple didn’t come up with the client-side scanning idea, by the way, William Barr and Trump’s justice department did (also read the follow-up):

The suddenness of this new push is alarming. Also noteworthy is that suddenly the main reason to demonize encryption is CSAM, with terrorism and other ills playing second fiddle. Even as recently as late July 2019, when Barr revived his predecessors’ habit of castigating encrypted service providers, it was drug cartels he invoked. But CSAM is the dominant focus now, suddenly and thoroughly.

It is beyond question that CSAM is a real and serious problem … It is radioactive, it is illegal everywhere, and no legitimate company wants it on their servers. Nevertheless, this new single-minded focus on CSAM in the revived anti-encryption push feels like an exceedingly cynical move on the part of the U.S. government. Out of the Four Horsemen of the Infocalypse (terrorism, drug trafficking, CSAM, and organized crime), terrorism didn’t work to turn public opinion against encryption, so the government has switched horse(men) midstream.

…

One proposal for enabling law enforcement access is to build a system where the provider … would check content, such as a photo attached to a message, before it’s encrypted and transmitted to another user – i.e. while the content is on the sender’s device, not traveling through the provider’s server – to try to figure out whether that content is or might be abusive content such as CSAM.

A long time coming

My disenchantment with Apple has been a long time coming.

It hasn’t helped that since we moved to Ireland and set up Small Technology Foundation, we haven’t been able to get Apple to move our developer account over from our old organisation based in the UK to the new one here in Ireland.

In fact, we were going to discontinue Better then as we couldn’t afford to keep two organisations going but we received an offer from Apple to sort things out. I naïvely took them at their word and announced it as “a happy ending.”

Right afterwards, Apple went entirely silent on us.

We haven’t heard a single word from them over the last two years even after we closed down the old organisation (that our Apple developer account is still technically linked to) and even though we contacted their offices in Ireland to follow up.

So there’s that.

(Understand that this is something they happily do for large companies. It’s just that a two-person not-for-profit is next to worthless in their eyes.)

As I wrote at the time:

[T]his is just how life is when you’re dealing with trillion-dollar faceless corporations. It’s just one reason why it’s so important that we fund and develop human-scale small tech as an alternative to the strangehold of big tech on our lives.

Then, they went and killed offline web apps.

And then they started using your data to enable third parties to target you (but said it was OK because all the data was on your own device¹).

And then they continued collecting more and more data about us that we were assured would stay on our own devices unless we wanted to share it.

And then they reneged on the last bit with their plans to implement client-side scanning.

And then I recently had a chilling chat with an ex-Apple employee and realised that the stuff we hear is just the tip of the iceberg. (Hey, Tim Cook, if you say “privacy is a human right” but withhold it from your own employees, does that mean you really don’t believe that it’s a human right or that you don’t believe your employees are human beings?)

Anyway, none of this makes me want to write another line of code for or spend another cent on the devices of such a company.

This apple is rotten to its core and I’m done with it.

So, what next?

We build the Small Web.

We support alternative operating systems.

We encourage and support folks like Pine64, Starlabs, Purism, and System76 who are building alternatives.

We envision and create a world where your only alternative is not to be shackled to the whims of fucking trillion-dollar corporations.

Who’s with me?

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Your own device tracking and profiling you is not OK even if the data never leaves your device if it is used not in your interests but in the interests of a third party. Apple News has been doing that for some time now. Does it matter whether Apple, Inc. has the data if some advertiser can still attempt to influence you based on your current emotional and mental state? No. Either your device works in your interests and your interests alone or it is not your device. ↩︎

Comet

mail@ar.al (Aral Balkan) — Sat, 11 Dec 2021 18:06:46 +0000

Comet. Wallpaper illustation by Margo de Weerdt

After several months of work, Comet version 1.0 is now available to install on elementary OS 6.

Comet is one of the artefacts from my three-month-long elementary OS 6 upgrade adventure and I will be writing more about the process of creating an app according to the elementary OS guidelines – what worked, what didn’t, and what can be improved – later.

I am also incorporating the lessons learned, workflows, etc., into Watson, my elementary OS 6 best-practices application template.

Features

Distraction-free interface

Separates your commit message from the comment generated by Git and helps you focus on writing better commit messages.

Spell check

Highlights spelling mistakes and offers suggestions. Automatically uses your current system language (you may need to install the dictionary for your language if it doesn’t come with the system.)

Emoji support

Express yourself with emoji. Heck, it can even count emoji properly (what, you think such things are easy?) 🤪️

Configurable

Configure your first line character limit with common defaults or a custom value according to the conventions used by you or your team.

Low effort

Can update your Git configuration for you. Remembers your previous Git commit message editor so it can restore it if you disable Comet.

I hope you enjoy it!

Install Comet from comet.small-web.org.

Comet is currently not available on the elementary OS AppCenter due to confusion around AppCenter policies regarding what constitutes a sandboxed app and what doesn’t as well as whether or not apps that are not sandboxed should be allowed. Follow this pull request and this issue for the latest news.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Ireland’s two-tier health insurance system

mail@ar.al (Aral Balkan) — Mon, 29 Nov 2021 16:34:17 +0000

Sorry, your browser does not support embedded audio. Download the recording.

An excerpt from an almost ten-minute phone call with a Vhi corporate advisor. (Transcript.)

Question

How do you get pre-existing conditions and waiting periods waived when getting health insurance from Vhi in Ireland?

Answer:

You start working at a multi-billion dollar American Big Tech company like VMWare.

We’re all equal…

In Ireland, we are told that by law everyone has access to the same health insurance plans, including company plans negotiated by Big Tech and other multinationals.

In reality, however, we have a two-tier system where the gentry who work at our Silicon Valley corporate overlords don’t have to put up with petty annoyances like waiting periods or pre-existing conditions while the rest of us hoi polloi do.

So, I can sign up for the same PMI 18 11 plan a VMWare employee can sign up for but with one crucial difference: the VMWare employee doesn’t have pre-existing conditions or waiting periods to worry about as they’re both waived.

…but some are more equal than others.

I first noticed this when, overwhelmed by trying to get my head around the sheer number of cryptic company plans, I started procrastinating by researching which companies they belonged to.

Using search engines, I was only able to uncover two: PayPal and VMWare. But it was on the VMWare healthcare benefits page (archive.org backup)¹ that I came across the following blurb:

Waiting Periods & Pre-Existing Conditions

Pre-existing conditions and waiting periods have been waived which provides you and your eligible dependents with coverage from your enrolment under the Company policy.

So I called Vhi’s group memberships team that deals with corporate accounts to find out how the two of us working on Small Tech for the common good at our tiny two-person not-for-profit could get the same deal as folks working to increase the profits of a multi-billion dollar US Big Tech company.

The answer, in short, was “you can’t.”

With apologies to Orwell, I basically learned that we’re all equal but that some of us are more equal than others.

What follows is a transcript of the relevant parts of that call², the most poignant moment of which was hearing the lovely person I was talking to express her resignation that nothing’s perfect and life’s not fair.

I truly hope, from the bottom of my heart, that we do whatever is in our power to change that. Because it doesn’t have to be that way.

We created this messed-up world.

We can create a better one.

Okay, all right, well that’s good to know… that’s good to know, alright… well I will… thank you for your time, I’ll definitely review the programmes that you have… the various options…

I also have a screenshot ;) ↩︎
I, of course, informed the Vhi representative that I was taping the call for my records, just like they taping it for their purposes. (And I have the full ~10-minute recording with that in it.) ↩︎

The magic one-line ImageMagick 7 AppImage installer

mail@ar.al (Aral Balkan) — Wed, 24 Nov 2021 15:36:09 +0000

To install ImageMagick 7 on any distribution that supports AppImage, copy and paste this one-line script into your favourite shell:

bash -lic "wget -O /tmp/magick https://download.imagemagick.org/ImageMagick/download/binaries/magick && chmod +x /tmp/magick && test \$(wget -qO- https://download.imagemagick.org/ImageMagick/download/binaries/digest.rdf | grep 'rdf:about=\"magick\".*' -A6 | sed -rn 's/.*<digest:sha256>(.*?)<\/digest:sha256>/\1/p') = \$(sha256sum /tmp/magick | sed -r 's/(.*)\s(.*)/\1/') && (sudo mv /tmp/magick /usr/local/bin/ && echo 'ImageMagick 7 successfully installed.') || (rm /tmp/magick && echo 'Installation failed. Security error: message digest verification failed for ImageMagick 7 AppImage binary.')"

What it does

Runs itself in bash, regardless of what interactive shell its being run on.
Downloads the ImageMagick 7 AppImage binary.
Sets the binary’s executable bit.
Downloads the ImageMagick 7 message digests RDF with the SHA-256 hash of the binary.
“Parses” the fucking RDF¹ using regular expressions to extract the SHA-256 hash.
Uses the sha256sum command to calculate the SHA-256 hash of the downloaded binary.
Compares the two hashes. If they match, it installs the binary by moving it to the /usr/local/bin folder. If they don’t match, it removes the downloaded binary. Either way, the script lets you know which action was taken.

Why it exists

Because a lot of folks don’t actually verify the hashes of apps that they download to their machines and that’s because doing so is an overly-convoluted and manual process. If we can automate it with a script, we can guarantee that everyone will be verifying their downloads.

This is not an issue when using app catalogues like the elementary OS AppCenter, but it is an important security hole for app binaries you download from the web.

If you have thoughts or suggestions on how to improve the script, please feel free to share them on this gist.

More image magic

See the post I wrote yesterday titled How to apply a chroma key using ImageMagick, which led me to create this script.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

The only reason I can think of for using an RDF for this purpose is to make sure that as few people as possible actually verify the signature. It means, among other things, that tools like xmllint fail. Please, folks, instead of intellectual purity and complexity maximalism, use the simplest thing that can possibly work. People downstream of you will thank you for it.

What does that mean in practice? Consider having a single file for each hash that just contains the hash. That would be easiest to consume. If you must have a single file for all hashes, consider using JSON. If you can’t do without a certain quota of angular brackets in your life, use XML. RDF? Fucking really? smh ↩︎

How to apply a chroma key using ImageMagick

mail@ar.al (Aral Balkan) — Tue, 23 Nov 2021 17:03:16 +0000

Yesterday, I wrote a short post on the fediverse with an overview of how to take a screenshot of an app with a context menu showing in elementary OS, while keeping its alpha channel and drop shadow using Krita.

Today, I decided that since I want localised screenshots for Comet in the elementary OS AppCenter, I have to automate the screenshotting process (as I have 5 screenshots per language and, even with the three languages Comet will launch with – English, Turkish, and Dutch – that still means 15 screenshots).

Automating regular screenshots is easy.¹ What’s harder is to automate the screenshot with the context menu I mentioned earlier as that requires the use of a chroma key.

To automate that, we need to use trusty old ImageMagick.

A little image magic

If you don’t already have it, you can easily install it in elementary OS using apt:

sudo apt install imagemagick

However, this will give you the legacy version of ImageMagick (version 6.x).

The easiest way to install the latest version (7.x), is to use the ImageMagick 7 AppImage.

The following one-line ImageMagick 7 AppImage installation script will do that for you while automatically verifying the message digest on any Linux distribution and on any shell:

bash -lic "wget -O /tmp/magick https://download.imagemagick.org/ImageMagick/download/binaries/magick && chmod +x /tmp/magick && test \$(wget -qO- https://download.imagemagick.org/ImageMagick/download/binaries/digest.rdf | grep 'rdf:about=\"magick\".*' -A6 | sed -rn 's/.*<digest:sha256>(.*?)<\/digest:sha256>/\1/p') = \$(sha256sum /tmp/magick | sed -r 's/(.*)\s(.*)/\1/') && (sudo mv /tmp/magick /usr/local/bin/ && echo 'ImageMagick 7 successfully installed.') || (rm /tmp/magick && echo 'Installation failed. Security error: message digest verification failed for ImageMagick 7 AppImage binary.')"

The commands in this post use ImageMagick 7 syntax but you can easily convert them to legacy ImageMagick 6 simply by removing the magick command at the start. They will then function identically, using the older convert command that’s part of ImageMagick 6.

Would it be alright by you if I degreenify you?

For the purposes of this post, let’s assume we already have the following screenshot of the app in a file called original.png and we want to apply a chroma key to remove the green background while keeping the lovely soft drop shadow.

Unlike a green screen in live action video, our background is a single, uniform colour: pure green (#00ff00). This will simplify things a lot.

First off, there’s quite a bit of solid green around the area we want to remove, even before we get to the semi-transparent drop shadow that we want to preserve and, of course, to the body of the window itself. So let’s start by trimming off the excess green:

magick convert original.png -trim +repage trimmed.png

This gives us a smaller canvas to work with, which should speed up future processing also.

Next, we run a command to remove the background while keeping the transparency:

magick convert trimmed.png -channel alpha -fx "1.0*b - 1.0*g + 1.0" image-with-alpha-and-colour-spill.png

We now have a new image with alpha transparency but the semi-transparent areas have a green tint to them. This is known as colour spill.

To remove the colour spill while maintaining the transparency, we have to do two things.

First, we extract the alpha channel from this new image:

magick convert image-with-alpha-and-colour-spill.png -alpha extract alpha-mask.png

That gives us the following alpha mask:

Now that we have a clean alpha mask and, given we know the background colour, we can use a clever little calculation² to remove the tint³:

magick convert original.png alpha-mask.png -alpha Off -fx "v==0 ? 0 : u/v - #00ff00/v + #00ff00" alpha-mask.png -compose Copy_Opacity -composite final.png

And that gives us our final image, with a beautiful alpha-channel drop-shadow without any colour spill.

I hope this helps you should you find yourself needing to do something similar.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

I just launch Comet and the elementary OS Screenshot app from the command-line and ask the latter to take a screenshot of the active window with a one-second delay. Then, I use xdotool to ensure that Comet is active and has focus before it is captured and close the window after a longer delay.

It looks something like this:

#!/usr/bin/env bash

function screenshot_and_close_comet {
  flatpak run io.elementary.screenshot --window --delay=1 &
  xdotool search --onlyvisible --class comet windowactivate windowfocus
  sleep 2
  xdotool search --onlyvisible --class comet windowclose
}

# Ensure we have a current build.
task/build

# Welcome Screen

# English
build/org.small_tech.comet &
screenshot_and_close_comet

# Turkish
LANGUAGE=tr_TR.utf8 build/org.small_tech.comet &
screenshot_and_close_comet

# Dutch
LANGUAGE=nl_NL.utf8 build/org.small_tech.comet &
screenshot_and_close_comet

# And so on…

↩︎

I didn’t have to go into the math this time around but I remember diving pretty deep into alpha blending/compositing about a decade ago while working on one of my iPhone apps.

If you want to understand the equation, start by reading up on the FX Special Effects Image Operator. The equation is used for each pixel in the images; u is the colour value from the first image and v is the colour value from the second image. What the equation is doing is calculating how much green is present in each pixel and removing it. ↩︎
Since I know that background colour is pure green, I hardcoded its hexadecimal value in the example to simplify the equation.

If you don’t know the colour off-hand, you can grab it from the image itself – e.g., from its top-left corner – by substituting u.p{0,0} in place of the hexadecimal colour value in the equation. ↩︎

My three-month-long elementary OS 6 upgrade adventure in three parts. (Part 1: Catts)

mail@ar.al (Aral Balkan) — Mon, 08 Nov 2021 15:20:41 +0000

I just upgraded my operating system from elementary OS 5 to 6.

It only took me three months.

The journey begins

elementary OS 6 (Odin) was officially released on August 10th, 2021.

Since there is no auto-update tool, you have to back everything up, wipe your computer, and install the new operating system from scratch. It took me a few days to build up the courage to do so. On August 14th, after publicly documenting some of the things I was backing up, I took the plunge.

A bumpy start

I hit my first hurdle almost immediately when I realised that I’d been using a custom task switcher called gala-alt-tab-plus that hadn’t been updated for Odin. Speaking with Mark, the maintainer, it didn’t look like it was going to be a trivial update either.

So I decided to bite the nail on the head and make an Odin-only version called Calmer Alt-Tab Task Switcher (Catts).

’Mentary songs for ’mentary Catts

I wanted Catts to eventually become the default task manager in elementary OS, not languish as an option for a handful of technically-savvy people for over seven years, like its precursor.

One way of achieving that might have been to take part in endless bikeshedding in GitHub issues, etc. That’s really not what I’m built for. So, instead, I knew what I needed to do was to show not tell:

Make it easy to install (and uninstall) so anyone, including the elementary team, could try it out easily.

This way, even if it wasn’t immediately accepted, people could still install and use it and, if enough people did, it might be included as the default later on.
Make the design case for why it is better than the default task switcher and do so visually in a manner that is difficult to ignore.
Make it easy to adopt. Don’t create obstacles to it being incorporated into the OS (so, for example, adhere to coding conventions).

Fast forward a couple of days and I had a rudimentary version working and ready to test on elementary OS 6.

And this is how I announced it in the readme, to cover the points above:

Why Catts?

A quick visual demonstration should suffice (if it doesn’t, please read on for the various reasons outlined below.)

Elementary OS stock task switcher:

active app icons during the task switching process.">

Catts:

a row of icons and an indicator shows which app will be active when you release the alt key. The window title of the selected app is also shown. Nothing animates.">

In other words, because:

alt + tab is a hidden, shortcut gesture for quickly switching between the various windows you have open.

There is already a graphically-heavy, slower alternative with the “Show Desktop” (⌘ + ↓) gesture that gives you an overview of the windows within a workspace with window previews that >can be used if differentiating windows based on their contents is important.

In elementary OS, however, the task switcher:

Overloads the dock (the dock is transformed to include icons of windows and the icons there used to indicate which window you’re switching to). This breaks the physicality of the dock and overloads its >meaning. That said, due to the amount of other animation going on, willing myself to concentrate on the dock is the only way I can use it at all.

Has excessive motion (animates windows backwards or forwards while dimming them in/out every time you press alt + tab). Imagine that happening with maximized or half-screen windows >on a 24" monitor. I don’t normally have issues with motion and it makes me feel seasick after a few uses.

Gets stuck. Sometimes it will just get stuck in a state where no window is selected. Pressing alt + tab again gets you out of it.

Is one-way (shift + alt + tab) doesn’t do anything.

Basically, the task switcher in elementary OS is unusable.

This one, despite its limitations, at least fixes the above issues.

Catts:

Is calm. It does not animate my windows. I don’t want cognitive complexity when I’m fast switching between apps. I want to select the app I want to switch to and switch to it. That’s it.

Uses icons. There is very little cognitive load to recognising an icon. There’s a reason we use icons of applications in menus, etc., instead of tiny thumbnails of them. The same principles apply here.

Enables you to tell apart different windows of the same app (simply, by displaying the window title in the switcher alongside the icon).

Uses the system colour scheme. Love Dark Mode? Catts does too.

I feel elementary OS would be far more usable in general – not to mention more familiar for folks just coming over from macOS or Windows – if we were to replace the default task switcher with Catts.

In fact, I’ve opened a feature request on the elementary OS Gala to do just that. If you agree, please give it a thumbs up to show your support.

The happy ending?

To cut a long story short, Catts was brought into elementary OS as the default Window Switcher last week and should be in November’s operating system updates.

I want to take this opportunity to thank David M. Hewitt for taking the initiative and his hard work in making this happen. I also want to thank Cassidy and Dani for their support and input throughout the process.

And yet…

So is it time to put Catts to rest?

Not just yet.

While I’m very happy to see a version of Catts incorporated into elementary OS, there is one major outstanding issue. The version that is being added to elementary OS is inaccessible. I was able to cobble together some basic accessibility for the widget using whatever scant documentation and code I could find in the GNOME/Gtk/Vala/elementary OS world and the current release of Catts does have that implemented.

However, the version being merged into the OS will be entirely inaccessible.

So, if you need an (even if imperfectly) accessible task switcher, please still keep using Catts until the elementary folks manage to implement it properly in Gala.

I’ve opened an issue to track this here: https://github.com/elementary/gala/issues/1301

I’ve also opened a discussion in the elementary OS Human Interface Guidelines asking for accessibility/inclusivity to be adopted as a core tenet of the operating system and for both the operating system itself and the developer documentation to be updated accordingly. Please feel free to share your thoughts on that here: https://github.com/elementary/hig/discussions/51

Once the task switcher in Vala is accessible, I will most likely retire Catts.

But that’s not the end of the story

Three months? Surely, Catts didn’t take three months to build (it didn’t). Laura and I took about a month off after two years of the pandemic to go see our parents right after we got our vaccinations. It took me another few weeks¹ to build out two other projects as the task switcher wasn’t the only thing that had issues.

You see, I had this little git commit message editor called Gnomit I wrote that I loved using everyday. With elementary OS 6, it stopped displaying correctly in dark mode. So, of course, this bugged me (because, hello, welcome to my life) so I decided to rewrite it specifically for elementary OS and to do so “properly” in Vala² to get a feel for the recommended development process.

As I worked through the developer documentation, I saw there was a huge amount of boilerplate that needs to be implemented for every elementary OS app in order to adhere to the elementary OS Human Interface Guidelines.

So I decided to create a best-practices application template for elementary OS called Watson and to use that to create my new git commit message editor, Comet.³

I’ll cover Watson and Comet in Parts 2 and 3.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

And we were in the process of finding a new place to live and moving to a new city throughout all this so it did all take longer than it otherwise could have. But that’s just life, I guess. ↩︎
I wrote the original in GJS. ↩︎
This, kids, is what we call “yak shaving.” ↩︎

How to count Unicode glyphs in Vala using Gtk

mail@ar.al (Aral Balkan) — Sun, 31 Oct 2021 21:26:17 +0000

Want to try something really scary this Halloween? Try counting the characters in a string in Vala.

For example, how many characters do you think there are in the following string?

var zombie = "🧟‍♀️️";

One?

You’re right!

So how would you go about getting this result in Vala? Use zombie.length, did I hear you say?

Oh, you poor, dear, naïve, sweet thing…

"🧟‍♀️️".length // 13

What’s that? What about char_count ()?

"🧟‍♀️️".char_count (); // 4

What do you think this is? Some toy language and standard library like Swift where you can just do zombie.characters.count and get the right answer (1) straight away?

This is Vala and Gtk, motherfucker, get ready for some pain!

So which is correct? 13, 4, or 1?

Yes.

(They’re all “correct” based on what question you happen to be asking and what you need to do with the answer.)

If we break it down, a female zombie (🧟‍♀️️) is, in UTF-8:

Count	Type	Details
1	glyph	as a typographer would call it
1	grapheme cluster	as Unicode calls it
4	code points	🧟️️ (`F09FA79F`) + Zero-width Joiner (`E2808D`) + ♀ (`E29980`) + Variation Selector 16 (`EFB88F`)
13	bytes	`F0 9F A7 9F E2 80 8D E2 99 80 EF B8 8F`

How to count (ha, ha, ha) in Vala

First off, let me tell you, I still don’t know what the equivalent of the Swift one-liner zombie.characters.count is in Vala/GLib but, after several days of banging my head against a wall, I figured out a way to count glyphs properly using a Gtk.TextIter with a Gtk.TextBuffer¹:

public int glyph_count (string text) {
    var text_buffer = new Gtk.TextBuffer (null);
    text_buffer.set_text (text);

    Gtk.TextIter glyph_count_iter;
    text_buffer.get_start_iter (out glyph_count_iter);

    var glyphs = 0;
    while (!glyph_count_iter.is_end ()) {
        glyphs++;
        glyph_count_iter.forward_cursor_position ();
    }

    return glyphs;
}

Using this function, you can now do:

glyph_count ("🧟‍♀️️"); // 1

And get the result approved by most human beings who think like human beings and not like computers.

Iterators and cursors are your friend

When working with Unicode in Vala/Gtk, iterators and cursors are your friend so remember to always work with cursor positions and not characters.

To add to the mindfuck, you don’t always get a cursor position when you ask for one. Sometimes you get a “character” offset instead.

For example, the cursor_position property on a Gtk.TextBuffer instance isn’t a cursor position, it’s a “character” offset. (Because they hate you and your little dog too.) So you cannot compare the cursor position from a text buffer directly with the count you get from the glyph_count () function, above. The former uses Vala “characters” while the latter deals in glyphs (what humans call characters).

Imagine the following is a string in a Gtk.TextView and the cursor is at the end of the string:

123🧟‍♀️️5│

In that situation:

glyph_count (text_buffer.text) // 5
text_buffer.cursor_position    // 9

So we can’t use the cursor_position property directly when we’re dealing with glyphs.

What we can do is to create and use an iterator based on the “character” offset stored in the cursor_position property:

var cursor_position = text_buffer.cursor_position;

Gtk.TextIter cursor_position_iter;
message_view_buffer.get_iter_at_offset (out cursor_position_iter, cursor_position);

Then, using its compare () method we can find out where in the string it is.

For example, to check if the cursor is within the first line of text as a person types in a Gtk.TextView:

var text_view = new Gtk.TextView ();
var text_buffer = new Gtk.TextBuffer (null);
text_buffer.set_text ("First line\nSecond line");
text_view.set_buffer (text_buffer);

text_view.end_user_action.connect (() => {
  Gtk.TextIter end_of_first_line_iter;
  text_view.get_iter_at_line (out end_of_first_line_iter, 0);
  end_of_first_line_iter.forward_to_line_end ();

  var cursor_position = text_buffer.cursor_position;
  Gtk.TextIter cursor_position_iter;
  message_view_buffer.get_iter_at_offset (out cursor_position_iter, cursor_position);

  if (cursor_position_iter.compare(end_of_first_line_iter) <= 0) {
    print ("Cursor is on the first line!\n");
  }
});

Finally, if you want to go to a specific position within a string, get the start iterator and use the forward_cursor_positions () method to move to your desired offset.

Hopefully, these tips will save you some time should you need to work with Unicode glyphs in Vala/Gtk².

Hope the dents in my head save you from ones in yours.

Oh, and happy Halloween!

If you know of an easier way, please do let me know. ↩︎
Perhaps while making apps for elementary OS (hint, hint!) ↩︎

How to change the colour of the underline in gspell

mail@ar.al (Aral Balkan) — Tue, 19 Oct 2021 18:00:53 +0100

gspell is GNOME’s spell-checking library.

When it discovers misspelld wrds, it underlines them in dark red.

(Only, it can’t do wavy underlines so it does solid ones.)

To change the colour, say to Dracula pink for dark mode, you set gspell_text_view.underline_color = "#ff79c6";

Haha, just kidding, you can’t.

There’s a 5-year issue open to ask for an option to change the colour.

Being the Henry Ford of library makers, GNOME likely sees this lack of customisability as a feature not a bug (it’s available in any colour as long as it’s dark red).

That said, it’s both a usability and accessibility issue, so here’s a workaround for wrestling control back from the library.

Workaround

The way I ended up managing to change the colour of the underline in gspell was to apply a tag to the entire text every time the text buffer changes or someone pastes into it.

The tag sets the underline colour and I make sure the priority of the tag is set to the highest in the buffer’s tag table, thereby making it override the colour that gspell uses.

Here are the relevant bits from my app, with a pink underline.

private string PINK_UNDERLINE = "pink-underline";

private Gtk.TextView text_view;
private Gtk.TextBuffer text_buffer;
private Gspell.TextView g_spell_text_view;

private Gtk.TextTag pink_underline_tag;

construct {
    text_view = new Gtk.TextView ();
    text_buffer = text_view.get_buffer ();

    g_spell_text_view =
        Gspell.TextView.get_from_gtk_text_view (text_view);

    g_spell_text_view.basic_setup ();

    Gtk.TextTag pink_underline_tag =
        new Gtk.TextTag (PINK_UNDERLINE);

    var pink = Gdk.RGBA ();
    pink.parse ("#ff79c6");

    pink_underline_tag.underline_rgba = pink;
    text_buffer.tag_table.add (pink_underline_tag);

    text_buffer.changed.connect (override_underline_colour);
    text_buffer.paste_done.connect (override_underline_colour);
}

// …

// Workaround for the underline colour
// not being configurable in gspell.
private void override_underline_colour () {
    Gtk.TextIter text_start;
    Gtk.TextIter text_end;
    text_buffer.get_start_iter (out text_start);
    text_buffer.get_end_iter (out text_end);

    text_buffer.remove_tag_by_name (
        PINK_UNDERLINE,
        text_start,
        text_end
    );

    text_buffer.apply_tag (
        pink_underline_tag,
        text_start,
        text_end
    );

    pink_underline_tag.set_priority (
        text_buffer.get_tag_table ().get_size () - 1
    );
}

Also, as a little bonus, if you’re using gspell in your own elementary OS apps, don’t forget that you have to specify it as a dependency both for Meson/Ninja and for Flatpak separately.

In your meson.build file:

executable(
    # …
    dependencies: [
      # …
      dependency('gspell-1', version: '>=1.8.0'),
    ],
    # …
)

For Flatpak declare it as a module in your Flatpak manifest. Below is the YAML version I’m using:

modules:
  - name: gspell-1
    config-opts:
      - '--disable-static'
      - '--disable-gtk-doc'
    sources:
      - type: 'archive'
        url: 'https://download.gnome.org/sources/gspell/1.8/gspell-1.8.3.tar.xz'
        sha256: '5ae514dd0216be069176accf6d0049d6a01cfa6a50df4bc06be85f7080b62de8'
    cleanup:
      - '/bin'

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

How to disable Gtk-Message warnings in your app

mail@ar.al (Aral Balkan) — Fri, 15 Oct 2021 21:11:56 +0100

So every elementary OS app at the moment, if you start it from Terminal, welcomes you with a variation of the following salutation:

Gtk-Message: … : Failed to load module "canberra-gtk-module"
Gtk-Message: … : Failed to load module "canberra-gtk-module"

Isn’t that friendly?

The journey begins…

I initially thought this was a long-standing issue with Flatpak (and I was a shady bitch about it too) but, thanks to a detailed write-up by Simon McVittie, we now know that it is more likely due to a configuration mismatch between the host OS (in this case elementary OS 6) and the runtime in your Flatpak.¹

So just turn off those warnings in your app if they bother you so much, I hear you say.

Well, reader, it’s not that simple.

After hitting my head on a brick wall (also known as GNOME/GTK documentation) for a few hours to no avail, I opened an issue on the GTK tracker, using it like Discourse as if I’d been raised in a barn like some sort of animal.

Thanks to Emmanuele Bassi’s hints in his response, I was able to finally find a way to disable the Gtk-Message warnings.

A happy ending

All you have to do², see, is create a custom log writer function.

Here’s an excerpt of the relevant bits from the app I’m building in case they help you too:

public class Application : Gtk.Application {
    public static string flatpak_id;
    public static bool is_running_as_flatpak;

    public Application () {

    public static int main (string[] commandline_arguments) {
        flatpak_id = Environment.get_variable ("FLATPAK_ID");
        is_running_as_flatpak = flatpak_id != null;

        if (is_running_as_flatpak) {
            Log.set_writer_func (log_writer);
        }

        return new Application ().run (commandline_arguments);
    }

    private static LogWriterOutput log_writer (LogLevelFlags log_level, [CCode (array_length_type = "gsize")] LogField[] fields) {
        return log_level == LogLevelFlags.LEVEL_MESSAGE ? LogWriterOutput.HANDLED : LogWriterOutput.UNHANDLED;
    }
}

Epilogue

Hopefully, knowing the folks at elementary OS, they’ll get to the bottom of the issue so we won’t need this workaround in the near future.

In the meanwhile, I plan on adding this to Watson – the best-practices application template for elementary OS 6 (Odin) that I’m working on, so that any apps created based on it will silence these warnings by default.

There is a lot to admire about Gtk+, really there is³, but sometimes it does feel like you have to construct a Rube Goldberg machine just to do the simplest things.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Also, according to Simon, as apps are ported to GTK4, the problem should go away as “GTK 4 no longer supports loading arbitrary modules specified in GTK_MODULES”. ↩︎
Sarcasm. ↩︎
Not sarcasm. ↩︎

Implementing dark mode in a handful of lines of CSS with CSS filters

mail@ar.al (Aral Balkan) — Tue, 24 Aug 2021 20:37:50 +0100

I finally got round to implementing dark mode for this site (the cobbler’s children have no shoes and all that…)

Here’s all the CSS I had to add:

@media (prefers-color-scheme: dark) {
  /* Invert all elements on the body while attempting to not alter the hue substantially. */
  body {
    filter: invert(100%) hue-rotate(180deg);
  }

  /* Workarounds and optical adjustments. */

  /* Firefox workaround: Set the background colour for the html
     element separately because, unlike other browsers, Firefox
     doesn’t apply the filter to the root element’s background. */
  html {
    background-color: #111;
  }

  /* Do not invert media (revert the invert). */
  img, video, iframe {
    filter: invert(100%) hue-rotate(180deg);
  }

  /* Improve contrast on icons. */
  .icon {
    filter: invert(15%) hue-rotate(180deg);
  }

  /* Re-enable code block backgrounds. */
  pre {
    filter: invert(6%);
  }

  /* Improve contrast on list item markers. */
  li::marker {
    color: #666;
  }
}

How it works

The real magic happens in the first rule:

body {
  filter: invert(100%) hue-rotate(180deg);
}

I run an invert filter alongside a hue rotate filter on the body element. That inverts the colours on the page while attempting to keep the hues as similar as possible.

You might be wondering why I didn’t apply this rule to the root html element. Thing is, originally, I did¹. However, unlike other browsers, Firefox does not apply the invert filter to the background of the root element.

So, instead, I have to set the HTML element’s background manually, in the section on workarounds and optical adjustments:

html {
  background-color: #111;
}

(Since the background colour of the page is #eee, I set the dark mode background to its inverse, #111, which matches what the invert filter does for the body.)

Next, I re-inverting media like images, videos, and iframes so they appear as originally intended:

img, video, iframe {
  filter: invert(100%) hue-rotate(180deg);
}

(As you can see, I’m simply running the same rule again to revert the invert.)

Finally, the rest of the rules are little opticals tweaks, to improve contrast on a few elements like the icons in the header and the backgrounds of code block that don’t fare well with the inversion filter:

/* Improve contrast on icons. */
.icon {
  filter: invert(15%) hue-rotate(180deg);
}

/* Re-enable code block backgrounds. */
pre {
  filter: invert(6%);
}

/* Improve contrast on list item markers. */
li::marker {
  color: #666;
}

If you’re looking to implement dark mode on your web site, you don’t use shadows/shades to denote hierarchy, and you don’t have a huge amount of time on your hands to craft a separate stylesheet, CSS filters are your friend.

Enjoy!

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Thanks to Kieran Barker, varx, Silmathoron, and others on the fediverse for pointing out the issue with Firefox and suggesting potential fixes. ↩︎

Omnibus GitLab Let’s Encrypt renewal error and fix

mail@ar.al (Aral Balkan) — Tue, 24 Aug 2021 19:39:19 +0100

I moved source.small-tech.org, our self-hosted GitLab instance, to Eclips.is a few months ago and noticed today that the Let’s Encrypt certificate had failed to renew.

When I looked on the server, the outdated Let’s Encrypt certificates were in /etc/gitlab/ssl as expected but, when I looked in the GitLab configuration file (/etc/gitlab/gitlab.rb), the Let’s Encrypt integration section was entirely commented out.

Having absolutely no memory of how I created the original certificates, I tried enabling those settings and reconfiguring GitLab (sudo gitlab-ctl reconfigure) and got the following errors:

Error executing action `run` on resource 'ruby_block[create certificate for source.small-tech.org]'

  RuntimeError
  ------------
  [source.small-tech.org] Validation failed, unable to request certificate

  Cookbook Trace:
  ---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'

Error executing action `create` on resource 'acme_certificate[staging]'

  RuntimeError
  ------------
  ruby_block[create certificate for source.small-tech.org] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [source.small-tech.org] Validation failed, unable to request certificate

  Cookbook Trace:
  ---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'

Error executing action `create` on resource 'letsencrypt_certificate[source.small-tech.org]'

    RuntimeError
    ------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 26) had an error: RuntimeError: ruby_block[create certificate for source.small-tech.org] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [source.small-tech.org] Validation failed, unable to request certificate

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'

There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[source.small-tech.org] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 26) had an error: RuntimeError: ruby_block[create certificate for source.small-tech.org] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [source.small-tech.org] Validation failed, unable to request certificate

The fix

Going by what worked for someone else, I did the following:

Deleted the /etc/gitlab/ssl directory.
Enabled the following two properties in /etc/gitlab/gitlab.rb under the Let’s Encrypt integration section:
```
letsencrypt['enable'] = true
letsencrypt['auto_renew'] = true
```
Reconfigured GitLab:
```
sudo gitlab-ctl reconfigure
```

And that seemed to fix things.

Documenting it here in hopes it might help somebody else (e.g., future me) ;)

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Key Mapper: a visual tool for remapping keys (and more) on Linux

mail@ar.al (Aral Balkan) — Mon, 16 Aug 2021 11:07:04 +0100

Mapping keys has never been easier in Linux.

I’m a bit particular when it comes to typing.

Specifically, I like to use good typographical habits whenever possible.

And, I use an external keyboard with a physical ANSI US Keyboard layout set to UK Macintosh layout on my Linux laptop¹. So, I need to reclaim my backtick/tilde key.

Previously, I used a combination of xmodmap, xev, and a .desktop in autostart to handle the backtick/tilde issue – which was about as fun as it sounds to figure out and implement – and the standard Alt Graph modifiers for things like typographical quotes. And while the AltGr modifiers are great and all, they weren’t entirely ergonomic.

When I was on a Mac, I used an awesome tool called Karabiner-Elements to handle all this with a minimum of fuss.

Surely it shouldn’t be this hard to remap your keys in Linux in 2021! Well, thanks to an app I just discovered called (wait for it) Key Mapper, it’s not.

Mapping keys for fun and pedantry

Using Key Mapper is fairly straightforward.

Once you’ve installed² and launched it, you’re greeted with a simple interface where you can select the device³ you want to create mappings for.

From there, you can create a new set of mappings, give them a name (confusingly in a separate text box called rename) and choose whether they should auto load.⁴

Finally, you create the mappings themselves by first pressing the key or key combination you want to map and then entering the codes of the key(s) you want them to map to.

Note that you are mapping keys to keys, not keys to characters. The distinction is important. So, for example, to map you’re mapping AltGr + [ to AltGr + Shift + v, not AltGr + [ to ‘.

Or, more precisely, using the key names used by Key Mapper, you’re mapping ISO Level3 Shift + bracketleft to ISO_Level_3_Shift + Shift_R + v.

Niggles

Key Mapper is great but it’s not perfect. Here are a few niggles I’ve noticed:

There is a noticable delay of about a second when a mapping fires. This could get annoying fast (and wasn’t an issue with the manual way I was doing mappings earlier).
The mappings didn’t survive suspend. It remains to be seen if that was a one-off or if it keeps happening.
There doesn’t seem to be a way to use a preset from one device on another. Case in point: I wanted to use the same mappings on my external keyboard and on the laptop keyboard so I had to create them twice. Not a huge problem for a handful of mappings but it could be more of a headache for larger lists.

A great start

Having a graphical key mapper in Linux is great. Here’s hoping that some of the niggles are not facts of life and can be improved upon with time.

Worst case scenario, I’ll have to hunt down how I was doing things in the past and reimplement that on my new elementary OS 6 installation.⁵

Hey, don’t judge me! ↩︎
I just grabbed the .deb file from their releases page. ↩︎
You’re not limited to keyboards; you can remap keys on mice and gamepads. Key Mapper also states that it works on X11 and Wayland, although I’ve only tested it under X11. And, though I haven’t used them yet, you can also use macros. ↩︎
In my limited testing so far, auto-loading seems to survive sleep for me but it did fail once after a suspend. Getting mappings to stick was also an issue with my homebrew system previously and I don’t remember how I finally managed to make it work reliably at both sign in and after sleep/suspend. I may have to hunt down my backup configuration files or re-read my blog posts that I linked above if this becomes a headache. ↩︎
I plan on writing a comprehensive post on elementary OS once I’ve got my laptop back to running the way I want it to under version 6/Odin. ↩︎

Apple is trying to redefine what it means to violate your privacy. We must not let it.

mail@ar.al (Aral Balkan) — Sun, 08 Aug 2021 16:36:02 +0100

Screeching for freedom. Illustration by Jérémie Fontana.

I was there when Steve Jobs unveiled the original iPhone (ironically, I gave a talk on Flash there right after his keynote). I started making apps for iOS from day one, learned Swift while it was still in alpha and updated my code with every new version. My work has been featured by Apple on its App Store and our app, Better Blocker, has maintained glowing reviews and a 4.7 star rating there for years.

If Apple goes ahead with its plans to have your devices violate your trust and work against your interests, I will not write another line of code for their platforms ever again.

Redefining what privacy means

Privacy is the fundamental human right to choose what you share with others and what you keep to yourself.

What Apple is doing with its plans to scan content on your own devices and notify third parties is an attempt to redefine what it means to violate your privacy.

It doesn’t matter where your privacy gets violated (on your own device or in the cloud). It matters that your privacy is being violated. The violation of privacy isn’t in the scanning of the content, it is in the notification of a third-party based on the scan.

Even if a third party isn’t notified, if the insight gleaned from the scan is used in the interests of a third party instead of in your interests and according to your wishes, this is also a violation of your privacy.¹

Violation of privacy by proxy is still violation of privacy.

Apple: are we the baddies?

Without privacy we don’t have civil liberties. If we make public the default, then anything we want to keep private has an association of guilt attached.

If you were to ask me who is a bigger threat to our human rights today, Google or Apple with these new plans, it’s Apple, without a doubt.

If you know anything about me or my work in the past decade you will understand exactly what that means.

I’m as shocked as you are.

I do not say this lightly or take any pleasure whatsoever in saying it.

I cannot stress enough how important it is that this line does not get crossed.

Slippery slope? Try cliff edge.

If Apple crosses this line, there’s no going back. It will set a dire precedent and shift the borders of personhood irrevocably.

Google will follow suit.

Soon any system that doesn’t implement on-device scanning/reporting will be deemed illegal (or carry an association of guilt).

It won’t be limited to phones and tablets. Your Mac is next. General computing as we know it could cease to exist.

It won’t be limited to the initial content either. Erdoğan, Putin, Bolsanaro and their brethren will decide the scope.

Just last week Hungary ordered shops to cover up LGBT-themed children’s books. What’s betting Orban adores Apple’s new plans? What happens when Márk receives a racy photo from Mátyás? (Oh, and did I mention, in Hungary they will eventually make Apple set the reporting age on iMessage photos so it applies to all under 18s.)

Mark my words: Apple’s plans to have teenagers’ phones snitch to their parents will result in honour killings in places like Turkey. Women will get murdered because of this. Tim Cook, you will have the blood of queer people on your hands. I truly hope you reconsider starting down this dangerous path.

This is not a slippery slope. It’s a cliff edge.

Beyond sorry

Apple backing down on this won’t be enough.² To maintain trust, it must make a contractual commitment that the data on your device is yours and yours alone and that Apple’s algorithms can only analyse it as permitted by you and only in your interests.

This is a greater struggle to protect personhood in the digital network age.

Today, we extend ourselves with technology. Not owning and controlling these aspects of ourselves is a violation of our personhood.

Protecting personhood in the digital network age

When I wrote The Universal Declaration of Cyborg Rights, I wanted to get people thinking about the kind of constitutional protections we would need to protect personhood in the digital network age.

I didn’t think we’d need them so soon. I thought we had more time.

But where do you even begin to get legislators to understand and act on such a thing in the here and now?

It’s not like I haven’t tried.

The screeching voices of the minority

Almost 5,000 people – security and privacy experts, cryptographers, researchers, professors, legal experts and Apple customers – have signed an open letter asking Apple to halt deployment of its content monitoring technology immediately and to issue a statement reaffirming their commitment to end-to-end encryption and privacy as a fundamental human right.

Although Apple hasn’t responded to us yet, it did circulate a memo internally calling us “the screeching voices of the minority.”

I urge you to join us.

We have a lot to screech about.

This is what Apple News has been doing for some time). Apple profiles you on your own device and enables advertisers to target you and attempt to manipulate your behaviour and emotional state for profit. Apple says that this does not violate your privacy. It absolutely violates your privacy by any meaningful definition of what privacy is.

Note that this is also what the Brave browser does. Does Brave violate your privacy? 100%! ↩︎
Whether or not Apple bows down to pressure on this and reverses course, it offers you precious insight into how they think.

It’s their phone, not yours.

Tim Apple is your daddy and as long as you live under his roof, you live under his rules. And he’s just made it clear he can enter your room whenever he likes and search your drawers.

It might be time to think about moving out.

Problem is, where do you go? Do you move in with creepy uncle Google next door? You’re screwed either way.

Oh and remember that your banking app only works on iOS and Android…

I’m seeing people say “just don’t use an iPhone.” It’s not that simple when everyday things like financial apps with two-factor authentication are locked into the two main platforms.

We need legislation to ensure critical services use open standards so you can use your PinePhone to buy lunch in the future.

It’s shocking how easily some folks jump to the technological equivalent of “just go live in a cave” as the solution. No, that’s not an acceptable alternative. We deserve to partake in modern life without sacrificing our human rights.

It’s victim blaming to tell everyday people they’re at fault for using one of the two main tech platforms instead of an (as of yet inaccessible) alternative. I have two PinePhones and my room overflows with open hardware. No, I don’t blame you for using an iPhone or an Android device. You’re the victim here.

Blame the actual culprits: clueless legislators/policymakers who allow these monopolies to continue and fail to protect our human rights. Blame Big Tech and those who enable it. ↩︎

Fish shell

mail@ar.al (Aral Balkan) — Sun, 25 Jul 2021 13:27:53 +0100

Fishing for a new shell? (Sorry.)

Beautiful defaults

Good design isn’t about removing the seams altogether¹; it’s about having beautiful defaults and layering the seams. And this is something that Fish shell absolutely nails.

To be fair, I have no idea why I didn’t give it a proper shot until yesterday. I guess because Zsh, or more precisely Oh My Zsh, has been perfectly adequate.

It’s not until you experience the intelligent auto-suggestions and completions in Fish, however, that you truly realise what you’ve been missing. And that’s before you learn just how easy it is to extend as nearly everything is a function.

Getting started

First off, download and install Fish shell.

I’m running elementary OS on my Star Labs notebook so I followed the Ubuntu instructions:

sudo apt-add-repository ppa:fish-shell/release-3
sudo apt update
sudo apt install fish

At this point, you can simply type fish in your current shell to open a Fish shell instance.

Out of the box, you get the excellent auto-suggest/completion.

I recommend you take a moment to at least skim through the comprehensive tutorial, documentation, and frequently-asked questions.

Sprucing it up

While Fish shell is perfectly usable without customisations, there are a few things I’ve gotten used to having in my shells that I wouldn’t want to do without. These range from the practical to the merely cosmetic.

The first thing I’d highly recommend you do is install Fisher.

Fisher

Fisher is a plugin manager for Fish. It does one thing and does it well.

You can install it by running the following one-liner (this is the script that it will run on your machine):

curl -sL https://git.io/fisher | source && fisher install jorgebucaran/fisher

From there, you can browse the list of Fisher plugins on awsm.fish.

Following is a brief list of the ones I installed.

Tide

Tide is a modern prompt manager for Fish.

After playing with it a little, I forked it to customise the Git prompt (I like to have my Git prompt tell me what the status is in words instead of cryptic symbols.)

A word is worth a thousand symbols. Or something.

Also, note that in the screenshot you can see a count of the files that fall into different git statuses. Normally, this would slow a prompt down but Tide is about as snappy as can be because it is asynchronous so you are never blocked from entering a command while waiting for your prompt to render.

Install it using Fisher:

fisher install IlanCosman/tide

Gills

Fish shell gives you gills (I need help, I know.)

Creating a plugin is so simple in Fish that I’ve already made a very simple one even though I only just started using the shell yesterday.

The plugin, which I call Gills, is about as simple as it gets: it just adds an empty line after your prompt and before the output of your command to balance the whitespace around them so you can more easily separate your prompts from your command output while skimming your terminal’s scrollback buffer.

It’s called Gills because it gives your output room to breathe. Get it? (Did I mention I was sorry?)

Technically, like basically everything else in Fish shell, it’s just a function. In this case, one that handles Fish shell’s fish_postexec event.

(It also handles a couple of special cases like cd, pushd, and popd that do not produce any output. In those cases, it doesn’t add the additional empty line. If you find any other edge cases, please feel free to open an issue or a pull request.)

Install it using Fisher:

fisher install small-tech/gills

Node Vesion Manager (nvm)

I work with Node.js everyday so, prior to Fish shell, I was using nvm-sh/nvm to manage my node versions.

While you can still use that script in Fish, there is a better option that’s called nvm.fish.

Designed for Fish, this tool helps you manage multiple active versions of Node on a single local environment. Quickly install and switch between runtimes without cluttering your home directory or breaking system-wide scripts.

Install it using Fisher:

fisher install jorgebucaran/nvm.fish

From there on, you can use commands like nvm install lts to quickly install and use different versions of Node.

nvm.fish also supports .node-version and .nvmrc files so pop one of those bad boys into your project directory to automatically have the Node version specified within them available when you switch to your project’s directory in your shell.

Puffer Fish

One feature of Zsh that I use all the time is to navigate back multiple directories simply by adding more dots to my command. So, for example, if I’m in the /a/b/c/d/e/ directory and I want to change to the b folder, I can simply write cd .... and Zsh will interpret that as cd ../../../ and save me some typing.

(Hey, I’m a developer, being lazy is part of the job description.)

So, anyway Puffer Fish enables Fish shell do the same thing. Actually, it’s even better because you get in-place expansion.

Install it using Fisher:

https://github.com/nickeb96/puffer-fish

z

While Puffer Fish makes it easy traverse backwards through a directory stack, z makes it easy to jump to any (previously visited) directory at any time.

z tracks the directories you visit. With a combination of frequency and recency, it enables you to jump to the directory in mind.

So, for example, if I visit ~/small-tech/small-web/domain and then, later, I want to get back there, I can simply type the following from anywhere:

z domain

And it’ll pop me back into my project folder.

I wasn’t sure if I’d end up using it when I first installed it but I find myself using it quite often to quickly switch to the directories of projects I work on often.

Install it using Fisher:

fisher install jethrokuan/z

Autopair

Autopair automatically matches opening/closing pairs of common punctuation like quotation marks and parentheses.

It might seem like a small thing, but it’s one of those things that gives you a little jolt of dopamine every time it Just Works™.

Install it using Fisher:

fisher install jorgebucaran/autopair.fish

Dracula theme

I like and use the Dracula theme pretty much everywhere and, of course, there’s a version for Fish.

Install it using Fisher:

fisher install dracula/fish

Configuration

How do you like your Fish?

If you’re coming from Bash or, like me, from Zsh, you might be wondering where to put your aliases, etc. The equivalent of ~/.bashrc or ~/.zshrc in Fish is ~/.config/fish/config.fish.

You can also configure your shell visually in your web browser by running the following command in your terminal:

fish_config

Also, while Fish does have aliases, it also has a much more powerful alternative called abbreviations.

Abbreviations

Abbreviations? DMIID (Abbreviation: don’t mind if I do)

Abbreviations are just regular Fish functions but you get a handy shortcut (abbr) for defining them. Unlike aliases, you get auto-suggestions for abbreviations and they are expanded in place so you can see the full command.

While I’ve converted most of my Zsh aliases to abbreviations, I’ve kept some (like code for codium) as aliases.

My advice is to convert all your aliases to abbreviations and then you will know right away whether you need to convert any back to aliases when you use them. (e.g., I didn’t want to keep seeing /usr/local/codium on my history so I decided to keep code as an alias.)

Here is my current config.fish file, in case it gives you some ideas. I’ve also listed links to the custom commands used to make it easier for you to find and install them.

if status is-interactive
    #
    # Aliases
    #

    # VSCodium
    alias code '/usr/bin/codium'

    #
    # Abbreviations
    #

    # Copy and paste to the clipboard by piping to these commands.
    # (Inspired by the default behaviour in macOS.)
    abbr --add --global pbcopy 'xsel --clipboard --input'
    abbr --add --global pbpaste 'xsel --clipboard --output'

    # Open files in their associated apps from Terminal.
    abbr --add --global open 'xdg-open &>/dev/null '

    # Git aliases to make git a bit more humane for everyday use.
    abbr --add --global git-log 'git log --graph --decorate --pretty=oneline --abbrev-commit'
    abbr --add --global git-log-dates 'git log --graph --decorate --pretty=format:"%h [%cr] %s'
    abbr --add --global git-tag 'git tag -n'
    abbr --add --global git-undo-last-commit 'git reset HEAD~'

    # Aliases for getting system and app information.
    abbr --add --global system-information 'neofetch'
    abbr --add --global disk-usage 'dust'
    abbr --add --global which-kernel 'apt-cache policy linux-generic'
    abbr --add --global node-v8-version 'node -p process.versions.v8'

    # Make rm a little safer (have it prompt once when deleting
    # more than three files or when deleting recursively).
    abbr --add --global rm 'rm -I'

    # A nicer ls that also shows the git status of files
    abbr --add --global l 'exa -lh --git --all'

    # A nicer ls that also shows the git status of files
    # (but not hidden files)
    abbr --add --global ll 'exa -lh --git'

    # Same nicer ls but in tree view.
    abbr --add --global lt 'exa -lh --git --all --tree'

    # lc = line count
    abbr --add --global lc 'wc -l'

    # Find out what’s running on port X
    abbr --add --global port 'lsof -i'

    # Better find
    abbr --add --global find 'fd'

    # Better ps
    abbr --add --global ps 'procs'

    # Package.json validator
    abbr --add --global validate-package.json 'pjv package.json'

    # Use ripgrep instead of grep
    abbr --add --global grep 'rg'
end

Tools mentioned in the configuration

XSel is a command-line program for getting and setting the contents of the X selection. Normally this is only accessible by manually highlighting information and pasting it with the middle mouse button.
xdg-open opens a file or URL in the person’s preferred application.
neofetch displays information about your operating system, software and hardware in an aesthetic and visually pleasing way.
dust is a more intuitive du for visualising disk usage.
exa is a modern replacement for ls that uses colours to distinguish file types and metadata and knows about symlinks, extended attributes, and Git.
fd is a simple, fast and user-friendly alternative to the find command for finding entries in your filesystem.
procs is a replacement for ps in a human-readable format and with some other additional features.
ripgrep (rg) is a line-oriented search tool that recursively searches the current directory for a regex pattern. It is similar to other popular search tools like The Silver Searcher, ack and grep.
package.json validator (pjv) verifies that your package.json files are correct.

Next steps

This introductory post doesn’t even begin to scratch the surface of what you can do with Fish but I hope that it sparks your interest and whets your appetite.

To learn more, view the documentation on the Fish shell website. You can also launch the documentation in a browser locally from Fish shell itself using the help command.

For example, to learn about abbreviations, run the following command:

help abbreviations

So, what are you waiting for?

Go Fish!

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

If you remove the seams altogether, you end up with the shiny locked boxes that Apple ships. Sure, they’re lovely to look at and use but you do so on the ever-evolving unilateral terms imposed by the corporation that makes them. This has considerable ramifications when it comes to protecting your freedom to own, control, and use your tools as you want to and impacts on your human rights (e.g., privacy), right to tinker, right to repair, etc. ↩︎

Remote: a little module for more elegant remoting with WebSockets

mail@ar.al (Aral Balkan) — Fri, 25 Jun 2021 13:59:29 +0100

Remote¹ is a tiny (< 50 lines of code) module that creates a very lightweight façade over a socket connection, using convention over configuration to give you an expressive interface with which to send outgoing messages and handle incoming ones.

You can use it both on the server and on the client.²

Here’s a simple Site.js example that performs some basic arithmetic on the server and keeps a count.

Install Site.js.
Create a folder to hold the example and the following directory structure inside it.
```
├── .dynamic
│  └── .wss
│     └── index.cjs
└── index.html
```
The index.cjs file is where you will put your server-side WebSocket route and index.html is where you will have your client.
Initialise an npm project.
```
npm init -y
```
Install the Remote module.
```
npm install @small-tech/remote
```

Create the client (in index.html).

<script type='module'>
  // Only load from Skypack for simple demonstration purposes. Don’t
  // use in production as it doesn’t support sub-resource integrity.
  // (See https://ar.al/2020/12/30/skypack-backdoor-as-a-service/)
  import Remote from 'https://cdn.skypack.dev/@small-tech/remote'

  const socket = new WebSocket(`wss://${window.location.hostname}/`)
  const remote = new Remote(socket)

  socket.addEventListener('open', () => {
    // Send remote events.
    remote.counter.increment.send()
    remote.addNumbers.request.send({firstNumber: 40, secondNumber: 2})
    remote.subtractNumbers.request.send({firstNumber: 44, secondNumber: 2})
  })

  // Handle remote events.

  remote.counter.update.handler = message => {
    log(`Counter is now ${message.count}`)
  }

  remote.addNumbers.response.handler = message => {
    log (`${message.firstNumber} + ${message.secondNumber} = ${message.result}`)
  }

  remote.subtractNumbers.response.handler = message => {
    log(`${message.firstNumber} - ${message.secondNumber} = ${message.result}`)
  }

  // DOM manipulation.
  const output = document.getElementById('output')
  const log = message => {
    const li = document.createElement('li')
    li.innerHTML = message
    output.appendChild(li)
  }
</script>

<h1>Simple remote arithmetic</h1>
<ul id='output'></ul>

Create the server (in .dynamic/.wss/index.cjs).

const Remote = require('@small-tech/remote')

let count = 0

module.exports = function (client, request) {

  const remote = new Remote(client)

  remote.addNumbers.request.handler = message => {
    remote.addNumbers.response.send({
      firstNumber: message.firstNumber,
      secondNumber: message.secondNumber,
      result: message.firstNumber + message.secondNumber
    })
  }

  remote.subtractNumbers.request.handler = message => {
    remote.subtractNumbers.response.send({
      firstNumber: message.firstNumber,
      secondNumber: message.secondNumber,
      result: message.firstNumber - message.secondNumber
    })
  }

  remote.counter.increment.handler = message => {
    count++
    remote.counter.update.send({ count })
  }
}

Run Site.js (in the root of your example folder).
```
site
```

Now hit https://localhost and you should see the following output:

Simple remote arithmetic

    • Counter is now 1
    • 40 + 2 = 42
    • 44 - 2 = 42

If you hit the page from other browser tabs, you should see the counter increase (it will reset if you restart Site.js as we are not persisting the value anywhere.³).

How it works

Messages are sent as JSON strings and received messages are parsed from JSON strings.

There are two special keywords: send and handler. You use the former when sending a message and you define the latter as a function to handle received messages.

Otherwise, there is nothing special about the path segments that make up the rest of the statements you see. Anything between the remote instance reference and either send or handler is used as the message type.

So, for example, addNumbers.request and addNumbers.response are message types, as is counter.update. Remote automatically creates the defined object hierarchies using some Proxy magic.

So far, Remote is working well for me as I build Domain. Give it a shot and see if it works for you too.

In building Domain (see a demo of it from last week’s episode of Small is Beautiful) using Site.js and SvelteKit, I use WebSockets¹ for calls between the client and server. This works really well and I find it much easier (less verbose and easier to maintain) than HTTP calls.

Even though the majority of the communication in Domain is request/response, I didn’t want to implement traditional RPC over WebSocket (which is quite restrictive). Instead, I wanted to retain the feel of a message-based API but give it some structure and an elegant, expressive interface.

Starting out, I didn’t want to use a heavyweight framework of any kind so I was simply passing messages back and forth, using plain old strings (POS; not a terrible acronym to describe the practice, if you think about it) for the message names. This works well enough as a quick and dirty way to get started but, as your app grows, you’ll likely start to feel the need to refactor.

During the first pass of refactoring, I replaced the POS with constants. This is fine, and gives you additional type safety at compile-time but it is also more verbose. But I just wasn’t happy with how the code read. So I decided to create a tiny class to standardise and abstract out the socket communication between client and server. ↩︎
It’s ‘isomorphic’, if you want to get fancy about it. ↩︎
If you want to persist the counter so it survives serve restarts, it is trivial to do so using Site.js.

Just update the server like this:
```
// …

if (!db.counter) {
  db.counter = { count: 0 }
}

module.exports = function (client, request) {

  // …

  remote.counter.increment.handler = message => {
    db.counter.count++
    remote.counter.update.send({ count: db.counter.count })
  }
}
```
(db is a global reference to the JavaScript Database (JSDB) instance that’s available for you to use in any HTTP or WebSocket route in Site.js.) ↩︎

Make anything a JavaScript module using Node.js ESM Module Loaders

mail@ar.al (Aral Balkan) — Thu, 27 May 2021 13:05:31 +0100

Disclaimer: you can’t load an actual bottle into Node.js (yet).

I have a message in a text file called bottle.txt. Here’s how I load it into Node.js and output it to the console:

import message from 'bottle.txt'
message()

Wait, what? How?

I’m glad you asked.

It’s thanks to Node.js’s experimental ESM Module Loaders feature. Here’s how you can implement this using the excellent node-esm-loader package which makes the Node ESM Loader API even easier to work with:

Message in a bottle(.txt file)

Create a new project directory, switch to it, and initialise a new Node project there.
```
mkdir message-in-a-bottle
cd message-in-a-bottle
npm init -y
```
Once you’re done, edit the package.json file to add the following key/value pair so Node knows to interpret .js files as ESM and not CommonJS:
```
"type": "module"
```
Create a text file called bottle.txt and add the following text to it:
```
This is a message in a bottle(.txt file)
```

Next, create the index.js file you saw earlier:

import message from 'bottle.txt'
message()

Now for the magic! Let’s make Node understand how to compile text files. First, install the node-esm-loader module to make it easier for us to work with the ESM Loaders feature:
```
npm i node-esm-loader
```

Next, let’s configure our loader to tell Node how to interpret text files. Create a file called .loaderrc.js with the following content:

import { URL } from 'url'

export default {
  loaders: [
    {
      resolve(specifier, opts) {
        if (specifier.endsWith('.txt')) {
          let { parentURL } = opts
          let url = new URL(specifier, parentURL).href
          return { url }
        }
      },
      format(url, opts) {
        if (url.endsWith('.txt')) {
          return { format: 'module' }
        }
      },
      transform(source, opts) {
        if (opts.url.endsWith('.txt')) {
          return {
            source: `export default function () {
              console.log(\`${String(source)}\`)
            }`
          }
        }
      }
    }
  ]
}

Finally, run your project, telling Node to use your new loader:
```
node --experimental-loader=node-esm-loader .
```

That’s it!

Alongside a warning that the ESM Loaders feature is experimental, you should see the following output in your terminal window:

This is a message in a bottle(.txt file)

How it works

In your loader, the resolve() and format() methods are where you tell Node that it should handle text files and treat them as JavaScript modules. And the transform() method is where you return an actual JavaScript module by compiling the text file (in this case, by wrapping its contents in a function that outputs them to the console).

Beyond the basics

So you’re most likely not going to use this for text files.

Instead, you can use it, as I am for example, to make a custom Express server that server-side renders Svelte files by using Vite as middleware.

I look forward to seeing all the creative uses you put this awesome feature to.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Archival cascades: a practical way to not break URLs

mail@ar.al (Aral Balkan) — Wed, 26 May 2021 11:05:56 +0100

Cascades are beautiful things.

This week, I reduced our DigitalOcean hosting costs for Small Technology Foundation from ~$90/month to $5/month by moving our canonical source code repositories as well as a few other servers to the complimentary hosting provided to our not-for-profit by the Eclips.is initiative by Greenhost and Open Technology Fund.

As part of the move, I had to decide what to do with the three servers that were still running various parts of the Ind.ie web site:

The latest version of the Ind.ie web site, with a notice that we were now Small Technology Foundation (this was a Hugo site).
The site we had from 2013-2017 (this was server-side-generated site using a custom engine I’d written in Node.js).
The labs site, which linked to a number of projects and held a few technical blog posts (this was a server-side-rendered site with a custom Express server I’d written in Node.js and which was deployed using dokku).

Memories

We haven’t been known as Ind.ie for the past three years now but that doesn’t mean we can just switch those servers off and be done with it. I’m hugely proud of our journey over the past eight years. Most of all in the fact that, as hard as it has been, we are still going. And that we will keep going for the foreseeable future. The Ind.ie site is an archive of the learning, iterating, and growing (in knowledge, understanding, and tools; not in size) that we’ve done through five of those years. Including…

The ethical design manifesto
The Ind.ie Summit and all of the videos from it (which I went through one-by-one to update and fix the video links on… my goodness we all look so young!)
Technical blog posts from the labs section, not to mention a photo of Oskar in a lab coat and red bow tie.¹

The plan

I knew I wanted to collate the three servers into one and use Site.js to host static versions of them.

The latest version of the Ind.ie web site was the easiest to deal with. Site.js has native support for Hugo so all I had to do was to create a .hugo directory in my new site and copy it there.

So my archived Ind.ie site looked like this:

ind.ie/
  ╰ .hugo
       ╰ (contents of the latest ind.ie site)

Next, I needed to serve the statically-generated contents of the site from 2013-2017. Now you’re probably thinking: “but Site.js is already serving the Hugo site and you need to serve that static content from the same namespace. How do you do that?”

Well, there are two ways I could have done it. Since Site.js simply serves any content in the root of your site as static content, I could have just copied the content there. But Site.js also has a feature specifically for this use case called archival cascades.

Archival cascades

If you have static archives of previous versions of your site, you can have Site.js automatically serve them for you.

Just put them into folder named .archive-1, .archive-2, etc.

If a path cannot be found in your current site, Site.js will search for it first in .archive-2 and, if it cannot find it there either, in .archive-1.

Paths in your current site will override those in .archive-2 and those in .archive-2 will, similarly, override those in .archive-1.

This was you create a cascade of archives where you can serve static snapshots of older content.

Using the archival cascade, old links will never die but if you do replace them with newer content in newer versions, those will take precedence.

So, after this step, my archive site structure looked like this:

ind.ie/
  ├ .hugo
  │    ╰ (contents of the latest ind.ie site)
  ╰ .archive-1
       ╰ (contents of the ind.ie site from 2013-2018)

Taking a static snapshot using wget

Since the labs site was server-side rendered, I needed to get a static snapshot of it.² The easiest way I could find of doing that was to use the handy wget command.

I simply ran the server on my local development machine (on port 3000) and ran the following command to save a static snapshot of it:

wget --recursive --domains localhost --no-parent --page-requisites http://localhost:3000

Once I had the static snapshot, I just added it to the archival cascade. So my final site structure looked like this:

ind.ie/
  ├ .hugo
  │    ╰ (contents of the latest ind.ie site)
  ├ .archive-1
  │    ╰ (contents of the ind.ie site from 2013-2018)
  ╰ .archive-2
       ╰ /labs
           ╰ (contents of the ind.ie labs site)

Then, I set up a server running Site.js, pointed the ind.ie domain to it, and synced the site over.

Don’t break URLs

It’s all too simple to just turn a server off and forget about it but the web doesn’t forget. What you leave behind will be a bunch of dead links. It’s not always practical to keep everything running forever but if you want to try and not break links, archival cascades and 404 to 302 support in Site.js should help.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Some of the links to articles on Labs might be broken as they refer to a forum that no longer exists. This was hosted by a commerical third-party and, sadly, we weren’t able to get a usable static export of our data at the time. ↩︎
As an alternative, if I had wanted to, I could have kept the server running somewhere and used the native 404 to 302 in Site.js to keep serving the old site. ↩︎

How to get clean analog audio from a Blue Yeti microphone into a Sony a6400 camera using a Raspberry Pi Zero

mail@ar.al (Aral Balkan) — Sun, 23 May 2021 20:36:13 +0100

Download the video.

A live demonstration from today’s S’update. Transcript)

Clean audio = clean USB power

You can connect the headphone out of a Blue Yeti microphone into the mic in of a Sony a6400 camera but the quality of the audio you get will depend on how clean the USB source powering your Blue Yeti is.

In order to get analog audio into your camera from a Blue Yeti, you need to plug the Blue Yeti into a computer¹ via USB and into your camera via an auxiliary cable. When I did this with the Blue Yeti plugged into a USB hub connected to a MacBook, I got quite a bit of noise, even after turning the input volume down to 1 on the Sony a6400 (which you should do).

When I tried plugging it into my daily driver development machine (a StarLabs LabTop), the noise was unbearably loud. Then, however, when I plugged it into an old MacBook Pro, the noise was almost gone. So I thought I’d give it a try with a Raspberry Pi Zero and — lo and behold – silence. It’s the cleanest USB power for a Blue Yeti that I’ve been able to find yet and it doesn’t hurt that it’s a $5 device.

Step-by-step

So, in case you have the same problem getting clean USB power from your existing devices to power a Blue Yeti microphone with a camera like the Sony a6400:

Get a Raspberry Pi Zero (with the official power adapter² and a good quality Micro USB male to USB A female adapter) and plug it in.
Turn the volume in your Sony camera down to 1 (Menu → Movie 2 → Audio Rec Level³).
Turn the gain on your Blue Yeti down and turn the headphone volume to somewhere half-way (you can tweak it using your level indicators but better to start lower and go higher rather than give yourself a loud shock).
Plug the Blue Yeti headphone out to the microphone in of your camera.
Plug the Blue Yeti USB out into the USB in on your Raspberry Pi Zero.

And that’s it. You should enjoy beautiful, clean audio with your Blue Yeti and Sony a6400⁴.

That should do it!

To see a live demonstration, watch today’s update, embedded above. A transcript of the stream is below.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Transcript

Hello and welcome to a special update you might have noticed if you watched the last few S’updates or the last Small is Beautiful that we did this week, this past week, that my audio was out of sync. In fact, if you’re paying attention right now, my audio should be out of sync. And I want to show you how I fixed that, how I found out what the problem was and how I fixed it. So let me show you, first of all, I’m also going to put on these headphones so I can hear what you’re hearing from my monitor over here.

So, all right. Now I can hear myself. The audio is fine. This is in terms of levels, et cetera. And let me show you my setup just slowly over here. Wait a minute. Actually, let me switch it up first, because I had it really nicely placed there. There we go. Let me pick up the camera now. So this is my setup and lots of screens. I know.

And in terms of the audio, so I have a Blue Yeti microphone up here and that Blue Yeti microphone, I’m going to stand up here and try not to snag my headphone cable off, has a USB connection here and a mic connection. And you can see apart from also it’s quite dusty. You can see that they’re both connected. What’s happening is I’m using the USB for power and I’m actually getting an analog signal out of the the headphone jack over there and where those are going to…

So if you look over here, I have a USB hub and this is the power coming in. So that’s just providing power to the Blue Yeti. Now, you can’t connect this to any power source. It has to be a computer. The Blue Yeti is very peculiar about that. And the headphone jack, if you follow that… is connected to the back of my Atem Mini Pro into the microphone input over here.

And now if you look at my Atem controller here, you’ll see in the settings that it’s the mic is set to line. OK, that’s important. I’m not going to put it on microphone right now because it would deafen you, but the headphone jack provides a signal that’s a little lower than line usually, but it’s still much higher than microphone powered like a microphone with power would be providing.

So that would be very, very loud if I switch to that. Now, the problem is and you can see that the sync is off, I’m assuming I can’t see that right now. But I’m assuming if you’re watching this, my lip sync is off. And that’s because let me show you my camera over here. So the video is coming out of this Sony Alpha 6400 and it’s coming out via HDMI that you can see over there.

So the problem is that HDMI has latency, all HDMI has latency. So the video is coming in slower than the audio. And that’s why there’s no lip sync. Now, with the Atem Mini Pro – with the app – which I’m running on a MacBook over here,

you can actually set the latency. So I can… if I go over here, over here, I can actually add a delay so I can add, I don’t know, let’s say [stuttering audio] four frames of delay, which is now making it very difficult for me to speak when I hear myself, because it’s coming in four frames delayed. So I’m just taking the headphones off. Now, this may or may not fix the problem. I am going directly from my microphone source in the mixer and previously four frames of delay is what I arrived at experimentally.

But recently something happened. I don’t know what it was. And I’ve done some tests and it seems like maybe when chroma key is on, the delay is different. Maybe this latest version of the the software has actually fixed it. I don’t know, because I only just updated. But the problem really is that this is not the way you should be doing it. Right? And so why was I splitting this up? Why was I taking… or why did I decide that I should put, like, the audio in directly to the mixer instead of into the mic – into the, sorry – into the camera, which is how I could have done it.

And that’s because when I tried it, I just got horrible static and horrible noise and I tried to narrow it down back then when I first set it up and I didn’t have time, so I was like, OK, I’ll do it like this. And then Atem came up with the delay and it was OK. So I’m going to turn that delay off again. So that will…

[stuttering audio] and there we go, the delay is zero now on my Atem controller… you can see over here the delay is zero. And let me show you what happens if I just plug the… instead of the line in… So instead of plugging it into the Atem, if I plugged it into the into the camera directly, let’s see what happens now.

I’m also going to… over here… momentarily… Let’s see. Let me put my headphones on as well just so I can hear what’s happening. OK. All right. So for one thing, I do have the audio levels on the camera. Let me try and get that here. The audio record level is set all the way down to one. OK, so when I do the switch, hopefully it won’t it won’t deafen your ears.

I’m taking out the audio. So are you going to lose my audio momentarily… [silence]

OK, and I just plug it in to

to the camera and you can hear it. I mean, it’s I’ve turned it down the doors. You can see on the on the camera, if I like to show you this, if I go to the menu, your record level, you see it’s that one. And if I’m quiet for a second…

You can hear that sound, right? If I actually turn the audio level up…

OK, so that’s the that’s the buzz that we’re getting and this is terrible, terrible audio. Of course, it’s terrible audio, but if you are watching now my lips, that should be in sync because, of course, the audio is now coming through the HDMI and it’s synched in the camera and it’s coming into the mixer. And at that point, it’s it’s mixed. So there’s not going to be any lip sync issue.

This is how you should be doing it. Now, this is not the horrible audio you want now. I mean, we can try and make it a bit better. So it’s much louder, of course, because remember, it was set to line on the other one. So you can… what you can do is… let me show you you can try turning down the headphone jack over here. And now that should be a bit better if you can still hear me. And also, there is, of course, a gain at the back of the… of the [Atem] let’s see…

Let’s try to get that in there. There’s a gain and you can turn that down as well. All right. So but this is not great audio by any means. You can hear that terrible humming. So I was like, OK, why why is this humming happening? And so today, what I did and I’ll do it now, it’s got to do with the USB connection with the power connection over here. So it’s got to do with this.

So what I tried was, OK, oh, it’s gone to sleep again. So let me turn this… or is it just… [wakes Mac from sleep] ok, there we go what I thought was OK, so what would happen if I changed… If I unplugged…the I wasn’t using this USB hub, but I plugged it into this old MacBook Pro, Laura’s old MacBook Pro that’s just lying around here. So look at what happens when I do that. You’re going to lose my audio for a second again and then it will come back. [silence]

All right, so now it’s being powered by the MacBook Pro. And can you hear that? That’s actually way better! There is it’s not the same sort of

the same sort of noise that we have. So if I go over here onto the camera and I just turn up. The audio level…

You can see you can hear when I turn it up…

There is some, but… It’s not… and of course, it’s going to be clipping.

But that is so much better, so, OK, so that was the “aha!” moment. All right, so this has to do with how much noise there is, how clean the USB connection is. So it’s it’s electro

electromagnetic interference, basically. So I was like, OK, I’m not going to use a MacBook Pro for this, not least of all, because that MacBook Pro, even if it’s just on at some time, sometimes the fan comes on and that’s really loud. So I was like, OK, what is? And again, you can’t connect it directly to just a USB power source. It needs to be a computer or the blue yeti doesn’t work. So it’s like, OK, what’s the simplest computer that I could have that’s also silent? That’s also really well built and shielded.

And that is what brought me to try…

the. Raspberry Pi Zero. So this is a Raspberry Pi Zero, one of a couple that I have hanging around that I was playing with and I was like, let’s try it with that. Now, every component matters in terms of the noise. And this is connected with the official cable, to… that’s actually an iPhone (or an iPad) power adapter there. I tried it originally with with a Raspberry Pi 4 with the original Raspberry Pi power adapter.

And that was very, very low noise as well. Much lower noise, actually, than the MacBook Pro. So I was like, OK, I hope it works with the with the Raspberry Pi zero as well. And spoiler: it did! So let me just show you what it’s like with that. But also remember, every component counts. So this cable here, for example, if I was to use so this cable here is where I’m going to connect the USB from the Yeti.

But I also tried it with this sort of cheap adapter that I had and there was a lot more noise with this. So every component really matters. So again, we’re going to lose audio for a second, but hopefully for the last time and I’m going to unplug the USB from the MacBook, oops that’s the power… [silence … struggles to plug in USB with one hand]

Right. OK, so I’ve just plug it in, that is that is harder to do than it looks with one hand to push you, apparently beyond my capabilities. All right. So…

Now, that is connected to a Raspberry Pi Zero and that is connected to power, so the Blue Yeti is on, and I have my levels. What’s interesting is I found that the levels are lower. So I’m just going to say, OK, here we go. Actually, there we go. Let me let me just adjust this… Oh, of course, the gain… is up so the levels are lower. So what I can do… is… I can turn the gain…

All right, wait a minute, sorry, let me switch to the camera so you know what I’m doing, I can turn the gain almost all the way down over here. Now, you’re not going to hear me very well. I’m just I’m very close to the microphone right now. And what I can do is I can then use… here… Let’s switch to this… I can then adjust the volume over here that I’m getting.

And and that’s pretty good. OK, I’m in the reds a little bit. So if you look at my monitor, you can see my levels. But I’m going to stop talking for a second and I just want you to take a look at those levels there, OK? [silence]

And that’s how little noise there is with the Raspberry Pi Zero with the Blue Yeti connected to the Raspberry Pi Zero. So again, I’m just doing this all live. So these are… not I’m going to get the levels just right. But that, in a nutshell, is how I’m going to be doing things from now on. And that should solve the lip-sync problem completely. And, yeah…

I can turn the mic off from the Atem Mini Pro and. Yeah. So it should be going in, synchronized into the Atem and this… So the real thing is that this little… If you’re having the same issue, for example, get a Raspberry Pi Zero. Get the official connector and everything, it might be a few bucks more, but just get the good stuff and I think this cable was from Pimoroni, I’m not sure in of their sets or I don’t know if it’s just a Raspberry Pi official cable, I don’t know…

But just get all good components for it, and, uh…

That is how you can get very, very clean audio out of the headphone jack and into the microphone, jack of the computer, and that way you you won’t have any audio sync issues whatsoever. So I hope you found this useful. And this has been a special little S’update, today, Sunday, May 23rd, 2021. Take care.

Be well. Bye bye!

A USB power supply won’t do, it has to be a computer. Otherwise, the Blue Yeti will power on but you will not get any sound from it. ↩︎
I used an iPhone/iPad power adapter with a good quality USB cable in the demo. ↩︎
And, of course, make sure Audio Recording is set to on in Menu → Movie 2. ↩︎
These instructions should work for the Blue Yeti and any other similar camera but I’ve only tested them with the Sony a6400. ↩︎

JSDB Migrations

mail@ar.al (Aral Balkan) — Sun, 16 May 2021 17:57:41 +0100

I’m busy working on Basil, the Small Web host, and while it’s nowhere near ready to use yet, I thought I’d try my hand at writing a database migration as they will be necessary once other people start using it.

Basil runs on Site.js and Site.js uses JSDB – JavaScript Database — as its database.

True to its name, JSDB is 100% JavaScript, so, of course, you write your migrations in JavaScript also.

Here’s what the migration I wrote ended up looking like:¹

// Migration of Settings table from version 1 to version 2.
//
// Payment providers is now an array instead of an object
// and is pre-populated with the defaults for the None and Token
// provider types.
//
// Also, adds the version attribute for the first time, thereby
// enabling future migrations.

const JSDB = require('@small-tech/jsdb')

const db = JSDB.open('.db')
const settings = db.settings

// Is migration necessary?
if (db.settings.version && db.settings.version >= 2) {
  console.log('Already migrated, exiting.')
  process.exit()
}

// Add version to settings table.
settings.version = 2

// Migrate the payment object to an array, copying over
// the Stripe-related details in version 1.
const payment = settings.payment

settings.payment = {
  // Set the payment provider to the new index of the only one
  // that existed previously (Stripe).
  provider: 2,

  // Create the providers array.
  providers: [
    {
      name: 'None',
      modes: null
    },
    {
      name: 'Tokens',
      modes: null,
      codes: []
    },
    {
      name: 'Stripe',
      modes: [ 'test', 'live' ],
      mode: payment.mode,
      modeDetails: payment.modeDetails,
      currency: payment.currency,
      price: payment.price,
      amount: payment.amount
    }
  ]
}

Neither Site.js nor JSDB has any built-in support for migrations yet so this is something you would write into the logic of your own applications at the moment.

As Basil is still under heavy initial development, I just made it a script and ran it directly.

I’m going to keep experimenting with migrations as I work on Basil and I wouldn’t be surprised if they work their way into JSDB and/or Site.js in time.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

The actual migration script has some more code to back up the table and handle errors, etc., but this is the gist of it. ↩︎

Scraping the latest EU VAT rates for e-services from the European Commission’s web site with Node.js

mail@ar.al (Aral Balkan) — Fri, 14 May 2021 15:14:36 +0100

So, you now know how to verify an EU VAT number with Node.js.

Lucky you! (Don’t say I don’t spoil you.)

But do you know what the latest VAT rates are every EU country?

(Do you? Huh, do you, punk?)

OK, so you can find them here. Yes, that’s an HTML page. No, I haven’t been able to find a simple JSON feed or anything. Yes, it is 2021.

Example

So anyway, here’s one way you can scrape it using Node.js:¹

Install the dependencies.
```
npm install cheerio node-fetch
```

Scrape it like you mean it.

import * as cheerio from 'cheerio'
import fetch from 'node-fetch'

const htmlLikeIts1999 = await (await fetch('https://ec.europa.eu/taxation_customs/business/vat/telecommunications-broadcasting-electronic-services/vat-rates_en')).text()

const $ = cheerio.load(htmlLikeIts1999)

const taxRates = {}

$('tr.table-vat-rates').each((index, row) => {
  const cells = $(row).find('td')
  const countryName = cells.first().html()
  const taxRate = parseInt(cells.last().html().replace(/\<br.*?$/, ''))
  taxRates[countryName] = taxRate
})

console.log(taxRates)

When you run it, you should see a nice hash table of the latest VAT rates resembling the one below appear in your console.

{
  Austria: 20,
  Belgium: 21,
  Bulgaria: 20,
  Cyprus: 19,
  …
  Sweden: 25,
  Slovenia: 22,
  Slovakia: 20
}

So, there you go.

The JSON endpoint for EU VAT rates that you never knew you wanted is now yours to cherish.

(Until they change the HTML structure, that is.)

w00t, etc.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

You feeling lucky? If you’re a real daredevil, why not use regular expressions instead? Go on… I double dare you…

import fetch from 'node-fetch'

const htmlLikeIts1999 = await (await fetch('https://ec.europa.eu/taxation_customs/business/vat/telecommunications-broadcasting-electronic-services/vat-rates_en')).text()

const taxRates = {}

htmlLikeIts1999.split('\n').filter(line => line.includes('<tr class=')).map(row => row.trim().replace(/^\<tr.*?\>/, '').split('td')).forEach(row => taxRates[row[1].replace('>', '').replace('</', '')] = parseInt(row[7].replace(/<br.*?$/, '').replace('>', '')))

console.log(taxRates)

(It’s ok, you can cry a little now.) ↩︎

Using the European Commission EU VAT Number validation API with Node.js

mail@ar.al (Aral Balkan) — Fri, 14 May 2021 13:46:20 +0100

You may know of the VIES site where you can manually validate EU VAT numbers but did you know that the European Commission also has an API for programmatically doing this?¹

So that’s the good news.

The bad news is that it’s SOAP.

I’ll wait until you’re done retching… ok, feeling better? Let’s move on.

Example

Here’s how you consume it in Node.js:

Install the soap package:

npm install soap

Create the web service by consuming the WSDL file and calling the right method:²

import soap from 'soap'
const wsdl = 'https://ec.europa.eu/taxation_customs/vies/checkVatService.wsdl'

const client = await soap.createClientAsync(wsdl)
const result = await client.checkVatAsync({
  countryCode: 'EE',
  vatNumber: '100244258'
})

console.log(result)

And that’s all there is to it.

Enjoy! ³

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

I didn’t until I stumbled onto it today while doing research into VAT OSS for Basil, which is part of our Small Web initiative at Small Technology Foundation. ↩︎
VAT Number in the example retrieved following a web search from SEB’s contact page. ↩︎
As much as that word can be used for anything tax-related. ↩︎

Hell site

mail@ar.al (Aral Balkan) — Mon, 10 May 2021 16:20:00 +0100

They have a word for Twitter on the fediverse.

They call it “hell site”.

It’s very apt.

When I first joined about 15 years ago, at the end of 2006, it was a very different place. A small, non-algorithmically curated space where you could have group chats with your friends.

What I didn’t know back then was that Twitter, Inc., was a venture-capital-funded startup.

It wouldn’t have mattered if I’d known, either, as I was clueless about funding or business models. I thought everyone in tech was just trying to make the new everyday things that improved people’s lives.

Even six years later, in 2012, I was still fixated on improving people’s experiences within the current system:

“Objects have value not because of what they are but because of what they enable us to do.

And, as the people who make objects, we have a profound responsibility. A responsibility to not take for granted the limited time that each of us has in this world. A responsibility to make that time as beautiful, as comfortable, as painless, as empowering, and as delightful as possible though the experiences that we craft.

Because this is all there is.

And it’s up to us to make it better.”

– This is all there is

What a fool, right?

Please take as much time as you need to point and laugh.

OK, done? Let’s move on…

Privilege is just another word for not having to care

Back then, I took for granted that the system in general was mostly good. At least I didn’t think it was actively evil.¹.

Sure, I was on the streets of London, along with hundreds of thousands of others, protesting the looming war in Iraq. And sure, I understood that we lived in an unequal, unjust, racist, sexist, and classist society (heck, I studied critical media theory for four years, so I had Chomsky coming out of my ears), but I somehow thought tech existed outside of this sphere. That is, if I thought about it at all.

Which clearly just meant that things weren’t bad enough to be affecting me personally to a degree where I felt I should even educate myself about it. And that, folks, is what we call privilege.

Sure, it felt odd whenever one of these startups did something that wasn’t in our best interests. But they told us they made mistakes and they apologised so we believed them. For a while. Until it became impossible to. And hey, I was just making “cool stuff” that “improved people’s lives” – first in Flash and then for the iPhone and iPad…

But I’m rushing ahead.

Back to me being entirely ignorant of business models and venture capital. Hey, you might find yourself at this point today. There’s no shame in that. So listen closely: here’s the problem with venture capital.

What happens in Venture Capital stays in Venture Capital

Venture capital is a high-stakes game of roulette and Silicon Valley is the casino.

A venture capitalist will invest, say, $5M in ten “startups” in full knowledge that nine of them will fail. What this gentleman needs (it’s almost always a “gentleman”) is for the remaining one to be a billion-dollar unicorn. And he (it’s almost always a he) doesn’t invest his own money either. He invests other people’s money. And they want 5×–10× their money back because this game of roulette is very risky.

So how does a startup become a unicorn? Well, there is one battle-tested business model that is known to work: people farming.

Here’s how it works:

Addict people.

Offer your service for free to your “users” and try to get as many people as possible hooked on your product.

Why?

Because you need to scale exponentially to get the network effects and you the need network effects to lock in the people that you originally attracted.

Heck, highly-celebrated folks have even written best-selling how-to guides about this step like Hooked: How to Build Habit-Forming Products .

That’s Silicon Valley for you.

Farm them.

Collect as much data about people as you can.

Track them on your app, across the web, and even in physical space to create detailed profiles of their behaviour. Use this intimate insight into their lives to attempt to understand, predict, and manipulate them. Monetise this with your actual customers, who pay you for this service.

This is what some folks call Big Data and what others call surveillance capitalism.

Exit (sell).

A startup is a temporary business and your endgame is to sell it to a wealthier startup or to an existing Big Tech company or to the public via an initial public offering (IPO).

If you made it here, congratulations. You might just become Silicon Valley’s next douchebag billionaire and Bitcoin philanthropist.

Many startups fail at the first step but as long as the VC gets their one unicorn, they’re happy.

Calling bullshit (part 1)

So I didn’t know that having venture capital meant Twitter had to grow exponentially and become a billion-dollar unicorn. I didn’t understand that those of us using it – and helping it improve – in those early days were ultimately responsible for its success. We were hoodwinked. (At least I was and I’m sure I’m not the only one who feels that way.)

All this to say that Twitter was destined to become the Twitter it is today the moment it took its first bit of “angel investment” at the very beginning.

That’s just how the Silicon Valley game of venture capital and unicorns goes. It is what it is. And what it is everything I’ve been spending the last eight years to raise awareness about, protect people from, and build alternatives to.

Here are some recordings of my talks that you can watch from this period:

The Camera Panoption (2014)
Decentralise Everything (2015)
Excuse Me, Your Unicorn Keeps Shitting In My Back Yard, Can He Please Not? (2016)
Ethical Design and Democracy (2016)
Small Technology (with Laura Kalbag) (2019)
Small Tech > Big Tech (with Laura Kalbag) – online (2020)

As part the “raising awareness” part, I was also trying to use platforms like Twitter and Facebook against the grain.

As I wrote in Spyware vs Spyware in 2014: “We must use existing systems to promote our alternatives if our alternatives are to exist at all.” Even back then, that was perhaps rather optimistic but one crucial difference was that Twitter, at least, didn’t have an algorithmic timeline.

Algorithmic timelines (or gaslighting 2.0)

What is an algorithmic timeline? Let me try and explain.

What you think happens when you tweet: “I have 44,000 people following me. When I write something, 44,000 people will see it.”

What actually happens when you tweet: Your tweet might reach zero, fifteen, a few hundred, or a few thousand people.

Based on what?

Fuck knows.

(Or, more precisely, only Twitter, Inc., knows.)

So an algorithmic timeline is a black box that filters reality and decides who gets to see what and when based on an entirely arbitrary set of criteria determined by the corporate entity it belongs to.

In other words, an algorithmic timeline is just a euphemism for mass socially-acceptable gaslighting. It is gaslighting 2.0.

The algorithm is an asshole

The nature of the algorithm mirrors the nature of the corporation that owns and authors it.

Given that corporations are sociopathic by nature, it should come as no surprise that their algorithms are too. In short, the algorithms of people farmers like Twitter and Facebook are shit-stirring assholes that get off on causing as much conflict and controversy as possible.

Hey, what did you expect exactly from one billionaire who has “#Bitcoin” as their bio and another that calls the people who ~~use~~ are used by his service “dumb fucks?”

These bastards delight in showing you things they know will upset you in hopes that you will retaliate. They revel in the resulting fallout. Why? Because the more “engagement” there is on the platform – the more clicks, the more time their addicts (“users”) spend on it – the more money their corporations make.

Well, enough of that, thank you very much.

Calling bullshit (part 2)

While I feel it’s important to raise awareness of the harms of Big Tech, I have probably said and written everything there is to say on the matter over the past eight years. I’ve given over a hundred talks at various conferences alone over that time, not to mention media interviews in print, and on radio, and television.

Here are some links to a relative handful of the things I’ve written on the subject during this period:

Schnail Mail (2013)
Beware of geeks bearing gifts (2013)
Trickle‐down technology and why it doesn’t work. (2013)
Towards an Indie Tech Manifesto (2014)
How Web 2.0 Killed the Internet (2014)
Spyware 2.0 (2014)
Privacy as Innovation (2014)
What is the ind.ie manifesto? (2014)
Why? (2014)
Alternatives (2014)
The Camera Panopticon (2014)
Europe, we need to talk about institutional corruption (2015)
Why I’m not speaking at CPDP (Hint: it’s the privacy-washing, stupid!) (2016)
The nature of the self in the digital age (2016)
Encouraging individual sovereignty and a healthy commons (2017)
We didn’t lose control, it was stolen (2017)
Decrypting Amber Rudd (2017)
Introducing the 7th pillar of DiEM25: An Internet of People – a progressive tech policy for a democratic Europe. (2017)
Constructive disobedience (2017)
Farewell, not goodbye: leaving DiEM25 (or “We need to talk about democracy, transparency, feminism, and Assange.”) (2017)
Web+ (2018)
Kyriarchy (2018)
Reclaiming RSS (2018)
Out of the frying pan and into the fire (2018)
Multi-writer Dat could power the next Web (2018)
Better Blocker: two year review and thoughts on the future (2018)
Extended Codice Interview With Rai 1 (2018)
Better, simpler, and more affordable (2018)
GDMR: this one simple regulation could end surveillance capitalism in the EU (2018)
Baby steps (2018)
Surveillance capitalism at the BBC (2018)
What does a private communicator look like? (2018)
Deployment-first development (2019)
Success criteria for the PC 2.0 era (2019)
The post-Web is single tenant (2019)
I was wrong about Google and Facebook: there’s nothing wrong with them (so say we all)
On the General Architecture of the Peer Web (and the placement of the PC 2.0 era within the timeline of general computing and the greater socioeconomic context)
Privacy is not a science, it is a human right (2019)
Small Technology (2019)
Slavery 2.0 and how to avoid it: a practical guide for cyborgs (2019)
The Do’s and Don’ts of Tech Regulation (2019)
Have you heard about Silicon Valley’s unpaid research and development department? It’s called the EU. (2019)
The Future of Internet Regulation at the European Parliament (2019)
In 2020 and beyond, the battle to save personhood and democracy requires a radical overhaul of mainstream technology (2020)
Apple just killed Offline Web Apps while purporting to protect your privacy: why that’s A Bad Thing and why you should care (2020)
What is the Small Web? (2020)
Clean up the Web (2021)
Ethics as PR (or the ‘Some Very Good People Work There!’ Fallacy) (2021)

Has any of it done any good?

I don’t know.

I hope so.

I also challenged countless folks at surveillance capitalists like Google and Facebook on Twitter and – before I left a few years ago – on Facebook, and elsewhere. (Anyone remember the time I got Samuel L. Jackson to call out Eric Schmidt for Google mining people’s email?) That was fun. But I digress…

Did any of that do any good?

I don’t know.

I hope so.

But here is what I do know:

Does calling people out make me miserable? Yes.

Is it nice? No.

Do I like conflict? No.

So enough is enough.

People come up to me sometimes to thank me for “speaking out.” Well, that “speaking out” comes at a very high price. So maybe some of those folks can pick up where I left off. Or not. Either way, I’m done with it.

In your face

One thing you have to understand about surveillance capitalism is that it is the mainstream. It is the dominant model. Every Big Tech company and startup is part of it². And being exposed to their latest bullshit and to hypocritical posts by people who proudly affiliate with them while purporting to work for social justice is not good for anyone’s mental health.

It’s like living in a factory farm owned by wolves where the loudest proponents of the system are the chickens that have been hired as line managers.

I’ve spent the last eight years, at least, responding to such things and trying to point out how Big Tech and surveillance capitalism cannot be reformed.

And it makes me miserable.

So I’m through doing so on platforms with shit-stirring asshole algorithms that get off on inflicting as much misery on me as they can in hopes of getting a rise out of me because that “makes the number go up.”

Fuck you, Twitter!

I’m done with your bullshit.

What next?

In a lot of ways, this decision has been a long-time coming. I set up my own place on the fediverse using Mastodon several years ago and have been using it ever since. If you haven’t heard of the fediverse, think of it like this:

Imagine that you (or your group of friends) own your own copy of twitter.com. But instead of twitter.com, yours is at your-place.org. And instead of Jack Dorsey, you make the rules.

You’re not limited to just talking to people on your-place.org either.

I also own my own place at my-place.org (let’s say I’m @me@my-place.org). I can follow @you@your-place.org as well as @them@their.site and @someone-else@some-other.place. It works because we all speak a common language called ActivityPub.

So imagine a world where there are thousands of twitter.coms and they can all talk to one another and Jack has fuck-all to do with it.

Well, that’s the fediverse.

And while Mastodon is just one way to get your own place on the fediverse, joinmastodon.org is a good place to start learning about it and get started in a way that is friendly and doesn’t require any technical knowledge.

As I already mentioned, I’ve been on the fediverse since the earliest days of Mastodon and I was already manually forwarding posts from there to Twitter.

I’ve now automated that process using moa.party and, going forward, I will no longer be checking Twitter or responding on it.³

As my posts on Mastodon are now automatically forwarded there, you can still use it to keep up with what I’m doing, if you like. Or, why not take the opportunity to join the fediverse and have a play?

Small is Beautiful

While I still believe that having good critiques of Big Tech is essential to influencing effective regulation, I don’t know if effective regulation is even possible given the levels of institutional corruption we have today (lobbying, revolving doors, public-private-partnerships, corporate capture, etc.)

What I do know is that the antidote to Big Tech is Small Tech.

We must build alternative technological infrastructure that is owned and controlled by individuals – not corporations or governments. This is a prerequisite for a future with personhood, human rights, democracy, and social justice.

Otherwise, we face a bleak tomorrow where our only agency is to beg some Silicon Valley king or other to “please sir, be kind”.

I also know that working on building such alternatives makes me happy while despairing at the state of the world just makes me utterly miserable. I know it’s privilege to have the skills and experience I have that enable me to work on such things. And I intend on making the most of it.

Going forward, I plan to concentrate as much of my time and energy as possible on building the Small Web.

If you’d like to talk about that (or anything else), you can find me on the fediverse. You can also chat to me on my S’update live streams here and during our Small is Beautiful live streams with Laura.

Here’s to brighter days ahead…

Stay safe.

Be well.

Love one another.

💕️

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Not that good and evil are the best ways to talk about all this. True evil is much more mundane and often far less intentioned that what you see in the movies. Usually, it simply involves remaining neutral in situations of injustice. In the words of Desmond Tutu, who I must have quoted a hundred times in the past eight years or so, “If you are neutral in situations of injustice, you have chosen the side of the oppressor. If an elephant has its foot on the tail of a mouse and you say that you are neutral, the mouse will not appreciate your neutrality.” ↩︎
If you’re not a venture-capital-funded new tech company, don’t call yourself a startup, that’s their brand and, if they’re in the same area as you, they have $5M in the bank to undercut and destroy your sustainable small tech business with. So don’t legitimise them by calling yourself that. ↩︎
I’m also planning on unfollowing everyone on Twitter so folks are not mislead into thinking I will be seeing their posts (which, given what we know – or, more precisely, what we don’t know – about Twitter’s algorithm, was never a given anyway). It might take some time for me to fully realise that as it’s not a priority. ↩︎

Ethics as PR (or the ‘Some Very Good People Work There!’ Fallacy)

mail@ar.al (Aral Balkan) — Sat, 08 May 2021 11:03:20 +0100

“Ethics in AI”

There’s been a lot of research into “Ethics in AI” recently and it’s being sponsored and led by… *checks notes*… surveillance capitalists like DeepMind¹.

Similarly, lots of people are working on “Ethics in Health” at Philip Morris and “Ethics in Environmentalism” at ExxonMobil.

OK, I made the last two up.

Did you notice?

(I guess it was pretty blatant.)

Because, clearly, that would be ethicswashing.

So how come we accept the first?

If someone told you they were working on environmental ethics at ExxonMobil, they would get laughed out of the room. Yet folks who work on “Ethics in AI” at surveillance capitalists like Google and Facebook are invited to talk to policymakers and sit at the table when crafting privacy and other technology regulation.

Environmental ethics at ExxonMobil

The only way I can make sense of all this is that we don’t see a Google or a Facebook as being as bad as a Philip Morris or an ExxonMobil.

I’ve been trying to explain why they are just as bad for the past seven years and I don’t know what more there is I can say.

My goal here isn’t to point fingers at the people who take these positions (although, goodness knows, if you have an alternative, please take it), but at the corporations themselves who are creating the positions to begin with as a cynical part of their lobbying and public relations initiatives.

Big Tech is following Big Tobacco’s rule book to the letter and it seems to be working splendidly so far.

This is why we can’t regulate effectively.

“The truly damning evidence of Big Tobacco’s behavior only came to light after years of litigation. However, the parallels between the public facing history of Big Tobacco’s behavior and the current behavior of Big Tech should be a cause for concern … We believe that it is vital, particularly for universities and other institutions of higher learning, to discuss the appropriateness and the tradeoffs of accepting funding from Big Tech, and what limitations or conditions should be put in place.”

– Mohamed Abdalla (PhD student, University of Toronto Center for Ethics) and Moustafa Abdalla (student, Harvard Medical School) (What Big Tech and Big Tobacco research funding have in common)

Institutional corruption

We are institutionally corrupt.

In addition to the massive influence of lobbying, revolving doors, and sponsorships, we also have corporate capture of academia.

Thanks to this neoliberal success story, university departments today are left at the mercy of funding by industry. If the only way an academic in the field can progress is to work with or secure funding from a Big Tech company that is not necessarily the academic’s fault, per se, but a systemic failure.

This is not to say that we should disregard those who make sacrifices to go against the grain and vocally oppose and refuse to take part in this system. On the contrary, we should be celebrating and supporting them.

“During a panel conversation, Black in AI cofounder Rediet Abebe said she will refuse to take funding from Google, and that more senior faculty in academia need to speak up. Next year, Abebe will become the first Black woman assistant professor ever in the Electrical Engineering and Computer Science (EECS) department at UC Berkeley.

‘Maybe a single person can do a good job separating out funding sources from what they’re doing, but you have to admit that in aggregate there’s going to be an influence. If a bunch of us are taking money from the same source, there’s going to be a communal shift towards work that is serving that funding institution,’ she said.”

– What Big Tech and Big Tobacco research funding have in common (emphasis mine)

When PR backfires

Big Tech is not infallible.

Far from it.

When ethicswashing goes wrong, it can backfire spectacularly and become a PR nightmare as we saw with the example of the ethicists hired by Google who got fired when they didn’t reach the conclusions the corporation wanted them to.

This is a relationship that is destined to fail as long as an ethicist does what their job title says instead of what the corporation wants them to do. Because surveillance capitalists like Google are fundamentally unethical.

And while fallout like this can undo the positive PR that Google is trying to cultivate, the presence of respected people in these positions is often all that’s necessary for the corporation to benefit from their legitimacy.

Maybe what we need is Big Tech’s own Surgeon General’s Reports moment. But that’s not going to be likely if the Surgeon General works for Philip Morris.

Greta Thunberg would never work at ExxonMobil

Being a privacy or AI ethicist at a Silicon Valley Big Tech company is the same as being an environmental ethicist at ExxonMobil: your very presence there legitimises the corporation.

There is a reason Greta Thunberg would never work at ExxonMobil.

There is a reason doctors would not work at Philip Morris.

There is a reason pacifists would never join the military.

Because no matter how well-intentioned you are, you are not going to change the fundamental nature of these institutions.

For corporations, their fundamental nature is dictated by their business model. For surveillance capitalists, the business model is based on mass violations of your privacy that are used to profile you with the goal of understanding, predicting and affecting your behaviour for profit. This is a fundamentally unethical business model within a fundamentally unethical socioeconomic system.

These corporations will employ you for as long as they can launder their reputations by using your legitimacy. And when you become too problematic, they will spit you right out again.

The sooner we all realise this, the sooner we tell them “we see what you’re doing”, the sooner no one else has to go through this. Because this isn’t good for anyone involved.

‘Ethics’, as far as a surveillance capitalist is concerned, is public relations.

And I know that’s not what anyone signed up for.

Lots of very good people…

Whenever there’s criticism of a surveillance capitalist like Google or Facebook, someone always says “but I know lots of very good people who work there.”

I don’t doubt it for a minute.

Heck, I know lots of very good people who work there!

In fact, the more wonderful they are, the more legitimacy they add to the unethical corporation they work at. It’s time we at least acknowledged that this is a problem even if we don’t have all the solutions yet.²

Acknowledging the problem is the first step in making these corporations socially unacceptable so we can effectively regulate them and secure funding for alternatives. Alternatives not just in technology but also in academia where these good folks can carry out truly independent research for the common good.

Eric Schmidt once told me “if we ever get too evil, we won’t be able to find anyone to work for us.”

I guess either he was wrong or we just don’t think they’re evil enough yet.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Owned by Alphabet, Inc., the same multi-trillion-dollar corporation that owns Google, YouTube, etc. ↩︎
And realise, I’m not talking about the folks who work in the cafeteria at Google but the ones with PhDs that sit on policymaking committees and the engineers on six-figure salaries with stock options that keep the machine ticking; the latter being the ones who might have other options even within this capitalist hellscape that we find ourselves mired in. ↩︎

Using Subresource Integrity (SRI) in Vite with @small-tech/vite-plugin-sri

mail@ar.al (Aral Balkan) — Mon, 19 Apr 2021 12:24:02 +0100

A few weeks ago I created a Subresource Integrity (SRI) plugin called @small-tech/vite-plugin-sri for Vite.

What is SRI and why should I care?

SRI is important if you don’t trust the place you’re loading resources (scripts and styles) from.

In your HTML file, you provide the hash of each external resource you’re loading. The browser then guarantees that the resource that’s loaded is the one you expected (or it doesn’t load it).

SRI was introduced on the Big Web because site owners don’t trust Content Delivery Networks (CDNs)¹. On the Small Web (see What is the Small Web), we don’t trust any server, not even the one our app is being loaded from. So we use SRI for all external resources loaded by our apps.²

How do I use it?

Just add the plugin to your vite.config.js file, like this:

import { defineConfig } from 'vite'
import sri from '@small-tech/vite-plugin-sri'

export default defineConfig({
  // …
  plugins: [sri()]
})

And run your build:

npx vite build

Is this for Vite only?

This plugin is specifically for Vite but Jonas Kruckenberg has a generic plugin for Rollup called rollup-plugin-sri that does the same thing, and which mine was inspired by.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

I talk more about what could happen if a CDN does not support SRI in Skypack: Backdoor as a Service?. ↩︎
So what about the initial index HTML file that’s served? Surely a malicious server could just serve anything it wants there and all our SRI stuff becomes security/privacy theatre, as the root of our trust model is broken, no?

If we limit ourselves to the context of the browser, yes.

That’s why, for the Small Web, we need an out-of-band means (e.g., a browser extension) of verifying that the source of the initial index file is also what we expect it to be (e.g., by verifying the hash and signature embedded in the source).

Whether or not you need to do any of this, of course, depends on your threat model. For most people, I would consider it overkill as compromising a web host to such a degree to carry out a targetted phishing attack on someone is something I’d consider a state-level attack.

And yet this is the only way with a web app to truly ensure that the place you’re going to enter your all-important passphrase into is what you expect it to be. So, of course, we are going to support it. ↩︎

Clean up the web

mail@ar.al (Aral Balkan) — Fri, 16 Apr 2021 19:45:42 +0100

Developers, it’s time for you to choose a side: will you help rid the web of privacy-invading tracking or be complicit in it?

🚮️ https://CleanUpTheWeb.org

Spread the word using the #CleanUpTheWeb and #FlocOffGoogle hashtags.

Passing data from layouts to pages in SvelteKit

mail@ar.al (Aral Balkan) — Sat, 03 Apr 2021 13:26:13 +0100

Data flow from layout to page (slot) in SvelteKit

I’m prototyping Basil, the free and open hosting client that’s going to power small-web.org, in SvelteKit and one thing I want to ensure from the outset is that the app is not hardcoded for our use so that anyone can easily set up a Small Web host simply by installing and configuring it.

Now, ideally, I want that configuration data to be loaded and rendered on the server (SSR/SSG) once.

Given that you cannot use Node APIs in layouts/pages even when they’re rendered on the server, the way to do this is to use an endpoint and consume it on the server in your main layout and then pass the data to any pages the layout has slotted into it using the context.

To cut a long story short, I struggled a bit to get this running initially as there are two separate “context” concepts in SvelteKit¹ and I couldn’t find any code samples when I was starting out with it. So here’s the code sample I wish I had.

Remember that SvelteKit is in beta and this might all change. There’s also the chance that, while this works, I’m still “holding it wrong” and there might be an even more straightforward way to achieve this that I do not know about.

The endpoint

The rest of this post assumes that you’ll be working with the following endpoint, defined in src/routes/config.js:

import os from 'os'
import fs from 'fs'
import path from 'path'

// Load the configuration from ~/.small-tech.org/basil/config.json
const config = JSON.parse(fs.readFileSync(path.join(os.homedir(), '.small-tech.org', 'basil', 'config.json'), 'utf-8'))

// Return the configuration in response to a GET request on /config
export async function get() {
  return {
    body: config
  }
}

Layout: load the data

You load the data in the load() handler.

For example, your layout in src/routes/$layout.svelte might resemble the following:

<script context='module'>
	import '../app.css'
	import Nav from '../lib/Nav.svelte'

	export async function load({ page, fetch, session, context }) {
		const url = '/config'
		const res = await fetch(url)

		if (res.ok) {
			const config = await res.json()
			return { context: { config } }
		}

		return {
			status: res.status,
			error: new Error(`Could not load ${url}`)
		};
	}
</script>

<Nav/>
<slot/>

Two key things to note here:

This is a context=‘module’ script, not a regular script tag.
We’re returning an object with a context property from the load() handler.

So what’s this context property?

According to the docs, it is exactly what we need:

This will be merged with any existing context and passed to the load functions of subsequent layout and page components.

This only applies to layout components, not page components.

The important thing to note is that it is not the same as the context you manipulate using the setContext() and getContext() methods. In fact, if you try to access those methods from context='module' scripts, you will get an error.

Page: load the data from the context

Now, let’s say you have a page at src/routes/index.svelte that gets slotted into your layout.

To use the configuration data, you just need to get it from the context you returned in your layout:

<script context='module'>
	let config
	export async function load({ page, fetch, session, context }) {
		config = context.config
		return true
	}
</script>

<main>
	<h1>{config.name}</h1>
	<h2>About this Small Web host</h2>
	<p>{config.description}</p>
</main>

<style>
	/* etc. */
</style>

That’s it!

Hope it helps in case you were stuck trying to do the same thing :)

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

You can actually also use the other type of context in Svelte/SvelteKit for this also but it is a far more convoluted process that involves returning a prop from the load() handler in your layout and storing it in the context using setContext() from a separate script tag (not your context='module' script tag) in the layout, and, finally, using getContext() to retrieve it from the script tag (not your context='module' script tag) of your page. Whew! ↩︎

Site.js starter template for vite + svelte

mail@ar.al (Aral Balkan) — Thu, 01 Apr 2021 16:35:45 +0100

Over the past few weeks, I’ve been experimenting with some initial clients for the Small Web clients for Site.js and Place using Snowpack and Svelte.

While I finally got everything working after some trouble (and a few discussions/pull requests, etc.), some recent regressions in Snowpack broke what I had again. So I thought that I’d give Vite a shot, both out of curiosity and to see if, possibly, things could be simpler.

Fast forward a couple of days and I’m ahead of where I was previously. This is not to poop on the Snowpack folks, they’ve been rather responsive and are a small team trying first and foremost to build a CDN (which I do have issues with, due to the lack of subresource integrity, but that’s a different conversation)¹.

site-vite-svelte

One of the things I wanted to do was to create a starter template for kickstarting projects that use this combination of client-side tech. So today I released the site-vite-svelte template for Site.js.

Installation

To install it, just do:

npm init using small-tech/site-vite-svelte my-project

That will use npm init using to download the contents of the site-vite-svelte repository to your my-project folder.

Then, just do:

cd my-project
npm install

And you’re ready to go!

Usage

You can choose to either run just the Vite + Svelte client using npm run vite or you can run Site.js + the Vite + Svelte client using npm run dev.

For full documentation, please see the readme.

I know this isn’t as smoothly integrated as Hugo is into Site.js but that’s on purpose. When it comes to Place, having the client decoupled from the Small Web protocol server is a feature, not a bug. As for Site.js, I want to see how I get on with these particular modules in a decoupled manner before deciding whether it’s worth integrating them more closely².

So feel free to have a play and let me know how you get on if you do and I hope you find it useful.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

On the topic of subresource integrity, I also released a vite plugin for subresource integrity yesterday. Enjoy! ↩︎
Also, I’m aware that while Place is lean (and will hopefully only get leaner as it shapes up), Site.js is quite bloated right now and I don’t want to add to that. ↩︎

npm init using

mail@ar.al (Aral Balkan) — Thu, 01 Apr 2021 16:04:54 +0100

Want to download a git repository to use as a starting point for your own Node.js app but you don’t want to clone the repository?

Try this:

npm init using <github-account>/<starter-project-name> <my-project>

What that will do is download the specified repository into the my-project directory on your local machine.

Then, you can just:

cd my-project
npm install

And you’re off!

For example, to start using the Site.js starter template for Vite + Svelte, do:

npm init using small-tech/site-vite-svelte my-site

How it works

Under the hood, this uses a tiged, a community-maintained forked of degit (see what they did there?) that I then subsequently forked and renamed to create-using.

The way npm init works, npm init something gets translated to npx create-something, which in turn downloads and runs the create-something module from npm, which is supposed to, well, create something.

In this case, create-using (being tiged) downloads and extracts the tarball of the referenced git repository on GitHub (other hosts are also supported) to the local directory you specify.

So, basically it’s a little hack so we can have more expressive syntax.

If you don’t care for such things, you can do exactly the same thing using tiged directly:

npx tiged small-tech/site-vite-svelte my-site

Personally, I’d rather remember npm init using but you do whatever floats your boat. Heck, you can even go old-skool and just clone the repository using git like some sort of cavaman! 😛️

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

References to methods are a JavaScript minefield

mail@ar.al (Aral Balkan) — Tue, 09 Mar 2021 14:22:56 +0000

Your code, doing exactly what you told it to (as always).

So, tell me what’s wrong with the following code:

//               ms  s  m  h
const ONE_DAY = 1000*60*60*24
const ONCE_A_DAY = ONE_DAY

class Certificate {
  #renewalDate = null

  constructor() {
    // Renewal is in thirty days.
    #renewalDate = new Date(Date.now() + ONE_DAY*30)

    // In a real app, you’d save the returned interval id
    // and clear it when you no longer need it.
    setInterval(this.checkForRenewal, ONCE_A_DAY)
    checkForRenewal()
  }

  async checkForRenewal() {
    console.log('Checking if certificate needs to be renewed…')
    const currentDate = new Date()
    if (currentDate >= this.#renewalDate) {
      await this.renewCertificate()
      console.log('Certificate renewed.\n')
    } else {
      console.log('Certificate does not need renewing yet.\n')
    }
  }
}

Nothing?

What if I told you it crashes when it tries to check for renewal the second time?

Not very obvious is it?

Checking if certificate needs to be renewed…
Certificate does not need renewing yet.

Checking if certificate needs to be renewed…
(node:3035) UnhandledPromiseRejectionWarning: TypeError: Cannot read private member #renewalDate from an object whose class did not declare it

(You might want to set the duration to something lower, like 3 seconds, if you don’t want to wait around for a day to see the error for yourself.) :)

References to methods in JavaScript are a runtime minefield

So the problem is on this line in the constructor:

setInterval(this.checkForRenewal, ONCE_A_DAY)

Can you see it yet?

(I don’t blame you, if not.)

Do as I think, not as I say

Debugging is the subtle art of reconciling what you think you asked a computer to do with what you actually asked a computer to do.

What we think we’re doing here is passing a reference to the checkForRenewal() method on the Certificate class to the setInterval() function.

But what we’re actually doing – because method references are not closures in JavaScript – is passing a reference to the checkForRenewal() function to setInterval().

What’s the difference? Oh, about one crash.¹

But wait a minute, if that’s the reason, isn’t the error message confusing? Yes, yes it is. (It’s so confusing that I’m writing this revised blog post because of it. – thanks, Tom!)

The red herring

Let’s remember what the error says:

TypeError: Cannot read private member #renewalDate from an object whose class did not declare it

This makes it sound like a private member called #renewalDate exists on the this reference but you don’t have permission to access it when actually, it doesn’t.

The actual issue

Let’s remove the use of private fields to make things clearer and easier to understand.

Remove all the octothorpes from the code and run it again, adding a console.log(this) statement to the start of the checkForRenewal() method.

When you run it, you should see something similar to the following when the first the renewal check is run as a method call:

Certificate { renewalDate: 2021-03-09T17:36:40.821Z }

Right, so that’s what we expect. But the second time we see any console output, we’re running the unbound function reference, not the method, so we get the following for this:

Timeout {
  _idleTimeout: 3000,
  _idlePrev: null,
  _idleNext: null,
  _idleStart: 3057,
  _onTimeout: [AsyncFunction: checkForRenewal],
  _timerArgs: undefined,
  _repeat: 3000,
  _destroyed: false,
  [Symbol(refed)]: true,
  [Symbol(kHasPrimitive)]: false,
  [Symbol(asyncId)]: 2,
  [Symbol(triggerId)]: 1
}

So our initially unbound function reference has been bound by the setInterval() method to a Timeout instance.

Also note what we don’t get: an error.

As far as our code is concerned, the certificate does not need renewing. This is because of the implicit type conversions JavaScript performs:

// Our original conditional
currentDate >= this.renewalDate

// Becomes:
Number(currentDate) >= Number(this.renewalDate)

// Becomes:
Number(currentDate) >= NaN // which is always false.

So without the use of private class fields, our code just fails silently (the silent killer).

Things are slightly better when you try to access a private class field from the callback function since at least it fails. But, sadly, we get a misleading error message that makes it sound like our initial unbound function reference was actually a closure all along.

The fix

The fix, simply, is to make sure you bind the function reference to its instance:²

setInterval(this.checkForRenewal.bind(this), ONCE_A_DAY)

Now, I can bet that the number of times you’re going to write that code out of the gate without getting bitten by this bug is exactly zero. Even after you’ve read this.³

But maybe you’ll remember this post and catch yourself after you’ve written it and fix it.

And it also shows you private class fields do help by at least not failing silently, even if the error they do fail with is confusing.

Stay safe out there, kids! :)

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

This little bug, looming in the latest Site.js code is what took down the Small Web site yesterday when its Let’s Encrypt certificate expired because of it.

I issued a new release of Site.js that fixes it a few hours after I was informed of it and the server auto-healed at around 3:30AM when it auto-updated to the latest release. ↩︎
As Mark Hughes pointed out on the fediverse, you can also use a closure instead of the bind() method. e.g.,
```
;(() => setInterval(this.checkForRenewal, ONCE_A_DAY))()
```
Personally, I find that using the bind() method expresses intent more clearly and is easier to understand but not everyone agrees. ↩︎
This is also exactly the kind of bug that is difficult to catch with tests. (I had tests for every method on there but I wasn’t calling them from unbound callbacks so I never triggered the issue.) ↩︎

fs-extra to fs

mail@ar.al (Aral Balkan) — Sun, 07 Mar 2021 12:46:48 +0000

I’m a big fan of the fs-extra module for Node.js. It has made my life much easier over the years. However, as I migrate my modules from CommonJS to ECMAScript Modules (ESM) and as I’m looking at bundling Place into a single JS file for deployment using esbuild, I’ve started removing third-party dependencies wherever I can easily replicate their behaviour using modules from the Node standard library.

For the move from fs-extra to fs, this is helped by Node.js implementing some of the features that were hitherto unique to fs-extra in version 14.x (LTS).

(There still isn’t an equivalent for recursive copying in the standard library. So you’re probably better off sticking to fs-extra is you need that functionality.)

So, without further ado, here’s a short summary of fs-extra methods and their equivalents in Node’s standard fs module that I’ve encountered, both as a reference for you and for my future self. (I’ll add to the list as I find more.)

Methods that need migrating

Not every fs method is re-implemented in fs-extra. It simply proxies calls to the built-in module for any methods it does not implement itself. These are the methods that have custom implementations that will need migration:

copy, emptyDir, ensureFile, ensureDir, ensureLink, ensureSymlink, mkdirp, mkdirs, move, outputFile, outputJson, pathExists, readJson, remove, writeJson

This post is not an exhaustive give to every method, just the ones I’ve had to migrate myself.

For simplicity, sync methods are shown, but the changes apply equally to the async methods.

Creating recursive directories

fs-extra

fs.mkdirpSync('/some/directory/structure/')

fs

fs.mkdirSync('/some/directory/structure/', { recursive: true })

Removing a directory and all its contents and not failing if the directory doesn’t exist

fs-extra

fs.removeSync('/some/directory')

fs

fs.rmSync('/some/directory', { recursive: true, force: true })

Regular expression find/replace

Find:

fs.removeSync\((.*?)\)

Replace:

fs.rmSync($1, {recursive: true, force: true})

Moving a directory on the same device (and overwriting the destination if it exists)

fs-extra

fs.moveSync(sourcePath, destinationPath, { overwrite: true })

fs

fs.renameSync(sourcePath, destinationPath)

Note: these are not strictly equivalent. The move methods in fs-extra act like the mv command and they will work across devices also. The rename methods in fs work like the rename system call and will not work across devices. Having Node’s rename methods work across devices is currently marked as a “won’t fix.”

Also, the overwrite: true behaviour is the default/only behaviour in the standard library.

Regular expression find/replace

Find:

fs.moveSync\((.+?), (.+?)(, .+?)?\)

Replace:

fs.renameSync($1, $2)

Copying a file

fs-extra

fs.copySync(sourcePath, destinationPath)

fs

fs.copyFileSync(sourcePath, destinationPath)

Copying a folder recursively

Sadly, there’s no simple way to do this using the standard fs module as it only provides a copyFile() method and does not have the equivalent of fs-extra’s copy() method.

Ensuring that a directory exists

fs-extra

fs.ensureDirSync('/some/directory/structure')

fs

function ensureDirSync (directory) {
  if (!fs.existsSync(directory)) {
    fs.mkdirSync(directory, { recursive: true })
  }
}

ensureDirSync('/some/directory/structure')

Note: fs-extra also supports an option that is either an integer or {mode: <integer>}. If you need that functionality, use the fs.chmodSync() method after creating the directory.

Writing a file

fs-extra

fs.outputFileSync(…)

fs

fs.writeFileSync(…)

Note that with fs-extra, if the parent directory doesn’t exist, it is created.

Bonus: promises and async/await

To use the asynchronous methods in Node’s fs module with async/await, import the fs/promises module instead. Note that the latter does not include the synchronous methods or stream methods.

e.g., To remove a directory recursively and not fail if it doesn’t exist:

import fsPromises from 'fs/promises'

fsPromises.rm('/some/directory', {recursive: true, force: true})

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Cache busting in Node.js dynamic ESM imports

mail@ar.al (Aral Balkan) — Mon, 22 Feb 2021 08:18:12 +0000

I’m porting JSDB to EcmaScript Modules (ESM) and one of the issues I had to look into was module cache invalidation.

JSDB is my little in-memory native JavaScript Database that writes JavaScript operations to append-only JavaScript logs that have UMD headers. And it loads in these tables either via a dynamic require() call, or, for very large tables, by streaming them in and evaluating them line by line¹.

When a table is loaded in, it is stored in the require cache. Then, during a session, data might be written to it. If the table is then closed, we must invalidate the old version in the cache in case the table is opened again in the same session (otherwise, all the changes that were written to it in that session will be lost as the old version is read in from the cache).

In the current CommonJS version, I lazily used a module called decache without giving it too much thought. The module works as described but is also not entirely necessary for my simple use case (as JSDB modules cannot include other modules so there isn’t a complicated cache hierarchy to navigate). I could easily have removed entries from the module cache manually by deleting keys on the cache property of the require function².

Module cache invalidation in CommonJS

To test this out youself, create a file called module.cjs, with the following content to display a random number:

module.exports = (() => Math.random())()

Then, create a file called index.cjs and require your module twice:

const path = require('path')
const modulePath = path.resolve('./module.cjs')

console.log(require(modulePath))
console.log(require(modulePath))

Run this and note that only one random number is displayed as the immediately-invoked function expression (IIFE) exported from your module is only run the first time your module is loaded in and then its value is cached.

To clear the cache, we simply have to remove its entry from the require.cache object.³ If you console.log(require.cache) after the initial call to the require() function in your code, you will see output similar to the following:

[Object: null prototype] {
  '/home/aral/sandbox/common-js-require-cache/index.cjs': Module {
  // …
  },
  '/home/aral/sandbox/common-js-require-cache/module.cjs': Module {
    id: '/home/aral/sandbox/common-js-require-cache/module.cjs',
    path: '/home/aral/sandbox/common-js-require-cache',
    exports: 0.6628156804529053,
    //…
}

To remove it from the cache, we simply delete its key. Add the following line of code between the two console.log() statements:

delete require.cache[modulePath]

(This is also why we used path.resolve() in the original example so we’d have the absolute path to the module.)

Now, when you run it, you should see two different numbers output as the module is loaded in fresh both times.

Cache invalidation in ESM with CommonJS-style requires

If I wanted to, I could still have JSDB output its tables in CommonJS or UMD modules even though I was using ESM for it. Why would I want to do that?

Well, for one thing, there is a very important difference in dynamic module loading between CommonJS and ESM: the former is synchronous while the latter is asynchronous. And refactoring previously-synchronous code to be asynchronous has cascading effects throughout our application and can complicate your library’s interface.

So if I was lazy or if I didn’t want to make my API asynchronous (which is less of an issue now that we have top-level await), I could simply use the createRequire() method in the built-in module module to create my own require() function that works under ESM. I detail how to do this in my previous CommonJS to ESM in Node.js post.

Here’s what the above example would look if that’s the direction you wanted to go.

Your module.cjs would stay the same and you’d load it in from an index.mjs:

import path from 'path'

import { createRequire } from 'module'
const require = createRequire(import.meta.url)

const modulePath = path.resolve('./module.cjs')

require(modulePath)
delete require.cache[modulePath]
require(modulePath)

Cache invalidation in ESM with dynamic imports

Note: This will leak memory and eventually crash your system. Garbage collecting stale ESM modules is not a solved problem as of Feb, 2021.

The ESM version of JSDB, however, will be 100% ESM and so it will write out ESM modules and use dynamic import() calls to load them in⁴.

Cache invalidation with dynamic imports is different and in some ways much easier. You don’t have a require.cache object to work with like you do in CommonJS. Instead, however, you can use a trick as old as the web itself (well, almost): just tack a random fragment to the URL of the module you’re loading.

So this is what cache invalidation looks like in pure ESM:

// index.mjs
const modulePath = './module.mjs'

async function importFresh(modulePath) {
  const cacheBustingModulePath = `${modulePath}?update=${Date.now()}`
  return (await import(cacheBustingModulePath)).default
}

console.log(await importFresh(modulePath))
console.log(await importFresh(modulePath))

One caveat is that the relative path I used in the example above works because the importFresh() function is in the same file. If I was to refactor it out to a common module, you would have to use absolute paths. Otherwise, module resolution would occur from whatever path the file containing your importFresh() function was itself imported from.

To npm install or not to npm install, that is the question

I actually went ahead and made a tiny Node module called @small-tech/import-fresh.

(But, as I mention in the readme, consider whether you really need to add yet another dependency to your project or whether you can get away with just using vanilla JavaScript based on what you know from above.)

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

If you’re using JSDB with datasets of these sizes you will also have to configure your system and Node.js to increase the heap size, etc. While there are legitimate use cases for such large datasets, it’s also possible that you’re up to no good. (Don’t use JSDB to farm people for their data or profile them or I’ll haunt you in your nightmares.) ↩︎
Remember that everything is an object in JavaScript, including functions. And yes, functions can have properties attached to them. This is because JavaScript is a prototype-based language. ↩︎
In more complicated applications, your modules will require modules and thus you’ll have to navigate the cache hierarchy and invalidate child modules, etc. Save yourself some trouble and use a ready-made module like decache, etc. (there are quite a number of them). ↩︎
In my tests with medium-sized datasets (~227MB), I didn’t perceive any substantial performance difference between dynamic imports in CommonJS vs ESM. The former is ever so slightly faster in real terms but they both have the same O(N) linear time complexity. ↩︎

CommonJS to ESM in Node.js

mail@ar.al (Aral Balkan) — Wed, 27 Jan 2021 10:30:02 +0000

Yesterday, I refactored Place¹, which is a non-trivial Node.js app, to use ECMAScript Modules (ESM). Here’s a diff of the full set of changes.

This is the approach I took and some of the issues I ran into, in case it helps someone else:

Find and replace

First off, I started by running the following regular expression² to quickly convert CommonJS require syntax to ESM import syntax:

Find

const (.*?)\s*?=\s*?require\(('.*?')\)

Replace

import $1 from $2

While this will do a lot of the work for you, you’ll still have to go in and add .js suffixes manually to any imports that refer to modules using a file path instead of the npm package name.

Next, I updated the export statements:

Find

module.exports = something

Replace

export default something

I did this manually as sometimes that something is an object you were assigning to module.exports and you cannot do that in ESM as what you’re exporting are references (aka live bindings. I’ll come back to this later with a specific example to demonstate what that means.)

Note that .cjs denotes a CommonJS file and .mjs denotes an ESM file in the following examples.

So, an object exported by CommonJS:

// index.cjs
const { name, age, breed } = require('./osky')
console.log(name, age, breed)

// osky.cjs
module.exports = { name: 'Oskar', age: 9, breed: 'Huskamute' }

Becomes the following when using ESM:

// index.mjs
import { name, age, breed } from './osky.mjs'
console.log(name, age, breed)

// osky.mjs
export const name = 'Oskar'
export const age = 9
export const breed = 'Huskamute'

Also, instead of exporting each constant separately in osky.mjs, we could have exported them all in one go:

const name = 'Oskar'
const age = 9
const breed = 'Huskamute'

export { name, age, breed }

These two approaches are functionally equivalent.

You could also be tempted to keep the default export and destructure it after import:

// index.mjs
import osky from './osky.mjs'
const { name, age, breed } = osky
console.log(name, age, breed)

// osky.mjs
export default { name: 'Oskar', age: 9, breed: 'Huskamute' }

While this will result in identical behaviour in this example, that’s because the live bindings we’re exporting are constants. Let’s see what would happen if we were exporting variables whose values could be changed after the import by the module we’re importing:

// index.mjs
import { name, age, breed } from './osky.mjs'

function updateStatus () {
  console.log(`${name} (a ${breed}) is now ${age} year${age > 1 ? 's': ''} old.`)
}

updateStatus()
setInterval(updateStatus, 1000)

// osky.mjs
export let name = 'Oskar'
export let age = 1
export let breed = 'Huskamute'

setInterval(() => age++, 1000)

When you run this, you will see that Oskar’s age increases by one every second and is reflected in the status update even though the updates take place in the module being imported and are logged in index.mjs. This is what we mean when we say ESM modules export live bindings. Basically, what you’re exporting is a reference to your original variable.

Now, change the export to the second approach we used:

// index.mjs
import osky from './osky.js'
const { name, age, breed } = osky
// …

// osky.mjs
let name = 'Oskar'
let age = 1
let breed = 'Huskamute'

setInterval(() => age++, 1000)

export default { name, age, breed }

When you run this version, you’ll notice that Oskar never ages. That’s because when we destructure, we no longer have a reference to the originally-imported variable and the updates taking place in the original module are not reflected in the console output.

If you did want to use a default export and keep the live bindings, this is how you’d do it:

// index.mjs
import dog from './osky.js'

function updateStatus () {
  console.log(`${dog.name} (a ${dog.breed}) is now ${dog.age} year${dog.age > 1 ? 's': ''} old.`)
}

updateStatus()
setInterval(updateStatus, 1000)

// osky.mjs
const name = 'Oskar'
const age = 1
const breed = 'Huskamute'

const osky = { name, age, breed }

setInterval(() => osky.age++, 1000)

export default osky

Note that I’ve purposefully stored the original values, as well as the osky object in constants in osky.mjs to illustrate the point even further of what’s being exported and what’s changing. (Remember that marking an object as a constant means that you cannot assign a different object to it, not that you cannot change the values of its properties or even add or remove properties from it.)

Anyway, all this to say that you should keep in mind that ESM modules are not just a different syntax, they are fundamentally different to CommonJS modules in how they work.

So, with the find and replace out of the way, I had to tackle a few other issues.

Dynamic requires and imports

In Place, I was using dynamic requires to load in routes for the server. I needed to change these to use dynamic imports instead so you could use ESM for the server as well as the client.³

This is not as straightforward as a simple find and replace because while dynamic requires are synchronous, dynamic imports are asychronous. And asynchronous code has a cascading effect⁴.

(Note that you can still do an old-school, synchronous require. You just have to create your require() function yourself. See A require by any other name…, below, for details.)

The asynchronous cascade

For example, refactoring function c(), below, to return a promise means that functions b() and a() become asynchronous also:

// Sync
function a () { console.log(b()) }
function b () { return c() }
function c () { return 'Hello' }

a()

// Async
async function a () { console.log(await b()) }
async function b () { await c() }
function c () {
  return new Promise (resolve => setTimeout(resolve, 3000)
}

await a()

Awaiting imports

The switch from synchronous dynamic requires to asynchronous dynamic imports was the biggest change in the refactor. Here’s a relevant section of the diff, showing the before and after:

-decache(routesJSFilePath)
-require(routesJSFilePath)(this.app)
+// Ensure we are loading a fresh copy in case it has changed.
+const cacheBustingRoutesJSFilePath = `${routesJSFilePath}?update=${Date.now()}`
+;(await import(cacheBustingRoutesJSFilePath)).default(this.app)

So requiring a default function from a module and immediately executing it goes from:

require(filePath)(args)

;(await import(filePath)).default(args)

Note the addition of .default. That’s how you access the default export of a dynamically-imported module.

Also note where the parentheses are around await. The promise must resolve before you can access the module’s properties.

And here’s something to watch out for: if you forget to await an asynchronous call when using dynamic imports, Node.js will throw a very unhelpful error message without a stack trace:

SyntaxError: Unexpected reserved word

If you encounter that error, chances are you should be checking for missing await statements in your code.

Cache busting with dynamic ESM imports

Note: This will leak memory and eventually crash your system. Garbage collecting stale ESM modules is not a solved problem as of Feb, 2021.

In the example above, you can see that I also had to change the cache-busting strategy for when routes are reloaded at runtime.

With dynamic requires, I was using the decache module.

With dynamic imports, you don’t have programmatic access to the module cache so I’m using an old client-side trick instead and appending a query string to the file path with the current timestamp in it.

If you want to test this out quickly for yourself to see that it works, create a file called loader.mjs with the following code in it:

setInterval(async () => {
  const cacheBustingPath = `./message.mjs?update=${Date.now()}`
  console.log((await import(cacheBustingPath)).default)
}, 1000)

Then, create another file called message.mjs that contains the following line of code:

export default 'hello'

Finally, run the loader:

node loader.mjs

You should see the message hello printed out to your console every second.

With that process still running, change the message in message.mjs and save the file and you will see the message update in the console.

Will no one think of the loops?

When using dynamic imports and asychronous calls in linear tasks, remember that you can’t just loop over promises like you can with synchronous calls, you must await each result.

So no forEach(), etc., for you.

Here’s a relevant diff from where I changed a synchronous forEach() loop into an asychronous loop using Sebastien Chopin’s handy asyncForEach() function:

-httpsPostRoutes.forEach(route => {
+asyncForEach(httpsPostRoutes, async route => {
-  this.app.post(route.path, require(route.callback))
+  this.app.post(route.path, (await import(route.callback)).default)
 })

And this is the asyncForEach() function itself:

async function asyncForEach(array, callback) {
  for (let index = 0; index < array.length; index++) {
    await callback(array[index], index, array);
  }
}

__dirname

Another issue that cropped up was the lack of __dirname support with ESM under Node.js. This is easily fixed by declaring __dirname as a global in your module:

import { fileURLToPath } from 'url'
const __dirname = fileURLToPath(new URL('.', import.meta.url))

This is such an easy, backwards-compatible fix that I don’t know why Node.js doesn’t automatically inject it into all ESM modules.⁵

Note: This snippet was previously using the .pathname property of the URL instance, which is wrong and won’t work on Windows. The updated snippet, above, properly converts the URL to a file path.

A require by any other name…

A problem with a similar solution to the missing __dirname functionality exists if you have code that uses require to load in JSON files. ESM modules do not currently support this. So what can you use, given you can’t use require with ESM modules?

Why, require, of course…

No, that’s not a typo.

You just have to create your own require function.

(I know, I know… I’m shaking my head too.)

import { createRequire } from 'module'
const require = createRequire(import.meta.url)

Then, you can require a JSON file just like you did before. e.g., along with the __dirname constant, above, the following code will work if you have a file called my.json in the same directory as your script:

JSON.parse(fs.readFileSync(path.join(__dirname, 'my.json'), 'utf-8')))

Again, unless there are some edge cases that the Node development team knows of that I don’t, I don’t see why this isn’t default behaviour that’s injected into ECMAScript Modules in Node.js.

The above two features – alongside doing the same for __filename, etc. – would go a long way to making the migration process from CommonJS to ESM a far more seamless one for many developers.

Little quirks

The final thing I had to do was to set the type property to module in my package.json. When you do this, all .js files are treated as ESM by Node.js. Which can cause some unexpected quirks.

For example, JSDB – my small, in-memory, streaming write-on-update JavaScript database – persists to JavaScript transaction logs and uses a CommonJS require to load them in.⁶ Even though I was importing JSDB in from node_modules, it crashed while trying to load data tables:

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /home/aral/small-tech/small-web/sandbox/sign-in.small-web.s
require() of ES modules is not supported.
require() of /home/aral/small-tech/small-web/sandbox/sign-in.small-web.org/.db/privateRoutes.js from /home/aral/small-.
Instead rename privateRoutes.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" .

    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1080:13)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at JSTable.load (/home/aral/small-tech/small-web/place/app/node_modules/@small-tech/jsdb/lib/JSTable.js:161:20)
    …

There seems to be some context leak between an app and the npm modules it imports whereby the type of the app overrides the type of the npm module declared in the module’s package file when using a dynamic require().

The quick workaround I hacked together for the moment was to add a default (of type CommonJS) package.json file to the database directory. In fact, I might just update JSDB to do that by default so that it can be used in both CommonJS and ESM projects with a minimum of fuss.

It feels hacky but I don’t think it’s worth the effort to create a non-hacky solution to something that is itself a huge hack (ESM in Node.js).

The same issue also cropped up when I used Snowpack’s loadConfiguration() method (which uses require() internally) to load the Snowpack configuration for Place. The easy workaround to that was just to rename snowpack.config.js to snowpack.config.cjs. The irony of having to make my Snowpack configuration file a CommonJS file when Snowpack exists to make ESM easy to use is not lost on me.

Other quirks

Error stacks are different between CommonJS and ESM

This is one you will probably not care about but it just bit me while converting Auto Encrypt to ESM as I have custom error handling that uses the information provided in stack traces.

To see the difference for yourself, create two files, index.cjs and index.mjs, both with the following line of code in them:

console.log((new Error().stack))

The output you’ll see in each will differ (the actuals paths you see will, of course, be based on the directory structure of your local system).

CommonJS:

Error
    at Object.<anonymous> (/home/aral/sandbox/stack-get-file-name-esm-vs-commonjs/index.cjs:1:14)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47

ESM:

Error
    at file:///home/aral/sandbox/stack-get-file-name-esm-vs-commonjs/index.mjs:1:14
    at ModuleJob.run (internal/modules/esm/module_job.js:152:23)
    at async Loader.import (internal/modules/esm/loader.js:166:24)
    at async Object.loadESM (internal/process/esm_loader.js:68:5)

Just be aware of this if you are doing anything based on the contents of the error stack, especially with paths as ESM returns URLs whereas CommonJS returns file paths.

Conclusion

Prior to native support for ECMAScript Modules in Node.js, I saw no reason to use them. Now, I take it as given for future Node projects and look forward to working more easily with isomorphic JavaScript.

I hope this post with my experiences from yesterday’s refactor are helpful for you if and when you undertake a similar task.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Place is a hard fork of Site.js for creating Small Web places that is not ready for use yet (it won’† even run for you yet as it depends on my fork of Snowpack; open pull requests). ↩︎
I use the ES6 const keyword to declare my imported modules as constants. If you use ES5 var, update the regular expression accordingly. ↩︎
The main reason for my refactor was because Place uses ESM modules via Snowpack for the client and I wanted the server to use them too so I can write isomorphic JavaScript that can be run on both the client and server. ↩︎
Making a sychronous function in a deep call stack asynchronous means that you have to make every function that calls it asynchronous and account for the change in execution nature in things like loops. ↩︎
The only reason I can think of is because there are some edge cases that I haven’t considered but which the Node team ran into. ↩︎
For larger databases, it streams the tables in but that’s outside the scope of this post. ↩︎

Snowpack Hot Module Replacement (HMR) from scratch with vanilla JavaScript

mail@ar.al (Aral Balkan) — Thu, 31 Dec 2020 17:13:17 +0000

If you watched Rich Harris’s SvelteKit preview, you were probably swooning over the state-maintaining source updates he demoes in his talk. These are thanks to Snowpack’s support for hot module replacement via esm-hmr. In this short post, I want to explain the concept from scratch using Snowpack and vanilla JavaScript.

You can either type in the examples or, if you’re feeling lazy, clone the source code and follow along.

Set up the project

First, set up your project folder and initialise it as a Node project.

mkdir snowpack-hmr
cd snowpack-hmr
npm init -f

Next, install Snowpack.

npm i --save-dev snowpack

Finally, initialise Snowpack to create a default snowpack.config.js file in your project folder:

npx snowpack init

(We’re not going to customise the configuration in this example so we could have skipped this step but it’s good for you to know how to do it for when you need to in your own projects.)

Define the entry point

Create a basic HTML page in your project folder called index.html with the following content:

<!doctype html>
<html lang='en'>
<head>
  <meta charset='utf-8'>
  <meta name='viewport' content='width=device-width, initial-scale=1.0'>
  <title>Snowpack Hot Module Replacement (HMR) from scratch with vanilla JavaScript</title>
</head>
<body>
  <h1>Snowpack Hot Module Replacement (HMR) from scratch with vanilla JavaScript</h1>
  <p>This page has been open for <span id='timer'>0</span> seconds.</p>
  <p id='message'></p>
  <script type="module" src="./index.js"></script>
</body>
</html>

Then, run the Snowpack development server:

npx snowpack dev

Now, go to http://localhost:8080 and you should see the basic web page rendered in your browser. If you change anything in the source and save, your page will do a full reload.

The timer is not active yet as we haven’t created the index.js module that we’ve asked the browser to include. So let’s do that next.

Add core functionality

Create the index.js module with the following initial content:

let timer = 0
let intervalId = null

export function setState(state) {
  timer = state.timer
}

function create() {
  intervalId = setInterval(() => {
    timer++
    document.getElementById('timer').innerText = timer
  }, 1000)

  message.innerText = 'Change this message and save the source file to see state-maintaining hot module replacement in action.'
}

function destroy () {
  clearInterval(intervalId)
}

create()

Refresh the page in your browser and you should see that the timer is now active and starts counting up the seconds since the page was loaded.

If you save the source file, you should see that the whole page reloads again and that the timer’s value resets. This is because we haven’t added any code to our module to handle hot module replacement. We shall do that a little later but first, how about we make things look a little less plain?

Add a spot of eye candy

Create a file called index.css in your project folder with the following content:

html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
}

body {
  max-width: 760px;
  margin-left: auto;
  margin-right: auto;
  text-align: center;
  font-size: 1.75em;
  padding: 0 0.5em;
}

h1 {
  font-weight: 100;
  font-size: 2em;
}

#timer {
  font-size: 2em;
}

#message {
  font-size: 0.75em;
  font-variant: small-caps;
  margin-top: 2em;
  padding: 0.5em;
  color: white;
  background-color: seagreen;
}

Import it by adding the following line to the top of your index.js file:

import './index.css'

Ah, that looks a bit better.

First taste of hot module replacement

Now, change the background-color property of the #message rule from seagreen to hotpink and save and you’ll get your first taste of hot module replacement with Snowpack as the styles update without a full reload of the page.

You didn’t have to do anything for this, Snowpack handles hot module replacement of CSS modules automatically for you. So that’s nice.

Given it’s New Year’s Eve as I’m writing this, let’s also install a Node module for creating some festive confetti.

npm i canvas-confetti

Next, import the new module at the top of your index.js file:

import confetti from 'canvas-confetti'

And then run it at the top of the create() method:

function create() {
  confetti()
  //…
}

Yay, confetti!

Also, did you notice… Snowpack magically enabled you to import your Node module using ECMAScript module syntax¹. If you want to learn more about how that works under the hood, check out the esinstall module.

Implement hot module replacement

Although Snowpack can handle hot module replacement automatically for your CSS imports, it needs your help with JavaScript modules.

To implement hot module replacement for your module, add the following code to the top of your index.js file:

if (import.meta.hot) {
  import.meta.hot.accept(({module}) => {
    // Restore state on the new module.
    module.setState(import.meta.hot.data)
  })
  import.meta.hot.dispose(() => {
    // Save state before destroying the current module.
    import.meta.hot.data = { timer }
    destroy()
  })
}

Now, whenever you save index.js, the timer will keep its state.

Wait, what?

Right, it might feel like I just pulled a “how to draw an owl” meme on you so let’s break it down to see what’s happening here.

Again, more slowly this time

Basically, Snowpack will not just update your module for you without your permission because it could lead to all sorts of issues (with state, dependencies, etc.) So you have to expressly tell it that you want hot module replacement for your module.

You opt into hot module replacement by calling import.meta.hot.accept().

In fact, if you want to, replace the code we just added with that one line, refresh your page, and see what happens when you save index.js a couple of times. The timer starts jumping around. That’s because the module is being loaded over and over so new timers are being created that start from zero and compete with each other.

That’s why we need the code we added earlier.

It has three parts:

The initial conditional
```
if (import.meta.hot) {
  // …
}
```
We first check if import.meta.hot is defined and only carry out the hot module replacement logic if it is. This means that not only will the code not run in production but, if you’re using a bundler like esbuild, it will also get tree shaken out of your final bundle.
Housekeeping and saving state
```
  import.meta.hot.dispose(() => {
    // Save state before destroying the current module.
    import.meta.hot.data = { timer }
    destroy()
  })
```
The dispose() method on import.meta.hot gets called before the current module is replaced and before accept() is called. This is a good place to store any state you want maintained and to do any housekeeping if you need to.

You forward state by setting the data property on import.meta.hot. This property is also available from the accept() method, which is where you can restore state.

For housekeeping, we call the destroy() method which clears the timer’s interval:
```
function destroy () {
  clearInterval(intervalId)
}
```
Restoring state
```
import.meta.hot.accept(({module}) => {
  // Restore state on the new module.
  module.setState(import.meta.hot.data)
})
```
In the accept() method, we use the reference to the new module we’re passed to call its setState() method with the state we saved earlier in import.meta.hot.data during the dispose() method.

In our simple example, the setState() function merely restores the current timer value:
```
export function setState(state) {
 timer = state.timer
}
```

And that’s it!

Limitations and caveats

Of course, this is just a basic, harcoded example to demonstrate the core concepts.

In a framework like SvelteKit, you will most likely never have to write code like this by hand as the framework will handle it for you.

Also, state preservation during hot module reloading is not foolproof. To demonstrate this, change the initial value of the timer variable and see what happens. e.g.,

let timer = 60

That’s right, nothing happens unless you refresh the page. That’s because our state saving and restoring code ignores that value. It was this issue that caused the svelte-hmr project to make state preservation an opt-in feature.

I look forward to seeing how all this is implemented in SvelteKit once it’s released.

In the meanwhile, I hope this gives you an overview of how hot module replacement works with ECMAScript modules.

Happy new year!

Now, save your index.js file, enjoy the confetti, and here’s wishing you and your loved ones a happy new year and a 2021 that is better than the trainwreck that was 2020.

💕️

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

You can also use Skypack, their Content Delivery Network (CDN) to import the package directly but I would strongly advise against using it for reasons that I detail in my previous post: Skypack: Backdoor as a Service? ↩︎

Skypack: Backdoor as a Service?

mail@ar.al (Aral Balkan) — Wed, 30 Dec 2020 12:48:57 +0000

There’s some exciting work being done with projects like SvelteKit to reduce complexity and improve the developer experience when building web applications.

At the heart of these efforts are basically three core elements:

A front-end framework like Svelte or Vue.
Native browser support for ECMAScript Modules.
Hot Module Replacement (e.g., see esm-hmr).
At deployment, bundling as usual with a tool like esbuild or rollup.

Two development tools that support this workflow are Vite, from the folks behind Vue, and Snowpack, from the folks behind a Content Delivery Network (CDN) called Skypack.

Snowpack is interesting in that it’s what Rich Harris is using to build the aforementioned SvelteKit.

What is your business model?

A few weeks ago, while looking into all this, I found a quote from Evan You, author of Vue and tweeted it:

“This also touches on something that bothers me a bit with Snowpack: that it’s owned by a startup that is using it as a strategy to promote its paid product. I just don’t feel comfortable betting the future of Vue tooling on Snowpack.”

In the ensuing discussion Evan asked about Skypack’s business model. As did Rich. I was also very interested in finding out as there was no mention of it on their About page. The reply from Fred K. Schott, their founder and CEO, didn’t really shed much light on it:

“Definitely! We’re building a business around the CDN & catalog discovery, but far from the the “pay $5 a month to unlock Snowpack Pro!” that Evan insinuated”

So I followed up:

“Hey Fred, I’m not sure I understand how you’ll be monetising the CDN (& I’m afraid I don’t know what ‘catalog discovery’ is). If you don’t mind, could you tell us who’ll be paying you and for what. Also, do you have any VC/angel investment and are you looking for an exit? Thanks!”

I didn’t get an answer.

Fast forward to yesterday when I stumbled onto a partial answer to my question in Skypak’s documentation for Skypack Pro, which “is currently in invite-only beta:”

Pro features let you customize Skypack to your organization, including: Private Registry Support Private Asset Hosting Custom Subdomain Support Service Level Agreement (SLA) Audit Logs & Customer Support

OK, so fair enough, it’s the npm model (core product is free, charge enterprises for pro features, then eventually get bought out by Microsoft).

Fair enough.

Given the swamp that is Silicon Valley surveillance capitalism, it’s not the worst business model a mainstream company could have. (Of course, it’s still part of the same rotten system but we’re talking about a mainstream project here, they’re out to make money and get rich, not save the world.)

So I kept looking into it a bit more and that’s when I realised that Skypack as a CDN has a huge glaring security hole that should really be the only thing anyone is talking about whenever its name is mentioned until it is fixed:

Skypack does not support subresource integrity (SRI).

Backdoor as a Service

There isn’t a single mention of subresource integrity in Skypack’s documentation and, given how it works, I’m not sure they can provide it without changing some of their core offering:

“Most JavaScript CDNs have to serve one file to all users … Skypack is the first CDN to automatically address this problem by skipping unnecessary compilation & polyfilling for modern browsers. When a user visits your site with a modern, up-to-date browser they’ll get a smaller, faster JavaScript response optimized for their exact browser.”

Why is subresource integrity such a big issue? Because without it, you can’t be sure what code you’re serving the people who use your app.¹

Remember that the code being served from Skypack becomes part of your app.

If I were In-Q-Tel right now, I’d be drooling as I wrote a check with lots of zeros in it for the Skypack folks because widespread use of Skypack would be any national security agency’s wet dream. Imagine being able to inject any code into any web application at any time to obtain login credentials, etc.

This isn’t even a backdoor. This is a wide open frontdoor. It’s basically Backdoor as a Service.

Packing it in

With the upcoming Snowpack v3.0 providing seamless support for Skypack via the experiments.source flag, more people will no doubt be interested in trying Skypack.

Until Skypack addresses this security issue in a satisfactory fashion (which I cannot imagine them not doing given the severity of it), I would highly recommend that you exercise caution and stick to only including code you can verify for yourself in your applications.

As for the Small Web stuff we’re working on, I’m still iterating on the developer experience and I’m going to continue looking into Snowpack (you can use Snowpack without Skypack), SvelteKit, Vite, etc., but I have a feeling that we will end up going with a custom solution based on integrating some of the core libraries (like Svelte, esm-hmr, etc.) into a fork of Site.js.

Update (Feb 4, 2021): Just stumbled on a blog post Terence Eden wrote a little over two years ago titled Dynamic JavaScript and SRI, within the context of a service called Polyfill.io, that voices the same concerns and reaches the same conclusions. Also, I just opened a discussion about this on the Snowpack forum in hopes that we can find a way to add SRI to Skypack.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Subresource integrity is especially important for the Small Web, where we do not trust servers at all and so keep secrets only on the client. It is, thus, of primary importance that we know exactly what code is served to the client. Not just from third parties but even from our own server. In fact, even subresource integrity isn’t enough by itself as it doesn’t verify the root source (think: index.html) so we cannot use it to create a root of trust. Instead we need to pair it with out-of-band verification of the root source, perhaps via the use of a browser extension. ↩︎

Why I wrote 152 extra lines of code just to do the same thing (and why I’d do it again today)

mail@ar.al (Aral Balkan) — Sat, 24 Oct 2020 19:09:17 +0100

Who else remembers printing out code on a dot matrix printer? Ah, those were the days… (Image courtesy Arnold Reinhold.)

At the end of the week, I added the following regular expressions to JavaScript Database (JSDB), to sanitise queries:

// Disallow list.
this.query = this.query.replace(/[;\\\+\`\{\}\$]/g, '')

// Allow list.
let sieve = this.query
  .replace(/valueOf\..+?\.toLowerCase()\.(startsWith|endsWith|includes|startsWithCaseInsensitive|endsWithCaseInsensitive|includesCaseInsensitive)\(.+?\)/g, '')
  .replace(/valueOf\..+?\.(startsWith|endsWith|includes|startsWithCaseInsensitive|endsWithCaseInsensitive|includesCaseInsensitive)\(.+?\)/g, '')
  .replace(/valueOf\.[^\.]+?\s?(===|!==|<|>|<=|>=)\s?([\d_\.]+\s?|'.+?'|".+?"|false|true)/g, '')
  .replace(/\|\|/g, '')
  .replace(/\&\&/g,'')
  .replace(/['"\(\)\s]/g, '')

They’re pretty straightforward.

(As far as regular expressions go, that is.)

The first expression strips out characters that can be used to carry out arbitrary code execution via injection attacks and the rest are used in a sieve to remove all the input we expect to be there.

We then check to see if anything is left behind in the sieve. If the sieve is not empty, we treat it as a possible attack and return an empty result set, as shown below.

if (sieve !== '') {
  // Sieve not empty, reject.
  this.#cachedResult = []
} else {
  // Run the query.
  this.#cachedResult = this.data.filter(valueOf => {
    try {
      return eval(this.query)
    } catch (error) {
      // Eval failed. Possible injection attack, return false.
      return false
    }
  })
}

I’ve simplified the comments in the code snippets but, otherwise, they are as I wrote them a couple of days ago.

JSDB, now with 20% more code!

Now, that a look at what that code looks like as of yesterday (and keep in mind that it does exactly the same thing).

The first thing you might notice is that it is now a 300-line file, including comments.

Wow, what happened?

Two things:

I am now composing my regular expressions from constants instead of using regular expression literals. e.g.,

static join() { return Array.from(arguments).join('') }
static anyCharacter = '.'
static oneOrMore    = '+'
static nonGreedy    = '?'

static get oneOrMoreCharactersNonGreedy () {
  return this.join(
    this.anyCharacter,
    this.oneOrMore,
    this.nonGreedy
  )
}

I now get the values for the allow list directly from the source (from QueryOperators.js which is the same code that is used by the query generation methods).

class QueryOperators {
  static STRICT_EQUALS = '==='
  static STRICT_DOES_NOT_EQUAL = '!=='

  static RELATIONAL_OPERATORS = {
    is: this.STRICT_EQUALS,
    isEqualTo: this.STRICT_EQUALS,
    equals: this.STRICT_EQUALS,

    isNot: this.STRICT_DOES_NOT_EQUAL,
    doesNotEqual: this.STRICT_DOES_NOT_EQUAL,

    isGreaterThan: '>',
    isGreaterThanOrEqualTo: '>=',
    isLessThan: '<',
    isLessThanOrEqualTo: '<='
  }

  static FUNCTIONAL_OPERATORS = [
    'startsWith',
    'endsWith',
    'includes',
    'startsWithCaseInsensitive',
    'endsWithCaseInsensitive',
    'includesCaseInsensitive'
  ]

  static get uniqueListOfRelationalOperators () {
  return Array.from(new Set(Object.values(QueryOperators.RELATIONAL_OPERATORS)))
  }
}

// …

class RE {
  // …
  static anyFunctionalOperator = this.anyOf(...QueryOperators.FUNCTIONAL_OPERATORS)
  static anyRelationalOperator = this.anyOf(...QueryOperators.uniqueListOfRelationalOperators)
}

// …

class Allowed {
  // …
  // Statements in the form valueOf.<property> === <value>, etc.
  // /valueOf\.[^\.]+?\s?(===|!==|>|>=|<|<=)\s?([\d\._]+\s?|'.+?'|".+?"|false|true)/g
  static relationalOperations = globalRegExp(
    RE.literal.valueOfDot,
    RE.oneOrMoreCharactersNonGreedyThatAreNotDots,
    RE.zeroOrMoreWhitespace,
    RE.anyRelationalOperator,
    RE.zeroOrMoreWhitespace,
    RE.anyValidRelationalOperationValue
  )
}

The effects, subsequently, are three-fold:

Two positives

The intent and clarity of the code is improved.
The maintainability of the code is improved and it is now safer as I can update the query generation code without worrying that I am going to forget to update the sanitisation code or that I might make typos while copying values over manually into regular expression literals.

And one negative

I’ve added 152 lines of extra code to the project.¹

So do the first two positive effects outweigh the cost of the third, negative one? Especially on a tool that had 755 lines of code to begin with? (The additional lines of code amount to a 20% increase of the codebase).

Definitely.

Fewer lines of code doesn’t always mean better code.

While less code generally means an easier to maintain and safer application, this axiom does not exist in a vacuum. In over three decades of programming, I’ve come to respect that clarity of intent is the most important factor when writing code.²

Is what I wrote what I had actually meant to write?

The following is confusing, especially to someone who isn’t versed in regular expressions:

/valueOf\.[^\.]+?\s?(===|!==|<|>|<=|>=)\s?([\d_\.]+\s?|'.+?'|".+?"|false|true)/g

Whereas the code below, that composes the regular expression from constants, can be read by almost anyone – even a non-programmer – to get the gist of what’s happening:

static relationalOperations = globalRegExp(
  RE.literal.valueOfDot,
  RE.oneOrMoreCharactersNonGreedyThatAreNotDots,
  RE.zeroOrMoreWhitespace,
  RE.anyRelationalOperator,
  RE.zeroOrMoreWhitespace,
  RE.anyValidRelationalOperationValue
)

Perhaps even more importantly, if you look at where the constants are mapped to their regular expression counterparts, you can immediately see whether the regular expression I wrote actually expresses what I thought I was writing because I use the following declarative form:

const thisIsWhatIWantToArchive = 'This is how I think I archive it'

So, now, anyone who knows regular expressions can read through the code and immediately see if there is a mistmatch between what I think I’m asking the computer to do and what I’m actually asking it to do. Furthermore, it is also easy to spot errors in the former for those cases where what I’m trying to do itself is not the right thing.

class RegularExpressionLiterals {
  static dot = '\\.'
  // …
}

class RE {
  // Concatenates into a string.
  static join () {
    return Array.from(arguments).join('')
  }

  // Creates negated set (e.g., [^xy]).
  static negatedSetOf () {
    return `[^${Array.from(arguments).join('')}]`
  }

  static literal = RegularExpressionLiterals
  static nonGreedy    = '?'
  static oneOrMore    = '+'

  // [^\\.]+?
  static get oneOrMoreCharactersNonGreedyThatAreNotDots () {
    return this.join(
      this.negatedSetOf(
        this.literal.dot
      ),
      this.oneOrMore,
      this.nonGreedy
    )
  }
}

In conclusion

If you’ve made it this far, I’d really appreciate it if you could take a quick look at the JSDB Query Sanitiser (and maybe even the much simpler String sanitiser in the JSDF parser) and see if you can spot anything I’ve missed. If you do notice anything amiss, I’d appreciate it if you could open an issue and let me know.

What? Did I write this blog post just to get more eyes on the security code for my new database?

Tsk, tsk…

Maybe ;)

PS. I do hope you enjoy using JSDB. I’m currently working on integrating it into Site.js for use in Small Web sites and apps.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

The statistics are compiled using scc. ↩︎
I am still happy to see that the whole of JSDB is just ~900 lines of code (not counting tests, and 1,443 lines counting tests with 100% code coverage). ↩︎

Introducing JSDB

mail@ar.al (Aral Balkan) — Tue, 20 Oct 2020 16:23:32 +0100

Yesterday, I released version 1.0 of JavaScript Database (JSDB), a new database for Node.js optimised for use with Small Web sites and apps.

It does things a little differently to other databases.

It’s an in-memory JavaScript database that persists to an append-only transaction log.

And it’s all JavaScript.

What I mean by that is that even the data is stored as JavaScript code. Not JSON. JavaScript.

(It’s also very fast and easy to use.)

Get started

You will need Node.js installed¹.

Set up your project and install JSDB.

# Create a folder to hold your project and switch to it.
mkdir lovecraft-country
cd lovecraft-country

# Initialise it with a package.json file.
npm init -f

# Install JSDB.
npm i @small-tech/jsdb

# Create a file to hold your code.
touch index.js

Open a database and write something to it.

const JSDB = require('@small-tech/jsdb')

// Open your database (creating it if it doesn’t exist).
// It will be stored in a directory called “db”.
const db = JSDB.open('db')

// Create a “table” on it. Tables can be arrays or objects.
// This is a list of the Lovecraft Country episodes and
// their ratings from IMDB. (It’s missing the last three.)
if (!db.episodes) {
  db.episodes = [
    {title: 'Sundown', rating: 8.4},
    {title: 'Whitey’s on the Moon', rating: 7.1},
    {title: 'Holy Ghost', rating: 7.7},
    {title: 'A History of Violence', rating: 7.6},
    {title: 'Strange Case', rating: 7.1},
    {title: 'Meet me in Daegu', rating: 8.4},
    {title: 'I am.', rating: 7.1}
  ]
}

Run it.

node .

In your terminal, you should see output similar to the following:

💾 ❨JSDB❩ No database found at …lovecraft-country/db; creating it.
💾 ❨JSDB❩ Creating and persisting table episodes…
💾 ❨JSDB❩  ╰─ Created and persisted table in 2.169 ms.
💾 ❨JSDB❩ Table episodes initialised.

The table is created and persisted into a file called db/episodes.js

Data as code

To dispel the magic and see how the sausage is made, take a look at what’s inside it this file by running the following command:

cat db/episodes.js

You should see something like this:

globalThis._ = [ { title: `Sundown`, rating: 8.4 }, { title: `Whitey’s on the Moon`, rating: 8.4 }, { title:
`Holy Ghost`, rating: 8.4 }, { title: `A History of Violence`, rating: 8.4 }, { title: `Strange Case`, rating: 8.4 }, { title: `Meet me in Daegu`, rating: 8.4 }, { title: `I am.`, rating: 8.4 } ];
(function () { if (typeof define === 'function' && define.amd) { define([], globalThis._); } else if (typeofmodule === 'object' && module.exports) { module.exports = globalThis._ } else { globalThis.lovecraftCountryEpisodes = globalThis._ } })();

Yes, that’s plain-old JavaScript. Not JSON. JavaScript.

And it’s a bit hard to read. If you were to prettify it, this is what you’d get:

globalThis._ = [
  { title: `Sundown`, rating: 8.4 },
  { title: `Whitey’s on the Moon`, rating: 8.4 },
  { title: `Holy Ghost`, rating: 8.4 },
  { title: `A History of Violence`, rating: 8.4 },
  { title: `Strange Case`, rating: 8.4 },
  { title: `Meet me in Daegu`, rating: 8.4 },
  { title: `I am.`, rating: 8.4 }
]
;(function () {
  if (typeof define === 'function' && define.amd) {
    define([], globalThis._)
  } else if (typeofmodule === 'object' && module.exports) {
    module.exports = globalThis._
  } else {
    globalThis.lovecraftCountryEpisodes = globalThis._
  }
})()

This is how JSDB stores data. It stores it as code in a format I call JavaScript Data Format (JSDF).

JavaScript Data Format (JSDF)

The first line is a single assignment of all the data that existed in the table when it was created or last loaded. In this case, it is basically the same array declaration you wrote to create the table.

The second line is a UMD-style declaration. What this means is that you could, if you wanted to, use require() in Node to load a JSDB table in directly or even load a JSDB table in using a script tag from the browser.

For example, create a file called index.html in the same folder with the following content:

<script src="db/episodes.js"></script>
<h1>Lovecraft Country</h1>
<p>Episodes and ratings from <a href='https://www.imdb.com/title/tt6905686/episodes?season=1'>IMDB</a>:</p>
<ul>
<script>
  episodes.forEach(episode => {
    document.write(`<li><strong>${episode.title}</strong> ⭐ ${episode.rating}</li>`)
  })
</script>
</ul>

Display this web page in your browser using something like Site.js and you should see the following.

A list of the first seven Lovecraft Country episodes from the first season.

Important security note

JSDF is not a data exchange format.

Since JSDF is made up of JavaScript code that is evaluated at run time, you must only load JSDF files from domains that you own and control and have a secure connection to.

To exchange data, use JSON, that’s what it’s for.

Updating data

So what happens when we update the existing data?

To see this, let’s add the missing episodes from the first season to our database and watch what happens to the contents of the table as we do.

Start by opening up an interactive Node.js session in one Terminal window (type node) and tailing (following) the database table from another one:

tail -f db/episodes.js

First, let’s open our database again in the interactive Node.js session:

JSDB = require('@small-tech/jsdb')
db = JSDB.open('db')

Then, let’s add the missing episodes for the first season, one at a time:

db.episodes.push ({title: 'Jig-a-Bobo', rating: 8.4})
db.episodes.push ({title: 'Rewind 1921', rating: 8.6})
db.episodes.push ({title: 'Full Crcle [sic]', rating: 7.0})
db.episodes.push ({title: 'Happy ending', rating: 10.0})

As you add them, you should see them appear at the end of the table. When you’ve added all four, your table should resemble the following:

// first line is the same as before
// second line is the same as before
_[7] = { title: `Jig-a-Bobo`, rating: 8.4 };
_[8] = { title: `Rewind 1921`, rating: 8.6 };
_[9] = { title: `Full Crcle [sic]`, rating: 7 };
_[10] = { title: `Happy ending`, rating: 10 };

And now you know how JSDB stores data that gets added or changed: it adds the code to affect that change to the end of the table. This is what makes it an append-only transaction log. And this is what keeps writes fast (in the single digit milliseconds for the most part on my development laptop) as they’re being streamed to the end of the file.

Ah, but we’ve made some errors, so let’s correct them.

First of all, there is no 11th episode called “Happy Ending”, so let’s remove that from the list:

db.episodes.pop()

Finally, we made an intentional spelling mistake in the title of episode 9. Let’s fix it in our interactive Node.js session:

db.episodes.where('title').includes('Full').getFirst().title = 'Full Circle'

Woah! What’s all that new stuff?

It’s a query in a language I call JavaScript Query Language (JSQL).

We’ll see a bit more of JSQL a little later on.² But for now, let’s take a penultimate look at the JSDF file for the episodes table. Here’s what it should look like now:

globalThis._ = [ { title: `Sundown`, rating: 8.4 }, { title: `Whitey’s on the Moon`, rating: 7.1 }, { title:
`Holy Ghost`, rating: 7.7 }, { title: `A History of Violence`, rating: 7.6 }, { title: `Strange Case`, rating: 7.1 }, { title: `Meet me in Daegu`, rating: 8.4 }, { title: `I am.`, rating: 7.1 } ];
(function () { if (typeof define === 'function' && define.amd) { define([], globalThis._); } else if (typeofmodule === 'object' && module.exports) { module.exports = globalThis._ } else { globalThis.episodes = globalThis._ } })();
_[7] = { title: `Jig-a-Bobo`, rating: 8.4 };
_[8] = { title: `Rewind 1921`, rating: 8.6 };
_[9] = { title: `Full Crcle [sic]`, rating: 7 };
_[10] = { title: `Happy ending}`, rating: 10 };
delete _[10];
_['length'] = 10;
_[9]['title'] = `Full Circle`;

Hmm, all our changes are there but do you see a problem?

Because I do.

We’ve deleted the last episode but the actual data for it is still in our table, on disk. In this case, that’s not much of a problem. But what if instead of a TV episode, it was a private piece of information that you wanted to delete? For the sake of privacy, it’s important that data we think is deleted is actually deleted. This is a problem that all append-only logs suffer from due to their inherent nature and it’s one that JSDB solves using compaction.

Compaction

By default, every time a table is loaded, any changes that were made to it in the last session are compacted into the single-line initialisation statement on the first line. This compaction process is important for various reasons:

It preserves privacy by actually deleting and updating changed data everywhere.
It makes subsequent loads faster as only a single assignment statement is run to create the data graph in memory instead of possibly hundreds or thousands of statements.

To see compaction in action, exit your current interactive Node.js session and start a new one. Then, open the database again:

JSDB = require('@small-tech/jsdb')
db = JSDB.open('db')

You should see output that resembles the following:³

💾 ❨JSDB❩ Loading table episodes…
💾 ❨JSDB❩  ╰─ Loading table synchronously.
💾 ❨JSDB❩  ╰─ Table loaded in 1.515 ms.
💾 ❨JSDB❩ Compacting and persisting table episodes…
💾 ❨JSDB❩  ╰─ Compacted and persisted table in 4.371 ms.
💾 ❨JSDB❩ Table episodes initialised

Now, if you take one final look at the episodes table, you should see that it’s back to being two lines and that the first line, shown below, includes all the changes we made so far:

globalThis._ = [ { title: `Sundown`, rating: 8.4 }, { title: `Whitey’s on the Moon`, rating: 7.1 }, { title:
`Holy Ghost`, rating: 7.7 }, { title: `A History of Violence`, rating: 7.6 }, { title: `Strange Case`, rating: 7.1 }, { title: `Meet me in Daegu`, rating: 8.4 }, { title: `I am.`, rating: 7.1 }, { title: `Jig-a-Bobo`,rating: 8.4 }, { title: `Rewind 1921`, rating: 8.6 }, { title: `Full Circle`, rating: 7 } ];

JavaScript Query Language (JSQL)

You got a tiny introduction to JavaScript Query Language (JSQL) in the previous section. Now that we have all the episodes correctly entered in our database, let’s use JSQL one last time to get the names of the episodes that have ratings higher than 8 stars.

In the same interactive Node.js session as before, enter the following query:

// Get the highly rated episodes.
highlyRatedEpisodes = db.episodes.where('rating').isGreaterThan(8).get()

// Print them out nicely.
highlyRatedEpisodes.forEach(episode => console.log(`${episode.title} (⭐ ${episode.rating})`))

You should see the following list:

Sundown (⭐ 8.4)
Meet me in Daegu (⭐ 8.4)
Jig-a-Bobo (⭐ 8.4)
Rewind 1921 (⭐ 8.6)

I hope this has whetted your appetite and that you’ll have a play with JSDB and consider how you could use it in your own apps.

Also, I’m in the process of integrating it into Site.js so you can use it when creating Small Web sites and apps.

You can find further details and examples in the JSDB documentation.

Enjoy and, no matter what you do, please do not use it to farm people for their data.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

If you don’t have Node.js installed, the easiest way to do so is by using nvm. ↩︎
Yes, we could have just written db.episodes[9].title = 'Full Circle' but where’s the fun in that? ↩︎
When running in the interactive shell, you might also see the following message: Expression assignment to _ now disabled. That’s just the shell telling you that the global _ variable is being used by JSDB. Usually, the shell uses this to contain the result of the last expression it evaluated. This is expected behaviour. ↩︎

What if data was code?

mail@ar.al (Aral Balkan) — Wed, 23 Sep 2020 16:05:49 +0100

Code? Data? Data? Code?

Update: This is now a thing and it’s not called WhatDB but JavaScript Database (JSDB). Read more about it in my Introducing JSDB post.

This is a little idea I’m throwing around for WhatDB, the transparent in-memory JavaScript database for local server data that I’m working on for Site.js, the Small Web construction set:

What if I don’t persist the database as a serialised JSON structure but as the series of changes that created it instead?

What do I mean by that? Take a look at this quick proof-of-concept:

Example

write.js

const fs = require('fs')

const writeStream = fs.createWriteStream('data.js')
writeStream.write('const data = []\n')
writeStream.write('module.exports = data\n')

for (let i = 0; i < 1000; i++) {
  writeStream.write(`data.push(${i})\n`)
}

writeStream.close()

read.js

const data = require('./data.js')
console.log(data)

First, run write.js, then run read.js. You should see the 1,000 item array logged to the console.

Why?

Even though WhatDB is for small amounts of data (since the Small Web discourages storage of privileged data on servers), having to execute a full JSON serialisation and write to disk on every write does not sit right with me.

Stringification is a bitch

Don’t get me wrong, it is definitely fast enough for smaller data sets. But even with 100,000 records of my test dataset I didn’t want to block the event loop in Node.js for the 40-50ms that it takes to stringify the database on every write.

When streaming isn’t enough (to lower CPU usage)

So I implemented streaming JSON serialisation using the excellent json-stream-stringify and that reduced the loop delay to the sub-ms region. But I was still maxing out CPU.

On a 1,000,000 record database (again, overkill for Small Web), that meant 100%+ CPU for the 40 seconds or so that serialisation was taking place. Actually writing the serialised data to disk, atomically, was trivial in comparison (a couple of seconds).

So at this point I have something that works for its intended use case and I can just call it a day and move on but I can’t. Even if it’s within acceptable parameters, it feels wasteful. (Is there such a thing as a code conscience? Who knows…)

Anyway, so today I started thinking out loud on the fediverse about how I could perform delta (partial) edits on the serialised data string instead of serialising the whole thing every time and that led to us throwing out ideas spanning CRDTs, piece tables, and even Kappa architectures before I came to realise that all that would be overkill and instead…

A simple idea for a simple database

What if I just stored all the instructions that are needed to create each table/object graph instead of a serialised copy of the data itself?

In fact, what if the data itself was a Node module?

This is the point that some folks will stare at me with horror in their eyes. The sacrilege… the heresy! What will they do, I wonder, when they learn that I’ve done far, far darker things…¹

Properties of such a system

Might be slow during initial object graph creation.

This is fine given that the initial object graph is created at server launch alongside other synchronous tasks. Performance isn’t a concern at that point.
Will take up more disk space than serialisation.

That’s ok, given disk space isn’t the bottleneck in an in-memory Node.js database, Node’s memory limitations are.
Fast reads.

This is a property of having an in-memory data store that is not affected by this decision.
Fast writes.

Since the file format is basically an append-only log, streaming writes are quick and easy on the event loop and system resources.
Require snapshots?

Since we’re losing the atomic writes due to the appends, it would make sense to have periodic snapshots.
Require compaction?

We could compact on initial load (on server start²) to ensure that deleted data is actually deleted. Compaction would basically result in a single-line JSON.parse statement on the full serialised table.

Can’t wait to get my hands dirty and experiment with this in WhatDB.

Thanks

Thanks to everyone on the fediverse (and that other proprietary platform) for being my rubber duck and for providing me with great resources and food for thought. Also, I just learned that what I’m doing has a name: object prevalence.

This is not the first time I’ve played with a similar concept. Back in the Flash days (yes, I’m that old), I made a data format called SWX that basically allowed you to load data wrapped in the native SWF format. Thank goodness this time I don’t have to clean-room reverse-engineer SWF bytecode. If you’re into history, you can read about SWX in the chapter R. Jon MacDonald wrote on it for our The Essential Guide to Open Source Flash Development book. ↩︎
Or every nth restart or on restart after n days, etc. ↩︎

What is the Small Web?

mail@ar.al (Aral Balkan) — Fri, 07 Aug 2020 13:03:28 +0100

Updated June 19th, 2023

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download Small Is Beautiful #23 directly, and watch it with your favourite video player.

Small Is Beautiful (Oct, 2022): What is the Small Web and why do we need it?

Today, I want to introduce you to a concept – and a vision for the future of our species in the digital and networked age – that I’ve spoken about for a while but never specifically written about:

The Small Web.

To understand what the Small Web is, let’s compare it to the Big Web. In other words, to the centralised Web we have today.

The Big Web

You are allowed to rent space on Megacorp’s servers in exchange for giving up your privacy, freedom of speech, and other human rights.

The Big Web is the centralised web; it is a web in the sense of a spider’s web. The spiders that sit at its centre waiting to suck you dry are Big Tech people farmers like Facebook, Google, etc.

The Big Web has “users” – a term Silicon Valley has borrowed from drug dealers to describe the people they addict to their services and exploit. We farm users in server farms. On the Big Web, we can fit thousands of users into a single server and Megacorps “scale” to run thousands upon thousands of servers in their farms.

On the Big Web, you never own your own home. You must rent your home from Megacorps. Most often, you don’t have to pay for your home using money. You pay for it by forfeiting your privacy, freedom of speech, and your other human rights. Collectively, we pay for it by forfeiting a democratic future.¹

The mass surveillance and factory farming of human beings on a global scale is the business model of people farmers like Facebook and Google. It is the primary driver of the socioeconomic system we call surveillance capitalism.

The Small Web

On the Small Web, you own your own home.

The Small Web, quite simply, is the polar opposite of the Big Web. It applies the Small Technology principles to the web.

The Small Web is Your Web

The Small Web is for people (not startups, enterprises, or governments). It is also made by people and small, independent organisations (not startups, enterprises, or governments²).

On the Small Web, you (and only you) own and control your own home (or homes).

The Small Web is the Single Tenant Web

Small Web applications and sites are single tenant. That means that one server hosts one application that serves just one person: you. On the Small Web, we do not have the concept of “users”. When we refer to people, we call them people.

Another fundamental difference between the Big Web and Small Web is that on the Big Web we trust servers and distrust clients whereas on the Small Web, we distrust servers and trust clients. We treat servers as dumb delivery mechanisms. The client – under the control of the person who owns the site or app – is the only trusted environment.

Our greatest usability challenge on the Small Web is making the ownership and control of your own web site or application as seamless as possible. It must be done in a manner that does not require any technical know-how whatsoever. Our goal is to make owning and maintaining your own home on the web as easy as renting from a Megacorp without all of the toxic ramifications the latter entails.

We are not there yet but the end (beginning?) is in sight.

Where we are today

Kitten is a complete Small Web development kit (including server).

The first step to building the Small Web is to build tools for developers to empower them to build the Small Web.

That’s why we’re building Kitten at Small Technology Foundation.

Currently, all our developer tools and technical infrastructure comes from Big Tech and the Big Web. They are optimised for creating Big Tech and the Big Web. While we can repurpose some of them for our own uses, we also need tools specifically optimised for building single-tenant web applications and the Small Web.

We’re currently building the tools developers (including us) need to build the everyday tools everyone will use.

In the words of Audre Lorde:

The master’s tools will never dismantle the master’s house.

(In other words, Big Tech’s tools will never dismantle Big Tech’s house. The best thing we can do as developers is to build our own tools so we can build our own houses in our own way.)

Single tenancy affords us a great reduction in complexity that we must take full advantage of if we are to make the Small Web experience as good as (if not better) than the Big Web experience.

The single tenant web is sustainable and scales differently. It does not have economies of scale. It does not scale vertically like the Big Web. It scales horizontally. As the Small Web scales, no single organisation or person scales alongside it. It does not centralise wealth and power.

Where we are heading

The Small Web is not a place; it is a public sphere. It’s the interconnections of individually-owned and controlled sovereign spaces on a global digital network.

Now that Kitten is reaching a level of maturity, we are using it to build a tool called Domain for hosting Small Web sites and applications built with Kitten with the aim of making it as easy to have your own Small Web site at your own domain³ as it is to sign up for a centralised social media platform.

We will be hosting our own instance of Domain at small-web.org. The domain has already been included in the Public Suffix List.

This work has begun and is in the early stages.

This service, once ready, will still initially be aimed at developers. However, with the infrastucture for instantly deploying any Small Web app or web site created, we (and other developers) can then turn our attention to creating beautiful, useful, and perhaps even delightful everyday things for everyday people on the Small Web that adhere to Small Tech principles.

All our work is free and open source (released under AGPL version 3.0) and we invite and encourage others to build similar services and to cooperate with us in igniting the spark of the Small Web.

Our vision for the future

You hear a lot of talk about blockchains and proof of work but this is not what the Small Web is about. Having a billion copies of the same database is not decentralisation. That’s centralisation. If you have a billion people, having a billion – two billion, three billion… – unique databases is decentralisation. In other words, decentralisation without topological decentralisation is bullshit.

When we say the Small Web is decentralised, we are talking about topological decentralisation.

(And in case talk of databases sounds dry, technical, and irrelevant to you, remember that each one of those databases is an artefact of a soul’s exploration of itself and the world around them.)

Our vision for the future is one where every person owns and controls their own place on our shared global network. This is the only way to ensure that we have a public sphere⁴, human rights, and democracy in the digital network era.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Of course, you can rent on the Big Web and from Big Tech by paying both with money and the forfeiture of your human rights. As surveillance capitalists grow in power, they can demand both forms of payment for access to their services. ↩︎
The Small Web can (and should) be funded from our common purse (our taxes) but the organisations building it must be independent of governments. Since that’s not going to happen from day one, we are doing our best to build a bridge from here to there within the limits of the capitalist system we find ourselves in. Our goal is to prove its feasibility and then ask entities like the European Union to support it in the interests of their citizens and the common good. ↩︎
If you want a commercial domain instead of a domain at the public small-web.org suffix, you will be able to register any domain and point it at your server. It is your server, after all. ↩︎
On a digital network, public space is not a place (like Facebook), it is the interconnection of sovereign spaces owned and controlled by individuals. Furthermore, I use ownership not in the capitalist sense of the word but in the very specific sense of “not owned by anyone else.” ↩︎

Live Stream: A web site on your phone with Site.js

mail@ar.al (Aral Balkan) — Sun, 12 Jul 2020 16:47:43 +0100

A sneak peek at hosting a web site on a PinePhone using Site.js 14.2.0 Alpha.

This event has now ended. You can watch the recording above.

Dear Apple, a little help here? How hard can it be to move our developer account to our new not-for-profit?

mail@ar.al (Aral Balkan) — Thu, 02 Jan 2020 11:36:02 +0000

Apple: Computer says no?

Dear Apple,

We’re a tiny two-person not-for-profit. We used to be based in the UK, where we were known as Ind.ie (and incorporated as a not-for-profit called Article 12). We left the UK (for reasons) and now we have a not-for-profit here in Ireland called Small Technology Foundation. It’s still just Laura and me (and our huskamute, Oskar).

We have an app called Better Blocker that’s on sale on the iOS and macOS app stores. Some months, the revenue we get from it helps us cover the rent. Better is listed on your Get Started: Safari Extensions page and has three years of history on the App Store.

The problem is that we are now faced with having to take it off the App Store as we don’t seem to have a way to migrate our developer account to our new not-for-profit. We also tried setting up a new developer account for Small Technology Foundation and moving Better over to it but we cannot since Better uses iCloud and apps that use iCloud cannot be moved between developer accounts.

I can see the logistical difficulty of moving apps between different developer accounts if they use iCloud if this is not a feature that’s already built into the platform at Apple. But what I don’t understand is why our developer account cannot be updated to belong to Small Technology Foundation instead of Article 12. Laura and I are co-founders of both not-for-profits (we are in the process of shutting Article 12 down) and we can prepare and sign any necessary contracts for this to happen.

Can we please update these two fields in your database?

So if anyone at Apple is reading this, it would really help a tiny not-for-profit out if you could help us with this. We have been trying to reach out but we haven’t had any luck in actually speaking to a human being about this.

If we cannot move our developer account over, we are either going to have to remove Better from sale and re-publish it under a separate Small Technology Foundation account (which means that people who bought Better will no longer get updates unless they buy the new app and that we will lose three years of history and most likely our placement on the aforementioned Safari Extensions article on Apple.com) or, if we want to avoid the possible backlash of having to do this, remove Better from sale permanently and focus all our energy on our current projects.

I do hope someone from Apple will get in touch and we can simply update our developer account to migrate it to Small Technology Foundation so we can continue to offer Better for sale and focus on building tools to safeguard human rights and democracy in the digital network age.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

In 2020 and beyond, the battle to save personhood and democracy requires a radical overhaul of mainstream technology

mail@ar.al (Aral Balkan) — Wed, 01 Jan 2020 15:50:00 +0000

As we enter a new decade, humankind faces several existential emergencies:

The climate emergency¹
The democracy emergency
The personhood emergency

In no small part thanks to Greta Thunberg, we’re definitely talking about the first. Whether we’re actually doing anything about it, of course, is very much up for debate².

Similarly, thanks to the rise of the far right around the globe in the shape of (among others) Trump in the US, Johnson in the UK, Bolsonaro in Brazil, Orban in Hungary, and Erdoğan in Turkey, we are also talking about the second, including the role of propaganda (so-called “fake news”) and social media in perpetrating it.

What we seem entirely clueless and ambivalent about is the third even though all the others stem from it and are symptoms of it. It is the emergency without a name. Well, until now, that is.

The personhood emergency

You cannot understand the personhood emergency without understanding the role that mainstream digital and networked technology plays in perpetuating it.

Your TV wasn’t watching you, YouTube is

Traditional – non-digital, non-networked – technology was a one-way broadcast medium. That’s the one thing that a book printed on the Gutenberg press and your analog TV set had in common.

It used to be that when you read a newspaper, the newspaper did not also read you. When you watched TV, your TV did not also watch you (unless you specifically allowed an audience measurement company like Nielsen to attach a ratings meter to your television set, that is).

Today, as you read The Guardian newspaper online, The Guardian – and over two dozen other third-parties, including the aforementioned Nielsen – also reads you. When you watch YouTube, YouTube also watches you.

This is not some tin-foil hat conspiracy theory, it’s simply the business model of mainstream technology. I call this business model people farming. It’s part of the greater socio-economic system we inhabit that Shoshana Zuboff calls surveillance capitalism³.

And it gets worse: Alphabet Inc., which owns Google and YouTube, doesn’t just watch you when you use one of their services but they also follow you around the Web as you go from site to site. Google alone has eyes on 70-80% of the Web.

But they don’t stop there, either. People farmers also buy data from data brokers, share data with other people farmers, and even know when you use your credit card in brick and mortar stores. And they combine all of this information to create profiles of you which are constantly analysed, updated, and improved.

We can consider these profiles to be simulations of us. They contain aspects of us. They can be (and are) used as proxies of us. They contain highly sensitive and intimate information about us. But we do not own them, the people farmers do.

It is not too much of a stretch to say that within this system, we do not fully own ourselves. In such a system, where our very thoughts are at risk of being read by corporations, our very personhood and the concept of self-determination itself is put at risk.

We stand at the precipice of reverting from being people to being property again, hacked via a digital and networked backdoor, the existence of which we continue to deny at our peril. The prerequisites for a free society hang in the balance of our understanding this basic reality.

If we extend ourselves using technology, we must extend the scope of human rights law to include this extended self.

If we cannot define the boundaries of a person properly, how can we hope to protect people or personhood in the digital network age?

Today, we are sharded beings. The boundaries of our selves do not end at our biological boundaries. Aspects of our selves live on bits of silicon that may reside thousands of miles away.

It is imperative that we acknowledge that the boundaries of the self in the digital network age have transcended the biological boundaries of our physical bodies and that this new boundary – the extended self; the sharded totality of the self – constitutes our new digital skin and that its integrity must be protected by human rights law.

Unless we do this, we are bound to flail around at the surface of the problem, making what are no more than cosmetic changes to a system that is quickly evolving towards a new type of slavery.

This is the personhood emergency.

A radical overhaul of mainstream technology

If we want to tackle the personhood emergency, nothing short of a radical overhaul of mainstream technology will do.

We must first understand that while regulating people farmers and surveillance capitalists is important to reduce their harms, it is both an uphill struggle against institutional corruption and that it will not, by itself, magically result in the emergence of a radically different technological infrastructure. And the latter is the only thing that can tackle the personhood emergency.

Imagine a different world

Humour me for a second and imagine this: Let’s say your name is Jane Smith and I want to talk to you. I go to jane.smith.net.eu and ask to follow you. Who am I? I’m aral.balkan.net.eu. You allow me to follow you and we start chatting… privately.

Imagine further that we can create groups – maybe for the school that our children go to or for our local neighbourhood. In such a system, we all own and control our own place on the Internet. We can do all the things you can do on Facebook today, just as easily, but without Facebook in the middle, watching and exploiting us.

What we need is a peer-to-peer system that bridges to the existing world wide web.

What we need is the opposite of Big Tech. We need Small Tech – everyday tools for everyday people designed to increase human welfare, not corporate profits.

Practical steps

At Small Technology Foundation, Laura and I have already started building some of the fundamental pieces of one possible bridge from surveillance capitalism to a radically democratic, peer-to-peer future. And we will continue to work on the other pieces this year and beyond. But there are practical steps we can all take to help move things in this direction.

Here are some practical suggestions for various groups:

Everyday people

Don’t blame yourselves; you are the victims here. When 99.99999% of all technology investment goes to people farmers, don’t let anyone tell you you should feel bad for being forced into using their services due to lack of alternatives.
That said, there are alternatives. Seek them out. Use them. Support the people who make them.
Understand that this problem exists. Call out the people responsible and defend others who do so. At the very least, don’t brush aside the concerns and efforts of those of us who are trying to do something about it.

Developers

Stop embedding the surveillance devices of companies like Google and Facebook in your web sites and apps. Stop exposing the people who use your services to surveillance capitalism.
Start looking at alternative ways of funding and building technology that do not follow the toxic Silicon Valley model.
Drop “growth” as your success metric. Build tools that individuals own and control, not your company or organisation. Build single-tenant web apps. Support free (as in freedom) and decentralised platforms (without getting mired in the blockchain swamp).

The European Union

Stop investing in startups and acting like Silicon Valley’s unpaid research and development department and invest in stayups instead.
Create a noncommercial top-level domain (TLD) open to anyone in the world where anyone can register a domain name (with an automatic Let’s Encrypt certificate) for zero cost with a single API call.
Build upon the previous step to offer every EU citizen, paid for by EU taxpayer money, a basic virtual private server with a basic amount of resources to host an always-on node in a peer-to-peer system that would unshackle them from the Googles and Facebooks of the world and create new opportunities for people to communicate privately as well as to express political will in a decentralised manner.

And, in general, at the very least, it’s time for every one of us to pick a side.

The side you pick will decide whether we live as people or as products. The side you pick will decide whether we live in a democracy or under capitalism.

Democracy or capitalism? Pick one.

If, like me, you grew up in the 80s, you probably unthinkingly accepted the neoliberal maxim that democracy and capitalism go hand-in-hand. This is one of the greatest lies ever told. Democracy and capitalism are polar opposites.

You cannot have a functional democracy and billionaires and trillion-dollar corporate interests and Silicon Valley’s Big Tech misinformation and exploitation machinery. What we’re seeing is the clash of capitalism and democracy and capitalism is winning.

Are we past a tipping point? I don’t know. Perhaps. But we can’t think like that.

Personally, I’m going to keep working to effect change where I feel I can be effective: in creating alternative technological infrastructure to support individual freedoms and democracy.

We’ve already laid the infrastructure of techno-fascism. We’ve already created (and are creating) the panopticons. All the fascists need to do is move in and take the controls. And they will do so democratically, before destroying democracy, just as Hitler did.

And if you think the 1930s and 40s were something, remember that the most advanced tools to amplify the destructive ideologies of the time were less powerful than the computers you have in your pockets today. Today we have machine learning and are on the brink of unlocking quantum computing.

We must ensure the 2030s are not like the 1930s. Because our advanced centralised systems of data capture, classification, and prediction plus a hundred years of exponential increase in processing power (note: I do not use the word “progress”) mean the 2030s will be exponentially worse.

Whoever you are, wherever you are, we have a common enemy: the nationalist international. The problems of our time transcend national borders. The solutions must also. The systems we build must be both local and global at once. The network we must build is one of solidarity.

We created the present. We will create the future. Let’s work together to ensure that that future is the one we want to live in ourselves.

My speech at the European Parliament at the end of last year at the Future of Internet Regulation event.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Or, more accurately, our habitat emergency as that’s what we’re at risk of losing unless we act decisively and do so now. ↩︎
Spoiler alert: we’re not. ↩︎
Although I am fond of surveillance capitalism as a term – it is precise and accurate in describing the latest iteration of capitalism that we inhabit – Shoshana and I differ on how we define it. Shoshana sees it as “a rogue mutation of capitalism”, somehow implying that capitalism is otherwise fine. I see surveillance capitalism as a natural evolution of capitalism. Systems of corporate surveillance do not corrupt capitalism, they amplify it and its extractive and exploitative nature. Surveillance capitalism, in my definition, is the interaction between surveillance and capitalism. Capitalism is about accrual of wealth. What happens when those with accrued wealth invest that wealth in mechanisms of surveillance that enables them to gather intimate insight about our lives which they then use to manipulate our behaviour to accue even greater wealth? You get the feedback loop between surveillance and capitalism that is surveillance capitalism. ↩︎

The Future of Internet Regulation at the European Parliament

mail@ar.al (Aral Balkan) — Fri, 29 Nov 2019 10:21:23 +0000

My speech at the European Parliament last week at the Future of Internet Regulation event.

Last week, I gave three talks¹ in Belgium, starting with one titled “Dear regulators, please don’t throw the baby out with the bathwater” at an event on The Future of Internet Regulation organised by the Greens and Pirates at the European Parliament on Tuesday.

You can watch it via the Peertube embed, above, thanks to the lovely folks at La Quadrature du Net who took the initiative to rip the recording of the live stream and host it on their own instance. They also edited together a version with my slides, captions, and translations. I’ve embedded that version at the end of this post.

The personhood and democracy crisis

During my talk, I tried to impress upon the audience of MEPs, policymakers, and bureaucrats the urgency of the personhood and democracy crisis that we are faced with due to surveillance capitalism.

We have an opportunity to do things differently in the EU. We must regulate surveillance capitalists – and move beyond data protection to legislate for algorithmic transparency and, ultimately, data minimisation.

Small Tech is the antidote to Big Tech

But that’s only one half of the story. Regulating to reduce harm is important but it must go hand-in-hand with regulating to incentivise an alternative ethical approach to building technology with different criteria for success.

We must fund small tech as the antidote to big tech.

The immediate feedback from the room ranged from shock and bruised egos² to enthusiastic hugs and MEPs asking for appointments to talk (sorry I haven’t been able to get back to you all, it’s hard being a tiny not-for-profit, but I will).

Here’s hoping the talk will have an effect.

We can only try, I guess.

A version of my talk with captions, slides, and subtitles in multiple languages.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

The other two were my opening keynote at Creative Ville in Antwerp and my talk to students at Howest International in Kortrijk. ↩︎
One of the panelists, a corporate lobbyist, actually tried to tone police me at the end of his own panel when I asked a question. I was apparently using my “outside voice” and should have used my “inside voice.” Uh-huh. You can see the exchange start at ~12:00:17 on the recording of the web stream, following my question on corporate lobbying and institutional corruption. ↩︎

Small Technology Foundation Personal Web Prototype-01: a mobile personal web server

mail@ar.al (Aral Balkan) — Wed, 13 Nov 2019 15:06:00 +0000

Small Technology Foundation Personal Web Prototype-01: an always-connected portable personal web server that fits in your pocket.

Imagine holding your personal web site in the palm of your hand. Imagine carrying the digital aspects of your self in your pocket instead of having them on some abstract cloud under the watchful eye of some faceless multinational corporation¹.

Personal Web Prototype-01 from Small Technology Foundation R&D achieves the former using widely available off-the-shelf parts and Site.js. Hopefully, it will also start you thinking in a similar vein about building things that enable the latter.

In this post, I will outline what Prototype-01 is and how you can build a similar personal web device yourself. The audience for this post is tinkerers and developers².

The hardware

Prototype 01: the hardware.

Here are the parts I used to build Prototype-01, as labelled in the first photo, above. The main ones are in boldface.

USB Micro Type-B (male) to USB Type-A (female) cable
14500 Lithium-ion battery (3.7v; rechargeable)
Piz-UpTime (the battery/UPS board)³
Raspberry Pi Zero W (with 32GB NOOBS SD-Card and solderless GPIO hammer header)⁴
Huawei E3372 LTE USB stick with mobile data SIM card⁵
Velcro strap (for attaching the LTE USB stick to the rest of it)⁶

In addition, you will need a domain name (I use and can recommend IWantMyName.com for mine) and you will need to know how to update your DNS settings for it.

To point your domain name to your device’s ever-changing IP address you will also need to use a service such as Ngrok (which is what I use here) or a dynamic DNS solution of some sort (which will likely need further setup)⁷.

Finally, to initially set up the Pi, you will also need a keyboard, mouse, and display to connect to it⁸.

Building it

Small Technology Foundation Personal Web Prototype-01: plug-and-play (at least on the hardware side).

This is the simplest step. Simply snap the Piz-UpTime with the 14500 battery to the Raspberry Pi Zero using the GPIO header³ and connect the LTE USB stick to the USB port on the Raspberry Pi⁴. Ensure that you use the power port on the PiZ-UpTime and not the one on the Pi itself to charge the battery and power the unit when connected.

Preparing the Pi Zero

To prepare your Pi to use as a headless web server, connect a keyboard, mouse, and screen to it. With the NOOBS SD-Card inserted, turn on your Pi. Once it boots, open up a terminal window and carry out the following steps.

Set up SSH access with public-key authentication

On the Raspberry Pi, select Preferences → Raspberry Pi Configuration from the Raspberry menu.
In the settings box that appears, set the subdomain.your.domain name you will serve your site from in the Hostname field.
In the Interfaces tab, select the Enable radio button next for the SSH field.
Click OK to save your settings and exit settings.
ssh into your Pi using the IP address of the Pi (which you can find by running ifconfig) and your Pi account name and password.
Add your SSH public key from your main computer (e.g., from ~/.ssh/id_ed25519.pub) to your Pi (e.g., into the ~/.ssh/authorized_keys file).

If you do not have SSH keys generated on your main computer yet, follow the instructions here to create a pair.
Log out of your ssh session and reconnect without using your password to test that the public-key authentication you just set up works.

Small Technology Foundation Personal Web Prototype-01: close-up of the LTE USB Stick.

Set up ngrok

You need to configure ngrok to create three secure tunnels to the Raspberry Pi:

HTTP tunnel for port 80 so Site.js can handle Let’s Encrypt challenges using the HTTP-01 challenge method.
TLS tunnel for port 443 that Site.js will use to serve your web site on.
TCP tunnel for port 22 so you can access your Pi at your domain name over SSH.

To do this, create a file called ngrok.yml in the ~/.ngrok2 directory (create the directory if it doesn’t already exist):

mkdir -p ~/.ngrok2
touch ~/.ngrok2/ngrok.yml

Then add the following configuration, customising it for your needs:

authtoken: <your-auth-token>
region: eu
tunnels:
  insecure-web:
    addr: 80
    proto: http
    hostname: <your-reserved-subdomain.your.domain>
    bind-tls: false
  secure-web:
    addr: 443
    proto: tls
    hostname: <your-reserved-subdomain.your.domain>
  ssh:
    addr: 22
    proto: tcp
    remote_addr: <your-reserved-tcp-address>

You can find your authtoken on your ngrok Auth page.

To learn how to reserve a custom domain, see the ngrok documentation for tunnels on custom domains and TCP Tunnels: listening on a reserved remote address. Also, alter the region to match the one you reserved your ngrok domains on.

Once you’ve customised and saved your ngrok configuration file, you can create your tunnels at any time using:

ngrok start --all

However, while that’s good for testing, what we want is for ngrok to start automatically when the Raspberry Pi does. This will allow us to SSH into the Pi so we can use it headlessly from now on and it will mean that once it has power, our web site will be reachable from anywhere in the world.

To do this, edit the /etc/xdg/lxsession/LXDE-pi/autostart file, add the following line to it, and save it:

@ngrok start --all

This will automatically run your ngrok tunnels in the background whenever the Pi boots up.

Install Site.js and run it as a startup daemon

Install Site.js

wget -qO- https://sitejs.org/install | bash

Create a basic web site:

# Create a folder to hold the web site.
mkdir -p ~/public

# Switch to that folder.
cd public

# Create a simple “Hello, world!” page.
echo 'Hello, world!' > index.html

Create a startup daemon using Site.js to serve your site (this will survive restarts and crashes, etc.):

site enable

Testing it

You should now be able to hit https://subdomain.your.domain and see your “Hello, world!” page.

Updating your site

Now that your site is live, how do you make it a bit more exciting?

On your main computer, create a more interesting web site than “Hello, world”.

If you want to get an idea for the types of things you can build with Site.js, see the tutorial I wrote recently on how to build a simple chat app with Site.js.
Make sure Site.js is also installed on your main computer and then issue the following command:
```
site --sync-to=subdomain.your.domain --exit-on-sync
```
Hit https://subdomain.your.domain again and you should see your new site.

To learn more about Site.js, read the Site.js documentation.

Small Technology Foundation Personal Web Prototype-01: all together now.

What next?

This is just the first prototype to come out of our work on Site.js as part of our research and development work at Small Technology Foundation. It doesn’t represent a specific product we are thinking of building but rather more the direction in which we’re thinking in general (which might result in products later on).

I hope, though, that it does help demonstrate two important points:

We all already carry much more powerful and refined versions of this prototype in our pockets (it’s your smart phone) and yet we cannot run our own web sites on them.

Why not?

Because the Big Tech corporations that make these devices don’t want us to.

I hope this gives folks like Purism and e.foundation some ideas about what they could do in this area to differentiate themselves from the likes of Apple, etc.
If we want the sort of freedom devices like this can offer people, we must ensure that they just work out of the box.

While I hope that you have found this post interesting if you’re a tinkerer or developer, I also hope that it highlights the mountain of work that must be done if we want all the various aspects outlined here to work seamlessly when our audience is everyday people who use technology as everyday things.

I hope this post has inspired to you think about what you can make that everyday people can use to take ownership and control of the digital and connected aspects of their lives.

As always, please feel free to let me know your thoughts.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Each one of us has the raw technology to achieve this in our pockets in the form of a smartphone but the smartphone manufacturers do not let us do this because they do not want us to have ownership and control the digital and connected aspects of our lives – they want to be able to filter our experiences, gatekeep our access, and manipulate our behaviour for profit. ↩︎
If we want simple everyday technology for everyday people to use, we must first empower those who make such things with the tools they need to do so without being beholden to the tools of Big Tech. To paraphrase Audre Lorde: the master’s tools will never dismantle the master’s house and neither will the master’s tools ever let you build your own house. ↩︎
There are several UPS / battery solutions for the Raspberry Pi Zero. The one I’m using here is the PiZ-UpTime 1.0. The latest, currently available version is the PiZ-UpTime 2.0).

Other power options include the PiSugar – which I’d love to get my hands on and the JuiceBox Zero.

If you do end up using the PiZ-Uptime, you will want to install a cron job to ensure that the board is cleanly shut down if the battery runs low on power. ↩︎
I’m using a Raspberry Pi W which doesn’t come with a GPIO header. You don’t need the GPIO header to connect the UPS (you can also use a USB Micro-B male-to-male cable/connector). If you do want to use the header, either get the Raspberry Pi WH model (which comes with it pre-soldered) or you can do what I did, which was to use a solderless GPIO hammer header from Pimoroni. ↩︎
If the LTE USB stick is not automatically recognised, see this. Also, make sure you put the SIM card in to the SIM card slot and not the SD-Card slot on the LTE USB stick. They’re easy to confuse. ↩︎
A wad of blue/white tack or a strong rubber band works just as well; just use whatever you have lying around. ↩︎
In this post, I will only cover Ngrok, with an Ngrok Pro account which is necessary for using custom domain names. ↩︎
Since the Pi Zero only has one USB port, you will either need a bluetooth keyboard/mouse or a combination USB keyboard/mouse or a USB hub.

If you don’t want to keep connecting a keyboard and mouse, look at Synergy. It enables you to share one keyboard and mouse across several devices. If you want a free and open source version, check out Barrier (which I learned about only later and haven’t tried myself yet).
↩︎

Introducing @small-tech/https, a batteries-included drop-in replacement for the Node.js https module

mail@ar.al (Aral Balkan) — Fri, 08 Nov 2019 18:54:12 +0000

@small-tech/https with globally-trusted Let’s Encrypt certificates

Today’s my birthday so I thought I’d give you a little present: @small-tech/https.

Plug-and-play https

This is essentially a drop-in, batteries-included version of the Node.js https module that:

Automatically provisions locally-trusted TLS certificates for localhost for development use (courtesy of mkcert seamlessly integrated via Nodecert).
Automatically provisions globally-trusted TLS certificates for your domain(s) for cross-browser testing, staging, and production courtesy of Let’s Encrypt.

Install

npm i @small-tech/https

Example

Here’s a basic HTTPS server that responds to every GET request with a simple “hello, world” page.

Set up

# Create the project folder and switch to it.
mkdir example && cd example

# Create a new npm module for the example.
npm init --yes

# Install dependencies.
npm i @small-tech/https

# Open up the main file in your default editor.
$EDITOR index.js

Code (index.js)

const https = require('@small-tech/https')

// Helpers
function html(message) {
  return `<!doctype html><html lang='en'><head><meta charset='utf-8'/><title>Hello, world!</title><style>body{background-color: white; font-family: sans-serif;}</style></head><body><h1>${message}</h1></body></html>`
}
const headers = {'Content-Type': 'text/html'}

let options = {}

// For globally-trusted Let’s Encrypt certificates uncomment options.
// To provision certificates, also remove “staging: true” property.

// options = {
//   domain: 'hostname',
//   staging: true
// }

// Default (no options): create HTTPS server at https://localhost
// with locally-trusted certificates.
const server = https.createServer(options, (request, response) => {
  // Respond to all routes with the same page.
  response.writeHead(200, headers)
  response.end(html('Hello, world!'))
})

server.listen(443, () => {
  console.log(' 🎉 Server running on port 443.')
})

Run
```
node index
```

Hit https://localhost and you should see your site served with locally-trusted TLS certificates.

@small-tech/https with locally-trusted certificates courtesy of mkcert

Let’s encrypt!

To provision globally-trusted Let’s Encrypt certificates instead, just uncomment the options object. Test it out with the staging flag set to true first and, once you verify that certificates are being properly generated, remove the flag to actually provision the certificates the first time your server is hit.

TL; DT

Too long, didn’t type?

You can find a version of this example in the /example folder. To download and run it:

# Clone this repository.
git clone https://source.ind.ie/site.js/lib/https.git

# Switch to the directory.
cd https

# Install dependencies.
npm i

# Run the example.
npm run example

Happy birthday to you!

I hope you enjoy my little birthday present and I hope that it makes your life easier.

Please let me know how you get on it with.

PS. I also got some lovely presents today ;)

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Site.js: now with auto updates in production

mail@ar.al (Aral Balkan) — Sun, 03 Nov 2019 11:12:31 +0000

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download this video directly, and watch it with your favourite video player.

A quick demonstration of the new auto-reload feature.

Site.js version 12.10.2 introduces automatic updates in production.

Why auto update?

Site.js is a personal web tool for individual developers – not startups or enterprises. It’s a tool for building everyday things for everyday people that do exactly what they say on the tin and nothing more.

In other words, Site.js is a tool for building small technology.

With that in mind, it goes without saying that the sites and apps you build and serve with Site.js will not have dedicated operations teams to keep them up to date and secure. And while you may be a dev-ops unicorn who also plays smashing electric guitar, not everyone is. And the people who will run their own instances of the things you build with Site.js definitely won’t be. So Site.js has to be your operations team and come with secure defaults out of the box.

Because security

One of the biggest security issues in tech is running outdated software. While I was running my servers on nginx, I couldn’t tell you when they were last upgraded as the process was so convoluted. With Site.js, running your own web server has to be deploy and forget. We cannot assume that folks are going to update their servers.

So auto updates of production servers is a crucial security feature.

Manual updates during development and testing

Automatic updates are a feature for production servers only (in other words, whenever you launch Site.js as a service/daemon using the enable command).

When running Site.js during development and testing, you can check for updates manually using the update command.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Site.js: now with auto server reload on source code changes

mail@ar.al (Aral Balkan) — Wed, 30 Oct 2019 19:21:35 +0000

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download this video directly, and watch it with your favourite video player.

A quick demonstration of the new auto-reload feature.

Auto reload

Site.js version 12.9.7 brings the second developer experience improvement to Site.js in as many days with an integrated auto reload feature that responds to source code changes on dynamic sites.

Live reload yesterday, auto reload today

Following yesterday’s addition of live reload for static sites, today you get auto server reloads when the source code of your dynamic site changes.

This is the sort of functionality you’d normally implement by using excellent third-party tools like Remy’s nodemon. With Site.js, you now get it out of the box. No external process manager required.

So now, if you change the code in a DotJS route or even if you add or remove a new node module, the server will automatically restart.

Note that during development, it’s the server that restarts, not your whole process. That’s faster than a full reload of the whole Site.js process.

Seamless restarts on deployment when necessary

While the auto reload feature is a great help during development, it also makes your life much easier during deployment.

If you run Site.js in production using the enable command, your server will now automatically restart when you sync your changes to it. (And do so only if it needs to.)

None of this changes the behaviour of static page and asset updates which have always been, and continue to be, immediately available on deployment without requiring a server restart.

Have a play!

Yes, I’m going to keep mentioning the tutorial until you try it!

If you’re wondering what Site.js is, check out the Site.js web site and have a glance at the Site.js documentation.

There’s also a tutorial I released a few weeks ago that takes you through building static and dynamic sites with Site.js as you build a simple chat app using WebSockets.

I hope these improvements make your experience of creating with Site.js more delightful.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Site.js: now with live reload

mail@ar.al (Aral Balkan) — Tue, 29 Oct 2019 18:28:29 +0000

Laura demonstrating the new live reload feature last night.

I just released version 12.9.6 of Site.js with live reload support for static pages.

There’s also a fix for the update command, so please update to this version so that you can keep updating using the update command when we move to 12.10.x and beyond.

Get Site.js version 12.9.6

Update to it

To update to Site.js version 12.9.6 from Site.js version 12.9.5, you can use the update command:

site update

Install it from scratch

If you’re running an earlier version of Site.js or if you are going to install it for the first time:

Copy and paste the following command into your terminal. Before you pipe any script into your computer, always view the source code (Linux and macOS, Windows) and make sure you understand what it does.

Linux

wget -qO- https://sitejs.org/install | bash

macOS

curl -s https://sitejs.org/install | bash

Windows 10

iex(iwr -UseBasicParsing https://sitejs.org/install.txt).Content

Live reload

Site.js is the easiest way to develop, test, and deploy a static web site but, until today, it wasn’t necessarily a delightful development experience as you needed to refresh your static pages manually to see the changes you made… like it was 2009 or something!

Well, take your hand off that refresh button because the latest version of Site.js comes with live reload out of the box.

Server-sent events to the rescue

Me, in my naïve younger days, earlier this week.

I implemented the live reload functionality using my fork of the excellent instant module, which in turn uses my fork of the also excellent sendevent module, both of which are by Felix Gnass and use server-sent events (SSE; also known as EventSource).

Rescuing server-sent events from a fox

Theory, meet practice.

While I initially thought that I’d be done implementing the feature in mere minutes thanks to the instant module, it actually took me two days. And we have Mr. Firefox Browser to thank for that.

Turns out, Firefox has a peculiar quirk¹ with EventSource when you’re testing a page and your host is localhost.² Basically, if you refresh the page from Firefox’s memory cache without doing a force refresh, your EventSource connection disconnects after 30 seconds.

Ooh, we’re half way there…

It took me about 20 minutes to integrate the instant module into Site.js and two days to narrow down and fix the issues with Firefox and contribute my fixes upstream. And that, ladies and gentlemen, is the wacky world of development that we all know and love.

It was worth it, though, because now Firefox behaves exactly like Chrom(ium) with EventSource connections over localhost when using the instant module³.

Development would be harder if you couldn’t bitch on Mastodon.

And a fix for the update command

Some days when you’re making things you have these “doh!” moments where you realise you did something really silly.

I had one of those today while attempting to use the Site.js update command to update from version 12.9.5 to this release, which was originally meant to be version 12.10.0.

Long story short, Site.js told me I was running a newer version than the latest released version because I, in all my glory, thought I was doing integer comparison while actually doing string comparison and, lexographically, 10 comes before 9.

Yay, me.

So that’s fixed now too⁴.

Getting started

I followed a tutorial and all I got was this lousy chat app…

If you’re wondering what Site.js is, check out the Site.js web site and have a glance at the Site.js documentation.

There’s also a tutorial I released a few weeks ago that takes you through building static and dynamic sites with Site.js as you build a simple chat app using WebSockets.

I hope the new live reload feature makes your life easier and, as always, if you have any thoughts or suggestions – or if you run into any issues – please let me know.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

See my merge requests for the instant (#17) and sendevent (#4) modules for further details. ↩︎
And I literally mean host, not even hostname – the quirk does not manifest, for example, if you’re testing over localhost:999, only localhost:443 (I did not test over port 80 because, well, it’s late 2019.) ↩︎
For now you can use my fork of instant but keep an eye on my pull request in Felix’s original repository and switch to using the original module once my changes have been merged. ↩︎
I’ll write up a short post with code snippets when I get a moment so we can all share a laugh. ↩︎

Fixing read-only file system errors after do-release-upgrade from Ubuntu 14.04 LTS to 16.04 LTS

mail@ar.al (Aral Balkan) — Thu, 24 Oct 2019 12:34:00 +0100

I upgraded an old server from Ubuntu 14.04 LTS to 16.04 LTS today and, when it restarted, I started getting “Read-only file system” errors on the root partition.

Ouch!

Here’s how I investigated and fixed the issue, including a list of the sites I found that helped me along the way.

Am I out of disk space?

I wasn’t sure if perhaps the installation had failed and corrupted something because I was out of disk space. So first I checked for that and saw that it wasn’t the case:

► df

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/vda1       61796348 44194416  14439820  76% /

Is the disk OK?

Next, I used fsck to see if the partition was all right. It was (phew):

► sudo fsck /dev/vda1

fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
DOROOT: clean, 289048/3932160 files, 11328157/15728640 blocks

Is the mount properly configured?

Since the partition looked OK, I wanted to see if the configuration instructions for the disk were correct so I asked to see a listing of the relevant file:

► cat /etc/fstab

And eureka, it looks like the upgrade changed the configuration to use a UUID:

# / was on /dev/vda1 during installation
UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 /  ext4  errors=remount-ro  0  1

So the question was, of course, is the UUID correct? So I checked what the UUID of /dev/vda1 was:

► ls -l /dev/disk/by-uuid/ | grep vda1

Which resulted in:

lrwxrwxrwx 1 root root 10 Oct 24 04:26 88f2ec31-89b7-4164-8fb8-c7736b549505 -> ../../vda1

So, there was our culprit. The UUIDs did not match.

Let’s fix it!

Since things were working previously with the disk specified as /dev/vda1 instead of via UUID, I thought I’d just edit the fstab file and be done with it:

► sudo nano /etc/fstab

But not so fast:

Unable to create directory /root/.nano: Read-only file system
It is required for saving/loading search history or cursor positions.

Doh!

Of course, the whole issue is that the file system is read-only so how am I going to update the configuration file.

Remount the drive

Again, since things were working properly and there didn’t seem to be a file system corruption, I thought I’d just try remounting the drive using a correct identifier:

mount -o remount,rw /dev/vda1 /

And that worked!

So I then updated the mounting configuration in /etc/fstab to:

/dev/vda1 /  ext4  errors=remount-ro  0  1

Turn it off and back on again

Finally, I rebooted.

And now I once again have a properly mounted, read-write root partition.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

References

How to migrate from VSCode to VSCodium (the best code editor ever minus the corporate bullshit)

mail@ar.al (Aral Balkan) — Thu, 24 Oct 2019 09:45:00 +0100

Blogception: a post on VSCodium as it’s being written in VSCodium.

I am writing this blog post in VSCodium. What? Is that like VSCode?

Yes, it’s basically VSCode minus the corporate bullshit like surveillance and proprietary-licensed binaries.

An ode to VSCode

VSCode is the best code editor I’ve ever used.

It’s actually rather delightful.

There, I’ve said it – and I’ve used a lot of editors across 30+ years of programming.

VS-what?

VSCode: it’s not just for code (ok, it is).

For those of you who are not programmers (yet!) or have been living under a rock, VSCode is an open source code editor from Microsoft.

I know, I was shocked too.

Enter, the bullshit

Even though it’s awesome, Microsoft is still Microsoft so – of course – it comes with telemetry (read: surveillance) by default. So while I love using it, I had to qualify every recommendation to others with “and remember to turn telemetry off.”

Well, no longer!

Beyond the bullshit

When I mentioned VSCode recently on my Mastodon, janči responded with the following toot:

It sounded great so I decided to check it out and I’m very glad I did (thanks, janči).

Enter VSCodium

The VSCodium project page on Microsoft GitHub.

So it turns out that some enterprising freedom-lovers created a project called VSCodium that takes the MIT-licensed source code, removes the telemetry (read: surveillance) from the codebase (along with the branding¹) and licenses the resulting binaries under the MIT license, just like the code itself².

How yummy!

Bonus: auto updates!

Also, given that my main development machine is a laptop running Linux (Pop!_OS³), I had to keep installing VSCode updates by hand as Microsoft does not make automatic updates available for my platform. VSCodium fixes that for me as they have an apt repository.

Shut up and don’t take my money!

So how does one install VSCodium? Also, given there’s a high chance you’re already running VSCode (because, like, who isn’t, amirite?), how do you migrate your extensions and other settings to it?

Easy peasy!

If you’re on a Debian-derived operating system, you can just copy/paste the following commands and be up and running in the next few seconds.

For other platforms, it’s just as easy if not easier (read the download/install instructions and the migrating from Visual Studio Code to VSCodium section of the documentation for details).

Install VSCodium with auto updates

On Debian-derived operating systems that have the apt package manager:

# Authorise Paul Carroty’s repository.
# (https://twitter.com/paulcarroty)
wget -qO - https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg | sudo apt-key add -

# Add Paul’s custom apt repository to your apt sources list.
echo 'deb https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/repos/debs/ vscodium main' | sudo tee --append /etc/apt/sources.list

# Update your apt package list and install VSCodium.
sudo apt update
sudo apt install codium

Copy your extensions and settings over from VSCode to VSCodium.

On Linux:

# Create the folder to host your extensions.
mkdir -p ~/.vscode-oss/

# Copy your extensions over from VSCode.
cp -R ~/.vscode/extensions ~/.vscode-oss/

# Create the folder to hold your settings.
mkdir -p $HOME/.config/VSCodium/

# Copy your settings over from VSCode.
cp -R $HOME/.config/Code/User $HOME/.config/VSCodium/User/

After you’ve followed the above instructions – or those for your own platform – you should be able to launch VSCodium by just typing codium in Terminal⁴

Need something to code?

According to sources within Number 10, a chat app with Site.js is the perfect thing to code using VSCodium.

I hope you enjoy using VSCodium as much as I do. And hey, if you’re looking for something fun to code with it, check out my recent tutorial on building a WebSocket-based chat app with Site.js.

Using the remote development feature in VSCode, you can even do it on a Raspberry Pi (yes, even on a Raspberry Pi Zero!).

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

VSCodium couldn’t keep the branding even if it wanted to as they would be violating Microsoft’s trademarks. ↩︎
The VSCode binaries carry a proprietary license. ↩︎
Pop!_OS is built on Ubuntu. ↩︎
To launch VSCodium using the same shortcut for VSCode, just alias it in your shell configuration file (e.g., ~/.zshrc for zsh⁵). ↩︎
You haven’t lived until you’ve tried zsh with oh-my-zsh and the agnoster theme with powerline fonts in tilix… just sayin’. PS. To install the powerline fonts on a Debian-derived Linux distribution, just use apt install fonts-powerline, and to activate the agnoster theme, set ZSH_THEME="agnoster" in your ~/.zshrc. ↩︎

The little Raspberry Pi that could (serve a web site)

mail@ar.al (Aral Balkan) — Tue, 22 Oct 2019 18:08:06 +0100

Yesterday, I asked folks following me on my Mastodon¹, if they’d help me blow up my Raspberry Pi Zero W:

A story in three toots…

Earlier this week, I got to test Site.js on the Pi Zero and saw that it runs flawlessly. Using ngrok, I exposed the Pi to the harsh and cruel Interwebs and then – for the fun of it – decided to see what would happen if lots of folks hit it at the same time. And…

Oh, the suspense…

Turns out, the little thing is a trooper.

The little Pi that could

During our little experiement, Site.js on the Pi Zero served over 7,000 requests. And since I made the ephemeral statistics URL public, folks creatively started using it to send me messages via 404 errors.

Wait, what’s Site.js again?

Site.js is a free and open source web tool for developers. I would argue that is the easiest way to develop, test, and deploy secure static and dynamic web sites. Check out this tutorial to get started and view the documentation if you want to learn every little detail.

It’s the first piece of the bridge we’re building at Small Technology Foundation between the centralised surveillance-based exploitative web we have and the peer-to-peer ethical web we want.

Site.js runs on Linux, macOS, and Windows (and, of course, Raspberry Pis²).

Why not have a play?

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

I also asked on (spit) the @aral Twitter account. The difference is that while I own and control my own Mastodon instance, Twitter, Inc., owns and controls my account on Twitter.com. On my Mastodon, I see everything that everyone sends me and I decide what I say and how (and I’m still bound my the laws of the jurisdiction that I live in and I don’t have the right to compel anyone to listen to me – a common misconception of free speech advocates). On Twitter.com, I see what their algorithms show me and I am able to say what Twitter, Inc. deems acceptable. For folks in Turkey, for example, that means they don’t see what the Turkish government doesn’t want them to see. ↩︎
Which run Linux on ARM. I’ve tested Site.js on the 3B+, 4B, and Zero W models. ↩︎

Site.js: now easier than ever to update

mail@ar.al (Aral Balkan) — Mon, 21 Oct 2019 11:48:00 +0100

Earlier today: the new update command seamlessly updating and restarting Site.js on SiteJS.org.

I just released Site.js version 12.9.5. This version brings with it some new commands, the most important of which is the update command¹.

# Update Site.js to the latest version.
site update

The update command does what it says on the tin and updates your copy of Site.js to the latest version.

If Site.js is running as a daemon, it is seamlessly restarted for you.

If you’re running instances of Site.js as regular processes, they will continue to run the older version until the next time you run the site command.

Better daemon control

The help command is a good way to discover what Site.js is capable of.

Until version 12.9.5, you were limited in your control over a Site.js daemon to the enable and disable commands.

The enable command both installs the systemd unit and starts the service and the disable command both stops the service and uninstalls it.

Today, you get three new commands – start, stop, and restart – that enable you to control the status of the service without affecting whether it is installed or not.

Learn Site.js the fun way

Learn about Site.js while building a chat app… you can even do it on a Raspberry Pi if you like!

If you’re wondering what Site.js is and what you can do with it, the easiest way to get started is a tutorial I wrote this month that takes you step-by-step through building a simple WebSocket-based chat app. And since Site.js version 12.8.0, you can even follow along on your Raspberry Pi if you have one.

Here are links to both resources:

Site.js is cross-platform

Site.js runs on Linux (x64 and ARM²), macOS, and Windows 10 so grab your computer, fire up Terminal and your favourite editor and discover the easiest way to create, test, and host your own static and dynamic web sites.

As always, please do let me know how you get on.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Given that the update command is released for the first time in version 12.9.5, you will have to wait for the next release before you can use it to actually update your version of Site.js. To update to version 12.9.5, please use the install command from SiteJS.org as before. ↩︎
Tested on my Raspberry Pi 3B+ and 4B. ↩︎

Scary git-lfs “bad object” error not so scary after all

mail@ar.al (Aral Balkan) — Sat, 19 Oct 2019 18:00:53 +0100

I just got the following error while trying to push to my git-lfs-enabled repository:

ref master:: Error in git rev-list --stdin --objects --not --remotes=origin --: exit status 128 fatal: bad object f895c581f0ff52b2272318105ad7bcc09b5fb8eb

Some quick ducking revealed scary stories of repository corruptions. Eek! Not what I wanted to hear.

Turns out, however, that the error message is not as scary as it looks.

Here’s when/why you get this scary error:

Your repository is head of master with a commit where the only changes are git-lfs changes and
Your repository is actually also behind master with non-git-lfs changes (these don’t appear in git status for some reason when your working copy is in this state) and
You do a git push.

The fix, of course, is to git pull and merge your non-git-lfs changes and then push again.

Whew!

Install git-lfs on a Raspberry Pi

mail@ar.al (Aral Balkan) — Sat, 19 Oct 2019 16:31:38 +0100

What is git-lfs?

Git Large File Storage is “an open source Git extension for versioning large files.” I use it on a number of our git repositories at Small Technology Foundation, including for the Site.js web site.

Why Pi?

As I announced yesterday, since version 12.8.0 of Site.js with Raspberry Pi support, I’ve started building Site.js on my Raspberry Pi 3B+. And it works perfectly, except for one thing: I can’t check the release binaries in without LFS and installing git-lfs on a Raspberry Pi wasn’t entirely straightforward.

How-to

Here’s how you can install git-lfs on your Raspberry Pi:

# Download the latest armv6l version of go (as of this writing, 1.13.3)
# You can check for the latest version here: https://golang.org/dl/
wget https://dl.google.com/go/go1.13.3.linux-armv6l.tar.gz

# Unarchive it.
sudo tar -C /usr/local -xzf go1.13.3.linux-armv6l.tar.gz

# Add the binary path to your system path.
# (Add this to your .bashrc or .zshrc if you want to persist it across reboots.)
export PATH=$PATH:/usr/local/go/bin

# Create and export your go path.
# (Again, add it to your shell configuration file if you want to persist it.)
mkdir ~/go
export GOPATH=~/go

# Download and compile git-lfs.
go get github.com/github/git-lfs

# Add the compiled binary to your path.
sudo cp ~/go/bin/git-lfs /usr/local/bin/

Test it out!

You should now be able to use git-lfs as usual. To test it out, cd into the working copy of the project you want to use git-lfs on and enter the following command:

git lfs install

If all goes well, you should see output resembling the following:

Updated git hooks.
Git LFS initialized.

Acknowledgements

Gist: Install Golang 1.8 on Raspberry Pi.

Site.js and Pi

mail@ar.al (Aral Balkan) — Fri, 18 Oct 2019 11:47:08 +0100

Chatting about Pi, on a Pi, with a chat server running on Site.js on the same Pi.

Yesterday, I released Site.js 12.8.0 which brings initial ARM support for Linux.

What that means is that it’s now easier than ever to get a static or dynamic (Node.js) web server up and running on a Raspberry Pi.

Site.js on my Raspberry Pi, serving the basic chat app in the first screenshot.

Have a play!

Do you have a Raspberry Pi 3B+ or 4B lying around?¹ Well then, grab it, fire up a terminal window, and get your first static site up and running in the next 30 seconds:

# Install Site.js.
wget -qO- https://sitejs.org/install | bash

# Create the most basic web site ever.
echo 'Hello, world! > index.html

# Run Site.js
site

Now launch a browser² on your Raspberry Pi and hit https://localhost in it.

My Raspberry Pi 4B, caught red-handed running the chat app in the first screenshot.

Please sir, may I have some more?

But, of course!

If this has whet your appetite, so to speak, head over to the Build a simple chat app with Site.js tutorial where you will learn lots more about making dynamic web sites with Site.js using DotJS (Node.js made simple, ala PHP) as you create the WebSocket-based chat example you see in the first screenshot.

Have fun and do let me know how you get on.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Those are the only ones I tested with so far. It might work on other ones too. Do let me know if you try it out. ↩︎
For example, type epiphany in a terminal window to launch the Epiphany browser – aka GNOME Web – that comes with Raspbian but isn’t available from the raspberry menu under Internet for some inexplicable reason. ↩︎

Build a simple chat app with Site.js

mail@ar.al (Aral Balkan) — Fri, 11 Oct 2019 17:59:42 +0100

Chat room

Status: Offline

Messages

The chat app we’re going to build together. (It’s live… open another browser window or hit this page from a different device to see it in action!)

We need to talk about Site.js

This weekend, I released Site.js version 12.7.0 with improvements to its WebSocket interface. Today, I want to take you step-by-step through building and running a basic chat app using Site.js.

It’s much easier than you think, so fire up a terminal window, grab your code editor, and let’s get started!

Install Site.js

If you don’t already have Site.js, you first need to install it.

Copy and paste the following command into your terminal. Before you pipe any script into your computer, always view the source code (Linux and macOS, Windows) and make sure you understand what it does.
Linux
```
wget -qO- https://sitejs.org/install | bash
```
macOS
```
curl -s https://sitejs.org/install | bash
```
Windows 10
```
iex(iwr -UseBasicParsing https://sitejs.org/install.txt).Content
```
Instructions for other platforms: Linux | macOS | Windows

The download and installation should take about 10-15 seconds depending on the speed of your Internet connection.

Unless otherwise stated, the remaining instructions should work verbatim across Linux, macOS, and Windows 10.
Test it

In your terminal¹, create a simple static “Hello, world!” web page and serve it:
```
# Create a folder to work in and switch to it.
mkdir demo
cd demo

# Create the simplest possible “web page”.
echo 'Hello, world' > index.html

# Run Site.js to serve the site.
site
```
To test that it’s working, fire up your browser of choice and visit https://localhost

Hello, world!

A very simple static page.

A couple of things to note:

You didn’t have to configure anything, it just worked.

Site.js is zero-configuration. It favours intelligent defaults over silly ones that leave you to do all the work.

You didn’t get a certificate warning in the browser.

Site.js uses the excellent mkcert tool to seamlessly create a certificate authority on your local machine and issue you a valid TLS certificate the first time you run a server at localhost.²

You have access to very basic, ephemeral statistics.

If you look at the terminal window, you will see an address you can hit in your browser to see basic statistics about your site.
```
📊 For statistics, see https://localhost/909b721d634e89c44754cc036fb379e1
```
Your statistics URL is cryptographically secure.
This is a cryptographically secure address that others cannot guess. So your statistics are initially private. If you want to share them with the world, just share the URL.

Also, these statistics are not stored anywhere and will reset when you restart the server (but your statistics URL will stay the same). They are there for you to discover how your site is being used so you can improve it and to see if there are any errors like missing pages, not to let you spy on people.

Once you’re done testing your shiny new site, press Ctrl C to stop the Site.js server.
Cha-cha-cha changes!

Static sites are all well and good but you were promised a chat app and you can’t build that with a fully static site. So let’s take a quick look at how we can create dynamic apps with Site.js.

I mentioned earlier that Site.js is zero-configuration. This means that it has certain conventions that it expects you to adhere to. For example, if you want to create dynamic routes in your web app, you must place them in a folder called .dynamic.

Before we move onto creating the chat functionality, let’s create the equivalent of our static “hello, world!” example but with some very basic dynamic functionality to display the current date and time.

A timely example

First, create a folder called .dynamic within your demo folder:
```
mkdir .dynamic
```
Next, open your code editor of choice and create a file called date.js in the .dynamic folder. Once you’re done, your project folder hierarchy should look like this:
```
demo/
    ├ .index.html    # static route
    └ .dynamic
          └ date.js  # dynamic route
```
In the date.js file, enter the following code:
```
module.exports = (request, response) => {
  const now = new Date().toString()
  response
    .type('html')
    .end(now)
}
```
Finally, run the site command in the demo folder and visit https://localhost/date.

A dynamic DotJS route. Refresh the page to see it update.

You should see the current date.

Refresh and you should see the date update. Congratulations, you just created your first dynamic web site using Site.js.

(Seriously.)

What magic is this?

I call it DotJS.

DotJS maps JavaScript modules defined in .js files (see what I did there?) to web routes on your web site in a manner that will be familiar to anyone who has ever used PHP.

In our example, DotJS knows that we want the file defined at .dynamic/date.js to be served at the address https://localhost/date.

If we’d wanted our dynamic page to be available from https://localhost/date-and-time instead, we would have defined our route in .dynamic/date-and-time.js.³

Using DotJS, all you have to do is write the logic for your web app. Everything else, including creating a secure HTTPS and WebSocket server for you and registering your routes, etc., is handled for you by Site.js.

Furthermore, there is no magic here. Under the hood, these are simply plain old tried-and-tested Express routes.⁴ Site.js contains Node.js so you can do anything in your dynamic routes that you can do with Node.js – including using node modules – without installing Node.js.

When you’re ready to move on, press Ctrl C to stop the Site.js server.
Did someone mention a chat app?

OK, OK, I know I’m taking the scenic route but I wanted to introduce you to the basic concepts of Site.js before getting to the chat app so you have a solid foundation to build on.

Site.js, as you might have guessed, is not limited to HTTPS routes. You can also create secure WebSocket routes. And you can use DotJS with WebSockets also.

Let’s start by creating and testing the back-end of the chat app and when we’re happy with it, we can cobble together a basic web interface for it.

The chat server

In your demo/.dynamic folder, create a new folder called .wss. This is the folder Site.js expects you to place WebSocket routes in.

Then, in your .wss folder, create a file called chat.js.

When you’re done, your directory structure should look like this:
```
demo/
    ├ .index.html
    └ .dynamic
          ├ date.js
          └ .wss
              └ chat.js
```
Now type the following code into that file:
```
module.exports = function (client, request) {
  client.room = this.setRoom(request)
  console.log(`New client connected to ${client.room}`)
}
```
The function you just wrote will be called any time a new client connects to our chat server at the /chat path (which will be available locally at wss://localhost/chat)⁵. In chat app parlance, what we’ve created is known as a “room.” So let’s try and join it and see what happens.

Room for improvement

To test our new room, run site in your demo folder and, once the server is running, open up a JavaScript console in your browser of choice and enter the following code:
```
socket = new WebSocket('wss://localhost/chat')
```
Now look in your terminal, you should see:
```
New client connected to /chat
```
We’ve connected… now what?
So we’ve just made a successful connection to the /chat room. Now we need the server to listen for messages sent from connected clients and broadcast them to every other client in the same room.

When you’re ready to move on, press Ctrl C to stop the Site.js server.

Broadly speaking

Modify the code in chat.js so that it matches the listing, below. The changed section is highlighted:
```
      module.exports = function (client, request) {
        client.room = this.setRoom(request)
        console.log(`New client connected to ${client.room}`)
```
```
        client.on('message', message => {
          this.broadcast(client, message)
        })
```
```
      }
```
What we’re doing here is creating an event handler that listens for message events and then uses the broadcast method that all DotJS WebSocket routes have to fan the message out to the other clients connected to the same room.

An important thing to note is that you should always use an anonymous function expression instead of an arrow function expression when creating your WebSocket routes to ensure that you can access methods like broadcast() using the this reference⁶.

Can you hear me now?

Our chat server is complete but does it work? Start the chat server again by running Site.js and let’s return to our JavaScript console and test it. This time, open two browser windows and let’s try and hold a conversation.

In the JavaScript console of the first browser window, enter the following, one line at a time, skipping the comments:

// Create the socket connection
socket = new WebSocket('wss://localhost/chat')

// Create the message handler to display incoming messages.
socket.onmessage = message => console.log(message.data)

In the JavaScript console of the second browser window, enter the following, one line at a time, skipping the comments:

// Create the socket connection
socket = new WebSocket('wss://localhost/chat')

// Create the message handler to display incoming messages.
socket.onmessage = message => console.log(message.data)

// Send a message
socket.send('Can you hear me now?')

Glance at the JavaScript console of the first browser window and you should see the message you just sent appear. You can reply from the first window using the same socket.send() method as before.

Functionally, our chat server is complete but let’s modify the code one last time just to add some logging and to document our work to make it easier to remember what we’re doing.

Before we move on, replace the code in chat.js with the following listing:

module.exports = function (client, request) {
  // New client connection: persist client’s “room”
  // based on request path.
  client.room = this.setRoom(request)

  // Log the connection.
  console.log(`New client connected to ${client.room}`)

  client.on('message', message => {
    // New message received: broadcast it to all
    // other clients in the same room.
    const numberOfRecipients = this.broadcast(client, message)

    // Log the number of recipients message was sent to
    // and make sure we pluralise the log message properly.
    console.log(`${client.room} message broadcast to `
      + `${numberOfRecipients} recipient`
      + `${numberOfRecipients === 1 ? '' : 's'}`)
  })
}

Restart Site.js so that it serves this latest version of your chat server.

Some housekeeping

If you remember, towards the start of this tutorial we created a dynamic HTTPS route that shows the current date and time. With Site.js serving the demo folder, try to access the /date route now.

4🤭4

Could not find /date

We don’t talk about my emoji problem…

Oops, you get the default Site.js 404 page⁷.

Site.js can’t file the file. Why?

Turns out ever since we created the .wss folder, Site.js has been ignoring our .dynamic/date.js route due to routing precedence rules.

That’s a fancy way of saying that if we want to use HTTPS and WebSocket DotJS routes together in the same web app, we must put our HTTPS routes in a folder called .https just like we put the WebSocket routes in a folder called .wss.

So create a folder called .https inside the .dynamic folder and move the date.js file into it.

When you’re done, your directory structure should look like this:
```
demo/
    ├ .index.html
    └ .dynamic
          ├ .https
          │   └ date.js
          └ .wss
              └ chat.js
```
Restart your Site.js server and hit https://localhost/date.

The route should now load correctly.

If you look at your terminal output, you will see that Site.js tells you exactly which routes it loads when it launches:
```
🐁 Found .https/.wss folders. Will load dynamic routes from there.
  🐁 Adding HTTPS GET route: /date
  🐁 Adding WebSocket (WSS) route: /chat
```
The console output from Site.js contains important details.
Room with a view

So our chat server works but it doesn’t have a web interface yet.

Let’s fix that!

Open up the static index.html we created in the very first exercise with the “Hello, world!” message in it and replace the contents of that file with the following:
```
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Basic chat app with Site.js</title>
</head>
<body>
  <h1>Chat room</h1>
  
</body>
</html>
```
With Site.js serving the demo folder, hit https://localhost in your browser and confirm that you see it say “Chat room”.

Create the interface

Our simple web interface is going to contain three main components: a connection status widget to show you whether you are connected to the server or not, a message form where you can identify yourself, compose messages, and send them, and, finally, a message display area where we can display both your sent messages and any incoming messages received from others.

Add the interface components to your web page by pasting the code below under the heading in the body of your page.
```
<p>Status: <span id='status' style="color: red;">Offline</span></p>
<form id='message-form'>
  <label for='nickname'>Nickname:</label>
  <input id='nickname' name='nickname' value='Anonymous'>
  <label for='message'>Message:</label>
  <input id='message' name='message' value=''>
  <button id='send-button' type='submit'>Send</button>
</form>
<h2>Messages</h2>
<ul id='messages'></ul>
```
Let’s also add some very basic styling to make our app a little easier on the eyes. Just before the end of the head tag, add a style tag:
```
<style>
  /* Code from the next step goes here. */
</style>
```
First, let’s style the major elements. Add the following to your style tag:
```
  /* Make CSS behave (as much as that’s possible) ;) */
  * { box-sizing: border-box; }

  body {
    font-family: sans-serif;
    padding: 1em;
  }

  h1 { margin-top: 0; }
```
Next, let’s style the form to make it display in a responsive two-column layout:
```
  form {
    background: #eee;
    display: grid;
    grid-template-columns: [labels] auto [controls] 1fr;
    align-items: center;
    grid-row-gap: 0.5em;
    grid-column-gap: 0.5em;
    padding: 0.75em;
  }

  form > label { grid-column: labels; }

  form > input, form > button {
    grid-column: controls;
    min-width: 6em;
    max-width: 300px;
    padding: 0.5em;
    font-size: 1em;
  }
```
Let’s also style the button so that it’s green when enabled, gray when disabled, and transitions smoothly between those states:
```
  button {
    text-align: center;
    cursor: pointer;
    font-size:16px;
    color: white;
    border-radius: 4px;
    background-color:#466B6A;
    border: none;
    padding: 0.75em;
    padding-top: 0.25em;
    padding-bottom: 0.25em;
    transition: color 0.5s;
    transition: background-color 0.5s;
  }

  button:hover {
    color: black;
    background-color: #92AAA4;
  }

  button:disabled {
    color: #999;
    background-color: #ccc;
  }
```
Finally, let’s hide the bullet points and add a background to the message list. We also want to give it a definite height so that it scrolls instead of growing forever:
```
#messages {
  height: 10em;
  overflow-y: scroll;
  background-color: #eee;
  padding: 0.75em;
  list-style: none;
}
```
Now, when you visit https://localhost in your browser, you should see that your (currently non-functional) chat interface resembles the one below:
Chat room

Status: Offline

Nickname: Message:

Messages
The web interface (non-functional).
Make the connection

OK, so let’s add some life to our room, shall we?

Under your interface code, right before the closing </body> tag, add a script tag and let’s get our status indicator working by making a WebSocket connection to our server, listening for the relevant events, and updating the interface accordingly:
```
<script>
  // Shorthand for basic DOM lookup via CSS selectors.
  const element = document.querySelector.bind(document)

  // Initialise web socket.
  const socket = new WebSocket(
    `wss://${window.location.hostname}/chat`
  )

  // Display the state of the connection.
  socket.onopen = _ => {
    element('#status').innerHTML =
      '<span style="color: green">Online</span>'
  }

  socket.onclose = _ => {
    element('#status').innerHTML = 'Offline'
  }

  // Code from the next step goes here.

</script>
```
Restart the Site.js server⁸ and you should now see the status indicator read Online when you reload the page:
Chat room

Status: Offline

Nickname: Message:

Messages
Live example, connected to wss://ar.al/chat.
Note that when making the WebSocket connection, we didn’t hardcode the URL like before. Instead, we used:
```
new WebSocket(`wss://${window.location.hostname}/chat`)
```
This ensures that the app will work regardless of which domain it is served from.

During development, windows.location.hostname will resolve to localhost, as before. When running in production – as it is here on my blog – it will resolve to the domain name of the site⁹.

Handle message sending

Now that our app can connect to the chat server and display its connection status, let’s implement the ability to send messages.

When we send a message, we won’t receive it back ourselves, so one of the things we must do is to add it to our local message list manually. Since we’ll also have to do this when we receive a message from someone else, let’s first create a function we can use for both these purposes.

Add the following code to the end of your current script:
```
// Helper: display a message object.
function displayMessage (message) {
    // Prepare the message HTML.
    const nickname = `<strong>${message.nickname}:</strong>`
    const text = message.text
    const messageHTML = `<li>${nickname} ${text}</li>`

    // Update the message list.
    element('#messages').innerHTML += messageHTML
}
```
Next, let’s create the handler that will be called when your message form is submitted by pressing the Send button.

Add the following code to the end of your script:
```
// Handle message sending.
element('#message-form').addEventListener('submit', event => {
  // Prevent the form from being submitted.
  event.preventDefault()

  // Get the nickname and text.
  const nickname = element('#nickname').value
  const text = element('#message').value

  // Clear the message
  element('#message').value = ''

  // Create a message object, serialise it as JSON, and send it.
  const message = { nickname, text }
  socket.send(JSON.stringify(message))

  // Update the local display
  displayMessage(message)
})

// Code from the next step goes here.
```
In our handler, we construct a message object that contains your nickname and the text of the message you want to send, serialise it into a JSON-formatted string, and then send it to our chat server.

Additionally, we clear the message box to make it easier for you to type your next message and we use our new displayMesssage() function to display the message we’ve sent locally so you can have a full timeline of messages, including your own.

Reload the page and you should now we able to send messages. Note that updates to static routes like our index.html do not require a server restart. Only changes to dynamic routes do.

To test that it is working, take a look at the Site.js console output in your terminal window and you should see the following message:
```
/chat message broadcast to 0 recipients
```
A message to no one.
So our message is being sent but no one is receiving it. This makes sense since the broadcast() method in our chat server does not send a copy of the message to the client it received it from and we don’t have any other clients connected.

Now open up a second browser window and load a copy of https://localhost in it and try sending another message.

This time, you should see the following console output in your terminal:
```
/chat message broadcast to 1 recipient
```
A message to someone.
Well that’s progress. So our messages are being broadcast successfully but we’re not doing anything to process them on the web interface yet.

Let’s implement that next.

Handle incoming messages

When a message is received on the socket, the onmessage event handler is invoked. We need to define this handler to parse the received JSON string and add it to the unordered list in our interface (remember, we serialise message objects in JSON format before sending them so we need to deserialise them when we receive them).

Add the following code to the end of your script, just before the closing </script> tag:
```
// Handle incoming messages.
socket.onmessage = message => {
  // Deserialise the message string and display it.
  message = JSON.parse(message.data)
  displayMessage(message)
}
```
Now when you test your app using two browser windows, you should be able to both send and receive messages.

You can test out what we have so far using the two browser windows below. They’re both running the code above:
Chat room

Status: Offline

Nickname: Message:

Messages
First chat window.
Chat room

Status: Offline

Nickname: Message:

Messages
Second chat window.

Spit and shine

So our chat room works but it’s not as elegant as it could be. Without going overboard (this is a basic tutorial, after all), there are a couple of little touches we can add that would improve its usability considerably.

Set initial focus (“don’t make me click”)

To start, when the app first loads, the first thing the person will most likely want to do is replace Anonymous with their own nickname. So let’s make it easier for them by focusing that field and selecting the text in it so all they have to do to get started is to start typing their nickname.

Right after the definition of the displayMessage() function, add the following code:

element('#nickname').focus()
element('#nickname').select()

Manage focus

While on the topic of focus, if the person types a message and presses Return to send it, the message text field maintains its focus. This is good as it means that they can send another message without doing any more work. However, if they use the Send button to send the message, the message text field loses focus. So it’s up to us to set its focus manually.

Under the element('#message').value = '' line in the form submit event handler, let’s set the focus after we’ve cleared the field:

element('#message').focus()

Auto-scroll the messages list

If you send more messages than will fit in the message list, the latest messages scroll off the screen.

Messages

Aral: Hey, so I have a joke for you…
Laura: Oh, yeah?
Aral: Knock, knock!
Laura: Who’s there?
Aral: Endangered species
Laura: Endangered species, who?
Aral: *silence due to extinction of species between delivery of lines*

Lack of auto scroll means we miss the punchline of the joke (and our species is going to be the punchline of the joke if we don’t get our act together).

This is less than ideal.

Instead, we want the message list to automatically scroll to the end every time a new message arrives so that you can read it without additional effort.

That’s easy enough to achieve by updating the displayMessage() function we wrote earlier to the following one. The changed section is highlighted, below:

      function displayMessage (message) {
        // Prepare the message HTML.
        const nickname = `<strong>${message.nickname}:</strong>`
        const text = message.text
        const messageHTML = `<li>${nickname} ${text}</li>`

        // Update the message list.
        const messageList = element('#messages')
        messageList.innerHTML += messageHTML
        messageList.scrollTop = messageList.scrollHeight

Client-side validation

Things are feeling a bit nicer now but the elephant in the room (his name is George!) is that we’re not performing any input validation. Someone could easily submit a message with no nickname and no message text and we would dutifully fan it out to the other people in the room leading to undue heartache and pain.

Try it out for yourself using the live example, above.

Let’s fix this by adding some client-side validation to our example. There is a listing of all of the code so far, including the instructions below, at the end of the section. Refer to that if you get lost in where to add the various code snippets below.

First, let’s create a validateForm() function we can call to ensure that the form is valid. That, in turn, will make use of a function called isValidString() that validates strings. If the form’s not valid, we’ll disable the Send button:

// Is the passed object a valid string?
function isValidString(s) {
  return Boolean(s)                // Not null, undefined, '', or 0
    && typeof s === 'string'       // and is the correct type
    && s.replace(/\s/g, '') !== '' // and is not just whitespace.
}

// Disables the Send button if the form isn’t valid.
function validateForm () {
  const nicknameIsValid = isValidString(element('#nickname').value)
  const messageIsValid = isValidString(element('#message').value)
  const formIsValid = nicknameIsValid && messageIsValid

  element('#send-button').disabled = !formIsValid
}

Now, we need to call our validateForm() function at certain times. First, we must call it when the page first loads so that the form is initially validated. Since there is no text in the message field, our interface will thus start out with the Send button disabled. This is what we want.

Next, we should validate the form after a message is sent. Why? Because we clear the old message field and so we want the Send button to be disabled again.

Finally, we must validate the form every time the text in the nickname and message text fields changes so that we can enable the Send button when there’s text in both of them:

element('#nickname').addEventListener('input', validateForm)
element('#message').addEventListener('input', validateForm)

This is the bare minimum of validation that we can get away with. Here’s all the front-end code we wrote together with the above improvements highlighted:

<!doctype html>
  <html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Basic chat app with Site.js</title>
      <style>
        * { box-sizing: border-box; }

        body {
          font-family: sans-serif;
          padding: 1em;
        }

        h1 { margin-top: 0; }

        form {
          background: #eee;
          display: grid;
          grid-template-columns: [labels] auto [controls] 1fr;
          align-items: center;
          grid-row-gap: 0.5em;
          grid-column-gap: 0.5em;
          padding: 0.75em;
        }

        form > label { grid-column: labels; }

        form > input, form > button {
          grid-column: controls;
          min-width: 6em;
          max-width: 300px;
          padding: 0.5em;
          font-size: 1em;
        }

        button {
          text-align: center;
          cursor: pointer;
          font-size:16px;
          color: white;
          border-radius: 4px;
          background-color:#466B6A;
          border: none;
          padding: 0.75em;
          padding-top: 0.25em;
          padding-bottom: 0.25em;
          transition: color 0.5s;
          transition: background-color 0.5s;
        }

        button:hover {
          color: black;
          background-color: #92AAA4;
        }

        button:disabled {
          color: #999;
          background-color: #ccc;
        }

        #messages {
          height: 10em;
          overflow-y: scroll;
          background-color: #eee;
          padding: 0.75em;
          list-style: none;
        }
      </style>
    </head>
    <body>
      <h1>Chat room</h1>
      <p>Status: <span id='status' style="color: red;">Offline</span></p>
      <form id='message-form'>
        <label for='nickname'>Nickname:</label>
        <input id='nickname' name='nickname' value='Anonymous'>
        <label for='message'>Message:</label>
        <input id='message' name='message' value=''>
        <button id='send-button' type='submit'>Send</button>
      </form>
      <h2>Messages</h2>
      <ul id='messages'></ul>
      <script>

        // Shorthand for basic DOM lookup via CSS selectors.
        const element = document.querySelector.bind(document)

        // Helper: display a message object.
        function displayMessage (message) {
          // Prepare the message HTML.
          const nickname = `<strong>${message.nickname}:</strong>`
          const text = message.text
          const messageHTML = `<li>${nickname} ${text}</li>`

          // Update the message list.
          const messageList = element('#messages')
          messageList.innerHTML += messageHTML
          messageList.scrollTop = messageList.scrollHeight
        }

        // Is the passed object a valid string?
        function isValidString(s) {
          return Boolean(s)                // Not null, undefined, '', or 0
            && typeof s === 'string'       // and is the correct type
            && s.replace(/\s/g, '') !== '' // and is not just whitespace.
        }

        // Disables the Send button if the form isn’t valid.
        function validateForm () {
          const nicknameIsValid = isValidString(element('#nickname').value)
          const messageIsValid = isValidString(element('#message').value)
          const formIsValid = nicknameIsValid && messageIsValid

          element('#send-button').disabled = !formIsValid
        }

        // Initialise web socket.
        const socket = new WebSocket(
          `wss://${window.location.hostname}/chat`
        )

        // Display the state of the connection.
        socket.onopen = _ => {
          element('#status').innerHTML =
            '<span style="color: green">Online</span>'
        }

        socket.onclose = _ => {
          element('#status').innerHTML = 'Offline'
        }

        // Validate the form whenever the nickname or message changes.
        element('#nickname').addEventListener('input', validateForm)
        element('#message').addEventListener('input', validateForm)

        // Set initial focus and selection.
        element('#nickname').focus()
        element('#nickname').select()

        // Validate the form when the app first loads.
        validateForm()

        // Handle message sending.
        element('#message-form').addEventListener('submit', event => {
          // Prevent the form from being submitted.
          event.preventDefault()

          // Get the nickname and text.
          const nickname = element('#nickname').value
          const text = element('#message').value

          // Clear the message text field.
          element('#message').value = ''

          // Focus the message text field.
          element('#message').focus()

          // Validate the form.
          validateForm()

          // Create a message object, serialise it as JSON & send it.
          const message = { nickname, text }
          socket.send(JSON.stringify(message))

          // Update the local display
          displayMessage(message)
        })

        // Handle incoming messages.
        socket.onmessage = message => {
          // Deserialise the message string and display it.
          message = JSON.parse(message.data)
          displayMessage(message)
        }

      </script>
    </body>
  </html>

Server-side validation

We just implemented front-end validation but that’s only half the story.

In a perfect world, everyone will use our lovely web page front-end and our front-end validation will catch all the issues, and no one will ever hit our back-end directly.

In the real world, watch as I fire up a browser and send you an empty message using the JavaScript console of my browser and your chat server dutifully delivers it to everyone in the room:

// All your front-end validation are belong to us.
socket = new WebSocket('wss://ar.al/chat')
socket.send(JSON.stringify({nickname: '', text: ''}))

Enter the above code into the JavaScript console of your browser, one line at a time, and check out the side-by-side browsers above. You should see your empty message display there:

Messages

Oops! That’s not good.

Front-end validation is a usability feature; back-end validation is a security feature.

In a real-world chat app we would have to implement a host of features to prevent abuse (like rate limiting, blacklists, blocking, etc.). That’s outside the scope of this basic tutorial but let’s at least add some basic validation to our chat server.

Add the following helper functions¹⁰ to your chat server at the bottom of your chat.js file:

// Is the passed object a valid string?
function isValidString(s) {
  return Boolean(s)                // Not null, undefined, '', or 0
    && typeof s === 'string'       // and is the correct type
    && s.replace(/\s/g, '') !== '' // and is not just whitespace.
}

// Is the passed message object valid?
function isValidMessage(m) {
  return isValidString(m.nickname) && isValidString(m.text)
}

Then, at the top of the message handler, let’s call the isValidMessage() function with a parsed instance of the serialised message and abort broadcasting it if the message isn’t valid:

// Perform some basic validation.
if (!isValidMessage(JSON.parse(message))) {
  console.log(`Message is invalid; not broadcasting.`)
  return
}

The live example at the very top of this tutorial connects to the final version of the chat server that contains our server-side validation.

You can test that it works by entering the following code into the JavaScript console in your browser, one line at a time, and verifying that your message does not show up:

// And I would have gotten away with it too,
// if it weren't for you meddling kids…
socket = new WebSocket('wss://ar.al/chat-final-version')
socket.send(JSON.stringify({nickname: '', text: ''}))

Here’s the final listing of the chat server, with this new feature highlighted:

module.exports = function (client, request) {
  // New client connection: persist client’s “room”
  // based on request path.
  client.room = this.setRoom(request)

  // Log the connection.
  console.log(`New client connected to ${client.room}`)

  client.on('message', message => {

    // New message received: broadcast it to all other clients
    // in the same room after performing basic validation.
    if (!isValidMessage(JSON.parse(message))) {
      console.log(`Message is invalid; not broadcasting.`)
      return
    }

    const numberOfRecipients = this.broadcast(client, message)

    // Log the number of recipients message was sent to
    // and make sure we pluralise the log message properly.
    console.log(`${client.room} message broadcast to `
      + `${numberOfRecipients} recipient`
      + `${numberOfRecipients === 1 ? '' : 's'}`)
}

  })
// Is the passed object a valid string?
function isValidString(s) {
  return Boolean(s)                // Not null, undefined, '', or 0
    && typeof s === 'string'       // and is the correct type
    && s.replace(/\s/g, '') !== '' // and is not just whitespace.
}

// Is the passed message object valid?
function isValidMessage(m) {
  return isValidString(m.nickname) && isValidString(m.text)
}

Remember that you have to restart Site.js for server-side changes to dynamic routes to take effect.

Congratulations, you did it!

Well, you were promised a basic chat app in Site.js and that’s exactly what we’ve just built.

Along the way, you also learned the basics of Site.js and how to use it to develop and serve not just WebSocket routes but regular HTTPS routes also using DotJS.

Going further

This tutorial just scratches the tip of the iceberg when it comes to what you can do with Site.js. As I mentioned earlier, anything you can do with Node.js, you can do with Site.js. What you get in addition is a zero-configuration secure static and dynamic web server.

So exactly what else can you do? Here are some ideas to explore on your own:

Test your app from any device

So far, you’ve only run servers at https://localhost and, behind the scenes, Site.js ensured that you didn’t get certificate warnings when you did. But what if you want to test your chat app from your phone or to have others test it with you? You could, of course, deploy it to a public Virtual Private Server, which is the topic of the next section, but you can also run your server at your hostname using the @hostname option¹¹ and provide outside access to your development machine using a tool like ngrok:
```
# Start Site.js at your hostname instead of at localhost.
site @hostname

# (In a separate terminal tab/window, start ngrok)
ngrok start --all
```
Having done this, you can then hit https://dev.my-domain.org from any device anywhere and access your site from your development machine¹². And, again, Site.js will work to seamless provision Let’s Encrypt certificates for you so you will not get any certificate errors.

Deploy to production

All you need to deploy to production is a Virtual Private Server running a flavour of Linux that has systemd and has a domain name pointing to it.

Once that’s set, if you ssh to your production server, you can install Site.js just like you did at the start of this tutorial and then, provided you’re in the directory that you want to serve, deploy a production server with the following command:
```
site enable
```
That will set up your server as a service that automatically restarts should it crash or should you reboot the server.

And, just like before, Site.js will automatically provision your Let’s Encrypt certificates the first time you hit your site via your domain name. All you have to do ensure is that your hostname is set properly on your server. To check and set your hostname, use the hostnamectl command.

Sync

So you have (a) your local copy of your site and (b) you’ve deployed a live production server. How do you get your site from A to B as you continue to work on it?

Simple!

On Linux and macOS¹³, you can use Site.js’s sync-to command to deploy your site like so:
```
site --sync-to=my-domain.org
```
The above command will work with no other information necessary if the account name on your development machine and on your production machine is the same and you’re in your site’s directory. Otherwise, check the docs for on how to specify any details you need to.

On your production server, to ensure that it supports sync, it’s a good idea to launch your server with the --ensure-can-sync option, which will install rsync – the tool Site.js uses behind the scenes to sync your files – if it doesn’t already exist.

And there’s more…

Check out the Site.js documentation for some of the other nifty things you can do with Site.js like not breaking links and contributing to an evergreen web by taking advantage of Site.js’s native support for cascading archives and the 404-to-302 method as well as little niceties like custom error pages.

I hope this tutorial has whet your appetite for Site.js and given you some ideas of what you can do with it.

Site.js and the single-tenant web

Remember that Site.js is a web tool for human beings, not startups or enterprises.

Site.js is for building single-tenant web sites and web apps.

What is a single-tenant web app (or site?)

A single-tenant web app is a personal web app. It’s an app (or site) that you own and control. It’s a step towards building a peer web.

That’s quite a radical concept for the web, which has so far been centralised.

We’re used to sites where you sign up for accounts with huge corporations. Turns out, when we do that, these corporations end up owning those accounts and they end up tracking and profiling us and attempting to influence our behaviour to make a quick buck (or billion).

When we examine the social impacts of this toxic business model, we find that they erode our personhood and threaten our democracies. (Wow, that escalated quickly!)

Flipping the web upside down

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download this video directly, and watch it with your favourite video player.

Watch Laura and me talk about moving beyond surveillance capitalism and flipping the web upside down with small technology.

If we want to own and control the digital aspects of ourselves that form part of who we are as people, we must turn the web on its head. We must flip it upside down.

Ultimately, we must build a web where each one of us has their own place). That’s what we’re working towards at Small Technology Foundation and Site.js is the foundations of our efforts.

I hope you find Site.js useful – even if you don’t care about its philosophical or ethical underpinnings – simply on the merits of what you, as a developer, can do with it and how easy it is to use.

Comments, questions? Yes, please!

If you have any questions about this tutorial or Site.js in general, please feel free to contact me via my mastodon or by email.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

If you don’t want to use the terminal, you can open up your graphical file browser and create the demo folder using that and then use your graphical code editor to create an index.html file in that folder. You will, however, need to run the site command from a terminal session with its current working directory set to the demo folder… there’s no getting away from that. ↩︎
Site.js can also function as a proxy server, so you can test any server locally over a secure connection. For example, if you are running a Hugo server at its default port (1313), you can test it from https://localhost by running Site.js like this:
```
# Proxy the server at port 1313 at
# https://localhost (port 443).
site :1313
```
Note that if you want live reload to work, you should add the --liveReloadPort=443 option to your Hugo server command. ↩︎
DotJS’s auto routing does not limit you to a flat structure. In our example, we could also have placed our code in a file located at .dynamic/date/time which would have made it available at the address https://localhost/date/time. ↩︎
If you want full control over your routing, including the ability to use regular expressions in your route names and accessing global state, etc., you can do everything you can in Express using the advanced routing feature by declaring a routes.js file in your .dynamic fodler. ↩︎
Note that the URL scheme is no longer https but wss since we’re talking about secure WebSocket routes now. If I had a penny for every time I wrote https:// when I meant to write wss://… ↩︎
You might have noticed that we use an anonymous function expression in the module.exports line whereas we used an arrow function expression in the previous (HTTPS) examples and even though we use an arrow function expression to define the event handler. This is not by accident; it has to do with scope. If you want to have access to the this reference in your DotJS routes and access methods like broadcast(), you cannot use an arrow function expression to define your module, you must use the function keyword. Inside of your module, you are free to use arrow function expressions to your heart’s desire. ↩︎
You can easily replace the default error pages with your own custom ones. And as far as 404 errors go, you can reduce the number of them on the web in general and contribute towards an evergreen web by making use of the native cascading archives and 404-to-302 support in Site.js. ↩︎
Site.js does not have LiveReload and does not automatically restart the server when dynamic routes change at the moment. When working with static content, this means that you have to manually refresh the browser and when working with dynamic content you have to manually restart the server whenever you make code changes. I realise this is less than ideal and both of these are high on my list of issues to address. ↩︎
The examples on this page are live and connect to the instances of the app I have running at https://ar.al/chat (corresponding to the initial example, without server-side validation) and https://ar.al/chat-final-version. Needless to say, this site is served by Site.js. ↩︎
You might have noticed that the isValidString() function is the same as the one we used to implement the validation on our front-end. Since both our client-side and server-side code in Site.js is JavaScript (remember, it uses Node.js under the hood), we can pull that function into its own module and re-use it in both places. ↩︎
On Windows (because Windows), you have to add quotes around the @hostname option, so the command becomes site "@hostname". Blame Bill Gates. ↩︎
To use ngrok with your own domain name, you will to subscribe for a Pro account from ngrok (at least this is what Laura and I use at Small Technology Foundation) and set up a subdomain on your own domain name per their instructions. ↩︎
On Windows (because Windows), the sync function is not available as there’s currently no free and open rsync implementation that we can use like we can on Linux and macOS. ↩︎

Site.js, now also on Windows 10

mail@ar.al (Aral Balkan) — Thu, 03 Oct 2019 11:30:00 +0100

Last week, I bought a refurbished 7-year-old ThinkPad 440p¹ so I could test Site.js under Windows.

Long story short, Windows is still shit². Plus, it’s now also a cesspit of surveillance³. And, over the weekend, I ended up adding native Windows 10 support⁴ to Site.js⁵.

So if you’re on Windows 10 – out of necessity or preference – follow along for a quick look at how you can create your own static and dynamic web site using Site.js in a minute or two.

1. Install Site.js (~15 seconds)

Skim the source code for the installer and, when you’re happy it’s not doing anything nefarious, copy and paste the following command⁶ into a PowerShell session running under Windows Terminal:

iex(iwr -UseBasicParsing https://sitejs.org/install.txt).Content

2. Create and test your first static site (~ 30 seconds)

Enter the following three commands into your terminal:

# Create a demo folder to work in and switch to it.
mkdir demo; cd demo

# Create a static index route.
Set-Content index.html 'Hello, world!'

# Start serving your site with Site.js.
site

Visit your new static site at https://localhost.

When you’re ready to move on, press Ctrl+C to stop Site.js.

3. Create and test your first dynamic site (~1 minute)

We’re going to use DotJS to create a dynamic route. In Site.js, dynamic routes are placed in a folder called .dynamic in your site’s root.

# Create the .dynamic folder.
mkdir .dynamic

# Create a dynamic route that displays the current time/date.
Set-Content ./.dynamic/date.js 'module.exports = (_, res)=>{res.end(`<pre>${new Date().toString()}</pre>`)}'

# Start Site.js.
site

Visit your new dynamic route at https://localhost/date.

Refresh to see that the date changes.

In the example above, I wrote the route in a rather obfuscated manner for brevity. Here is the same route, re-written for clarity and better compatibility:

module.exports = (request, response) => {
  const now = new Date().toString()
  response
    .type('html')
    .end(`<pre>${now}</pre>`)
}

There’s no magic…

DotJS simply maps regular .js files (see what I did there) to Express routes. All you have to do is write the logic for your routes. Site.js takes care of all the plumbing.

Using DotJS, you can specify HTTPS GET and POST routes as well as secure WebSocket routes.

If you need full flexibility in your routing, you can do anything you can with Express and Express-WS using advanced routing.

I hope you enjoy using Site.js on Windows 10 and remember that you can use everything you’ve just learned on Linux and macOS too. In fact, there are a couple of things you can do on those platforms that you cannot on Windows 10.

Windows 10 Limitations

On macOS and Linux, you can sync your local site to your remote server. This is not possible on Windows 10 as there isn’t a free and open rsync implementation that we can use.

Also, on Linux distributions with systemd (e.g., Ubuntu 18.04 LTS), you can run production servers. In fact, that’s how this post is being served to you right now.

This blog, Laura’s site, and the Small Technology Foundation web site are all powered by Site.js.

Next steps

To learn more about Site.js, read through the Site.js web site (and try out the interactive demo in the header), read the Site.js documentation, and feel free to hit me up on my Mastodon if you have any questions.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

It was the cheapest way I could test under Windows, given that a license of Windows 10 Pro costs basically what I paid for it. And, I have to say, it runs like a champ. ↩︎
Part of “having been around” is having started out on DOS at age 7 and stubbornly spent twenty odd years on DOS/Windows before overcoming my Stockholm syndrome. ↩︎
But that’s a story for a different post. And yes, I have the screenshots. ↩︎
Note that Windows Subsystem for Linux (WSL) is not supported. Site.js uses mkcert to seamlessly provision development-time TLS certificates. The way mkcert does this is to unceremoniously create a certificate authority on your local machine. The problem is that under WSL, the certificate authority gets created under Linux, not Windows. And then you go and run your browser of choice under Windows and, oops, you get a certificate error. This is not a trivial issue to fix. If the mkcert folks add support for WSL in the future, Site.js should also benefit. Until then, Site.js won’t be supporting WSL. ↩︎
Now, you might be asking yourself: “Aral, if you hate Windows so much, why did you spend time supporting it?” And my answer to that is when you’re building a bridge to a better place, you can’t have both its abutments on the shore you want to get to. You have to plant one of them on the shore you want to get away from.

Case in point: I wouldn’t have been able to switch to Mac in 2006 if it hadn’t been for cross-platform free and open source tools making the transition easier and Parallels enabling me to run Windows alongside OS X until I was ready to cut the cord entirely.

This is also why I talk about Tincan being a bridge from the centralised web to the peer web.

It’s no use building islands of utopia if the only people who can reach them are master swimmers. Our biggest challenge is building bridges. And that sometimes means spending time in places we would rather not. ↩︎
On Linux and macOS, you can install Site.js with a single-line installation command that you copy from the web site. I wanted the Windows version to be as easy.

This does mean the installer does a couple of “clever” things out of necessity like re-downloading itself and then launching the saved script in an elevated process to carry out the installation with administrator privileges. As far as the experience is concerned, it should be seamless. Please do let me know if it isn’t for you. ↩︎

Introducing Small Technology Foundation, Site.js, and Tincan

mail@ar.al (Aral Balkan) — Tue, 27 Aug 2019 12:05:00 +1000

Say “hello” to Small Technology Foundation.

Today, Laura and I want to introduce you to Small Technology Foundation, where we will be continuing the work we started at Ind.ie five years ago.

In those five years, we’ve developed a strong understanding of the problem (surveillance capitalism) and we’ve been iterating on solutions to it.

Our work led us to leave the UK, move to Sweden, and finally, last year, to settle in Ireland.

As part of our relocating to Ireland, we are in the process of shutting down our not-for-profit in the UK, Ind.ie (Article 12), and we’ve set up a new not-for-profit in Ireland.

We’ve also refined how we articulate both the problem and the solution we are exploring. The result is Small Technology Foundation and the concept of Small Tech as the polar opposite of – and the antidote to – Big Tech and Silicon Valley’s church of startups and exponential growth.

What is Small Technology?

The tenets of Small Technology.

Small Technology are everyday tools for everyday people designed to increase human welfare, not corporate profits.

What are you working on?

Site.js: what would a web development tool look like if it was designed for individuals, not startups or enterprises?

I’ve personally spent the last six months creating Site.js – which I’m excited to announce here alongside the setup of our foundation. Site.js is the first step in our plan to evolve a peer web based on the DAT ecosystem.

Site.js makes it simple to set up your own single-tenant, secure personal web site (static and dynamic). How simple? Here’s how you’d deploy a “Hello, world!” site on your production server:

# Install Site.js
wget -qO- https://sitejs.org/install | bash

# Create the site
echo 'Hello, world' > index.html

# Enable a secure production server
site enable

And what if you wanted a dynamic site?

OK, create a file called counter in .dynamic/counter.js:

let counter = 0

module.exports = (request, response) => {
  response.type('html').end(`You’ve visited this page ${++counter} time${counter === 1 ? '' : 's'}`)
}

Then run:

site

And hit https://localhost/counter.

Refresh to see the counter update.

I call this DotJS (think PHP for JavaScript) and it enables you to create secure, dynamic JavaScript servers with the full power of Node.js and Express without having to build any of the plumbing.

And it’s as easy to use WebSockets with it too.

Oh, and Site.js already powers this site, the Small Technology Foundation site and Laura’s site. So we’re eating our own hamster food.

The Small Technology Foundation web site is also an example of how to extend a static site (in this case, one that uses Hugo), with dynamic functionality. Check out the Fund Us page for an example of the WebSocket functionality powering the patronage pages (hint, hint) ;)

To see more, run the demo on SiteJS.org, try some other examples, and read through the documentation.

What’s next?

Tincan: bridges a peer-to-peer network with the centralised web.

The next step is to continue developing Site.js and to build on top of it a peer-to-peer network that bridges to the centralised web. We call this Tincan (the latest evolution of Hypha, Indienet, and Heartbeat).

Tincan: two people communicating.

There are no big reveals here, we are not Silicon Valley (we’re the opposite of Silicon Valley). We’re in this for the long haul. We’re not a startup, we’re a stayup. As such, we will continue to iterate and build on our work from the past five years.

Happy we exist? Enable us to continue to exist.

In the past five years, my family has invested three homes to enable our work. We don’t have any more homes to invest.

If you’re happy we exist, please become a patron and support us to ensure that you can still be happy we exist in the next five years.

We are going to be moving more things over from Ind.ie to Small Technology Foundation in the coming days (like our source code repository, etc.) and we’ll let you know when we do. And we’re going to continue to polish things up on the web site and improve Site.js. Just like everything else we do, this is yet another iteration.

We will also be cancelling all patronages on Ind.ie so if you want to continue supporting us, please become a patron of Small Technology Foundation.

Partner with us

We are also looking for VPS hosts and domain name registrars to partner with. If you share our social mission of a kinder Internet built for people, not corporations and governments, please contact us.

There’s a Turkish saying that goes “from drops, are lakes formed…”

After five years of working on this problem, I am more excited than ever with where we are and the progress we are making. I hope you will join us, as allies, on this journey and enable us to realise our dream of creating technology that works for people, not startups or enterprises; technology that protects human rights and democracy instead of eroding them.

Visit Small Technology Foundation today and let us know how you can help.

Small Technology at the Interactive Future Exhibition in Darmstadt

mail@ar.al (Aral Balkan) — Wed, 10 Jul 2019 11:19:57 +0100

Small Technology: my opening slide.

I’m about to take off for Frankfurt via Zurich to talk about Small Technology at the Interactive Future Exhibition in Darmstadt tomorrow. This is the graduation event of students on the Interactive Media Design programme at the Hochschule Darmstady University of Applied Sciences. It’s an honour to have been invited by the students to speak at their graduation event and I’m very much looking forward to seeing their projects and speaking with them.

ar.al is now running on a new server

mail@ar.al (Aral Balkan) — Sat, 06 Jul 2019 17:11:22 +0100

Just a quick note that this site is now running on a new (as of yet unreleased) server. Most of the functionality should be the same as before with the caveat that the DAT version is temporarily unavailable. I’ll be re-enabling it once I’ve added native DAT support to the server itself.

More info soon.

Have you heard about Silicon Valley’s unpaid research and development department? It’s called the EU.

mail@ar.al (Aral Balkan) — Sat, 22 Jun 2019 06:42:04 +0100

You… yes you.

Who should you thank for Facebook’s Libra?

And, if you’re an EU citizen who pays their taxes,

You.

Surprised? Don’t be.

None of this was unforeseen

Today, the EU acts like an unpaid research and development department for Silicon Valley. We fund startups, which, if they’re successful, get sold to companies in Silicon Valley. If they fail, the European taxpayer foots the bill. This is madness.

Slavery 2.0 and how to avoid it: a practical guide for cyborgs

So what should we do instead?

Let’s instead invest in many small and independent not-for-profit organisations and task them with building the ethical alternatives. Let’s get them to compete with each other while doing so. Let’s take what we know works from Silicon Valley (small organisations working iteratively, competing, and failing fast) and remove what is toxic: venture capital, exponential growth, and exits.

Instead of startups, lets build stayups in Europe.

Instead of disposable businesses that either fail fast or become malignant tumours, let’s fund organisations that either fail fast or become sustainable providers of social good.

…

The EC must stop funding startups and invest in stayups instead. Invest €5M in ten stayups in each area where we want ethical alternatives. Unlike a startup, when stayups are successful, they don’t exit. They can’t get bought by Google or Facebook. They remain sustainable European not-for-profits working to deliver technology as a social good.

Furthermore, funding for a stayup must come with a strict specification of the character of the technology it will build. Goods built using public funds must be public goods. Free Software Foundation Europe is currently raising awareness along these lines with their “public money, public code” campaign. However we must go beyond “open source” to stipulate that technology created by stayups must be not only public but also impossible to enclose. For software and hardware, this means using licenses that are copyleft. A copyleft license ensures that if you build on public technology, you must share alike. Share-alike licenses are essential so that our efforts do not become a euphemism for privatisation and to avoid a tragedy of the commons. Corporations with deep pockets must not be able to take what we create with public funds, invest their own millions on top, and not share back the value they’ve added.

Slavery 2.0 and how to avoid it: a practical guide for cyborgs

Sounds great, wish you’d told someone about it…

When I mentioned this plan several years ago at the European Parliament, my words fell on deaf ears. It’s still not too late to try. But every day that we delay, surveillance capitalism becomes ever more entrenched within the fabric of our lives.

Slavery 2.0 and how to avoid it: a practical guide for cyborgs

But sometimes you have to repeat things before people take notice…

We must also start to fund ethical, decentralised, free and open alternatives from the commons for the common good. We must ensure that these organisations have social missions ingrained in their very existence that cannot be circumvented. We must make sure that these organisations cannot be bought by surveillance capitalists. Today, we are funding startups and acting as an unofficial and unpaid research and development arm for Silicon Valley. We fund startups and, if they’re successful, they get bought by the Googles and Facebooks. If they’re unsuccessful, the EU taxpayer foots the bill. It’s time for the European Commission and the EU to stop being useful idiots for Silicon Valley, and for us to fund and support our own ethical technological infrastructure.

Given how much information Silicon Valley, and thus the US government, has on EU citizens, this is not just a matter of our human rights and the future of our democracy, but also the national security of European countries and that of the EU. This isn’t to say that we must wall ourselves off or create a European silo. On the contrary, we must ensure that the technological infrastructure we fund and build is free and open, decentralised, and interoperable so that anyone, anywhere in the world can use it.

Aral Balkan and Laura Kalbag: We’re not sleepwalking into a dystopian future, we’re there today

If only we knew how to fix this…

If only there was an Ethical Design Manifesto that initiatives like DECODE could have adopted that would have prevented this.

If only we knew the general shape of the solution so we could invest EU taxpayer funds in projects that fit it.

If only there were Small Technology principles that clearly stated how to fund and build technology that Facebook would never want to buy.

Oh, wait, we did know but we were so busy being arrogant…

“Small tech” cannot afford to be so small-minded.

Evgeny Morozov

… that we ended up being the useful idiots that helped out Facebook.

A key component in Libra comes from an EU-funded academic startup recently acquired by Facebook. (It was apart of the Decode Project, on whose advisory board I sit). Ask yourself how Europe can compete with US/China when Facebook can simply snatch such R&D

Evgeny Morozov

Oops!

Rerun cloud-init on multipass

mail@ar.al (Aral Balkan) — Sat, 15 Jun 2019 22:17:58 +0100

Today, I had the need to experiment with rerunning cloud-init on a virtual machine created with multipass. You can use cloud-init with multipass by specifying a cloud-init.yaml file when creating your instance. e.g,

multipass launch --name my-instance --cloud-init ./cloud-init.yaml

This is all well and good and works as you would expect.

However, today, I wanted to experiment with running cloud-init on an already-provisioned instance. My use case is that the lovely folks at Eclips.is (a project by Greenhost and the Open Technology Fund) who have given Ind.ie a generous amount of free hosting do not yet support cloud-init when provisioning a server. So I wanted to see if I can run my cloud-init script after I provisioned a server and I wanted to test that out locally using multipass.

How to rerun cloud-init

Ordinarily, it is actually quite simply to rerun cloud-init after a server has been provisioned.

First, you use the clean command to remove the current build artifacts:

sudo cloud-init clean

Then, you specify your meta-data and user-data files in the /var/lib/cloud/nocloud-net/ directory, which you must create:

sudo mkdir /var/lib/cloud/nocloud-net/
sudo touch /var/lib/cloud/nocloud-net/meta-data
sudo nano /var/lib/cloud/nocloud-net/user-data

Then, paste your cloud-config file into the editor and save (in nano, Ctrl-o + Ctrl-x).

Finally, run the init command to re-initialise your instance using cloud-init:

sudo cloud-init init

Now, in theory, this should work.

In practice, however, we end up with a /var/lib/cloud/instance/user-data.txt file that only contains the vendor-data provided by multipass, not our user-data.

Multipass

Say you have created a multipass instance without specifying a cloud-init.yaml file, like this:

multipass launch --name my-instance

To run your cloud-config file, you first have to SSH into the box. You can either do this by running multipass ls and finding the IP address and then using the account and key that multipass automatically creates to ssh to it (e.g., ssh multipass@<ENTER THE IP-ADDRESS OF YOUR INSTANCE HERE> -i $(locate multipass | grep .*id_rsa)) or you can use the handy shortcut that multipass provides:

multipass shell <NAME OF YOUR INSTANCE>

Then, to find the culprit, run the status command in verbose mode:

sudo cloud-init status --long

Which should give you something along the lines of:

status: running
time: Sat, 15 Jun 2019 21:11:55 +0000
detail:
DataSourceNoCloudNet [seed=/var/lib/cloud/seed/nocloud-net,/dev/sr0][dsmode=net]

The first seed source you see is the one you defined. The second one is the one that multipass passes in. You can cat it to see what it has:

sudo cat /dev/sr0

You should see the vendor-data specified and, at the end of that, you should see an empty cloud-init section:

#cloud-init
{}

This is also what I saw at the end of /var/lib/cloud/instance/user-data.txt and that led to me to think that perhaps our datasource was being ignored or overwritten.

You can find the data sources defined in the file /etc/cloud/cloud.cfg.d/90_dpkg.cfg. To see its contents:

cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg

And you should find a long list similar to:

datasource_list: [ NoCloud, ConfigDrive, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Bigstep, Scaleway, AliYun, Ec2, CloudStack, Hetzner, IBMCloud, None ]

I had a suspicion that the ConfigDrive data source might have been overriding my user-data, so I edited that file and reduced the list down to:

datasource_list: [ NoCloud, None ]

Then, after running sudo cloud-init clean and sudo cloud-init init again, I had multipass successfully using my cloud-config from /var/lib/cloud/nocloud-net/user-data.

Information about cloud-init seems hard to come by so while I’m mostly documenting this for my own sake, I hope it also ends up helping someone else out in the future.

Useful resources

From Cloud-init docs

Cloud-init docs

Other

Export/import issues with GitLab CE

mail@ar.al (Aral Balkan) — Sat, 08 Jun 2019 16:04:16 +0100

GitLab CE (the free/open source version of GitLab) has an import issues feature but doesn’t have an export issues feature (because, not enterprise, apparently).

So if you fork a project and want to transfer the issues also, you’re out of luck. Unless you use the API, that is.

So I ducked around and found that a kind soul by the name of Joseph Heenan had created a Perl script to export your GitLab issues in CSV format. Spoiler: do not run this as-is and import the resulting CSV into GitLab CE as you will get corrupted issues. Because apparently GitLab CE has its own, incompatible CSV format compared to GitLab EE (because, not enterprise, apparently). So keep reading…

Perl like it’s 1999

Of course, since I’ve never gotten anything written in Perl ever to run first time in the last three decades of working with computers, I first had to fix the following error:

Can't locate Text/CSV_XS.pm in @INC

Which I did by installing the Text::CVS_XS module:

/usr/bin/perl -MCPAN -e'install Text::CSV_XS'

This is a good point to grab a coffee while Perl installs half the known Internet onto your computer.

I say CE, you say EE

Now, if you update the script with your GitLab installation, repository, and access information and run it, it should work but you cannot import this export into your GitLab CE instance. When I did, I ended up with a bunch of issues that the URL of the issue on the original project for the title and the body. Because apparently GitLab CE’s import CSV format isn’t compatible with GitLab EE’s export format. So, instead, I had to hack the script to leave just the two fields documented in the GitLab CE import CSV alert box (title and description). Honestly, this is all I wanted anyway.

Finally

Here’s the updated script that should work. It creates exports a CSV file from your GitLab CE instance called issues.csv in the current directory that you can import back into GitLab CE to get the titles and bodies of issues transferred to your fork.

Update: FFS, I just realised that it imports all issues as open. Seriously, GitLab, can you please put both the fucking import and export feature into CE and stop making people jump through hoops for such basic functionality? Thankfully, I only have a handful of closed issues so I’m going to close them manually. I don’t have more time to waste on this right now but I hope this helps you tweak it further if you need to.

#!/usr/bin/perl -w

use strict;

use LWP::UserAgent;
use Text::CSV_XS qw( csv );
use JSON::PP qw(decode_json);
# Uncomment these for debugging
# use LWP::ConsoleLogger::Easy qw( debug_ua );
# use Data::Dumper;

my $PROJECT_ID="<project-id>"; # numeric project id, can be found in project -> general settings
my $GITLAB_API_PRIVATE_TOKEN='<your-token>'; # obtained from https://gitlab.com/profile/personal_access_tokens
my $baseurl = "<your-gitlab-domain>"; # or https://gitlab.com if using Gitlab.com

$baseurl .= "api/v4/";
my $issuesurl = $baseurl."projects/".$PROJECT_ID."/issues";

my @issues = ();
my $page = 1;
my $totalpages;

do
{
  my %query_hash = (
      'per_page' => 100,
      'page' => $page
  );

  print "Fetching page $page".(defined($totalpages)?" (of $totalpages)":"")."\n";

  my $ua = LWP::UserAgent->new();
  $ua->default_header('PRIVATE-TOKEN' => $GITLAB_API_PRIVATE_TOKEN);

  my $uri = URI->new($issuesurl);
  $uri->query_form(%query_hash);
  my $resp = $ua->get($uri);
  if (!$resp->is_success) {
      die $resp->status_line;
  }
  $totalpages = int($resp->header("X-Total-Pages"));

  my $resptext;
  $resptext = $resp->decoded_content;

  my $issuedata = decode_json($resptext);

  push(@issues, @{$issuedata});
}
while ($page++ < $totalpages);

my $outputfname = "issues.csv";
my $csv = Text::CSV_XS->new ({ binary => 1, eol => $/ });
open my $fh, ">", $outputfname or die "$outputfname: $!";

my @headings = [
  "Title",
  "Description",
];
$csv->print ($fh, @headings) or $csv->error_diag;

foreach my $i (@issues)
{
  my @values = [
      $i->{'title'},
      $i->{'description'},
  ];
  $csv->print ($fh, @values) or $csv->error_diag;
}

close $fh or die "$outputfname: $!";

print "Issues saved to $outputfname\n";

The Do’s and Don’ts of Tech Regulation

mail@ar.al (Aral Balkan) — Sat, 11 May 2019 09:19:36 +0100

The Al Jazeera segment from yesterday. Source (Warning: The video is embedded here without trackers. If you watch it on Al Jazeera you will be tracked, including by Facebook).

Yesterday, I was interviewed on Al Jazeera news about the Macron-Zuckerberg meeting (Warning: the linked page on Al Jazeera includes privacy-eroding trackers, including Facebook.)¹ in which they apparently decided on a framework of “co-regulation.”

This is not what I mean when I talk about the need to regulate surveillance capitalists.

Co-regulation is bullshit

Co-regulation is basically saying “hey, Big Corp, come, sit at the table and let’s decide together how to regulate you”. Alongside lobbying, revolving doors, and public-private partnerships, it is part and parcel of institutional corruption.

It is not the job of democratically-elected governments to elevate unelected corporate bodies to their own level to help write the rules of their own regulation. It is the job of such governments, in line with their democratic mandate, to regulate corporations in the public interest to reduce their harms.

But what else do we expect of Macron exactly? He is a neoliberal who is on the record for wanting to make France “a start-up nation.”

Furthermore, we exist within a climate where our democratic governments are so clueless, corrupt, or powerless (or all of the above) that instead of protecting the interests of their citizens, they are sending ambassadors to Silicon Valley corporations.

Piece Of The Pie Regulation

When governments do manage to find the will to regulate it is usually for one of two reasons:

Government surveillance and censorship
Business concerns (antitrust and competition)

In other words, for almost entirely² the wrong reasons.

From now on, I’m going to call this type of regulation Piece of The Pie Regulation because it essentially boils down to either governments or the greater business community wanting a piece of the pie while ignoring the human rights concerns at the heart of surveillance capitalism and the business model of people farming.

Government surveillance and censorship

“The NSA didn’t wake up one morning and say let’s spy on everybody. They woke up one morning and said all these companies are spying on everybody, let’s get ourselves a copy.”

Bruce Schneier

The first form of Piece of the Pie Regulation is where governments regulate tech companies because they want access to their data, algorithms, and platforms for their own surveillance, policing, and censorship desires.

We can see this aspect feature in yesterday’s Macron-Zuckerberg deal that sees the government of France gain unspecified insider access to Facebook’s data and platform.

While ideally Big Tech would love to have zero government interference in its affairs (this is a power struggle, after all), it is not fundamentally opposed to this sort of regulation. As Mark Zuckerberg said after yesterday’s meeting: “I’m encouraged and optimistic about the regulatory framework that will be put in place.” If a tech CEO is enthusiastic about a piece of government regulation, the one thing you can be sure of is that that regulation is not in your interests as an individual.

“I wake up every morning and I fight regulation, it’s what I do, it’s my job.” — Eric Schmidt, then-CEO of Google

The last thing we want are regulations that make corporations like Facebook better censors, better filters of our reality, and the authority on what is and isn’t acceptable and who is and isn’t a terrorist. And the only thing worse than a corporation with that sort of power is a government with access to it when it has a police force and and army attached.

We live in a world where autocrats like Turkish president Erdoğan maintain a stranglehold on the media and imprison people who criticise them as terrorists. How long do you think it will take for the Turkish government to knock on Facebook’s door demanding the same sort of access that the government of France now has?

Antitrust and competition

In monetary terms, antitrust regulation has netted surveillance capitalists the largest losses. Especially in Europe, regulators like Margrethe Vestager have levied fines ranging from millions to billions of euros on companies like Google and Facebook.

This is good. Yes, fine them. Yes, tax the shit out of them³. Yes, break them up.

But we must not limit our regulatory frame to antitrust and competition law. Because there is also a darker desire by regulators that approach the issue solely from this angle: they want all businesses – especially new startups – to have a piece of the tasty data pie held by companies like Google and Facebook.

Instead of calling for companies like Facebook to be broken up – like Elizabeth Warren in the US is doing – Vestager, for example, advocates that “we should get access to data from tech companies to push back and get into the market.” Which market? The surveillance capitalism market, where it is taken for granted that companies created centralised services that collect and monetise your data. No, this market is the problem.

Vestager’s call mirrors those of a recent government-commissioned report in the UK that suggests we “force the largest tech companies to open up to smaller firms by providing access to key data sets.” In other words, smaller firms should have a piece of the pie.

No.

The pie is toxic.

We need to bin the pie and bake a different, non-toxic pie.

Giving more people the ability to sell toxic pies does not solve the problem of toxic pies. If anything, it perpetuates the myth that there is no alternative to surveillance capitalism. Such a failure of imagination is what is currently starving ethical alternatives from the attention and investment they need to exist.

Regulation Do’s

So proper regulation is one half of tackling the problem of surveillance capitalism (the other half being investment in ethical alternatives). But what would proper regulation look like?

In order of effectiveness:

Algorithmic transparency: we should be calling for surveillance capitalists to open up their algorithms so we know exactly what they are doing with our data. Politicians like Marietje Schaake have recently started to flirt with the idea even if they’re not calling it by name yet.
Data minimisation: If we’re serious about regulating surveillance capitalists, this is the route we must take. See my proposal for a General Data Minimisation Regulation for an example that could kill surveillance capitalism tomorrow (good luck finding the political will, backbone, or imagination to implement that today).

That’s it.

We need regulation that limits the ability of surveillance capitalists to gather data, profile us, and grow. We must tax the shit out of them, break them up, and limit the data they can collect. What we must not do is open up their ill-begotten data to other companies and governments or implement wide-scale government surveillance and censorship alongside the existing corporate one.

We must go beyond Piece Of The Pie Regulation and beyond the “terrorists and pedophiles” (fear/security) and “business, business, business” frames to a human rights frame for regulation.

As long as we judge success based on what’s good for a “Digital Single Market”, we face an impossible challenge in getting policymakers to consider human rights, human welfare, and the health of our democracies as success criteria. Just as with the climate crisis, whether or not we can effectively regulate Big Tech depends on whether or not we can move beyond judging success based solely on economic indicators.

This is why I was happy to be part of the Reframing Digital Europe initiative alongside the Commons Network. Our work there has resulted in an alternative vision and policy frame for technology in the EU that we call Shared Digital Europe.

Surveillance capitalism cannot be reformed. No amount of regulation will magically transform a factory farm for human beings – which is what Facebook, Google, etc., are – into an animal sanctuary. What effective regulation can do is limit their harms and give ethical alternatives a chance to get off the ground.

So regulation is one half of the solution. The other half is incentivising the ethical alternatives.

Beyond regulation; beyond surveillance capitalism

An ethical alternative to surveillance capitalism in which we own and control our own data will not arise magically out of business as usual. It will not be funded by the same institutions that fund surveillance capitalism. It will not have the same success criteria. It will require investment from the commons in creating a peer web and Small Technology.

If you want to understand just what’s at stake (personhood and democracy), how things are at the moment (not good), what alternatives and stopgaps exist today (a few, even with zero funding), and the shape of the alternative ethical technological infrastructure we must invest in and build, watch this bootleg recording of my recent Bucerius Lab Lecture on Small Technology in Hamburg.

You can use a tracker blocker like our own Better Blocker or uBlock Origin to protect yourself. ↩︎
I have nothing against antitrust regulation as long as it is not the only frame by which we approach regulating surveillance capitalists. Breaking up monopolies is great. Let’s do that. Let’s fine them for anticompetitive behaviour, of course. But we must move beyond an competition frame to a human rights frame in order to effectively regulate surveillance capitalists. ↩︎
Given its de facto violation of human rights and toxic ramifications on our democracies, it’s time to start talking about taxing Big Data like Big Tobacco. ↩︎

Slavery 2.0 and how to avoid it: a practical guide for cyborgs

mail@ar.al (Aral Balkan) — Thu, 02 May 2019 15:09:33 +0100

The cover of Issue 32 of the Kulturstiftung des Bundes magazine.

This is the original English version of an article that I wrote for Issue 32 of the Kulturstiftung des Bundes (The German Federal Cultural Foundation) magazine. You can also read the German version. Thanks to Mauro Morales, there is now also a Spanish translation.

You are most likely a cyborg and you don’t even know it.

Do you have a smartphone?

You’re a cyborg.

Do you use a computer? Or the web?

Cyborg!

More generally, if you use digital and networked technology today, you are a cyborg. You don’t have to implant yourself with microchips. You don’t have to resemble Robocop. You are a cyborg because you extend your biological abilities using technology.

Reading that definition, you might take pause: “But wait, human beings have been doing that for far longer than digital technology has existed”. And you’d be right.

We were cyborgs long before the first bug crawled into the first vacuum tube of the first mainframe computer.

The caveman brandishing a spear and lighting a fire was the original cyborg. Galileo gazing into the heavens with his telescope was both a Renaissance man and a cyborg. When you pop in your contact lenses in the morning, you’re a cyborg.

Throughout our history as a species, technology has enhanced our senses. It has afforded us greater mastery and control of both our own lives and the world around us. Equally, technology has been used to oppress and exploit us – as anyone who has ever stared down the barrel of their oppressor’s rifle will readily attest.

‘Technology,’ according to Melvin Kranzberg’s first law of technology, ‘is neither good nor bad; nor is it neutral.’

So what decides whether technology contributes to our welfare, human rights, and democracy or erodes them? What separates good technology from bad technology? And, while we’re at it, what separates Galileo’s telescope and your contact lenses from Google and Facebook? And why does it matter whether we see ourselves as cyborgs or not?

We must all try to understand the answers to these questions. The price of not doing so may be very high indeed. These are not merely questions about technology. They are questions that cut to the heart of what it means to be human in the digital and networked age. How we choose to answer these questions holds fundamental consequences for our welfare, both personally and as a society. The answers we choose will determine the character of our societies and, in the long term, possibly even impact the survival of our species.

Ownership and control of the self in the digital and networked age

Imagine a world where you’re assigned a device at birth that watches and listens and follows you from that moment onwards. It can also read your mind.

Over the years, this device records your every thought, every word, every movement, and every interaction. It sends all this information about you to a powerful mainframe computer owned by a multinational corporation. There, these aspects of your self are collated using algorithms to create a simulation of you. The corporation uses your simulation as a digital proxy to manipulate your behaviour.

Your digital proxy is invaluable. It is everything that makes you who you are (apart from your physical body). The corporation understands that it does not have to own your physical body to own you. Detractors call the system Slavery 2.0.

All day long, the corporation subjects your simulation to tests. What do you like? What makes you happy? What makes you sad? What do you fear? Who do you love? What are you going to do this afternoon? It uses the insights it derives from these tests to get you to do what it wants. Perhaps to buy a new dress or vote for a certain politician.

The corporation is political. It must continue to survive, grow, and thrive. It cannot be hindered by regulations. So it must influence political discourse. Thankfully, everyone who is a politician today was also assigned the same device you were at birth. So the corporation owns their digital proxies too. This makes it much easier for the corporation to get its way.

All this said, the corporation is not omniscient. It can still make mistakes. It might infer incorrectly – based on your thoughts, words, and actions – that you are a terrorist when you are not. When the corporation gets it right, your digital proxy is an invaluable tool for manipulating your behaviour. When it gets it wrong, it might get you sent to prison. Either way, you lose.

Sounds like a cyberpunk science-fiction dystopia, doesn’t it?

Replace ‘the corporation’ with ‘Silicon Valley.’ Replace powerful mainframe computer with ‘The Cloud.’ Replace ‘the device’ with ‘your smart phone and your smart home assistant and your smart city and your smart this, that, and other.’

Welcome to Earth, circa present day.

The German version of the article in Kulturstiftung des Bundes magazine.

Surveillance Capitalism

We live in a world where a handful of multinational corporations have unfettered streaming access to the most intimate details of our lives. Their devices watch and listen and track us on our persons, in our homes, on the Web, and (increasingly) on our sidewalks and on our streets. These are not tools we own and control. They are the eyes and ears of the socio-techno-economic system that Shoshana Zuboff calls ‘surveillance capitalism’.

Just as in our fictional cyberpunk dystopia, the robber barons of Silicon Valley are not merely content with watching and listening. For example, Facebook announced at their developer conference in 2017 that they had 60 engineers working on literally reading your mind¹.

Earlier, I asked what it is that separates Galileo’s telescope and your contact lenses from the wares of Facebook, Google, and other surveillance capitalists. Understanding the answer to that question is crucial to understanding the extent to which the very concept of personhood in under threat by surveillance capitalism.

When Galileo used his telescope, only he saw what he was seeing and only he knew what he was looking at. The same is true for when you wear your lenses. If Galileo had bought his telescope from Facebook, Facebook, Inc., would have recorded everything he saw. Similarly, if you get your contacts from Google, they will come with embedded cameras and Alphabet, Inc., will see what you’re seeing. (Google doesn’t make those lenses yet, but they have a patent for it². In the meanwhile, if you can’t wait, Snapchat makes spectacles with cameras in them.)

When you use a pencil to write in your diary, neither the pencil nor your diary know what you’ve written. When you write your thoughts into a Google document, Google knows every word.

When you send a letter to a friend by regular post, the post office does not know what you’ve written. It’s a crime for anyone else to open your envelope. When you send your friend an instant message on Facebook Messenger, Facebook reads every word.

When you sign into Google Play Services on an Android phone, your every move and interaction is meticulously logged, sent to Google, stored forever, analysed, and used against you in the court of surveillance capitalism.

It used to be that we read newspapers. Today, newspapers read us. As you watch YouTube, so does YouTube watch you.

You get the idea.

Unless we (as individuals) own and control our technology, ‘smart’ is a euphemism for ‘surveillance.’ A smart phone is a tracking device, a smart home is an interrogation cell, and a smart city is a panopticon.

Google, Facebook, and other surveillance capitalist are factory farms for human beings. They make their billions by farming you for your data and exploiting that intimate insight into your life to manipulate your behaviour.

They are scanners for human beings. They exist to digitise you, to own that digital copy, and to use it as a proxy to grow even larger and more powerful.

We must understand that these corporations are not anomalies. They are the norm. They are the mainstream. The mainstream of technology today is a toxic spill of American crony capitalism that threatens to engulf the entire planet. We are hardly immune to its fumes here in Europe.

Our politicians are quickly entranced by the millions these corporations spend in the lobbies of Brussels. They are beguiled by the wisdom of Singularity University (not a university). Meanwhile, our schools stock Chromebooks for our children. Our taxes are lowered, so that surveillance capitalists are not unduly burdened should they want to order another Guinness. And our institutionally-corrupt policymakers are too busy organising data protection conferences keynoted by Google and Facebook to protect our interests. I know, because I spoke at one last year. The speaker from Facebook was fresh out of his job at the French data protection office, renowned for the beauty and efficiency of its revolving doors.

Something has to change.

And I’m increasingly convinced that if that change is to come at all, it must come from Europe.

Silicon Valley is not going to solve the problem it created. Mainly because companies like Google and Facebook do not see billions in profit as a problem. Surveillance capitalism isn’t broken by its own success criteria. It works perfectly for companies like Google and Facebook. They are laughing all the way to bank while laughing in the faces of regulators whose comical fines barely exceed a couple of days of revenue. It’s been said that “punishable by fine” means “legal for rich people”³. That’s doubly true when it comes to regulating trillion-dollar multinational corporations.

Similarly, venture capital is not going to invest in the solutions that would destroy the immensely lucrative business model it helped fund.

So when you see initiatives like the so-called Centre for Humane Technology with venture capitalists and ex-Google employees on board, ask some questions. And maybe spare a few more questions for organisations that purport to be creating ethical alternatives while being funded by surveillance capitalists. Mozilla, for example, takes hundreds of millions of dollars from Google every year⁴. It has taken more than a billion dollars from them in total. Are you happy to entrust them with building the ethical alternatives?

If we want to chart a different path forward in Europe, we must fund and build technology differently. We must have the courage to diverge from our friends across the pond. We must have the self-confidence to tell Silicon Valley and their lobbyists that we’re not buying what they’re selling.

And we must base all this on a solid foundation of human rights law. Did I say ‘human rights?’ I meant cyborg rights.

Cyborg rights are human rights

We are faced with a crisis in human rights law that goes all the way back to what we mean by ‘human.’

Traditionally, we draw the boundaries of the human self at our biological boundaries. Furthermore, we have a system of laws and justice that aims to protect the integrity of those boundaries and thus the dignity of the self. We call this system human rights law.

Sadly, this definition of the self is no longer adequate to fully protect us in the digital and networked age.

In this new age, we extend our biological abilities using digital and networked technologies. We extend our minds and our selves using modern technology. Therefore, we must extend our understanding of the boundaries of the self to include the technologies by which we extend our selves. By extending the definition of the self, we can ensure that human rights law covers and thus protects the entirety of the self in the digital and networked age.

As cyborgs, we are sharded beings. Parts of us live on our phones, parts on a server somewhere, parts on a laptop. The integrity of the self in the digital and networked age is the sum total of the integrity of those shards.

So cyborg rights are human rights as applied to the cyborg self. What we do not need are a separate set of – probably lesser – ‘digital rights.’ This is why The Universal Declaration of Cyborg Rights⁵ is not a self-contained document but an addendum to The Universal Declaration of Human Rights.

While constitutional protection of cyborg rights is a necessary long term goal, we do not have to wait for constitutional change before acting. We can and must start protecting ourselves by creating ethical alternatives to mainstream technology.

Ethical technology

An ethical technology⁶ is a tool that you own and control. It is a tool designed to make your life kinder and easier. It is a tool that enhances your abilities and improves your life. It is a tool that only acts for – and never against – your interests.

Conversely, an unethical technology is a tool owned and controlled by corporations and governments. It furthers their interests at the expense of yours. It is a shiny trap designed to capture your attention, addict you, track your every move, and profile you. It is a factory farm disguised as a playground.

Unethical technology is toxic for our human rights, welfare, and democracy.

Planting better seeds

Ethical technology does not grow on trees; you have to fund it. How you fund it matters.

Unethical technology is funded by venture capital. Venture capital doesn’t invest in a business, it invests in the sale of the business. It also invests in very risky businesses. A venture capitalist in Silicon Valley will invest, say, $5 million in 10 different startups, knowing that 9 of them will fail. So he (he’s usually a he), needs that 10th one to be a billion-dollar “unicorn” so he can get five to ten times his money back. (It’s not even his money, it belongs to his clients.) The only business model we know in technology that delivers that sort of growth is people farming. Slavery paid well. Slavery 2.0 pays well too.

We should not be surprised that a system that values cancer-like growth above all else has resulted in tumours like Google and Facebook. What is astounding is that we seem to be celebrating the tumours instead of treating the patient. And more perplexing still, we appear doggedly determined to infect ourselves with the same disease here in Europe.

Let’s not.

Let’s fund ethical alternatives.

From the commons.

For the common good.

Yes, that means with our taxes. It’s kind of what they’re for (to build shared infrastructure for the common good that advances the welfare of our people and our societies). If the word “tax” scares you or sounds too old-fashioned, just replace it with “mandatory crowdfunding” or “democratised philanthropy”.

Funding ethical technology from the commons doesn’t mean that we get governments to build, own, or control our technologies. Neither does it mean that we nationalise companies like Google and Facebook. Break them up! Sure. Regulate them! Of course. Do anything and everything that limits their abuses as much as possible. But the only thing worse than a corporate panopticon is a state one (not that these two things are mutually exclusive).

Let’s not replace one big brother with another.

Let’s instead invest in many small and independent not-for-profit organisations and task them with building the ethical alternatives. Let’s get them to compete with each other while doing so. Let’s take what we know works from Silicon Valley (small organisations working iteratively, competing, and failing fast) and remove what is toxic: venture capital, exponential growth, and exits.

Instead of startups, lets build stayups in Europe.

Instead of disposable businesses that either fail fast or become malignant tumours, let’s fund organisations that either fail fast or become sustainable providers of social good.

When I mentioned this plan several years ago at the European Parliament, my words fell on deaf ears. It’s still not too late to try. But every day that we delay, surveillance capitalism becomes ever more entrenched within the fabric of our lives.

We must overcome this failure of imagination and base our technological infrastructure on those principles that are the best of humanity: human rights, social justice, and democracy.

Today, the EU acts like an unpaid research and development department for Silicon Valley. We fund startups, which, if they’re successful, get sold to companies in Silicon Valley. If they fail, the European taxpayer foots the bill. This is madness.

The EC must stop funding startups and invest in stayups instead. Invest €5M in ten stayups in each area where we want ethical alternatives. Unlike a startup, when stayups are successful, they don’t exit. They can’t get bought by Google or Facebook. They remain sustainable European not-for-profits working to deliver technology as a social good.

Furthermore, funding for a stayup must come with a strict specification of the character of the technology it will build. Goods built using public funds must be public goods. Free Software Foundation Europe is currently raising awareness along these lines with their “public money, public code” campaign. However we must go beyond “open source” to stipulate that technology created by stayups must be not only public but also impossible to enclose. For software and hardware, this means using licenses that are copyleft. A copyleft license ensures that if you build on public technology, you must share alike. Share-alike licenses are essential so that our efforts do not become a euphemism for privatisation and to avoid a tragedy of the commons. Corporations with deep pockets must not be able to take what we create with public funds, invest their own millions on top, and not share back the value they’ve added.

Finally, we must stipulate that the technologies built by stayups are peer to peer. Your data must remain on devices that you own and control. And when you communicate, you must communicate directly (without a “man in the middle” like Google or Facebook). Where this is absolutely technically infeasible, any private data controlled by a third party (for example a web host) must be end-to-end encrypted and you must hold the only key.

Even without any statistically-relevant investment in ethical technology, there are already tiny groups working on alternatives. Mastodon⁷, a federated ethical alternative to Twitter was created by one person in their early twenties. A couple of people got together to create a project called Dat⁸ that could be the basis of a decentralised web. For over ten years, volunteers have been running an alternative non-commercial domain name system called OpenNIC⁹ that could empower everyone with their own place on the Web…

If even without any support these seeds are beginning to sprout, imagine what we could achieve if we actually started watering them and planting new ones. By investing in stayups, we can start a fundamental shift towards ethical technology in Europe. We can start to build a bridge from where we are to where we want to be. From a world in which corporations own us by proxy to a world in which we own ourselves. From a world in which we are again becoming property to a world in which we remain as people. From surveillance capitalism to a peerocracy.

Indie Web Server 9.1.0: Better error handling

mail@ar.al (Aral Balkan) — Tue, 30 Apr 2019 13:16:29 +0200

Indie Web Server version 9.1.0 is a minor release that handles the following two, related (and common), errors more gracefully:

Trying to enable a web server daemon when one is already enabled

Previously, this would succeed even though the earlier server would continue to be served. Now, it gives an error and instructs you to disable the existing server.
Trying to enable a web server daemon when some other service was using port 443

Previously, this would succeed but your server would only start running once (and if) the service blocking port 443 exited. Now, it gives an error and exits.
Trying to start a regular web server process when some other service is using port 443

Previously, this would fail but show a stack trace. Now, it shows a friendly error and no longer shows the stack trace.

Indie Web Server 9.0.0: Housekeeping

mail@ar.al (Aral Balkan) — Mon, 29 Apr 2019 19:59:24 +0200

I just released Indie Web Server version 9.0.0.

This is mostly a housekeeping release and nearly all the changes are under the hood.

Maintainability

I refactored the command-line application quite extensively to pull out all the commands into their own modules and remove any redundancy in the command-line argument parsing.

The whole thing is far more robust now and ready for the two new features I’ve been itching to add.

Breaking changes

There is a major version bump, however, and that’s due to two reasons:

The syntax of the local proxy option has changed.

You now have to specify the proxy command explicitly (providing a URL instead of a path is no longer automatically interpreted as a request to create a proxy server):
```
web-server proxy http://localhost:1313
```
This is to ensure that there is no ambiguity when I implement one the new features I’m planning that will also take a URL as its argument.
I’ve dropped Windows support.

Indie Web Server should continue to work under Node.js in Windows but it will no longer be supported and could break in the future.

If you want to use Indie Web Server on Windows, I highly recommend that you use Windows Subsystem for Linux to do so (with, for example, Ubuntu).

Now that the codebase is much more maintainable, I look forward to implementing two new features in the coming days. I’ll announce them once they’re ready.

Indie Web Server 8.2.0: Cascading archives for an evergreen web

mail@ar.al (Aral Balkan) — Sat, 20 Apr 2019 16:52:58 +0100

Cascading archives for an evergreen web.

I just released version 8.2.0 of Indie Web Server. This version brings with it a cascading archives feature to make it easier than ever for you to support an evergreen web and not break existing links as you evolve your sites.

Easier than the 404 to 302 technique for static archives

Indie Web Server already had native support for the 404 → 302 technique where you convert 404s into 302 redirects to earlier versions of your site, thereby preserving those links. However, that does require that you keep the older version of your site online separately. If the older version of your site is a dynamic one, that makes sense. But what if it is a static site or what if you can take a static backup of it? It would be so much simpler if Indie Web Server could serve that static archive for you automatically as a fallback if a path cannot be found on the latest version of your site.

This is exactly what the new cascading archives feature does.

And, just like everything else with Indie Web Server, it requires zero configuration, relying instead on a naming convention.

How it works

Let’s say you are serving your site from the my-site folder.

To use the cascading archives feature, just put the archive of your site into a folder named my-site-archive-1 so that the directory tree looks like this:

|- my-site
|- my-site-archive-1

That’s it!

If a path cannot be found in my-site, it will be served from my-site-archive-1.

You’re not limited to a single archive either (and hence the “cascading” bit in the name of the feature). As you have multiple older versions of your site, just add them to new folders and increment the archive index in the name. e.g., my-site-archive-2, my-site-archive-3, etc.

Paths in my-site will override those in my-site-archive-3 and those in my-site-archive-3 will, similarly, override those in my-site-archive-2 and so on.

Your old links will never die but if you do replace them with never content in never versions, those will take precedence.

I hope you enjoy the new cascading archives feature of Indie Web Server and I hope it makes it easier and more economical for you to contribute to an evergreen web where we try not to break links unless we absolutely have to.

Fixing the icon regression in Pop!_OS 19.04

mail@ar.al (Aral Balkan) — Sat, 20 Apr 2019 14:05:12 +0100

Ah, consistent icons.

After installing Pop!_OS 19.04 yesterday, my desktop experience became an eyesore as the previously consistent icon set was replaced with what I can only describe as icon vomit soup.

If you want to get your pre-19.04 consistent icon set back, do this:

Method 1 (update)

Pop!_Planet has released a “Pop Classic” icon theme that you can download and install.

You should use this method.

Method 2

Update: This was how I originally got the icons back. The Pop Classic method, above, is better as you can switch back and forth more easily. I’m now using that.

# Backup the current icons and get them out of the way.
sudo mv /usr/share/icons/Pop /usr/share/icons/Pop-19.04

# Clone the older branch and install it.
git clone --single-branch --branch=master_cosmic --depth=1 https://github.com/pop-os/icon-theme.git
cd icon-theme
sudo make install
sudo make post-install

Note: the post-install step failed for me but the consistent icons are back and my eyes are happier.

Consistency is the #1 problem with desktop Linux

Here’s hoping that Linux distributions understand that the biggest problem with desktop Linux today isn’t lack of apps, functionality, or performance… it’s consistency.

Distributions must enforce consistency. System76 had the right idea originally. It’s sad to see this regression in their approach.

Remember that the right of millions of people who use your operating system to a pleasing functional and aesthetic experience far outweighs the vanities of developers to subject people to their proud app icon concoctions in Gimp. We’re developers. We make things that we and other people to use. It’s not about us. It’s about them.

Indie Web Server 8.1.1: Reverse proxy (local mode)

mail@ar.al (Aral Balkan) — Thu, 18 Apr 2019 15:25:42 +0100

I just released Indie Web Server version 8.1.1 which introduces a reverse proxy feature.

What’s Indie Web Server? Watch this quick 2-minute demo video to find out.

Reverse proxy

In local mode (for development use), if you run the server with an HTTP URL instead of a path to serve, Indie Web Server will start as a reverse proxy.

It proxies:

HTTP ↔ HTTPS
WS ↔ WSS

I added this feature as we use Hugo quite a bit at Ind.ie for generating our static sites (e.g., this blog and the Ind.ie web site) and, by default, Hugo runs its server over HTTP on 1313. This is fine for development but if you run anything on localhost over HTTPS, most browsers have trouble letting you run anything else over HTTP again.

So if you’re going to test anything over HTTPS on localhost, it’s worth testing everything over HTTPS. Heck, with how easy Indie Web Server makes it, you really have no reason not to. Especially now with its reverse proxy feature.

Hugo cooking on https://localhost

So, for example, if you want to test Hugo over HTTPS on localhost, just run Hugo’s built-in web server as usual. e.g., for this blog:

hugo server --buildDrafts --renderToDisk --baseURL=https://ar.al --appendPort=false

Then install Indie Web Server 8.1.1 and start it as a reverse proxy using:

# Install it
wget -qO- https://ind.ie/web-server/install.sh | bash

# Run it
web-server http://localhost:1313

Note that the reverse proxy feature currently only works in local mode (not with the global or enable commands).

Enjoy! :)

This site now runs on Indie Web Server

mail@ar.al (Aral Balkan) — Wed, 17 Apr 2019 17:38:18 +0100

In the interests of eating my own hamster food¹, I just switched this site from nginx to Indie Web Server.

The only complication in the process was that I had to update the hostname of the server to match the domain name.

Otherwise, the whole process was basically as follows:

# Disable nginx.
sudo systemctl disable nginx

# Copy the web directory into a more human-sounding place
# (not necessary but I like it better, so there!)
cp -R /var/www/live.ar.al/html/ ~/site

# Update the hostname to match the domain.
sudo hostnamectl set-hostname ar.al

# Install Indie Web Server.
wget -qO- https://ind.ie/web-server/install.sh | bash

# Stop nginx.
sudo systemctl stop nginx

# Launch Indie Web Server as a startup daemon.
web-server enable

I also had to update the path in my rsync-based deployment script² since I’d changed the path of the web site files on the server.

And that was it! Within seconds, the site was up and running again on Indie Web Server.

Mmm, hamster food is yum!

🐹🥜

PS. Watch this two-minute video to see just how easy it is to install and run Indie Web Server from scratch.

What, dogs have a monopoly on the concept or something? ↩︎
Part of my Web+ system. ↩︎

Set up a live static personal web site in seconds with Indie Web Server 8.0.0

mail@ar.al (Aral Balkan) — Tue, 16 Apr 2019 20:20:20 +0100

Watch as I set up a secure static personal web site in seconds from installation to live.

On April 1st (no joke), Indie Web Server 7.1.0 introduced the ability to set up a live static web site in seconds on any server that had Node.js installed.

Two weeks of hitting my head on a wall and multiple rewrites later, Indie Web Server 8.0.0 now lets you do that without Node.js thanks to native binaries for Linux.

You can now literally go from installing the server to serving a fire and forget secure personal web site in seconds:

# Make a web page
echo 'Hello, world.' > index.html

# Install Indie Web Server
wget -qO- https://ind.ie/web-server/install.sh | bash

# Start your secure web site
web-server enable

Version 8.0.0 also introduces native binaries for macOS although you can only run regular processes, not daemons on that platform. (Linux systems with systemd are currently the only ones supported for production use.) And those of you on Windows can still install Indie Web Server via npm and use it as a development server with locally-trusted certificates:

npm i -g @ind.ie/web-server

I hope Indie Web Server makes your life easier. It is just one of the earliest modules to come out of the Hyhpa research and development project. It’s what I’ll be using to power the untrusted web node in Hypha.

See the Indie Web Server homepage and the source code repository and documentation for more details, including how you can use it as the basis of your own dynamic servers with Node.js.

As always, if you like and want to support the ongoing work of our tiny two-person not-for-profit, please fund us.

Indie Web Server 7.1.0: Launch a live secure static site with a single command

mail@ar.al (Aral Balkan) — Mon, 01 Apr 2019 11:10:51 +0100

Deploying a secure static web server now takes just one command and a few seconds.

You have: a VPS with a domain name pointing to it and Node.js installed.

You want: to deploy a secure, live static site.

You do: npm i -g @ind.ie/web-server && web-server --live

Hit your domain name in a browser.

The first time you do it, it will take a few seconds to load as your Let’s Encrypt certificates are being provisioned for you by ACME TLS. Then you’re up and running (with an A on the SSL Labs SSL Server Test), serving a static site from the folder you were in when you issued the command.

Your web server is running as a daemon thanks to the seamlessly integrated pm2 process manager. If you restart the server, your web site will be launched automatically. Ditto if it should crash, etc.

Monitor your live server via the seamlessly integrated pm2 process manager.

To monitor it:

web-server --monitor

To view the logs:

web-server --logs

To take it offline and stop it from launching at startup:

web-server --offline

For full details, please see the relevant section in the documentation.

Where’s all this heading? Read this.

Indie Web Server: now with native 404 to 302 support for an evergreen web

mail@ar.al (Aral Balkan) — Sun, 31 Mar 2019 12:31:06 +0100

Seamlessly handle links to earlier versions of your sites.

What if links never died? What if we never broke the Web? What if it didn’t involve any extra work? It’s possible. And easy. Just make your 404s into 302s.

Indie Web Server now has native support for 404 to 302.

Learn more at 4042302.org.

Custom error pages for Indie Web Server

mail@ar.al (Aral Balkan) — Sat, 30 Mar 2019 20:12:45 +0000

A custom 404 message.

I just released Indie Web Server version 6.3.0 with new default 404 and 500 error pages and support for custom ones.

A custom 500 message. Poor little baby monster is really taking it badly.

To create a custom error page for your static site, just create a folder at /404 or /500 in your web content and add, at a minimum, an index.html file in it.

Any assets you put in those folders can be addressed using standard relative links from the index.html file.

On your custom 404 error page, you can use the template variable THE_PATH to include the missing path that the person tried to access and on your custom 500 error page, you can use the template variable THE_ERROR to include the body of the error message.

For example, here’s an excerpt from the sample custom 404 error page that is used in the unit tests¹:

<h1>Hmm…</h1>
<img src="hmm-monster.svg" alt="Green monster, thinking.">
<p><strong>Sorry, I can’t find</strong> THE_PATH</p>

Your error pages will be served at the URL of the error itself and using the correct error codes (not, for example, using redirects).

New default error pages

The default 404 page.

I also added a tiny bit of life to the default error pages.

The default 500 page showing the test page at /test-500-error

Hope you enjoy them! :)

You can find both of the sample custom error pages pictured here in the test/site directory used by the unit tests. ↩︎

Upcoming talks: April

mail@ar.al (Aral Balkan) — Fri, 29 Mar 2019 12:22:34 +0000

In April, I will be speaking at two events, starting with The Straits Times Education Forum debate on entrepreneurship to be held at Singapore Management University on April 6th, 2019.

ST Education Forum

Some of the participants at this year’s ST Education Forum.

The Straits Times (ST) is Singapore’s most popular paper. I remember reading its Malaysian counterpart¹ as a kid in Kuala Lumpur and eagerly awaiting its tech supplement every Thursday to find out – among other things – exactly how many floppy disks the next version of King’s Quest would require.

At this year’s ST Education Forum, I will be joined by Sumitra Pasupathy, country director of Ashoka Singapore and Malaysia, to argue for the motion that “Entrepreneurs today do more harm than good.” We will be debating Professor Timothy Clark, SMU’s new Provost and Associate Professor Reddi Kotha, Academic Director of Master of Science in Innovation at SMU, who will be arguing against the motion. The debate will be chaired by Chua Mui Hoong, Opinion Editor at ST.

ST recently ran two articles on the event². Here’s some background on what I will be presenting:

The Silicon Valley model of entrepreneurship is often celebrated.

However, Mr Aral Balkan takes issue with the business model underpinning tech-based start-ups, termed “surveillance capitalism”, in which technology companies monetise the data captured through monitoring their users' online behaviour.

The co-founder of Indie, a social enterprise striving for social justice in the digital age, says that supporting sustainable businesses is in the common interest - arguing that the world needs “stay-ups”, instead of Silicon Valley’s disposable and surveillance-based “start-ups”. He also maintains that the antidote to surveillance-based Big Tech is “Small Tech” made by humans for humans.

EuropaCamp: React. Act. Democracy!

EuropaCamp: React. Act. Democracy! in Hamburg.

On April 26th, I will be speaking on Small Technology and the peer web at EuropaCamp, organised by ZEIT-Stiftung.

This will be the first time I’ve been back in Hamburg since I was there for some constructive disobedience with Diem25 during the G20 in 2017 and it follows my Bucerius Lab Lecture on Digital Emancipation that I gave at the same venue in 2016.

My talk is titled Small Technology: the antidote to surveillance capitalism for protecting human rights and democracy in Europe and beyond and here’s the description:

Big Tech, with its billion-dollar unicorns, has robbed us of the potential of the Internet. Fueled by the extreme shortsightedness and greed of venture capital and startups, the utopic vision of a decentralised and democratic commons has morphed into the dystopic autocracy of Silicon Valley panopticons that we call surveillance capitalism. This status quo violates our human rights, threatens our democracies, and casts doubt on the integrity of personhood itself.

Aral Balkan is a designer and programmer who has been making things with computers for the past 35 years. He’s spent the last five of those working on the problem of technologically regulating the abuses of surveillance capitalism as well as designing freedom-respecting alternatives to it. In this talk, he presents his latest thinking on how we can create alternatives to surveillance capitalism that cannot be co-opted. His suggestion is simple:

Think small.

Here’s to hopefully seeing some of you in person this month.

The New Straits Times. ↩︎
The second one includes a video of some of the local participants and students talking about the issue, which I’d highly recommend you watch if you have a few minutes. It’s awesome to see the next generation aware of and caring about the issues. ↩︎

How to trim an MP4 video without re-encoding it

mail@ar.al (Aral Balkan) — Sat, 16 Mar 2019 14:36:02 +0000

You have an MP4 file and all you need to do is trim a few seconds off from either the start or the end (or both). And you don’t want to spend time cutting it in a non-linear editor and re-exporting it and waiting for it to encode again.

If you do not need frame-level precision¹, there is a near-instant way you can do it using ffmpeg².

For example, I trimmed the last few seconds off my previous video – which I’d saved as obs-demo.mp4 – using the following command to take everything from the very beginning until the 2 minute 13 second mark and to create a new file called obs-demo-trimmed.mp4 with it:

ffmpeg -ss 00:00:00 -i obs-demo.mp4 -to 00:02:13 -c copy obs-demo-trimmed.mp4

Hope this saves you some time.

Source

George Chalhoub via StackOverflow.

This technique has keyframe-level precision. ↩︎
If you don’t have it, you can install it on Debian-esque systems with:
```
sudo apt install ffmpeg
```
↩︎

Using the Advanced Scene Switcher Plugin with manual overrides in OBS Studio

mail@ar.al (Aral Balkan) — Sat, 16 Mar 2019 14:26:00 +0000

Advanced Scene Switcher plugin: General Tab.

Open Broadcaster Software Studio comes with a built-in Automatic Scene Switcher that can switch between different scenes¹ based on the title of the active window². Sadly, at least on my Linux laptop running Pop_OS! 18.10, Chromium is not detected. As I’m going to be running Chromium to display my slides during my talk next week in Lille, this is a problem.

Enter the Advanced Scene Switcher OBS plugin, which can detect Chromium by window title and also offers a boatload of other criteria for automatically switching scenes. It does, however, require a tiny bit of configuration if you want to retain the ability to manually override the currently-displayed scene using the Scenes list in OBS and/or the multiview window.

Install Advanced Scene Switcher

To install the plugin on Linux, download and unzip it. Then, copy the advanced-scene-switcher.so file from the SceneSwitcher/Linux directory to your OBS Studio plugins directory. For me, that was:

cp SceneSwitcher/Linux/advanced-scene-switcher.so  /usr/lib/obs-plugins/

Instructions for other platforms are available on the plugin’s web site.

Disable Automatic Scene Switcher

Make sure you disable the built-in Automatic Scene Switcher (Tools → Automatic Scene Switcher → Stop) and restart OBS Studio.

Set up your window-title-to-scene mappings

Applications are detected using regular expressions and the Window Title.

Use the Window Title tab to add regular expressions to match and map them to the scenes you want them to trigger.

For example, when Scene Switcher detects that a window with a title that starts with “Tilix³” gains focus, it switches to the scene called Tilix.

Enable manual override

To enable manual overrides, you must tell the Scene Switcher to pause when OBS and the Multiview window are in focus.

This bit is important and it tripped me up initially. If you want to retain your ability to manually override the shown scene using the Scenes list or the multiview window in OBS, you must pause Automatic Scene Switcher for these windows. To do so, switch to the Pause tab in the plugin’s dialogue window and add the following two entries to the Pause the Scene Switcher when … is in focus list:

Multiview.*
OBS.*

That’s it. You should now have the Advanced Scene Switcher plugin properly set up to detect and switch to the corresponding scene for you when you switch applications and you should still be able to manually switch to you scene of choice using the Scenes list and the multiview window in OBS Studio.

Enjoy!

Scenes are combinations of sources (inputs). For example, if I want a screen that has a full-screen recording of my web browser with a picture-in-picture overlay of my camera, I would make a scene with two sources: one, a Window Capture source set to capture video from my web browser’s window and the other a Video Capture Device set to use my webcam. Then, I can scale and position the webcam video over the browser. ↩︎
More specifically, you can match a window title using regular expressions. ↩︎
In other words, a window with a title that matches the regular expression ^Tilix.*, which reads “starts with (^) the literal string Tilix (Tilix) and followed by zero or more characters of any type (.*). ↩︎

Open Broadcaster Software Studio is amazing!

mail@ar.al (Aral Balkan) — Sat, 16 Mar 2019 12:58:33 +0000

Can I be any more excited about discovering OBS Studio? (Hint: no.)

Open Broadcaster Software Studio (OBS Studio¹) is an amazing free software realtime switcher for live streaming and recording that works across all major platforms including Linux.

I used it to record my Introducing Indie Web Server video this week.

Right after that, I also recorded the second short video above to show you my setup².

OBS, alongside slides.com and reveal.js is also what I’m using to prepare for my opening keynote next week at Le Grand Barouf Numérique organised by The European Metropolis of Lille³. I’ll be documenting that setup as I go.

Which, I just realised, I was erroneously referring to as Open Broadcasting Studio until now. ↩︎
On these first two videos, I set the size of my output based on the size of the maximized app windows I was recording, which means that the aspect ratio is a little messed up (it’s not exactly 16:9). That’s why the video has black bars on the sides. I’ve now updated my setup and record the apps in fullscreen so that this will no longer be an issue in future videos. ↩︎
The theme is “Who runs The world?” The event aims to be “a two-day debate about our digital empowerment.” Find out more about my upcoming talks in Lille and Copenhagen in March. I’ll also post details on my upcoming talks in Singapore and Hamburg in April soon. ↩︎

Introducing Indie Web Server (video)

mail@ar.al (Aral Balkan) — Thu, 14 Mar 2019 18:29:11 +0000

Watch as I set up a secure development and production web server with a single command using Indie Web Server.

I just recorded a short video demonstrating just how simple and seamless Indie Web Server really is.

To install and run a secure local development web server and serve the current directory without certificate warnings:

npm i -g @ind.ie/web-server
web-server

To install and run a secure production web server and serve the current directory without certificate warnings, on the production machine (e.g., VPS server) that you have already pointed a domain name to, do:

npm i -g @ind.ie/web-server
web-server --global

Then hit your site at your domain name. The first time you hit the site, it will take a few seconds to load as Let’s Encrypt certificates are seamlessly provisioned for you and used. That’s it.

Note: If you want to run your production server so that it restarts should it crash or should the computer restart,you should use a process manager like pm2. Using pm2, the number of commands you need to set up a fire and forget personal web server balloon to a whopping four:

# Install pm2.
npm i -g pm2

# Set pm2 to run itself and your servers at startup.
# (This will output a command and ask you to copy and paste it. Do that.)
pm2 startup

# Install Indie Web Server.
npm i -g @ind.ie/web-server

# Run Indie Web Server using pm2.
pm2 start web-server -- --global

That’s it! You now have a production web server that restarts should it crash and is automatically launched on server restarts. It has never been this easy to run your own secure personal web server. Enjoy and please do let me know you get on with it.

For full details of the command-line syntax (and the API), please see the documentation.

Reclaiming your backtick/tilde key with a UK Macintosh key layout on an ANSI US Keyboard in GNOME

mail@ar.al (Aral Balkan) — Tue, 12 Mar 2019 01:36:46 +0000

Thonk, thonk, thonk!

Let me begin by acknowledging that this is most likely a niche use case. I am documenting this for my own future reference as much as anything else. That said, the technique can be used to remap or swap or alter your keyboard’s layout to your heart’s content.

As a recent owner of a Topre Realforce 87UW 55g keyboard¹ who uses UK Macintosh layout on his Linux box (🎶sosumi🎶), my one pain point is missing my beloved backtick²/tilde key: `~.

Now, of course, I hear you mutter, “but, Aral, you could just hold down your handy AltGR³ key and tap the ] key twice to create a tilde by abusing the diacritic⁴” Goes without saying really – but that’s no fun when you’re running a Unix-based system⁵ and you’re a programmer who loves template strings in JavaScript.

So, instead, here’s how to set the key marked as the `~ key on an ANSI US keyboard to actually be that key by assigning it the same values as the `~ key that falls between the Shift and Z keys on a UK Macintosh layout and which doesn’t exist on an ANSI US keyboard.

Backup your current keyboard mapping⁶:
```
xmodmap -pke > ~/xmodmap_original
```
Find the keycodes you need by running the following command and pressing the keys you’re interested in. You are going to override the values of the key you want to set with the non-existent key (in my case, I had to press the non-existent key on my laptop’s ISO UK keyboard):
```
xev -event keyboard
```

Create a file in your home directory (e.g., called my-keyboard-customisations) with your Xmodmap setting:

! Make the useless section key in the Mac/UK layout
! into the priceless backtick-shift-tilde key.
keycode  49 = grave asciitilde less greater bar brokenbar bar brokenbar grave asciitilde bar brokenbar backslash bar bar brokenbar

Make a simple script that will run xmodmap with your customisations file (e.g., in ~/bin/modmap):
```
#!/bin/sh
xmodmap ~/my-keyboard-customisations
```
Remember to make the script file executable:
```
chmod +x ~/bin/modmap
```

Create a GNOME Desktop file to run at startup (e.g., _~/.config/autostart/xmodmap.desktop) with the following contents:

[Desktop Entry]
Type=Application
Exec=/home/{REPLACE-WITH-YOUR-ACCOUNT-NAME}/bin/modmap
Hidden=false
X-GNOME-Autostart-enabled=true
Name=xmodmap
Comment=xmodmap script

Now log out and log back in again and your new settings should take effect.

And voilà⁷, just like that, I have my backtick-shift-tilde key back and can enjoy my beloved UK Macintosh layout but with extra thonk-thonk-thonks! 🤓

~~~ fin ~~~

Sources

Oh my goodness these keys are amazing to type on! ↩︎
Alt Graph (Right Alt). ↩︎
Or the accent grave, if you will. ↩︎
And similarly, you can create a backtick by layering two accent grave diacritics like this:

AltGr | | ↩︎
There’s no place like ~. ↩︎
You can return to your original mapping at any time using:
```
xmodmap ~/xmodmap_original
```
↩︎
In case you’re wondering what the accent grave diacritic is useful for, I just used it to write the a-grave in voilà:

AltGr | a

Of course, now that I have my backtick key back, I can use my Compose Key instead:

Compose ` a ↩︎

Setting multiple key bindings for the same action in GNOME

mail@ar.al (Aral Balkan) — Mon, 11 Mar 2019 21:44:37 +0000

Editing keybindings in dconf Editor.

In GNOME, you can only set one key binding for a given action using the Settings app (under Devices → Keyboard) even though the settings data structure itself accepts an array.

You can, however, set multiple key bindings per action by installing the dconf Editor app or through the command-line using the gsettings command.

To install dconf Editor:

sudo apt install dconf-editor

To change key bindings via the command-line (e.g. for switching the windows of an application):

gsettings set org.gnome.desktop.wm.keybindings switch-group "['<Alt>Above_Tab', '<Super>Above_Tab']"
gsettings set org.gnome.desktop.wm.keybindings switch-group-backward "['<Shift><Alt>Above_Tab', '<Shift><Super>Above_Tab']"

Indie Web Server

mail@ar.al (Aral Balkan) — Sun, 10 Mar 2019 13:15:20 +0000

Indie Web Server serving my blog over a TLS tunnel via ngrok.

Indie Web Server¹ is a secure and seamless Small Tech personal web server.

Use it to seamlessly serve your personal static web site in development and production or build your own dynamic web app on top of it using JavaScript and Node.js.

Indie Web Server is as easy as it gets.

Features

Zero-configuration – It Just Works 🤞™.
Develop and test with seamlessly-provisioned locally-trusted mkcert TLS certificates via Nodecert.
Stage and deploy with automatically-provisioned globally-trusted Let’s Encrypt TLS certificates via ACME TLS²

Install

$ npm i -g @ind.ie/web-server

Use

Command-line

Start serving the current directory at https://localhost:

$ web-server

Start serving the site directory at your hostname:

$ web-server site --global

For example, if you run the command on a connected server that has the ar.al domain pointing to it and ar.al set in /etc/hostname (on Unix/Linux/macOS), you will be able to access the site at https://ar.al. The first time you hit it, it will take a little longer to load as your Let’s Encrypt certificates are being automatically provisioned by ACME TLS.

API

You can also use Indie Web Server programatically as the basis of you own web applications.

Examples

Serve the current directory at https://localhost using locally-trusted TLS certificates:

const webServer = require('@ind.ie/web-server')
const server = webServer.serve()

Serve the current directory at your hostname using globally-trusted Let’s Encrypt TLS certificates:

const webServer = require('@ind.ie/web-server')
const server = webServer.serve({global: true})

Create a custom server:

const webServer = require('@ind.ie/web-server')
const express = require('express')

const app = express()
app.use(express.static('.'))

const options = {} // to use globally-trusted certificates instead, set this to {global: true}
const server = webServer.createServer(options, app).listen(443, () => {
  console.log(` 🎉 Serving on https://localhost\n`)
})

For full details of the command-line syntax and API, please see the documentation.

Previously known as HTTPS Server. ↩︎
Receives an A on the SSL Labs SSL Server Test. ↩︎

HTTPS Server: now with seamless Let’s Encrypt support

mail@ar.al (Aral Balkan) — Fri, 08 Mar 2019 10:19:57 +0000

Note: HTTP Server is now Indie Web Server.

You can now also use globally-trusted Let’s Encrypt TLS certificates.

Yesterday, I introduced HTTPS Server as a development server. What a difference a day makes! Today, with seamless Let’s Encrypt support via the excellent Greenlock.js module¹, HTTPS Server is ready for use as a secure Small Tech personal web server for development and deployment. It is a human-scale tool for creating and hosting single-tenant/personal web applications using Node.js.

With HTTPS Server you can:

Develop using seamlessly-provisioned locally-trusted certificates.
Develop/stage using seamlessly-provisioned globally-trusted certificates via a secure tunnel (e.g., with a pro or business plan at ngrok with custom domains).
Deploy using seamlessly-provisioned and renewed globally-trusted certificates (e.g., via pm2).

I plan to expand on each of the above use cases in future posts.

In the meanwhile, we have one more small piece of the puzzle in our fledgling effort to build a bridge from surveillance capitalism to peerocracy.

Although, what’s up with telemetry and tracking in a Node module? These “features” are turned off in HTTPS Server. ↩︎

Upcoming talks: March

mail@ar.al (Aral Balkan) — Thu, 07 Mar 2019 19:27:01 +0000

Who runs the world? The topic of Le Grand Barouf Numérique.

I’m giving two talks this month; at Le Grand Barouf Numérique in Lille, France on the 20th and at the Art, Technology, and Change conference at CPH:DOX, the Copenhagen International Documentary Film Festival in Copenhagen, Denmark on the 26th.

The Antidote to Big Tech (is Small Tech) – my session description at CPH:DOX.

At both events, I will be talking about how the bridge from surveillance capitalism to peerocracy will be paved by Small Technology and the peer web.

HTTPS Server

mail@ar.al (Aral Balkan) — Thu, 07 Mar 2019 14:16:15 +0000

Note: HTTP Server is now Indie Web Server.

HTTPS Server (source, GitHub mirror, npm) is a development server that uses nodecert to automatically provision and use locally-trusted TLS certificates.

Install

npm i -g @ind.ie/https-server

Use

Command-line:

https-server [folder-to-serve] [--port N]

All arguments are optional. By default, a secure HTTPS server will be created to serve the current folder over port 443.

If you do not already have TLS certificates, they will be created for you automatically using nodecert.

All dependencies will be installed automatically for you if they do not exist if you have apt, pacman, or yum (untested) on Linux or if you have Homebrew or MacPorts (untested) on macOS.

API

HTTP Server can also be used programmatically (e.g., with Express).

Example

const httpsServer = require('https-server')
const express = require('express')

const app = express()
app.use(express.static('.'))

const options = {} // (optional) customise your server
const server = httpsServer.createServer(options, app).listen(443, () => {
  console.log(` 🎉 Serving on https://localhost\n`)
})

For more details on the API, please see the readme.

Help wanted

If you get a chance to test HTTPS Server with the following setups, please let me know by opening an issue on the GitHub mirror:

Linux with yum
macOS with MacPorts

Nodecert

mail@ar.al (Aral Balkan) — Thu, 07 Mar 2019 14:01:37 +0000

Nodecert (source, GitHub mirror, npm) is a Node.js module and command-line tool that automatically provisions locally-trusted TLS certificates for your development environment using mkcert.

Install

npm i -g @ind.ie/nodecert

Use

Command-line:

nodecert

JavaScript (Node.js):

require('nodecert')

(Synchronous.)

Certificates

The generated certificates are placed in the ~/.nodecert directory.

Small Technology

mail@ar.al (Aral Balkan) — Mon, 04 Mar 2019 10:19:01 +0000

The antidote to Big Tech is Small Tech.

Big Tech, with its billion-dollar unicorns, has robbed us of the potential of the Internet. Fueled by the extreme shortsightedness and greed of venture capital and startups, the utopic vision of a decentralised and democratic commons has morphed into the dystopic autocracy of Silicon Valley panopticons that we call surveillance capitalism. This status quo threatens not just our democracies but the very integrity of our personhood in the digital and networked age¹.

While ethical design unambiguously describes the criteria and characteristics of ethical alternatives to surveillance capitalism, “ethics” itself is being co-opted by Big Tech in public relations initiatives that misdirect from the core systemic issues² to highlight superficial symptoms³.

We need an antidote to surveillance capitalism that is so anathema to the interests of Big Tech that it cannot possibly be co-opted by them. It must have clear and simple characteristics and goals that are impossible to misinterpret. And it must provide a viable, practical alternative to Silicon Valley’s strangehold on mainstream technology and society.

That antidote is Small Tech.

Small Tech

Is built by humans for humans⁴.
Is not for profit⁵.
Is built by individuals and organisations without equity capital⁶.
Is not sponsored by Big Tech/surveillance capitalists⁷.
Is private by default⁸.
Is peer-to-peer⁹.
Is copyleft¹⁰.
Favours small over big, simple over complex, and modular over monolithic.¹¹
Respects human rights, effort, and experience¹².
Is human scale¹³.

These criteria mean that Small Tech:

Is owned and controlled by individuals, not corporations or governments.
Respects, protects, and reinforces the integrity of personhood, human rights, social justice, and democracy in the digital and networked age.
Encourages hitherto impractical non-hierarchial political organisation and agency at scale.
Nurtures a healthy commons.
Is sustainable.
Will one day be funded from the commons, for the common good.
Will never make anyone a billion dollars.

Background reading: The nature of the self in the digitial age, Encouraging individual sovereignty and a healthy commons, and We didn’t lose control, it was stolen. ↩︎
We have a system in which 99.99999% of investment goes into funding surveillance-based businesses tasked with growing exponentially by violating the privacy of the general population. ↩︎
“Attention” and “addiction”. While it is true that surveillance capitalists crave to capture our attention and addict us to their products, they do this not as an end in and of itself but because the more we use their products, the more they can farm us for our data. Companies like Google and Facebook are factory farms for human beings. Their products are the farming machinery. They must provide a shiny façade to keep our attention and addict us so that we – the livestock – will willingly allow ourselves to be farmed. These institutions cannot be reformed. Big Tech can only be regulated in the same way we regulate Big Tobacco to reduce its harms on society. And we can – and should – invest in the ethical alternative: Small Tech. ↩︎
Small Technology is human-to-human (H2H) in nature. Specifically, it is not built by for-profit corporations to exploit individuals – what we call business-to-consumer (B2C) technology. It is also not technology built by corporations for other corporations – what we call “enterprise” or “business-to-business” (B2B) technology. ↩︎
We build Small Tech primarily for the common good, not to make a profit. This does not mean we disregard the economic system we find ourselves mired in currently or the fact that the alternatives we build must be sustainable. While we hope that one day Small Tech will be funded from the commons, for the common good, we cannot wait for our politicians and policymakers to wake up and implement such social change. While we find ourselves having to survive within capitalism, we can sell and make a profit from Small Tech. But that is not our primary purpose. Our organisations are primarily concerned with sustainable methods for creating tools that empower people without exploiting them, not with making a profit. Small Tech is not charity but it is not-for-profit. ↩︎
Organisations with equity capital are owned and thus can be sold. Contrast this to organisations without equity capital (for example, companies limited by guarantee in Ireland and the United Kingdom) that cannot be sold. Furthermore, if an organisation has venture capital, we can consider that it has already been sold at investment-time as, if it doesn’t fail, it must exit (be bought by a larger corporation or by the public in general in an IPO). The exit is what venture capitalists invest their clients’ money in. The exit is how those investors make their return on investment. We avoid this toxic poison pill in Small Tech by creating organisations without equity capital that cannot be sold. Silicon Valley has disposible businesses they call Startups. We have sustainable organisations working for the common good that we call Stayups. ↩︎
The revolution will not be sponsored by those we are revolting against. Small Tech rejects sponsorship by surveillance capitalists. We will not allow our efforts to be used as public relations to legitimise and whitewash the toxic business model of Big Tech and help them stave off effective regulations to curb their abuses and give the ethical alternatives a chance to flourish. ↩︎
Privacy is having the right to decide what you keep to yourself and what you share with others. Therefore, private-by-default is the only definition of privacy that matters. What this means is that we design Small Tech so that people’s data stays on their devices. If there is a legitimate reason why this is not possible (for example, we need an always-on node in a peer-to-peer system to guarantee findability and availability), we ensure that data is end-to-end encrypted and the individual who owns the tool has sole possession of the keys to their private information and sole control over who the “ends” are (to avoid the spectre of Ghosting). ↩︎
The core topology of our technology is peer to peer – a system without centres in which every node is equal. Nodes that individuals do not have direct control over (e.g., the always-on node in the peer-to-peer system mentioned in the previous footnote) are untrusted and unprivileged dumb relay nodes that never have the keys to people’s private information. ↩︎
In order to ensure a healthy commons, we must protect the commons from exploitation and enclosure. Small Tech uses copyleft licenses to ensure that if you benefit from the commons, you must give back to the commons. This also stops Big Tech from embracing and extending our work only to eventually lock us out of it using their vast concentration of wealth and power. ↩︎
Small Tech is influenced in no small part by the wealth of existing work by those inspirational designers and developers in the JavaScript community that birthed the DAT and Scuttlebutt communities. Their philosophy of creating pragmatic, modular, minimalist, human-scale components results in technology that is accessible to, maintable by, and at the benefit of individuals. Their approach, and ours, in turn, is based on the UNIX philosophy. ↩︎
Small Tech adheres to The Ethical Design Manifesto. ↩︎
Small Tech is built by humans, for humans; it is staunchly non-colonial in approach. It is not built by smarter humans for dumber humans (e.g., by “developers” for “users” – we do not use the othering term “user” in Small Tech. We call people, people.) We build our tools as simply as possible so that they can be understood and maintained and improved upon by the greatest number of people. We do not arrogantly expect people to put in undue effort to learn our tools. We invest undue effort ourselves in making them intuitive and easy to use. We implement beautiful defaults and layer the seams. Remember: complexity happens, simplicity you have to strive for. In Small Tech, “too smart” is a euphemism for dumb. In the words of Brian Kernighan: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.” We take the spirit of Brian’s quote and apply it across the board – from funding, to organisational structure, to the design of the product, to its development, deployment, and beyond.

Photo credit: Small Things, Big Things by Sherman Geronimo-Tan. Released under Creative Commons Attribution. ↩︎

Hypha: Glossary

mail@ar.al (Aral Balkan) — Mon, 18 Feb 2019 10:05:48 +0000

This is a work-in-progress glossary of terms for Hypha.

Term	Description
Hypha	A small technology peer web platform (work-in-progress) that provides individual sovereignty over those aspects of your self that you choose to express digitally and a means of communicating with others on your own terms without third-party mediators or filters. Hypha is free software and peer to peer. It aims to create a bridge from the centralised web to the peer web. Hypha is based on the Dat ecosystem.
Small Technology	Technology created by organisations for which exponential growth and making a billion dollars or “becoming a unicorn” is a failure state. Technology designed for individuals that respects, protects, and strengthens the integrity and dignity of the person and thus provides the basis for a healthy commons. Technology designed to further social justice.
A hypha	A single database that replicates according to the protocols of the Hypha platform. A hypha is addressed on the centralised Web by a domain name and on the peer Web by its hyphalink.
Node	A node within a hypha is any location that replicates that particular hypha.
Initialised node	A node that has a local copy of a hypha set up (even if it has not replicated yet).
Uninitialised node	A node that does not have a local copy of a hypha set up yet.
Passphrase	A secret Diceware phrase generated with a process that has at least 100 bits of entropy. All key material used in a hypha, including the hyphalink, are generated from this passphrase. The passphrase is never stored anywhere (people are highly encouraged to store it in their password managers).
Domain	A centralised ICAAN domain that points to the Relay Node.
The Forever Node ∞	An unprivileged always-on node that can either be hosted first-party or (more commonly) by an untrusted third-party. The Forever Node: a) provides distribution of the latest Hypha source to browser nodes (and perhaps, later, native nodes), b) acts as a bridge between the centralised web and the peer web by i) providing findability, ii) bridging replication between browser and native nodes, c) provides signalling etc. (e.g., for WebRTC), and d) provides availability. The domain points to the relay node.
Hyphalink	An Ed25519 public key that uniquely identifies a Hypha. A discovery key generated from this key is used with Hyperswarm to provide peer-to-peer findability for hyphas.
Authorised node	An initialised node that has write privileges for a hypha.
Unauthorised node	An initialised node that has read-only access for a hypha.
Signed-in node	An authorised node with an unlocked local writer.
Signed-out node	An authorised node with a locked local writer.
Origin node	The initial database that’s created for a hypha. The only purpose of the origin node is the authorise the first writer.
Writer	Any node with write privileges. Synonymous with authorised node.
Reader	Any node with real-only privileges. Synonymous with unauthorised node.

Privacy is not a science, it is a human right

mail@ar.al (Aral Balkan) — Thu, 14 Feb 2019 12:17:00 +0000

Given the levels of institutional corruption in academia and in the regulatory bodies and advocacy institutions that should be protecting our privacy, very few things shock me these days. So hats off to Bart van der Sloot for managing the impossible and finding a new low by framing institutional corruption as scientific neutrality in his article Dubbele petten in de privacywetenschap.

The gist of Mr. van der Sloot’s argument can be summarised with this doozy of a quote from his article¹:

Should privacy science be pro-privacy, or is it an undermining of the neutrality of privacy science? If privacy science should be neutral, why is there so much commotion about the sponsorship by commercial parties like Google, Facebook and Palantir and are there few words wasted on sponsorship by activist civil rights organizations such as the Electronic Privacy Information Center (EPiC), Privacy First and Bits of Freedom, which are outspoken pro-privacy? Does this not indicate that the criticism of sponsorship by commercial parties comes from persons who are not themselves neutral and objective, but actually pursue a pro-privacy agenda?

Where does one begin to dissect such a juicy turd?

“Privacy science” is not a thing

First off, let’s get this straight: privacy is not a science, it is a human right. There is no such thing as “privacy science”. There never was. There never will be. It’s not a science any more than human rights is a science. I can see what you’re trying to do, Mr. van der Sloot, and it’s duplicitous as all hell.

Let me tell you why privacy is not a science: because there is no scientific reason for us to have privacy any more than there is a scientific reason why we should not be slaves. Mr. van der Sloot purposefully conflates ethics, which asks “what is good?” and “how should we live?” with evolutionary biology, and perhaps sociology, which study the way things are and how they came to be the way they are and maybe even extrapolate to how they might be in the future given a certain set of constraints. The latter do not make value judgements. The former is all about value judgements. To the extent that studies in the latter follow the scientific method, we call them sciences. The former is not science, it is philosophy.

It is science that tells us how a projectile can be propelled from a hand-held device at such velocity as to cause terminal damage to another human being and exactly how it causes that damage. It is ethics that tells us we should not shoot people. Only people interested in providing some sort of pseudoscientific justification for their desire to shoot people conflate the two.

If you need Mr. van der Sloot to make my case himself, this next snippet of utter depravity should do the trick:

…the question is whether this does not make the image of the privacy scientist too one-sided and whether the name of the science is not being used or abused to take a personal standpoint. An example is the petition from January 2014 entitled ‘Academics against mass surveillance’, following the Snowden disclosures, in which it seems unequivocally suggested that mass surveillance is unlawful and unethical.

But are these statements sufficiently scientifically substantiated, or are they rather one-sided, emotional views? From a legal point of view, the assessment of these kinds of programmes is not so simple, which also applies, for example, to an ethical evaluation.

Thus, the name (‘Academics against mass surveillance’) of science is used to add lustre to a personal view, so the scientists who refrain from signing such petitions believe. They also ask the question whether ‘right-wing’ scientists should then start their own petition: Other Academics for Mass Surveillance. Moreover, in the area of privacy, you can continue to sign petitions. So here too, there are a number of question marks to be placed: why should one petition sign and another not?

In case reading the above paragraphs didn’t make your blood boil as it did mine, re-read them and replace “mass surveillance” with “genocide”. And remember that science is “neutral” on that subject too.

Heck, even Mr. van der Sloot’s core postulate on science being neutral doesn’t stand up to scrutiny. Science isn’t neutral, it is entirely biased in favour of facts. It is the specific purpose of the scientific method to separate fact from fiction. However, what it does not then do is to assign a subjective value to those facts. That, once again, is the purview of philosophy and ethics.

But wait a minute, why are organisations that purport to defend your privacy silent on this topic?

Having dismissed the kindergarten philosophy underpinning Mr. van der Sloot’s argument, let’s turn to a more practical matter because there is one question that Mr. van der Sloot raises that I would also like an answer to:

Why are, in Mr. van der Sloot’s words, so “few words wasted on sponsorship by activist civil rights organizations such as the Electronic Privacy Information Center (EPiC) [sic], Privacy First and Bits of Freedom, which are outspoken pro-privacy?”

Is it because these organisations are, as Mr. van der Sloot would like them to be, neutral when it comes to privacy? Are they objective bystanders, taking notes? If so, it would be helpful for us to know so we do not ascribe to them any sort of ethical or moral position on the subject of privacy so that we are not disappointed later. If they do, however, hold a position on the matter, now would be a good time for them to find their voices and their spines and tell us exactly what that position is.

I look forward to reading your responses to Mr. van der Sloot, EPIC, Privacy First, and Bits of Freedom.

A dangerous new attempt at reframing that we must nip in the bud

When I first became aware of the problem of surveillance capitalism and the threat it poses to personhood in the digital/networked age, I admit that I did naïvely believe that people who called themselves “privacy professionals” were there to protect our privacy, just like I like to believe that doctors are there to protect our health. In truth, the key word in that label isn’t privacy, it’s professional.

Privacy professionals are just people who get paid to work on the topic of privacy. Whether they work to protect or erode your privacy depends on who pays them. Given who holds the money and what their business model is (big tech and people farming), it is not surprising that most people who call themselves “privacy professionals” today work to erode, not protect, your privacy.

Even in such a corrupt status quo, the suggestions Mr. van der Sloot provides in his conclusion (“rules needed”) manage to convey a thirst for an even newer low:

What to do. Privacy science can connect to existing regulations of for example the KNAW and sector-specific regulations, such as in the medical sector. Or with the Dutch Code of Conduct for Scientific Practice of the VSNU, which states that a scientist must avoid personal relationships ‘that could raise reasonable doubt about the objectivity of his decisions’, that research must be carried out ‘scientifically soundly’ and methodologically correct on commission and that scientists must be transparent about their relationships, financial flows and any agreements with third parties.

All this is not enough, because it is too general and noncommittal. It would therefore be advisable to draw up further standards and guidelines for the integrity and independence of privacy science, but perhaps also more widely. Privacy is often found in ethical environments and privacy science permits ethical judgements more than once; this is precisely why it must conduct the ethical discussion about its own position and actions.

What Mr. van der Sloot is suggesting here is dangerous. It is a brazen attempt to further institutionalise the corruption in an already highly-corrupt field while shaming those already few who actually stand up for human rights and privacy. It is an attempt to get ethical privacy advocates de-platformed and potentially fired for being “unscientific”² and biased.

Mr. van der Sloot wants people working in privacy to be neutral. In the words of Desmond Tutu, “If you are neutral in situations of injustice, you have chosen the side of the oppressor. If an elephant has its foot on the tail of a mouse and you say that you are neutral, the mouse will not appreciate your neutrality.”

By this point, you might be wondering, “who is this Mr. van der Sloot you are talking about?” Who is this man who wants us to think privacy is a science (it’s not) and punish those privacy scientists (not a thing) who aren’t ‘neutral?’ What does he do? Well, he is the chief organiser of Amsterdam Privacy Conference sponsored by Google and Palantir. Oh, and he is also a docent at the Tilburg University on “Privacy and Big Data.”

Enough bullshit

In many ways, and without such intention, Mr. van der Sloot’s article shines an impossible-to-ignore spotlight onto the outstanding level of rot and corruption in academia and non-governmental organisations today. It also lays bare a new and dangerous effort to further cement such corruption institutionally by those that benefit financially from it.

Perhaps such a spectacular display of disdain of ethics, privacy, and human rights will be enough to overcome even the herculean efforts of suspension of disbelief practiced by some in our field to deny that we have a gigantic problem on our hands.

Or, to put it more simply, perhaps Mr. van der Sloot has produced a turd of such magnificence that even those who usually remain silent might see it fit to stand and comment.

I sure hope so.

Because the silence has been deafening as of late.

All quotes are translated from the original with DeepL. ↩︎
Not surprisingly, this newfound love for “science” is the same tactic used by the US factory farming industry in their attempts to push lower food standards on the EU, and most recently, on post-Brexit UK. ↩︎

On the General Architecture of the Peer Web (and the placement of the PC 2.0 era within the timeline of general computing and the greater socioeconomic context)

mail@ar.al (Aral Balkan) — Wed, 13 Feb 2019 14:30:00 +0000

There have been three eras of general computing… the fourth remains to be written.

A highly-compressed overview of the history of general computing

The first era of general computing was the mainframe era and it was centralised. The second, the personal computing (PC 1.0) era, was decentralised. The third is the Web era (let’s call it Mainframe 2.0). And it is, once again, centralised. Today, we find ourselves at the end of the Mainframe 2.0 era and on the precipice of a new chapter in the history of general computing: the Peer Web. Let’s call it the Personal Computing 2.0 (PC 2.0) era.

On taxonomic criteria and the greater context

The primary characteristics that determine our taxonomy of the four eras of computing are ownership and control, reach, and network topology. However, these neither exist in a vacuum nor do they emerge spontaneously.

Technology is an amplifier. What technology amplifies – in other words, the character of our technologies – is determined by the social, economical, and ideological context that funds and produces it to realise its objectives. Any taxonomy of technological artifacts not grounded in socioeconomics is akin to a classification of life devoid of evolutionary theory. Without understanding the social, economical, and ideological forces that forge our technologies, we cannot understand their true purpose beyond that which is stated by the marketing departments of the organisations that produce them.

Furthermore, given that technology is a multiplier, a crucial question for the future of humanity is what exactly is it multiplying? Given that the rate of change in technological progress is exponential, and given that this translates to an exponential increase in our ability to impact our own habitat, it is essential that we are able to answer this question correctly. And if we find that the feedback loop between our ideologies and their realisations (as enabled by the technologies they sponsor) are leading to outcomes that range from undesirable to catastrophic – from radically reduced freedom and human welfare at the optimistic end of the scale to extinction of our species at the other – then we must alter our ideologies and socioeconomic success criteria and ensure that technological infrastructure exists to amplify those more sustainable alternatives.

I must also state that I am not optimistic about our chances of effecting the change we need to stave off the worst outcomes. I am, at most, cautiously hopeful. This is part of a far more fundamental battle that rages inside of each of us: the struggle between our base and higher-order evolutionary traits (both of which have made us who we are today). Our base evolutionary traits once facilitated our survival in times when resources were scarce and our agency within our environment was limited. Yet, today, left unrestrained, they lead us to obesity and self-harm at the species scale. And yet we also possess higher-order evolutionary traits that manifest as our ability to forgo instant gratification. We have been known to reject the next readily available source of glucose and dopamine and engage in acts of enlightened self interest. These have translated into our forming societies, caring for others, and the creation of social safety nets. These higher-order evolutionary traits have had as much (if not perhaps a far greater in later times) impact on the success of humanity as a social species. If we care about our own freedom and welfare as individuals and as a society at the lower end of the scale and the survival of the species at the higher end, we must shun the former and embrace the latter traits. It remains to be seen if we are capable of doing so or whether the current levels of inequality within our systems have already crossed a tipping point beyond which they self perpetuate ad perniciem.

Finally, any analysis that overlooks the irrationality woven into the very fabric of our society cannot hope to make sense of the nonsense that shapes much of our current ideology, and, thus, gets translated into policy. These nonsensical aspects are, ironically perhaps, as scientifically grounded in the same evolutionary process I mentioned earlier. Irrationality simply wasn’t a negative evolutionary trait when our technological capabilities limited its potential fallout to a statistically-insignificant percentage of our habitat or our species. Back when, we could at most destroy cities and, at our most barbaric, kill some millions. Even with our relatively modest technology during the colonial era, however, it turns out that we managed genocide on such a scale that it led to a period of global cooling. Today, we have the technological capability to destroy our habitat and wipe out our species several times over. And it would be folly to assume that the ideologies that shape our policies and determine our success criteria, which fund our technologies, which realise our ideologies are in any way based on reason, enlightened self interest, or even a fundamental desire to survive or thrive in anything but the extreme present. Our current feedback loop incentivises, rewards, breeds, and elevates to great power and wealth the most sociopathic/psychopathic, shortsighted, and self-destructive of behaviours and the people that embody them. To break such a loop, we must look not only at which rational levers to adjust to alter our socioeconomic success criteria but also which irrational clutches prevent us from doing so. To put it simply, we cannot avoid catastrophe as long as we believe that limitless, exponential growth is possible in a closed system with scarce resources. And we cannot impart the responsibility necessary to alter our behaviour as long as we believe that we do not actually possess catastrophic agency over our habitat because we ultimately attribute agency at such scale to a magical sky fairy.

All that said, all we can do is to concentrate on the change that we can effect. Blissful ingnorance will not get us anywhere but neither will abject hopelessness. We must compose a rational understanding of the problem and concoct sustainable alternatives that can be adopted should the will arise. Whether or not our alternatives are adopted or used is unknowable and, to a great extent, irrelevant. Because what we do know is that unless the alternatives exist – even in some basic prototypical form – it is guaranteed that they will not be adopted or used and that disaster is inevitable. All we can do is to make our case as clearly as we can, attempt to the best of our abilities to inform and educate, build the tools that can fix the problem, and generally be the change we want to see in the world. This does not require optimism, only realism with imagination. This is not philantrophy. It’s humanity’s struggle to survive past its a painful adolescence and emerge as a responsible adult. This is about getting us from celebrating sociopathically-extreme shortsightedness and greed as success to embracing enlightened self interest.

Now put all that aside (but always keep it in your little finger) and let’s return to our taxonomy of technology.

Mainframe 1.0

In the original mainframe era, ownership and control of computers was centralised to a handful of wealthy organisations (military, academic, and corporate). The direct reach of the network was similarly limited to the people in those organisations. That is not to say, of course, that the impacts of mainframe computing did not extend beyond the walls of those organisations. One need look no further than IBM’s partnership with the Third Reich for evidence of its worst ramifications. (If IBM could enable the Holocaust using punch-card mainframes, imagine what Alphabet, Inc. could do today with the technology it has.)

PC 1.0

The first personal computing era began circa 1976 with the Apple I. It was the first time that individuals got to own and control computers. It was also the last decentralised era of technology. While the primary reach of computing expanded far beyond institutional walls and entered our homes, what was noticeably lacking (beyond the rudimentary bulletin board systems – BBSes – that were the purview of a strict subset of enthusiasts) was a network to tie all these personal computers together and enable them (and us) to communicate. Information in the PC 1.0 era was exchanged the old-fashioned way… hand-to-hand using floppy disks. In the PC 1.0 era, individuals had ownership and control of their machines. If, for example, you installed an app and it used your 9600 baud modem to phone home and report on your behaviour, we called it a Trojan – and we understood it to be malware. Today, we call a far more sophisticated and ubiquitous version of this the extremely lucrative business model of Silicon Valley.

Centralised Web (Mainframe 2.0)

There is a common misconception (perhaps owing to its stated aims or a conflation with the nature of some of the fundamental protocols of the Internet) that the World Wide Web is decentralised. It is not and never has been.

The fundamental architecture of the Web has always been client/server. In other words, centralised. The Centralised Web era was, in effect, a return to mainframe computing. The only difference is that the mainframes are now global in reach. Our new mainframes are household names like Google, Facebook, and Snapchat (and lesser-known data brokers and surveillance companies that lurk behind the scenes like Acxiom and Palantir).

The Web today is an oligopoly of multinational corporations with business models based on surveillance and ownership of people by proxy. It was the Centralised Web that ushered in the socioeconomic system we call surveillance capitalism. Capitalism, of course, has always relied on some level of surveillance and ownership of people, all the way back to its basis in slavery. What is different today is the nature and scope of the surveillance and ownership.

It is crucial that we understand that even in the early days, the Web was centralised. The centres (the servers) were just closer to each other in size. That changed when the Web was commercialised. The injection of venture capital with its expectation of exponential returns for Vegas-style high-risk betting provided the enzymatic pool that incentivised the centres to grow in tumour-like fashion until we got the monopolies of Google, Facebook, and their ilk. A new era of people farming was born. And while we diagnosed the tumours early, instead of recognising them as a threat, we started celebrating them and paving the way for a new type of slavery by proxy to sneak in through this digital and networked backdoor. This new slavery, let’s call it Slavery 2.0, is not as crude as Slavery 1.0. In Slavery 2.0, we no longer need physical possession of your body. We can own you by proxy by obtaining and owning a digital copy of you.

Here’s how it works: The so-called consumer technologies of today, with few exceptions, have two facets. There is the face you see: the addictive, potentially useful one that you interact with as “the user” and the one you don’t see. The one that’s watching you and taking notes and analysing your behaviour so that it can use the intimate insight it gleams from this constantly-evolving profile of you as a digital proxy with which to manipulate and exploit you. In retrospect, the World Wide Web is a most fitting name for the construct that enabled this. It’s a web with a giant spider in the middle. The spider goes by many names… Google, Facebook, Snapchat…

It’s also no coincidence that the centralised Web evolved alongside a period of unprecedented global concentration of wealth and power within the hands of a tiny group of billionaires. Surveillance capitalism, after all, is the feedback loop between capitalism (accumulation of wealth) and surveillance (accumulation of information). Surveillance capitalism is what you get when those with accumulated wealth invest that wealth in systems that result in the accumulation of information within the same hands which they then exploit to accrue further wealth.

The power differential between the haves and the have nots in surveillance capitalism is compounded not just by a widening gap in the wealth of the former versus the latter but also by the information the former has on the latter. To put it simply, if I know everything about you and you know nothing about me, I essentially own you by proxy. If, further, I dictate the tools you use to experience the world around you, I get to filter (and thus create) your reality. In the film The Matrix, people’s minds inhabit a virtual reality while their bodies are farmed in physical space. On Earth, circa 2019, we inhabit a physical space while our minds are farmed from a virtual reality. But, as in any good science fiction story, there is hope that a band of plucky rebels might just turn the tide in the face of overwhelming odds… and that brings us to the present day where we find ourselves witnessing and helping shape the next era of technology: the Personal Computing 2.0 era.

Peer Web (PC 2.0)

Where the Centralised Web is client/server, the Peer Web is peer to peer. Unlike traditional peer to peer, however, the Peer Web makes use of unprivileged always-on nodes to solve two of the usability problems that have plagued traditional peer to peer systems: findability and availability.

In order to bootstrap the Peer Web, we must make the provision of personal always-on nodes at a universally-reachable and human-readable address a seamless process. In other words, we must make it trivial for anyone to sign up for and own a node (hosted either by a third party or themselves) that guarantees universal findability via a domain name as well as close-to-constant availability. These two requirements are non-negotiable as they form the bare minimum necessary to compete with the usability of centralised systems. On top of this, however, we can layer those aspects made possible by a peer-to-peer and end-to-end encrypted architecture: privacy and censorship resistance.

Inverting the web

On the Peer Web, the always-on node, likely hosted by a third party, must be an unprivileged node. What I mean by this is that it must not know/have the paraphrase (and the secret keys derived from it) that the nodes that are physically under your control do. The always-on node acts as a signaling service, as a dumb relay (for example between browser-based nodes and native nodes) and as a software delivery mechanism for browser apps.

That last point requires clarification as its implications run contrary to the traditional best practices of web development. In traditional web development, sites are rendered on the server and delivered to the client. In the earliest days, web sites were entirely static. Then, we started adding functionality through JavaScript. This functionality was implemented as enhancements to the page (“progressive enhancement”). While this traditional model has been upended in recent years with the evolution of so-called single page apps (SPAs) that are rendered on the client using JavaScript, traditional best practice for the Centralised Web still holds that if your site doesn’t work without JavaScript, it is broken. For the Peer Web, we invert that rule:

On the Peer Web, if your always on node works without JavaScript, it is centralised (and thus broken).

Trust but verify

An always-on node on the Peer Web is an unprivileged and untrusted node. Thus, when used to deliver an application to a browser, we must be sure that it is delivering exactly what we expect from it and nothing malicious. The way we do this is to make sure that we can verify the content that it serves. And that means that that content must be predictable. The method we use to verify the content is a combination of subresource integrity of the script and third-party verification of the integrity hashes in the HTML (e.g., via a browser extension). What this precludes is the implementation of any server-side implementation of app functionality. All browser node functionality must be implemented client-side.

The importance of the above paragraph becomes clear once you realise that aspects of a person’s identity are tied to pieces of information that only they possess and which they enter into their trusted nodes, including browser nodes. These passphrases (and the secret signing keys and encryption keys that are derived from them behind the scenes) must be kept secure as they are what secure the integrity of the person they belong to in the digital/networked age.

On the Peer Web, a strong passphrase (eg., a randomly-generated Diceware phrase with >= 100 bits of entropy) is the key to a database that’s associated with a domain name you own as well as an uncensorable identifier (a hash that addresses your database within a distributed hash table).

In terms of usability, we can (and should) carry forward familiar concepts and flows from the centralised world (such as “sign up” and “sign in” and “authorise device/node”) that abstract away the more confusing technical details of decentralised authentication and replication.

Building bridges

I mentioned earlier that the always-on node acts as a “dumb relay.” In this capacity, the always-on node works not just as a means to ensure findability and availability but also as a bridge between the browser and native apps (including, possibly, operating systems).

In technical terms, the always on node bridges between native and web by replicating with native nodes over TCP and bridging with browser nodes via WebSocket. Native nodes, of course, replicate among themselves via TCP, and browser nodes via WebRTC. The end result, from the perspective of the people who use such systems as everyday things, is a seamless and continuous experience across their various devices and between their browser and native apps.

I have begun to explore the genera architecture outlined here with a series of spikes in a project I’m calling Hypha (RSS, Source).

Hypha: a work-in-progress.

Hypha

The core technologies I’m basing Hypha on are the hyper* modules (hypercore, hyperdb, and hyperdrive) from the Dat project. While Hypha is the logical evolution of Heartbeat, and the continuation of our work alongside the City of Ghent last year on the Indienet project, it is an evolution of (and differs substantially from) both in many details.

Heartbeat, while it had the same overall goals, was limited in scope, design, and execution. It had a single privileged centralised node for signaling whereas in Hypha you get your own unprivileged always-on node. Heartbeat was also limited to a single device; Hypha is not. There are, also, aspects that are very similar. Heartbeat had a kappa architecture and replicated data using a modified version of Syncthing. Hypha will implement a kappa architecture (see this workshop and the design of Cabal for references) and use hyper* for replication. However, unlike Syncthing, which is a file backup tool that I was trying to shoehorn into being the engine of a decentralised web, Dat and the hyper* family are designed to solve the same problem I’m tackling. And, unlike Heartbeat, Hypha is not limited to a single platform and has both the browser and native apps as first-class citizens.

Hypha is also an iteration of our Indienet project with the City of Ghent last year (here’s a talk I gave at Eurocities describing it right before preparations for a new right-wing local government meant that our funding was cut for the project). The Ghent experience made me realise a number of things. First, that anything politically-funded is fragile and should be treated as such (as we should remember before putting our trust in city governments while designing systems. Your government may be progressive today and fascist tomorrow. Does the design of your system protect people against the worst-possible government that might ever come to power?) It also made me realise that I must follow a first-principles approach and that I need to take my time to fully grok and implement the fundaments of this system. (Specifically, that I cannot lead a team to think through these aspects for me.) In other words, slow down and (re)start small. That’s the approach I’m taking with Hypha and it is starting to really pay off. What Hypha will become is slowly evolving from a series of iterative spikes.

For any interested developers out there, the latest completed spike is Multiwriter-2 (Source), where I implemented and integrated an end-to-end encrypted secure ephemeral messaging channel that is used to provide seamless device/node authentication. The latest work-in-progress spike is Persistence-1 (Source), where I am now moving from using random access memory as storage to random access IndexedDB in the browser and random access file on native. This iteration is also going to be the first time I implement a flow similar to what will eventually be used.

The Persistance-1 Spike will get us closer to the actual flow in Hypha proper.

Hypha has no release date, no big reveal, and makes no promises. It’s what I’m calling my current work on as I continue to tackle the general problem I’ve been working on in one shape or other for the past six years. At this point, I’m not looking for collaborators as I’m still working through the basic concepts on my own. But if you’re a developer and you want to start playing with some of the code, please do. Although it would probably be a more useful introduction to the space if you take the Kappa Architecture Workshop and the Learn Crypto Workshop (find my work files here), start hanging out in the Dat chat room, exploring the Dat project, checking out the Dat blog and reading through the excellent documentation and reference material.

We’re at the very beginning of the Peer Web (PC 2.0) era. I saw yesterday that my blog, which is also available over Dat (use Beaker Browser to view that link), is currently the most popular Dat site on the Internet. Out of the ten that the researcher was able to find by scouring the top 2.4 million domains, that is. And Ind.ie’s web site (dat link) is the third most popular. That tells you just how nascent this all is. If you want to help write the next chapter of the Internet, now is the time to pick up your quills and join us in hyperspace.

As always, if you want to support my work, you can help fund our not-for-profit, Ind.ie, or just tell your friends about Better Blocker, the free and open source tracker blocker that Laura and I develop, maintain, and which we sell on macOS and iOS.

Hypha Spike: Persistence 1

mail@ar.al (Aral Balkan) — Tue, 12 Feb 2019 14:18:04 +0000

Note: Unlike previous spikes, I’m leaving this spike in a non-functional state and instead starting from scratch on the main project. See post-mortem

Source

source.ind.ie/hypha/spikes/persistence-1 (canonical location)
Github mirror (pull requests, issues, etc. welcome)

Scope

Following on from Hypha Spike: Multiwriter 2 this spike aims to:

Use persistent storage
Evolve the sign up / sign in processes accordingly

Design

Implementing persistance and getting closer to the actual flow.

Domain registration + hosting setup (out of scope, although see Hypha Spike: Deployment 1)
Origin node setup: an uninitialised always-on node is hit: the setup page is delivered. In browser (and, later, also via native app), person chooses a generated Diceware passphrase. The node that the person is using becomes the origin node.
Origin node replicates to the unprivileged always-on node via WebSocket. At this point the always-on node is initialised and set to replicate that particular database only.
The person sees themselves as signed in and can post entries. (For the purposes of this spike, simple text-only public posts – just so we have some data and can see things replicate.)
On some other node (browser or native), person accesses the domain. They can see the public posts as we set up a read-only local database and replicate them. We know which database to replicate based on a Dat-DNS lookup via the .well-known/dat location on the domain of the always-on node. See security note 1.

They do not see the positing interface. Instead, they see a passphrase field (not shown in above whiteboard sketch) and a sign in button. Entering a passphrase creates the deterministic keys and proceeds to request authorisation via the secure ephemeral messaging channel. The person is asked to authorise the request from an existing node (e.g., the browser(/native app*) they set up from). _* out of scope for this spike._
On a previously-authorised node (e.g., the origin node), the person approves the authorisation request.
On the newly-authorised node, the person sees the posting interface and can now write to the database.

Iteration plan

✔ (tag) Implement random-access-idb in the browser node.
✔ (tag)Implement random-access-file in the always-on node.
Update interface and flow according to the design notes above.

Future plans

Create a higher level Hypha authentication library with a simpler API that abstracts away the messaging aspect (@hypha/auth)
Tie in with Hypha Spike: Deployment 1
Add options to interface to selectively enable replication over WebSocket or WebRTC or both for testing.
Clean up the interface and carry out some general housekeeping on the code.

Upcoming spikes

multiwriter hyperdrive (carried over from multiwriter-2)

Development notes

Although the random-access-storage project states that the interfaces of the various random-access-* projects are the same, this is not entirely true for random-access-memory and random-access-idb (IndexedDB). This tripped me up while migrating from one to the other. I noted the discrepancy and suggested that we document it.
If you’re testing IndexedDB persistence on Firefox, make sure you are not browsing in private mode as it will fail silently.
I’ve decided to use the hostname as the identifier of the browser node database. As browser nodes will have to be served from a domain and as they will have the domain available regardless of whether they are online or, later, offline (PWA support), it feels like the correct identifier to use.
Initial load flow:

Initial load flow for browser nodes.
Regarding sign-ins: if a database is lost for any reason, we will need to recreate it. So I’ve reopened the reproducible writers pull request I was preparing for hyperdb. Update: On further thought, having a reproducible unique ID for browser nodes is a hard problem and this feels like a premature optimisation right now. Instead, I realised I can separate the sign in process (the provision of the passphrase) from the database creation (something that also becomes clear once the flow accounts for unauthorised, read-only nodes – i.e., people who are just browsing your public content). Also, we can encrypt the local secret key and persist the encrypted key in the browser. The combination of these should mean that we can get away with sacrificing a writer if the database is lost and create a new one instead. We’ll need to keep an eye on real-world usage down the road but it doesn’t feel like this should be a showstopper. The Multiwriter DEP, for example, states that “The design should easily accommodate dozens of writers, and should scale to 1,000 writers without too much additional overhead.” I will write a separate post to document the sign up/sign in and database creation flow.

Separating sign in from database initialisation. Legend: rK = read key (public key), wK = write key (secret key), eK = (symmetric) encryption key (secret).
I’m also thinking that the origin database shouldn’t be used for anything else but to create and authorise a secondary writer.

Setup flow where the origin database is used just to authorise the first writer.
Did some testing on hyperdb/hypercore (writer) recreation in a separate recreate-hypercore spike.

Post-mortem

Working on this spike has made a couple of things clear:

If I don’t want to introduce new concepts into the web experience (I don’t) and keep the flow to the traditional (sign up/sign in), I have to use two hyperdbs per instance: one signed out and one signed in:

A person is signed out if they haven’t entered their passphrase and/or their write key was not saved (unencrypted) localStorage. The latter will be an option (“keep me signed in”). When an authorised node is signed out, the write key is stored encrypted in local storage and decrypted once the person has entered their passphrase.

Since we must have the read and write key for a local writer in order to recreate a hyperdb, we cannot recreate a hyperdb when the write key is encrypted and the person has not entered their passphrase yet. In this case, we need a separate hyperdb, with a separate read and write key. This will allow us to implement the familiar (signed in/signed out) experience from the centralised web.
The spikes have morphed (especially during the last few ones) into iterations and have now grown to an unmanageable state. I’ve hit diminishing returns in this spike while trying to alter the core interactions. It’s time to start the project proper from scratch and using more maintainable practices. As such, I am going to leave this spike in its non-functional state and start the main project. I will still be using spikes in the future to explore specific issues but from here on, I will be iterating on the main project and on modules used by it.

Security considerations

Regarding design step 5: Remember that the always-on node is untrusted and unprivileged. We could easily set it up so that it returns the Dat URL in the rendered source but we won’t be doing that. The unprivileged node will return unaltered source that we will verify using subresource integrity (out of scope for this spike). We will also implement trusted third-party audits of the source (this could, for example, be handled by a browser extension that compares the hashes received as well as the hash of the source code with the trusted hashes from the source code repository).

Other: See Hypha Spike: Multiwriter 2

Reference

random-access-idb: Random-access-compatible indexedDB storage layer.
random-access-file: Continuous reading or writing to a file using random offsets and lengths.
DEP-0004: Hyperdb
DEP-0008: Multiwriter

Also see Hypha Spike: Multiwriter 2 Reference

Hypha Spike: Multiwriter 2

mail@ar.al (Aral Balkan) — Fri, 01 Feb 2019 15:04:03 +0000

Source

source.ind.ie/hypha/spikes/multiwriter-2 (canonical location)
Github mirror (pull requests, issues, etc. welcome)

Scope

Following on from Hypha Spike: Multiwriter 1 this spike aims to:

Simplify the node authorisation flow to a simple authorisation alert on already-authorised nodes.

Design

Peer-to-peer node authorisation: early proof of concept.

This is the onboarding and new node authorisation (sign up/sign in) flow:

Person signs up either via a native app or the web. The app or the browser becomes the origin node. The database is only writable on this node at this point.
Person signs in using their password on a second node (either native or web). A database is created on this node and replicates with the origin database but it is read only at creation. The person is asked to sign into an existing node and approve the new node.

Whiteboard sketch showing sign up (origin node) and sign in (read-only node) and the means available for peer-to-peer authorisation request of nodes via an encrypted ephemeral messaging channel.
The second node uses an encrypted ephemeral messaging channel extension to the Dat protocol to ask for authorisation. This request is sent browser-to-browser via WebRTC, native-to-native via TCP, and browser-to-native and native-to-browser via WebSocket (the always-on node proxies the requests via WebSocket but it cannot see the contents of the messages as it is an unprivileged node and doesn’t have the secret key).
On the origin node (or, later, on any other authorised node), the person is prompted to authorise the new node. When they do, the new node becomes able to both read and write to the common database.

Iteration plan

~~✔ Refactor to create reproducible local writers~~
~~✔ (tag) Authenticate (manually) using the node name and reproducing the remote node’s local keys~~
✔ (pull request) Update DEP-0000: Ephemeral Message (Extension Message) to support hyperdb.
✔ (tag) Implement an ephemeral messaging channel between nodes and use a JSON request to ask for authorisation of new nodes (WebRTC)
✔ Also add the ephemeral messaging channel to replication over WebSocket
✔ (tag) Also add the ephemeral messaging channel to replication over TCP
✔ (tag) Add ephemeral message deduplication to the browser and native clients as messages may be received more than once due to the always-on node relay.
✔ (source) Implement a secure ephemeral messaging channel as a Dat extension based on DEP-0000: Ephemeral Message (Extension Message).
✔ (tag) Implement the secure ephemeral messaging channel in the spike. (Currently only on WebRTC replication.)
✔ (tag) Extend the secure ephemeral messaging channel to support unprivileged relay nodes (the always-on nodes).
✔ (tag) Implement secure ephemeral messaging channel on WebSocket connection. (Encrypted messages are now relayed by the always-on node.)
✔ Update the native app (mock) to accept a secret key and set up the secure ephemeral messaging channel over a TCP connection.
✔ (tag) Fix: allow any authorised node to authorise any other node.

Pushed to later spikes:

Create a higher level Hypha authentication library with a simpler API that abstracts away the messaging aspect (@hypha/auth)
Add options to interface to selectively enable replication over WebSocket or WebRTC or both for testing.
Clean up the interface and carry out some general housekeeping on the code.

Future plans

Release the authentication functionality as a stand-alone module that can be used across projects – e.g., @hypha/auth

Upcoming spikes

persistence
multiwriter hyperdrive

Security considerations

Authentication is stateless and handled via encrypted messages over an encrypted connection. Valid handled messages could be stored in the persistent database and used to protect against replay attacks (this would require a man in the middle attack on an encrypted connection with very little possible gain as the message can only be acted upon from a writeable node. If an adversary has the writeable node already, they would not need to request authorisation. The only scenario I can think of is if they somehow came across a read-only node that had requested authorisation but hadn’t been granted it.)
As the messages are sent over an ephemeral messaging channel they cannot be used to denial of service the database as they are not persisted. They also cannot be used to denial of service the person’s experience (e.g., by causing endless alerts) as any messaged not encrypted with the secret key is simply discarded. Other techniques (rate limiting, etc.) can also be employed in a manner similar to how they are used in traditional APIs.
Hypha is app and browser-based. If a device containing authorised nodes is lost and stolen that is outside of Hypha’s jurisdiction. Proper device security should be employed (i.e., auto lock, password on lock, full-disk encryption). That said, if an unlocked device with an authorised Hypha node is compromised, we could consider having a ‘compromised’ message replicate to the other nodes. If the legitimate owner is still in possession of multiple nodes, they could send a compromised message from all those nodes, strengthening the trust in the message. Regardless, a single compromised message should be cause to not trust the database any longer. Additionally, the always-on node can be updated with a new database backed-up from the old one and Dat DNS could be used to point to the new database to enable the person to continue where they left off.
The secret keys are never persisted on nodes. They are regenerated as necessary from the passphrase.
If the always-on node is compromised, the adversary cannot write to the person’s database as the always-on node is an unprivileged node and has read-only access. If we decide to use Dat DNS (either host-based or DNS-record based), it could be used by an adversary to spoof the person’s database and replace it with another for anyone relying solely on the lookup. However, clients could check for changes in the read key and consider that a compromise unless the change is, for example, written into the database itself.
If both a writable node and the always-on-node are compromised that particular aspect of the person should be considered compromised.

Postmortem

In this spike, I added hyperdb support to Paul Frazee’s dat-ephemeral-ext-msg module and created an end-to-end encrypted secure messaging channel for use between nodes owned by the name person: secure-ephemeral-messaging-channel.

Any authenticated node can now securely authorise any other node as a writer.

Up to this point, the spikes have been using random-access-memory as the storage layer. In the next spike, I will expore persistence. My plan is to then look at the auth module refactor once the sign up/sign in process more closely implements real-world behaviour.

Reference

Historic links (Heartbeat – 2014)

Heartbeat was the initial precursor to Hypha. We were limited by lack of control over the replication engine we had chosen (syncthing). Other limitations were: it was single writer/device, it had a privileged centralised signalling server. That said, it basically used kappa architecture (although the term was independently being coined at about the same time) and did solve some of the same challenges we need to solve now using the Dat protocol/ecosystem. The design is very close to that of Cabal but with the addition of authentication and private messaging.

Interesting links

Hypercore protocol deep dive

mail@ar.al (Aral Balkan) — Thu, 24 Jan 2019 07:29:23 +0000

2019/01/24: This is a Work In Progress (WIP). I will be live-updating this post as I work on the spike. If you want to get streaming updates without having to refresh your browser, open the DAT version in Beaker Browser and toggle the live reloading feature. Please feel free to talk to me about this on the fediverse as I work on it, perhaps via Mastodon.

Last modified: Thu, 24 Jan 2019 07:29:23 UTC

At the heart of Dat is the hypercore protocol. Understanding it is fundamental to grokking how and why the higher-level libraries and tools in the ecosystem work the way they do and have the properties they have.

This is a documentation of my study of the hypercore-protocol code. It’s meant to be a personal reference. For introductory material as well as general references, please see the references section.

Overview

The hypercore protocol is defined using protocol buffers in schema.proto. From this, message.js is generated via npm run protobuf.
Initial message sent when establishing a connection is a Feed message that includes the discovery key and a nonce¹. If both parties have the read key (public signing key of the hypercore), then they can replicate from the second message onwards via an encrypted connection that uses XSalsa20 to encrypt the messages with the read key and the nonce.

Limitations

Lack of general ephemeral messaging channel

For certain use cases (e.g., ability to share public keys in multi-writer replication/presence), it would be useful to have a generic and ephemeral (non-persisted) messaging channel. There is currently a DEP by Paul Frazee for using the extension message feature in the protocol to implement this (DEP-0006: Session Data)

Also see: DEP: Ephemeral message extension pull request. This PR was closed but I’m not sure why exactly as – unless I’m missing something – the privacy concerns can be addressed by signing the messages with the Feed key as part of the extension protocol itself (comment).

The discussion on the DEP’s pull request is very valuable and concerns the use cases that we have for Hypha also.

Note: there are ephemeral messages within the protocol (e.g., keepalive) but no general means to send arbitrary ephemeral messages.

References

How Dat works (blog post): documentation for the Dat protocol
DAT protocol spec
Protocol buffers

Number only used once. ↩︎

Hypha Spike: Multiwriter 1

mail@ar.al (Aral Balkan) — Tue, 22 Jan 2019 13:00:03 +0000

Source

source.ind.ie/hypha/spikes/multiwriter-1 (canonical location)
Github mirror (pull requests, issues, etc. welcome)

Design

Following on from Hypha Spike: WebRTC 1 and Hypha Spike: DAT 1, this spike aims to explore:

Get multiwriter working with hyperdb.

Flow

Note: the flow is currently limited by not being able to pass a custom read and write for the local writer to hyperdb. I will be adding this functionality to hyperdb in the next spike. Once that’s in there, I should be able to reduce this flow to two steps: request permission + grant permission (via an interaction familiar to anyone who has ever authorised one device from another). I did not tackle that first as I wanted to get hyperdb and multiwriter working as-is myself before anything else. Taking it a step at a time.

Step 1

Load app in two different browsers. Press the “Sign up” button in left browser.

Step 2

Left browser initialises origin hyperbd and writes to it. Copy its hyphalink to the second browser and press the “Sign In” button.

Step 3

This creates a hyperdb for the same hyphalink in the right browser. It replicates the latest entry from the left browser (see issues, below) and writes to its local hypercore but those writes do not replicate as it has not been authorised to write to the main hyperdb by the origin node (left browser).

Step 4

We write a few more entries using the Write button in the origin (left) browser and note that they replicate as expected to the right browser.

Step 5

We write a couple of new entries into the right browser and note that, as expected, they do not replicate to the origin browser because the node in the right browser has not been authorised yet.

Step 6

We copy the local read key from the right browser and paste it into the Other node read key field in the left browser and press the Authorise button.

Step 7

We note that the right browser is now authorised as its last item replicates to the origin node (left browser). Again, see issues (I don’t know yet why only the last-written item is replicating and not the whole hypercore. I might have configured it incorrectly but I haven’t had a chance to look into it yet.)

Step 8

We add a few more items to the right browser and note, as expected, that they now replicate to the origin node from the newly-authorised second writer.

Backend

We also note that some of the entries have replicated to the always-on node over WebSocket. But not all (see issues).

Issues

To-do: investigate:

Only the last item added before and any items added after replication begins are replicated via WebRTC (the latter requires {live: true} in the options.) This does not manifest in replication over WebSocket (although, see below).
Above also appears to manifest over TCP. Why?
Also, recently noticed that only some items are replicating over WebSocket. Why?

Notes

Signalhub is now integrated into the server. You do not have to run it separately for WebRTC.
Although we are not creating a persistent database in this iteration (we’re using random-access-ram instead of, say, random-access-i(ndexed)db), we must validate as if we were. The actual app will persist the database so it would never present the option to authorise a node from the same platform/app combination (e.g., Browser X on Platform Y). What this means practically is that you must test with two different browsers when testing on the same machine.

Iteration plan

✔ (tag) Implement multi-writer via hyperdb
✔ (tag) Integrate Signal Hub*
✔ (tag) Get multiwriter working with hyperdb automatically generating the local key material and with manual local key copy/paste between web clients
✔ Mirror the spike to GitHub

This is about as far as we can come with the current state of hyperdb.

Once this is working, I have to extend hyperdb so that you can specify your own keys for the local writer so that we can have reproducible keys for writers.

Once that’s done, I’ll spike:

cross-node authorisation via ephemeral messaging over hyperswarm and public-key authorisation (this can be released as a stand-alone module that can be used across projects – e.g., hyperauth)

Pushed to later iteration:

For ‘sign-in’ feature, generate a hypercore based on the reproducible node id
Integrate WebRTC client*
Implement multi-writer via multifeed

* These are to make it simple for others to clone and run the spike with minimal effort.

General: document what’s necessary to implement proper multiwriter (i.e., with ability to both authorise and de-authorise writer nodes). We need to get the ball rolling on this if it isn’t already.

Upcoming spikes

Extend hyperdb to accept a hypercore factory in its constructor
hyperdrive

Limitations

It’s currently not possible to specify your own key material for the root hypercore in hyperdb. This means we cannot use hyperdb out of the box with the keys generated from our diceware passphrase (Issue #158). I will be resolving this issue as the next spike.

Further thoughts on device authorisation

Whiteboard sketch: thoughts on device keys and authentication in Hypha

As initially posted in the Datproject discussion room:

Been giving device authorisation in multiwriter some more thought and, contrary to my initial knee-jerk reaction, I think it can be done within the limitations of the current system. It should be possible to handle lost/stolen devices as long as the write key (secret key/private key) is never stored on any device. Since I’m using key derivation from a Diceware passphrase salted with the unique domain in Hypha, this should be easy to support in a usable manner (the passphrase has to be stored in a password manager – or a brain better than mine – anyway).

So I’m thinking that the master passphrase will not be tied to any hypercore and every device’s writeable hypercore will derive its keys from the master key with the name of the device used as the salt. This means that on any node where the person enters the passphrase and the name of the device to be authorised is known, its public (and private) key can be calculated and the device authorised in the local hypercore.

If a device is lost/stolen, lack of the passphrase will disallow further writing. Of course, this requires that the device is properly secured at other layers in the stack (i.e., auto lock, password on lock, full-disk encryption).

Postmortem

See flow, issues, iteration plan, and upcoming spikes for a good summary of how the spike went, what issues I ran into, and how I plan to resolve them.

Reference

Historic links (Heartbeat – 2014)

Interesting links

Hypha Spike: WebRTC 1

mail@ar.al (Aral Balkan) — Sun, 20 Jan 2019 14:08:58 +0000

A hypercore replicating over WebRTC between two browsers

Source code

Design

Following on from (and in conjunction with) the Hypha Spike: DAT 1, this spike aims to:

Replicate a hypercore from browser-to-browser using WebRTC.

Notes

Iteration plan

Create a simple single-page web application that replicates a passed read key (equivalent to the native replication script)

Usage

The servers

Run the Hypha DAT 1 Spike (blog post) in a separate browser and copy the hyphalink.

Run a local instance of signalhub

npm install -g signalhub
cd <dat-1-spike-directory>/server
signalhub listen -p 445 --key localhost-key.pem --cert localhost.pem

Paste the hyphalink into the hyphalink field and press the Connect button.

Postmortem

No special notes/gotchas. Worked out of the box.

Reference

Joe Hand’s Hyperdb web example

Hypha Spike: Diceware

mail@ar.al (Aral Balkan) — Tue, 15 Jan 2019 21:34:17 +0000

Pulled out from Hypha Spike: DAT 1.

Source code

Design

Screenshot of the completed spike.

Use Diceware for passphrase generation to ensure a high-entropy process.
Use budo in the spikes to enable lightweight use of bundling/modules/etc.

Postmortem

Pulled this out into its own Spike so that it doesn’t clutter the DAT 1 spike.

Starting with the Aspect Setup 1 spike, in this spike I:

Refactored to use budo so I can use requires, etc.
Was getting an error in budo due to Chokidar on Linux:
```
Error: Cannot find module 'fsevents' from '…/node_modules/chokidar/lib'
```
Ended up upgrading dependencies for budo (pull request) and also squashing this error as the error is in error (pull request). We will, of course, not be using budo in production but it’s fine for the purposes of this spike.
Started generating passphrases using Diceware and EFF’s New Wordlists for Random Passphrases

References

xkcd: Password Strength
Excellent explanation of above xkcd. Quote: “Security at the expense of usability comes at the expense of security.” – AviD
Passphrases that you can memorize — but that even the NSA can’t guess
Diceware (Source code)
EFF’s New Wordlists for Random Passphrases
eff-diceware-passphrase

Hypha Spike: DAT 1

mail@ar.al (Aral Balkan) — Mon, 14 Jan 2019 22:42:02 +0000

Screenshot of data replicated between the browser and the always-on node, and between the always-on node and a native client.

Source code (See iteration plan for links to specific tags.)

Design

Following on from Hypha Spike: Diceware, this spike aims to explore:

Creating an in-browser DAT data store using the keys generated in the previous spike
Replicating that datastore over a web socket connection with the always-on node and making it available over UTP

Notes

Iteration plan

Note: since the original spike, a further iteration has also added WebRTC support.

In-browser hypercore gotcha

When creating a hypercore in browser by manually specifying the read and write keys, you must convert the keys to Node.js’s Buffer type, you cannot use ArrayBuffer. Just submitted a pull request to the hypercore readme to make this explicit as this caught me out initially:

The [key] and secretKey are Node.js buffer instances, not browser-based ArrayBuffer instances. When creating hypercores in browser, if you pass an ArrayBuffer instance, you will get an error similar to key must be at least 16, was given undefined. Instead, create a Node.js Buffer instance using Feross‘s buffer module (npm install buffer). e.g.,
const storage = someRandomAccessStorage
const myPublicKey = someUint8Array

const Buffer = require('buffer').Buffer
const hypercorePublicKeyBuffer = Buffer.from(myPublicKeyAsUint8Array.buffer)

const hypercore = hypercore(storage, hypercorePublicKeyBuffer)

Callback in onwrite hook gotcha

If you implement the onwrite hook in the options passed to the hypercore constructor, you must explicitly call the passed callback at the end of your handler (cb()) or your appends will not work. (Pull request to update docs.)

WebSocket/replication gotcha with budo and livereload

Initially, I was getting the following errors while trying to replicate over the web socket connection:

Firefox:

The connection to wss://localhost/livereload was interrupted while the page was loading.
The connection to wss://localhost/hypha/f86a223b93b19929eee4e402480ac4d69ad4d8342b2f39b03f3dfd7d0fafbe93 was interrupted while the page was loading.

GNOME web:

[Error] WebSocket connection to 'wss://localhost/livereload' failed: Compressed bit must be 0 if no negotiated deflate-frame extension
[Error] WebSocket connection to 'wss://localhost/hypha/78575ce623d7e7ef8e55c7d888e36f64c4fcea9404b1073ca517f94cc32b08b4' failed: Compressed bit must be 0 if no negotiated deflate-frame extension

The issue is that budo’s web socket server and mine are clashing. It looks like Jim’s had it turned off also in his Shopping List Example (it defaults to off).

Postmortem

A strong passphrase is generated via EFF’s Diceware Word List.
From the passphrase, Ed25519 (signing) and Curve25519 (encryption) key material is derived via session25519.
The signing keys are used to create a hypercore in the browser.
The hypercore is replicated via web socket to the unprivileged always-on node (server)
The always-on node joins the hyperswarm and announces the hypercore via its discovery key.
A native client is run with the read key of the hypercore. It calculates the discovery key from the read key and uses hyperswarm to find and replicate the hypercore from the always-on node that originated in the browser.

Limitations

This spike proves only a subset the absolute basics of the Hyphanet design. See areas for future study.

Future improvements

✔ Specify read key as a command-line argument on the native node
Integrate signalhub

Areas for future study

✔ Browser-to-browser discovery and replication via WebRTC. See Hypha Spike WebRTC 1
Multi-writer/multi-feed
CRDT

Reference

Jim Pick’s DAT Shopping List demo (Tokyo version)
How DAT works

I was wrong about Google and Facebook: there’s nothing wrong with them (so say we all)

mail@ar.al (Aral Balkan) — Fri, 11 Jan 2019 12:17:06 +0000

It’s always difficult admitting you’re wrong. But sometimes, it’s exactly what you have to do in the face of overwhelming evidence to the contrary. So, today, I admit that I was wrong about Google, Facebook, and surveillance capitalism in general being toxic for our human rights and democracy.

You see, it simply cannot be true given how they are endorsed by some of the most well-respected groups and organisations in the world. All the evidence points to Google and Facebook being good actors who are not a threats to our privacy.

FSF and The Software Freedom Conservancy

CopyLeftConf 2019 is brought to you by Google, Microsoft, and the FSF

The FSF is the world’s foremost defender of software freedom. The Software Freedom Conservancy “is a not-for-profit charity that helps promote, improve, develop, and defend Free, Libre, and Open Source Software (FLOSS) projects.” This month, The Software Freedom Conversancy is organising CopyLeft Conference sponsored by Google, Microsoft, and the FSF.

In fact, Google is such a force for good in the world that they are allowed to sponsor a CopyLeft conference even though they ban CopyLeft licenses at their company.

If even the FSF doesn’t have a problem with having their logo right next to Google and Microsoft’s, who am I to criticise these corporations?

How can I blame any policymaker who points to that and says “Aral, you’re being a bit melodramatic about these companies, aren’t you?” They’re right. I must be. What am I trying to do? Out-Stallman Stallman? Madness!

Mozilla

Firefox defends your privacy. That’s why its default search engine is Google.

Mozilla is a not-for-profit for-profit organisation working to ensure “an Internet that truly puts people first, where individuals can shape their own experience and are empowered, safe and independent.” Mozilla makes Firefox, a browser that is Fast. Private. Fearless. That’s why its default search engine is Google¹.

Mozilla doesn’t have a problem with Google, frequently partners with them, and even uses Google Analytics.

If such an upstanding and ethical not-for-profit for-profit that cares so deeply about protecting our privacy has no problem with having Google as its primary search engine or taking hundreds of millions of dollars from them, who am I to criticise Google on privacy?

Apple

Apple’s devices protect your privacy. Google is the primary search engine on their browser. Google must also protect your privacy. (Photo: Chris Velazco, Engadget)

Apple’s business model is to sell products to people. They are proud that this separates them from companies that sell people to people. As their building-tall ad at CES says, “What happens on your iPhone, stays on your iPhone.”

As they state on their privacy page:

At Apple, we believe privacy is a fundamental human right … Every Apple product is designed from the ground up to protect that information. And to empower you to choose what you share and with whom.

Apple’s CEO, Tim Cook, has personally endorsed this commitment to privacy and this is why Apple has Google as the default search engine for their browser and why they welcome the 12 billion dollars of revenue that this deal brings in. Because Google is just like Apple and designs their services from the ground up to protect your privacy. Why else would Apple allow them on their phones and jeopardise your privacy?

If Tim Cook is happy having Google on the iPhone then surely there must be something wrong with me.

GNOME

GNOME makes it easy to use Gmail (by advisory board member Google). If you use FastMail, however, you have to do some work.

If Apple is too commercial an example for you, then there’s GNOME, led by the non-profit GNOME foundation. They make a popular and usable display environment for UNIX-like systems. It’s what I use on my daily driver.

GNOME doesn’t see a problem with Google. In fact, Google is on their advisory board and apps on GNOME have first-class support for Google’s services.

Geary, the email application, for example, has Gmail listed in first place in the account setup dropdown and makes it trivial to get started with a simple interface that’s far easier to use than setting up a generic IMAP provider.

If Gmail was bad for your privacy – if, for example, Google read all your messages and used it to profile you (I know, there I go with the nutty conspiracy theories again) – then GNOME foundation would not promote it in their software.

If they were forced to support Gmail just because it’s popular but absolutely hated doing so, they would put up a warning message to protect you. Something like “When you use Gmail, Google, Inc. will use the contents of your messages to profile you. Please only proceed if you understand the dangers.” But they don’t do that. On the contrary, they put it first and make it as easy as possible to set up and use so Gmail must be fine.

In fact, I think we have to start asking questions about some of these other email providers who say they don’t spy on you. Like FastMail, the service I use.

My fork of Geary has FastMail as a first-class citizen. This is because I must be a communist.

When I added support for FastMail to Geary, my changes were rejected. If FastMail was an ethical provider of email, I’m sure that this would not have been the case. No doubt the team would have fallen over themselves to promote an ethical email service over an unethical one that reads people’s emails and profiles them.

So now I’m worried about what GNOME knows about FastMail that I don’t. What are those sneaky FastMail people up to? Maybe we should all move to Gmail. It’s what GNOME recommends by default, after all.

Nordic Privacy Arena

We learned how to make Facebook ads at the Facebook keynote at the Nordic Privacy Arena. I was not clever enough to appreciate it.

The Nordic Privacy Arena is a yearly gathering of data protection officers and privacy professionals. At this year’s event, Facebook had a keynote and I was told by the organisers to be nice to Facebook and Google, keep my points to my own talk and not challenge the speaker with questions after his talk like I did at Mozilla’s session.

They were so right! What was I thinking? Thankfully, they cut my question from the video of Mozilla’s talk so you don’t have to be subjected to my radical conspiracy theory that if you’re getting hundreds of millions of dollars from a corporation you perhaps might not be working against its interests.

I apologise profusely for being such a silly little pain in the ass. After all, if the data protection officers and privacy professionals organising and attending this event see no problem with Facebook or Google, who am I to disagree? These people are tasked with protecting our privacy. Surely, they have our best interests at heart and understand the basics of their job well enough to know exactly what they’re doing when they invite Google to keynote their next event.

Furthermore, the Facebook keynote was by Nicolas de Bouville whose previous job was at the French data protection office (CNIL), famous for its magnificient revolving doors. So if Nicolas chose to go work at Facebook after CNIL, surely Facebook can’t be that bad.

And it’s not just Nordic Privacy Arena. RightsCon by AccessNow has no problem with Google or Facebook. As they state on their web site:

RightsCon is the world’s leading conference on human rights in the digital age, brought to you by Access Now … RightsCon offers organizations and businesses the perfect opportunity to generate exposure, goodwill, and growth. So when you consider whether to invest in RightsCon, remember sponsorship can positively impact your organization and business by demonstrating leadership, building your brand, engaging your community, and more.

Neither does Amsterdam Privacy Week. Nor CPDP. Nor FOSDEM.

No, it’s clear. It must be me. I must just be deluded. This is the only conclusion that makes sense given the avalanche of evidence.

War is Peace, Freedom is Slavery, Surveillance is Privacy

So, in light of the overwhelming support for surveillance capitalism by generally well-respected organisations that say they work to protect our human rights, privacy, and democracy, I have decided that I must be the one who’s wrong.

If Google, Facebook, etc., were even half as bad as I make them out to be, these organisations would not be partnering with or endorsing them.

I just want to apologise for being such a “Negative Nancy” and take this opportunity to thank the FSF, The Free Software Conservancy, Mozilla, Apple, The Nordic Privacy Arena, AccessNow, RightsCon, Amsterdam Privacy Week, CPDP, and GNOME for showing me the error of my ways. I now know, thanks to their moral leadership on this issue, that it’s perfectly fine for people working to protect human rights and democracy to take millions and billions from companies like Google and Facebook and to partner with them.

I must admit, I feel a little silly. The last five years would have been so much easier if only I’d understood this earlier.

Here’s wishing you all a world full of freedom sponsored by Google, privacy sponsored by Facebook, and democracy sponsored by Palantir.

And, if for some wild reason that’s not the world you want to live in, then maybe it’s time for some organisations to take a fucking stand. Start calling surveillance capitalists “surveillance capitalists.” Say no to their money. And stop legitimising these bastards.

Mozilla is now trying to diversify its revenue stream and, in some markets, has different default search engines. For example, it partners with privacy-championing Chinese search engine Baidu in China. ↩︎

Hypha Spike: Aspect Setup 1

mail@ar.al (Aral Balkan) — Thu, 10 Jan 2019 12:27:36 +0000

Design

My Boogie Board notes on the general design on Hypha

Philosophy

Your identity – your self – is a sharded aggregate of information¹. For an organism to have integrity it must have ownership and control over the aggregate of these various elemental shards that, combined, constitute its being.

In Hypha (subscribe via RSS), I will call these shards aspects².

For the purposes of Hypha, an aspect is defined by a secret known only to the person who owns it.

From this secret, we derive two keys³:

A key to obtain and read this aspect (“read key”)
A key to write to this aspect (“write key”)

Anyone with the read key can replicate your aspect and read any unencrypted information in it. The root of an aspect is public information. It is how other people find you.⁴

Your write key is used both to add to your aspect and, for private information, to encrypt it.

The owner of an aspect can write to it from any device that they own.

The aspect acts as a root index that links to both public and private collections of data. These collections may be interactions and include contributions by multiple people on multiple devices.⁵

Scope

This spike will focus on the basics of aspect setup:

Enter strong password on web interface
Generate ED25519 signing keys (read key and write key⁶) from password (via Argon2)
Generate Curve25519 encryption keys from signing keys
~~Generate root aspect DAT archive using the keys generated in Step 2.~~
~~Replicate the aspect from a separate node to test that it works as intended (e.g., command-line DAT running on a native client)~~

Steps 4 & 5 moved to the next spike (see postmortem).

All key generation happens on the client. The untrusted server (unprivileged always-on node) must never have the secret key.⁷

This spike is related to the Indienet publickey auth spike from last year. However, I no longer feel that publickey authentication is necessary for the client. The DAT archives are our source of truth, they’re what we replicate, and they already handle authentication. Visibility of sensitive data does not have to be controlled at the web interface level but handled through end-to-end encryption. I am therefore also leaning towards the web interface being a single-page application.⁸

Spike notes

Iteration 1

Branch

Review what we had with the Indienet project.

Iteration 2

Screenshot of Iteration 2

(Master branch.)

Use session25519 to generate the DAT keypair from a strong passphrase and the domain name as salt. Since domain names are globally unique, this is a strong salt, regardless of the fact that it is not random and could be short (e.g., ar.al).

Postmortem

I’m going with the approach in Iteration 2.
In the next spike, I will look at generating a DAT using the generated key material and replicating it to the always-on node via web socket.

Also see

Update: (2020-03-16) All Indienet links removed as that project is now over and we no longer own the domain. Our work continues at Small Technology Foundation. See Site.js and our ongoing research and development.

Indienet general cryptography policy
Indienet security spikes (docs, source)
Indienet configuration information docs
Hypha Spike: Deployment 1

References

I include biological aspects in the definition of information because our biology is also, at its fundaments, information like all else. ↩︎
In DAT terms, an aspect corresponds roughly to a multiwriter/multifeed hypercore collection. Neither are entirely fit for purpose in their current state although the former is a better fit for implementing a root aspect while the latter is better suited for implementing individual/group interactions (see kappa architecture). ↩︎
This is an ED25519 keypair. The public key is the DAT read key. From the signing keys, we also generate Curve25519 encryption keys. ↩︎
On your always-on node, it is analogous with your domain name. That said, and crucially, it also exists separately from your domain name (which is still a centralised and commercially-governed identifier). Your read key (more precisely, its hash, which we call the discovery key) is the canonical address for your aspect. Even if your domain name changes/goes away, your aspect will remain as long as at least one host is available on the network for it to be found and replicated from. ↩︎
Each collection/interaction is also a multifeed kappa architecture data store. ↩︎
While both the read and write keys are considered secrets for collections/interactions, for the root aspect, the read key considered public. It is possible to create a fully private aspect by not linking it to a domain name and not advertising the read key but that is not an initial use case for Hypha. It would be interested to see how it could be implemented with a native DAT browser (currently, Beaker Browser). ↩︎
It’s outside the scope of this spike but for future reference: We must protect the system from Evil Host/Evil App Store (or Evil Maid at Good Host, etc.) attacks. Since the setup of the private key happens on the client and must stay on the client unless we are to compromise all security and privacy of the system, we must ensure that the client is not compromised. For native clients, this can be achieved via combination of open source and reproducible builds. For web clients, we can use subresource integrity along with third-party validators/a web of trust. ↩︎
This is because the always-on node/web interface must be an unprivileged node. While the core of that requirement concerns the node not having the person’s secret, it also applies to not privileging it through additional functionality. I often say that we are building a bridge between the centralised web (the Mainframe 2.0 era) but what I’ve so far failed to state is that this is a one-way bridge. The goal isn’t to encourage travel in both direction but travel from Mainframe 2.0 to PC 2.0. So it makes sense to invest as much effort as possible in the forward-looking (native) clients physically owned and controlled by everyday people that we are trying to move them towards instead of the bridge itself. The bridge is essential but it is a means to an end. ↩︎

The post-Web is single tenant

mail@ar.al (Aral Balkan) — Wed, 09 Jan 2019 12:38:52 +0000

Hypha (subcribe to updates via RSS) is an exploration of what personal technology means in the digital/networked age. The goal is to create a bridge from the Mainframe 2.0 era to the Peer Computing (PC 2.0) era¹. When we talk about scale in peer computing, our focus is on creating systems that are human-scale.

To ensure that the systems we design are human-scale, we must favour:

Small over big
Simple over complex
Clarity over cleverness
Inexpensive over expensive

This is not an exhaustive list. But you get the idea.

For Hypha, it means that the always-on node is a single-tenant application/server.

This focus means that we can remove a host of complexity from the design and keep things small, manageable, and inexpensive.

When designing peer technology, we must nurture a profound respect for the limitations of individuals: whether that is time, knowledge, ability, or psychology. Not because of an elitist preconception about ‘the human condition’ that presupposes that some ill-defined ‘majority’ are lacking in any of those areas but because they constitute scarce resources for all of us.

It is no longer permissible to perpetuate silly rights of passage based on the myth that people must work hard to obtain, understand, and therefore deserve the tools that we create. We must put to rest the toxic myth that those who do not use the overly-complicated contraptions we create do so because of a lack of intelligence, ability, or desire. I do not use your crappy confusing dog’s arse of an application not because I do not care about my privacy or because I lack the necessary technical knowledge but because my name is Amanda and I’m a brain surgeon who works over 60 hours a week and has three kids at home. I simply do not have enough hours in the day to devote to deciphering the diarrhoeic mess you just dumped on my lap to save yourself the effort of thinking about anyone else but yourself while developing it. I think it’s time we laid to rest the stereotype of ‘even your grandmother’ and started designing for Amanda instead.

A system is only as simple as its most complicated part.

Simplicity is our greatest competitive advantage. And a system is only as simple as its most complicated part. This is why we must, at all times, think holistically about the overall simplicity of the system we are designing. Which necessitates that we think about the whole experience from the outset.

Now, I know from first-hand experience that this can be debilitating and result in developer’s block. So we must also proceed with the caveat of “for what is feasible given the current stage of development.” My goal at this point is to make it as simple as possible for other developers to get up and running with Hypha on their own domain and on their own VPS.

We will then iterate on making this a seamless process for everyday people who want to use Hypha as an everyday thing.

My goal is to keep development as modular as possible so that we will, hopefully, get lots of small modules and tools as Hypha progresses.

Success criteria for the PC 2.0 era

mail@ar.al (Aral Balkan) — Wed, 09 Jan 2019 12:23:52 +0000

The major success criteria for the Peer Computing (PC 2.0¹) era, as I see them:

Own and control your own identifiers. You and you alone should decide on your identifiers and your identifiers should be accessible as long as at least one of your devices is accessible on the network. For a concrete example of how this would work, see how data is addressed under the DAT protocol.
No privileged nodes. The network must not have any privileged nodes, especially ones hosted by third-parties. This does not mean it cannot have third-party hosted nodes. In fact, it is a prerequisite for bootstrapping such a system in an accessible manner that we have always-on nodes hosted by third parties (see always-on node, below). It does mean, however, that such nodes must be less-privileged than nodes that are under the physical control of people. Basically, this means that such nodes must never have the person’s private key.
Effortlessly own and control your own always-on node² on the Internet. For PC 2.0 to be adopted as an everyday thing by everyday people, it must provide the same (if not better) levels of findability and availability as centralised systems. This is simply not possible in a purely peer-to-peer network, at least in its early stages. The always-on node is training wheels for the peer-to-peer network we are building. It is the supports we print when doing 3D printing. We might be able to remove them one day (and we should design it so that it can be removed) but without it, the whole thing would collapse from the start.

You can either physically host your always-on node yourself³ or have it hosted by a third-party⁴. The always-on node is part of a personal peer-to-peer network of other more precariously-connected nodes that you have physical ownership of (e.g., your phone, laptop, home assistant, car, etc.)

Have the always-on node, unless it is hosted physically by you, be less privileged than the other nodes. (You must be the sole holder of the keys to the system so only devices you physically control will ever have your password/private key).
Have this system be as accessible, usable, and pleasurable as possible in totality. That includes criteria such as affordability⁵, ease of getting started (‘onboarding’), usefulness (functionality and usability), etc.

Deployment-first development

mail@ar.al (Aral Balkan) — Wed, 09 Jan 2019 11:59:46 +0000

Independent technology – ethical technology – must be as accessible as possible for its intended audience at every step of the process. That doesn’t mean it must be accessible as possible to everyone at every stage in its development but rather it should be accessible as possible for the people that are working on it or with it at any given point.

Hypha is currently at the start of its development stage and thus must be as accessible as possible to developers who want to follow along with its development, run it themselves, and possibly fork it off and try new things with it.

One of the greatest barriers to trying out new free and open source web-related projects is the difficulty in deploying them. This is because nearly all web-related projects are designed as multi-tenant services. They are designed to scale to being used by thousands if not millions or billions of people. And that brings with it a whole slew of complexity.

Case in point: I spent hours and failed the first time I tried to deploy Mastodon¹. One of my developers on the Ghent project last year gave up after struggling with it for most of a day. While I eventually managed to deploy Mastodon for myself initially via a “serverless” Heroku-ish service and, eventually, from source, today I do not maintain my own Mastodon instances. Instead, I offload that complicated and involved task to the wonderful Hugo who runs masto.host.

Which is why I’m looking into deployment-first development with Hypha.

Hypha Spike: Deployment 1

mail@ar.al (Aral Balkan) — Sat, 05 Jan 2019 23:40:29 +0000

Source code

Wait, what, we’re deploying Hypha (subscribe via RSS) – but we haven’t even built it yet?!

Exactly.

Philosophy

Design and scope limitation for the spike

There are two interrelated processes to deploying your own instance of Hypha:

Domain registration and/or DNS setup
VPS server setup
TLS setup

There is a somewhat cyclic relationship between these three steps as they each depend on the other for certain information.

The DNS setup requires the IP address of the server and the server needs to know the domain that it will be responding for. To complicate things a little more, the domain name has to propagate before we can obtain a free TLS certificate from Let’s Encrypt.

Also, steps 1 and 2 have a commercial aspect.

For the purposes of this spike, I want to concentrate only on Step 2: automating the VPS server setup.

Cloud-init

VPS accounts are available for a couple of euros per month these days and many support cloud-config syntax (examples) via the cloud-init standard by Canonical as part of the new instance provisioning process via a ‘user data’ field on their online forms or via their APIs. Supported operating systems include Ubuntu, Fedora, Debian, RHEL, CentOS, and others.

In this spike, I’m going to explore using cloud-init to set up a server so that we can automatically:

Install the latest Long-Term Support (LTS) version of Node.js in a manner that would make it easy to perform updates on it.
Clone and run an empty (‘hello world’) version of Hypha.

Thankfully, Canonical has a tool called multipass that lets you easily spin up Ubuntu instances locally and pass them a cloud-init file. I’ll be using that to iterate on the cloud-init script.

Notes

Spike source code repository
cloud-init documentation
cloud-init format supports Gzip compression as user-data is limited to ~16,384 bytes.

Add an account so you can ssh into the instance

#cloud-config
users:
  - name: <INSERT ACCOUNT NAME HERE>
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh-authorized-keys:
      <INSERT SSH PUBLIC KEY HERE>

Replace <INSERT ACCOUNT NAME HERE> with the account name you want (e.g., this is the <account name>@<your instance ip> that you will use to SSH into the instance).

Replace <INSERT SSH PUBLIC KEY HERE> with your public SSH key, which you can most likely find in ~/.ssh/id_rsa.pub.

For example, if your account name is indie, you want the instance to be called hypha, and you save the above file as cloud-init.yaml, you can start up a new instance and connect to it over SSH:

Create an launch the hypha instance:

multipass launch --name hypha --cloud-init cloud-init.yaml

List the available instances to find the IP address of the new hypha instance (e.g., 10.83.214.166):
```
multipass list
```
Connect via SSH:
```
ssh indie@10.83.214.166
```

Here is a good article on users and groups.

For the final cloud init file, with many more tasks, see cloud-init.yaml and read the comments.

Thoughts/to-dos/questions

Since TLS setup with Let’s Encrypt depends on domain name propagation, it is the last thing we must do (and is thus outside the scope of this spike). See dns-01 verification (examples). Can be used along with Lexicon for manipulating DNS records in a standardised way across providers. e.g., See this Dehydrated hook for Namecheap + Let’s Encrypt. Also interesting, IWantMyName has a Dynamic DNS interface which could possibly be used for this.
Node.js is perfectly capable as its own server and does not need to be proxied (e.g., by nginx) for single-tenant use.
✓ ~~Add link to spike source code repository~~
How long does server setup take?

About ~2 minutes 30 seconds. 2 minutes of that is our custom initialisation and 30 seconds the generic server setup. That incudes apt update/upgrade, Node.js install, PM2 install, etc.
Test the cloud-init script with a number of different hosts.

To explore in future spikes

Domain registration and DNS setup via third-party service and API.
Native app that handles onboarding (domain registration, DNS setup, VPS setup, TLS setup, and app setup) in a seamless experience.
As above but with the integration of a payment step for the domain registration and hosting.

Postmortem

We can get a server up and running with a Node.js app in ~ 2 minutes 30 seconds without any optimisations. This could be hugely optimised for everyday use later by having prebuilt server images but it is entirely acceptable as-is for use by developer to deploy their own copy of Hypha. Even when TLS is supported, the longest part of a developer getting up and running with their own node of Hypha will be the DNS propagation.

References

TLS

mkcert: a simple zero-config tool to make locally trusted development certificates with any names you’d like

Server setup

multipass: orchestrate virtual Ubuntu instances (supports cloud-init)
cloud-init: the standard for customising cloud instances
NodeSource Node.js binary distributions: for Ubuntu, etc.
PM2: Node.js Production Process Manager with a built-in Load Balancer.

Promising discoveries

(Unused in current spike but might be useful in the future.)

greenlock-express: Free SSL and managed or automatic HTTPS for node.js with Express…
nodenv: for managing node versions in production

Getting Green Recorder running on Wayland under Gnome on Ubuntu 18.10-based systems

mail@ar.al (Aral Balkan) — Tue, 01 Jan 2019 20:17:41 +0000

Green Recorder is an app for recording your screen on Linux which, as far as I know, is the only such app at the moment that works with Wayland. It’s what I used to capture the video of my segment on Al Jazeera News today¹.

I had trouble trying to install it on my laptop running Pop!_OS 18.10² with Gnome 3.30.

Here’s a brief summary of the issues I encountered and the workarounds I implemented to get it running.

E: Unable to locate package green-recorder

I was unable to install Green Recorder via apt³.

To fix:

I installed it from source:

sudo pip install pydbus
git clone https://github.com/foss-project/green-recorder.git
cd green-recorder
sudo python setup.py install

The build succeeded but running the green-recorder binary resulted in a series of errors.

Failed to open file “/usr/share/green-recorder/ui.glade”: No such file or directory

Traceback (most recent call last):
  File "./green-recorder", line 262, in <module>
    builder.add_from_file("/usr/share/green-recorder/ui.glade")
gi.repository.GLib.Error: g-file-error-quark: Failed to open file “/usr/share/green-recorder/ui.glade”: No such file or directory (4)

To fix:

sudo mv /usr/local/share/green-recorder /usr/share/

Could not load ‘/usr/share/pixmaps/green-recorder.png’

(green-recorder:12589): Gtk-WARNING **: 15:59:28.105: Could not load image '/usr/share/pixmaps/green-recorder.png': Failed to open file “/usr/share/pixmaps/green-recorder.png”: No such file or directory

To fix:

Move it to /usr/share/pixmaps/:

sudo mv /usr/local/share/pixmaps/green-recorder.png /usr/share/pixmaps

ConfigParser.NoOptionError errors

You encounter a number of errors as certain app options do not exist:

ConfigParser.NoOptionError: No option 'videoswitch' in section: 'Options'
ConfigParser.NoOptionError: No option 'audioswitch' in section: 'Options'
ConfigParser.NoOptionError: No option 'mouseswitch' in section: 'Options'
ConfigParser.NoOptionError: No option 'followmouseswitch' in section: 'Options'

To fix:

echo 'videoswitch = True' >>  ~/.config/green-recorder/config.ini
echo 'audioswitch = True' >>  ~/.config/green-recorder/config.ini
echo 'mouseswitch = True' >>  ~/.config/green-recorder/config.ini
echo 'followmouseswitch = False' >>  ~/.config/green-recorder/config.ini

(You can toggle all of these defaults in the interface later but they need to be set in the config file for the app to run properly for the first time.)

With those fixes, the app ran and I was able to get a fullscreen recording⁴.

Outstanding issues

Although full-screen recording works, the Select a Window button is disabled.

Selecting the Select an Area button results in a selection window but clicking the Apply button results in the following error:

xwininfo: error: No window with name "Area Chooser" exists!
Traceback (most recent call last):
  File "./green-recorder", line 453, in areasettings
    output = subprocess.check_output(["xwininfo -name \"Area Chooser\" | grep -e Width -e Height -e Absolute"], shell=True)[:-1]
  File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['xwininfo -name "Area Chooser" | grep -e Width -e Height -e Absolute']' returned non-zero exit status 1

I haven’t had a chance to look into those issues yet.

I also used Pitivi to edit it. It’s hosted on a paid-for Vimeo account and presented on my site via a simple video tag without any tracking. ↩︎
Based on Ubuntu 18.10. ↩︎
I also tried via Gnome Software and although it said it was installed, it wouldn’t launch. ↩︎
The only setting to record computer audio that worked for me was Monitor of Built-in Audio Analogue Stereo. ↩︎

Al Jazeera News interview: French “tech tax”

mail@ar.al (Aral Balkan) — Tue, 01 Jan 2019 16:55:12 +0000

Who’s more powerful today? Democractically-elected governments or multinational tech companies? Corporate taxation is just one element of a power play that has fundamental impacts on democracy and human rights.

I did a live interview on Al Jazeera News today on France pushing forward alone with a new tax on big tech companies.

Sometimes you have to stick a screwdriver in it (or how to liberate a Chromebook in ten easy steps)

mail@ar.al (Aral Balkan) — Mon, 31 Dec 2018 12:38:26 +0000

Google is about as open as a clam.

Over the holidays, I found a Chromebook that Samsung had given me to evaluate about six years ago and which had been gathering dust ever since. Coincidentally, Laura’s sister Annie had just told me that she needed a laptop. Hmm… Well, there was no way I was going to give her a Google spy device, so I decided to liberate the Chromebook from Google’s surveillance-based operating system (ChromeOS) and gift it to her.

Now, you would think given how people just love to harp on about how damn open Google is, that this would be easy to do. Just install a lightweight Linux distribution and be done with it, right?

Oh, you poor, naïve, dear. No, not even close.

Instead, what you have to do¹ is to physically open up the computer, short a pair of jumpers to disable the write protection and flash the firmware with something that isn’t designed to stop you from protecting yourself from Google’s surveillance machine.

Google is closed

Google is anything but open² and Chromebooks are not computers; they are corporate surveillance devices. A Chromebook is an inexpensive data milking device and you are the cow.

It’s no coincidence, for example, that they have tiny hard drives. Why do you need local storage when you can just put all your data on Google’s machines and use Google’s services for everything? And what if you decide to foil Google’s cunning plan and install a larger hard drive? Computer says, “no!” You can’t. Why? Because “security”, of course. Wink, wink!

Similarly, you cannot install a different operating system. And if you have the gall to try and dual boot, you are greeted with a nag screen on every boot. Why? Because “security”, of course. Nudge, nudge!

Giving the owner of a computer control over who can and cannot update the hardware or operating system is a valid security concern. Giving the manufacturer such control and making it as difficult as possible for the owner isn’t.

If Google really cared about people’s security, they’d have designed Chromebooks to ship with hardware keys which, when inserted, would enable the hardware or software to be updated. Boom! Problem solved. And they wouldn’t have designed a malicious firmware that tries to get you to revert to factory defaults on every boot once you’ve modifed your own system. This is one of the most owner-hostile features I’ve seen yet in a piece of consumer technology³. But then again, you never really own a Google Chromebook… it owns you.

How to liberate a Chromebook

Here are the instructions for liberating a Samsung Series 5 550 Chromebook (lumpy) and installing GalliumOS on it:

Push the developer switch to the right. This is a small switch at the back of the laptop on the right-hand side.
Place the laptop on its lid and disconnect the battery by sticking a paperclip into the battery disconnect hole.
Remove all eight screws using a Phillips screwdriver. Four of the screws are hidden under the pads for the feet, so you will have to remove those first by gently prying them loose using a small flat-head screwdriver.
Pry open the lid by sticking a small flat-head screwdriver in and gently moving it all around the edges. You will hear clicks as the plastic clamps come loose.
Locate the write protect jumper and short it. The way I did this was to find a small flat-head screwdriver and stick it into the plastic head of the jumper. I put non-conductive tape under and above the screwdriver to make sure it didn’t short anything else.

Screwdrivers for freedom.
Put the cover back on (but don’t clamp it shut) and start up the machine. You will see a scary warning screen. Press CTRL+D. Once ChromeOS boots, press CTRL+ALT+T to get to Crosh and type shell to drop into a terminal running Bash.
Enter sudo -s, followed by flashrom --wp-disable. You should see a success message. If you don’t check that you’ve disconnected the battery and that your screwdriver is properly shorting the right jumper.

One step closer to freedom.
Install SeaBIOS (Full ROM):
```
cd;bash <(curl https://johnlewis.ie/flash_cb_fw.sh)
```
Flashing firmware for freedom.
Reboot. If you see the SeaBIOS screen, then you can turn the computer off, remove the screwdriver, and close the computer up.

Freedom is a BIOS, best served cold. Or something.
Prepare a USB key with GalliumOS on it. Insert it into your USB slot and reboot. You will need to download the Sandy / Ivy Bridge version of Gallium OS. You can burn it to a USB key using a tool like Etcher.

Playing with i3-gaps on GalliumOS. (Gallium comes with xfce and xfwm.)

Your computer should boot into the GalliumOS installer. Follow the instructions to set up your free and open operating system and you should find yourself the proud owner of a fully-liberated Unchromedbook.

Enjoy having the freedom to do what you want with your own machine without being tracked and profiled by Google.

On the machine I prepared for Annie, I ended up covering up the corporate branding with some colourful unicorn and cloud stickers I found at the local stationary store⁴.

Because fuck you, Silicon Valley, you don’t get to own unicorns and clouds.

Sources

This is true for the Samsung Chromebook 550 model, at least. On other models, apparently you can open up the machine and remove a screw from the motherboard, etc. On all models, you have to flash the firmware to stop Google from nagging you on every boot. ↩︎
To prove me wrong, please just email me a link to the source code for Google Search, Google Mail, Google Docs, Google Play Services, and all of their proprietary, surveillance-based services as well as the free/open source licenses for all of the above. Thanks! ↩︎
Apple is engaged in similar shenanigans with its latest T2 chip. But then again no one ever accused Apple of being open. Still, at least they make their money by selling products to people instead of selling people to corporations like Google does. ↩︎
I was originally going to write something like “unchromedbook” or “liberated” on the case using some scrabble letters but ended ruining the tiles when the superglue exploded out of the tube, ruining the tiles and almost sticking my fingers together. That was fun (wasn’t fun). ↩︎

Tilingnome

mail@ar.al (Aral Balkan) — Sat, 29 Dec 2018 10:10:58 +0000

Tilingnome in action

Update 2019/01/02: I’ve discovered Slinger and started using that instead. Tilingnome gets in the way too much.

I like the idea of a tiling window manager like i3 (or i3-gaps, or sway) not necessarily because of their lightweight nature when compared to a fully-fledged desktop environment like GNOME but because of their potential organisational and navigational value¹.

Tiling up is hard to do

The problem with the current crop of tiling window managers, however, is that they are difficult to set up and configure and they suffer from a host of issues such as no out-of-the-box HiDPI support, screen tearing, etc.² You are also bereft of some of the everyday things you take for granted with desktop environments, such as having a built-in way to change your desktop image or manage locking on suspend, etc.³

So, yesterday, after getting i3-gaps to work as best as I could on my machine, I went looking to see if anyone had made an extension that gives you the operational aspects of a tiling windowing manager within my desktop environment of choice, GNOME. And lo and behold, I found Tilingnome.

GNOME 3.30 updates

Sadly, it didn’t work on GNOME Shell 3.30, so I first had to update it. Now it does⁴. The joys of free software.

It was also not clear immediately how it should be used because the keyboard shortcuts were not documented or exposed anywhere. Unless I’m missing something, this seems to be the norm for keybindings in GNOME extensions. The only way I could find of changing the bindings was to alter the source, recompile the schema, and reload the extension. The least I could do was to document the keyboard shortcuts in the readme.

In addition to the documented keyboard shortcuts, you will probably want to use the full-screen shortcut (on my distribution, F11) to focus on your current task from time to time.

To tile or not to tile…

Tilingnome has its own idiosyncrasies and gremlins. It fails to resize and position certain apps and can flake out when you leave full-screen mode in an app. I’m not sure if I will keep using as it seems to get in the way more than it improves my workflow at the moment. That said, it remains an option for managing workflow with multiple windows. And if you want to improve it, you can.

I’d love to hear your thoughts on tiling window managers and Tilingnome so feel free hit me up on my Mastodon.

This is a very personal and subjective measure and not a blanket statement. Depending on how comfortable you are with remembering and using keyboard shortcuts for navigating between windows, you will either love or hate a tiling window manager. Also, GNOME, out of the box, gives you the ability to use your pointing device to snap a window to the left, right, top, or bottom of your screen (or to maximize it). It also has keyboard shortcuts for doing all of the above. For many people, that will probably be enough. In fact, I’m not entirely sold on the benefits of a tiling window manager myself but I’m exploring the experience and their effect on my workflow out of academic interest. ↩︎
Most of which have workarounds. But again, you have to be willing to invest time in seeking them out and implementing them. As with most such things, we cannot expect everyday people to do this (because they might have two jobs and five kids, not because they are not smart enough) but it is a great learning experience if you want to understand how your computer and operating system works better. If nothing else, it leaves you with a profound respect for the amount of work a desktop environment like GNOME does. ↩︎
Again, you can add all of these things and more yourself. ↩︎
Or, more precisely, will do when my pull request is merged. Until then, you can use my branch. ↩︎

What does a private communicator look like?

mail@ar.al (Aral Balkan) — Sun, 23 Dec 2018 19:22:13 +0000

The SnapOnAir Lora CommunityCator lets you message without a SIM card.

Hypha is not about building a single product. It’s about exploring the possibilities and problem domain of private communication and what it means to have technology that enables privacy (and therefore personhood). At the same time, it isn’t about designing from the inside out. It’s not about building the protocols and waiting for the tools to happen. It’s about experimenting with both. And, in the process, hopefully sparking one or more everyday things that enable people to communicate with privacy.

So what could some of those everyday things be?

A purely ephemeral private P2P communicator.
A private communicator with stored/replicated history.
A personal web “site” that enables public publishing alongside acting as an always-on, always, available node to enable findability, signalling, and availability in an otherwise peer-to-peer system.
…

Given that we have to build a bridge, in every sense, from the centralised world we live in to the peerocratic world we want, the initiatives we embark on have the unenviable requirement of being successful in both worlds.

They must be economically viable, at least on a small scale, in the current world (so that, at the very least, we can feed ourselves and continue to work on them¹), and they have to be successful in enabling a new peerocratic infrastucture for society to safeguard personhood and usher in equitable, just, and sustainable peerocratic governance structures. In order to do either, the way people use and experience these platforms must be through convenient, usable, everyday things.

But what will these things look like?

Will they be purely virtual and accessed via apps or browsers on mainstream platforms? (Will mainstream platforms allow them to exist should they become successful?)

Will they be physical and self-contained? Possibly with control over the whole stack? Will they resemble the SnapOnAir Lora communicator by Philippe Cadic or the prototype communicator shown below by arturo182?

arturo182’s communicator prototype in action

Will we snap them together ourselves from readily available and/or free and open components, like the little touch-screen device below that I was snapped together this weekend that’s running a basic P2P replicated chat app in Node.js that uses kappa-core?

A home-made communicator.

Will they be combinations of the above?

With Hypha, it’s clear what shape our core protocols will have². What I’m looking forward to exploring in the coming year is the shapes the tools will take as we explore the problem space to solve specific use cases.

The protocols, without the convenient everyday things that enable everyday people to utilise them, are simply the theoretical ejaculate of masturbatory academics.

The tools, without the protocols that ensure privacy are not fit for purpose.

Here’s what I do know:

Whatever we build must be…

Small
Focused
Verifiable
Usable
Convenient
Useful

Where the centralised mainstream’s advantage is scale, complexity, and secrecy, ours is lack of scale, simplicity, and openness.

We must think small.

I look forward to exploring the small things in 2019.

And/or we must try to educate policymakers and nurture the political will fund the independent organisations that make these platforms from the commons for the common good and ensure that their products remain in the commons and cannot be enclosed. But I’m not holding my breath on this one. Not least because lobbying, revolving doors, public-private partnerships, and multistakeholderism – collectively, instutional corruption – makes it very difficult to get politicians and policymakers to embrace what is right instead of what is good for their wallets or what they’re exposed to day in, day out from corporate public relations departments. ↩︎
Kappa architecture (append-only logs with streaming views) with end-to-end encrypted content for private messages, replicated via the DAT protocol. Depending on the use case, a CRDT like LSEQ can be used at the content layer. ↩︎

A simple Node transform stream with the new Node 10 pipeline() function

mail@ar.al (Aral Balkan) — Thu, 20 Dec 2018 12:54:13 +0000

As part of my research for Hypha (RSS), I just completed the Kappa Architecture Workshop and I’m continuing to dive deeper down the stack to brush up on the fundamental concepts I need to be comfortable with going forward.

Next up is Node streams.

Streams are used everywhere in Node and while I’ve made extensive use of them in the past, there’s always been parts that seemed magical¹. I don’t really grok them and I’m working to change that.

So this morning I whipped up a little custom transform stream and got up to speed with the new pipeline() function in Node 10 that adds the functionality of Matthias’s pump module to Node proper².

The simple example takes the raw stream of characters (Buffer objects) you type from stdin, pipes them through a transform stream that uppercases the lowercase characters (and handles CTRL + C for exit) and then pipes it to stdout to display in your console.

const { pipeline, Transform } = require('stream')
const { StringDecoder } = require('string_decoder')

// Handle the raw output from standard input
// (characters, not lines, as is the default).
process.stdin.setRawMode(true)
process.stdin.resume()

class UppercaseCharacters extends Transform {
  constructor(options) {
    super (options)

    // The stream will have Buffer chunks. The
    // decoder converts these to String instances.
    this._decoder = new StringDecoder('utf-8')
  }

  _transform (chunk, encoding, callback) {
    // Convert the Buffer chunks to String.
    if (encoding === 'buffer') {
      chunk = this._decoder.write(chunk)
    }

    // Exit on CTRL + C.
    if (chunk === '\u0003') {
      process.exit()
    }

    // Uppercase lowercase letters.
    if (chunk >= 'a' && chunk <= 'z') {
      chunk = chunk.toUpperCase()
    }

    // Pass the chunk on.
    callback(null, chunk)
  }
}

// pipeline() is new in Node 10
pipeline(
  process.stdin,
  new UppercaseCharacters(),
  process.stdout,
  err => {
    if (err) {
      console.log('Pipeline failed: ')
    } else {
      console.log('Pipeline succeeded.')
    }
  }
)

You can achive the same thing on earlier versions of Node using the pump module simply by replacing the pipeline function with the pump function:

const pump = require('pump')
pump(
  process.stdin,
  new UppercaseCharacters(),
  // …
)

Especially duplex/transform streams – a.pipe(b).pipe(a) is still a royal mindfuck, made a little more sane via pump syntax. ↩︎
See Why use pump? from the Gulp documentation for an explainer. ↩︎

Boogie Board: a beautiful, modern, portable take on the blackboard

mail@ar.al (Aral Balkan) — Sun, 16 Dec 2018 16:00:11 +0000

The Boogie Board Blackboard by Kent Displays

I’m always on the look-out for tools that help me think things through with quick sketches and notes. Until recently, nothing really beat a traditional whiteboard and a good paper notebook and pen. So you can imagine that I was suitably excited to discover the Boogie Board Blackboard by Kent Displays.

Green on black. Shown with the dotted paper backing template.

The Boogie Board Blackboard is a no-frills electronic writing instrument that has a cholesteric liquid crystal display (ChLCD). Or what their marketing department calls Liquid Crytal Paper™.

In practice, it means that images are created by a mechanical/chemical process when you write on the Boogie Board with a stylus. No electricty required. The only time the device’s watch battery is used is when you want to erase something. This also explains why the battery, which is replacable, is stated to last around five years.

Prior to this model, erasing the screen was an all or nothing affair for Boogie Boards. That changes with the Blackboard. The included stylus has an eraser side that you can use to rub out portions of your work. And it even looks like you were rubbing out pencil on paper, with slight smudges. Far from being a problem, I feel it adds even more to the analogue feel of the device.

And feel is what it’s all about.

Laura using the Blackboard.

The Boogie Board Blackboard feels like you’re writing on an analogue medium. It doesn’t feel exactly like paper or a blackboard. It has its own feel. And it feels good.

The Blackboard comes in two sizes (note and letter¹). I got the larger letter-sized one. And I love it.

Given that this is a no-frills instrument, there is no fancy way to transfer what you’ve created to the digital realm. There is, however, an app you can download that lets you save your work by taking a photo of the screen with your phone. It has the ability to detect the corners of the Blackboard but, in practice, it has never done so correctly for me. Instead, I set the corners manually. It’s not a hassle. The app can then optionally apply an enhancing filter (I make it convert the green on black image to dark gray on white) and store it, catagorizing it by title and date/last update. All in all, this process works perfectly well for me.

Here is a sample of the saved output. You will probably start to see me post images with the aesthetic as I work on Hypha. So now you know where they come from.

An image saved on the Blackboard app.

The Blackboard is the first electronic writing instrument I’ve used that just works. It gets out of the way and lets me think through something without worrying about the medium I’m using to do so.

I couldn’t find this particular model for sale in the EU so I ended up ordering it from the US. It got here in two days for around €50. In case you’re confused, that’s the price of the device and the shipping, not just the shipping. In the US, the smaller version sells for $35 and the larger goes for $45 making this one affordable little electronic writing tablet.

And did I mention it feels great to write on it?

Because when have Americans ever used rational measurements? (The note is smilar to A4 and the letter is larger than that.) ↩︎

Kappa Architecture workshop

mail@ar.al (Aral Balkan) — Sat, 15 Dec 2018 18:04:33 +0000

Kappa Architecture Workshop is an excellent online resource by Stephen Whitmore (of Cabal fame), Mathias Buus (one of the cornerstones of the DAT Project), et al., that gives you an introduction to Kappa Architecture using modules from the DAT Node.js ecosystem like hypercore, multifeed, discovery-swarm¹ and kappa-core².

The examples take you from the very basics – such as how to make peer to connections and send simple ephemeral chat messages (code below) to P2P replicated feeds with multiple writers and beyond.

You can follow along with the workshop online, view my working files as I do, and also submit any issues you may run into or improvements you might want to suggest on the workshop’s source code repository.

const discovery = require('discovery-swarm')
const swarm = discovery()

const nickname = 'person' +  Math.floor(Math.random() * 42) + 1

swarm.join('my-very-very-simple-p2p-app')

swarm.on('connection', function (connection, info) {
  console.log(`Found a peer: ${info.host}:${info.port}`)

  process.stdin.on('data', function (data) {
    connection.write(JSON.stringify({
      type: 'chat-message',
      nickname,
      text: data.toString().trim(),
      timestamp: new Date().toISOString()
    }))
  })

  connection.on('data', function (data) {
    data = JSON.parse(data)
    console.log(`${data.timestamp} ${data.nickname}: ${data.text}`)
  })
})

See the new, improved version called hyperswarm, which you should be able to replace discovery-swarm with in the examples in the workshop. ↩︎
For an example of kappa-core in use in a real-world library, see cabal-core, the – uhum – core of Cabal. ↩︎

Remember directory from last session in Tilix with zsh

mail@ar.al (Aral Balkan) — Sat, 15 Dec 2018 17:46:40 +0000

If you’re using the excellent Tilix terminal with the wonderful zsh shell (in my case, via oh-my-zsh), you might notice that opening a new session (e.g., splitting your current session vertically or horizontally or opening a new window), doesn’t start you at the directory you were in in the previous one as you would expect but rather returns you to your home directory.

I’ve been getting increasingly fed up with this but not so much that I actually felt bothered enough to do something about it. Until today.

To fix it, you apparently have to source file called vte.sh in your shell configuration¹. Why? To be honest, I couldn’t care less. This is one yak I really do not need to shave right now. Some yaks look better with hair. It’s a fact. Don’t @ me.

On my machine, I did the following to find the file:

find / -type f -name "vte.sh"

That located several copies – the most promising of which seemed to be:

/var/lib/flatpak/runtime/org.gnome.Platform/x86_64/3.28/6d1d0ebbd72404c61d109307eb2240542b7ad82608bc6428bba6f3eebcfc8bf3/files/etc/profile.d/vte.sh

So, in my .zshrc file, I added the following line to source it:

# Make Tilix remember the path from previous sessions.
source /var/lib/flatpak/runtime/org.gnome.Platform/x86_64/3.28/6d1d0ebbd72404c61d109307eb2240542b7ad82608bc6428bba6f3eebcfc8bf3/files/etc/profile.d/vte.sh

I opened a new window, changed into a directory off of home, split the window, and boom, it remembered my path.

Case closed.

What lovely hair you have. Good yak! Down boy!

Source: https://github.com/gnunn1/tilix/issues/1208 ↩︎

Surveillance Capitalism at the BBC

mail@ar.al (Aral Balkan) — Mon, 10 Dec 2018 18:41:49 +0000

Recently, I recorded a video on surveillance capitalism with BBC Ideas.

You can watch the video on the BBC Ideas web site and you can also embed it on your own site, as shown above. All you need is a tiny bit of code that looks something like this:

<iframe
  src="https://www.bbc.com/ideas/videos/surveillance-capitalism-has-led-us-into-a-dystopia/p06p0tdy/player"
  width="640" height="500"
  scrolling="no" style="overflow: hidden"
  allowfullscreen frameborder="0"></iframe>

But what happens when you add that code to your site?

Do you just get the video, as you would expect, or do you also get something you didn’t bargain for?

To find out, I ran our inspector tool from Better on this very page to see if any third-party trackers get included with a BBC Ideas video when you include one on your own site.

Better Inspector inspecting this page.

Turns out there is a third-party tracker included with the embed: Chartbeat¹.

And if you visit the BBC Ideas page to watch the video there, you will also be tracked by AT Internet², in addition to Chartbeat.

How’s that for irony?

This is not just ironic. It’s unethical.

When someone embeds a video on their page, they do not expect to expose their visitors to third-party tracking and profiling by some random corporation like Chartbeat that they’ve never heard of.

So, what is surveillance capitalism?

It’s this.

It’s the mainstream.

It’s the Web.

It’s the BBC exposing you to third-party tracking by two companies – Chartbeat and AT Internet – when you watch a video about surveillance capitalism on their site. It’s people becoming unwitting accomplices to perpetuating the reach of surveillance capitalism on the Web by sharing a video on their own sites to raise awareness about the dangers of surveillance capitalism.

Here’s hoping that when the folks at the BBC see this, they will do some soul searching and revise their policies. At the very least, please do not include a third-party tracker in video embeds. Innocent people who just want to share videos should not find themselves unknowingly complicit in web tracking and profiling.

As for all of you looking for ethical alternatives to surveillance-based video services for your own content, check out Peertube.

Better already blocks Chartbeat. ↩︎
We weren’t aware of AT Internet, but we are now and they will be blocked in this week’s Better update. ↩︎

Baby steps

mail@ar.al (Aral Balkan) — Fri, 07 Dec 2018 12:14:28 +0000

Apple didn’t start out by making the iPad Pro.

Five years ago, when I decided to devote myself to tackling the problem of surveillance capitalism, it was clear what we needed: convenient and beautiful ethical everyday things that provide seamless experiences¹ on fully free-as-in-freedom stacks.

This is as true today as it was then and it will remain so. The only way to compete with unethical products built by organisations that have control over hardware + software + services is to create ethical organisations that have control over hardware + software + services and thus have at least the possibility to craft competitive experiences. We remove our eyes from this goal at our peril.

That said, it also clear that this is a huge, if not impossible, undertaking for individuals and small, independent organisations. We find ourselves in a world where 99.99999% of all investment goes into funding centralised surveillance-based startups and thus where 99.99999% of the technology infrastructure that exists and that we have to work with – including open source and web standards – has been created to support the needs of centralised, surveillance-based organisations and their products, incentivise their topology, and perpetuate their goals.

Small, independent groups can and do create great things with next to no resources². But we cannot rely on the fluke of individual sacrifice and genius to cobble together alternative ethical mainstream platforms to effect systemic change from whatever open source scraps they manage to scavenge from surveillance capitalism while striving to survive within its incentive structures.

In other words, the ethical alternatives will not grow on trees. They must be funded. And given that they cannot and will not be funded by the same interests that created the problem to begin with (venture capital), we need alternative, ethical funding to create alternative, ethical infrastructure. The technological infrastructure of our societies must be funded from the commons, for the common good. And that requires political will and a system that’s not institutionally corrupt. Neither of which we have today.

Given all these givens, it’s no surprise that I often find myself frustrated. But frustration alone doesn’t get us anywhere. Action can.

So here’s what I’m doing and what we’re doing at Ind.ie currently and into 2019. Our approach can be summarised by the phrase “regulate and replace”.

We cannot get the political will to regulate effectively or fund the alternatives as long as surveillance capitalists like Google and Facebook are deemed good actors; as long as they and their business model are socially acceptable. So we will continue to raise awareness and fight whitewashing attempts by so-called “privacy professionals” and organisations like Mozilla (who get nearly all their money – hundreds of millions of dollars – from Google). We will continue to call out the doctors in the cigarette ads in technology.
We will continue to technologically regulate surveillance capitalists and the adtech industry by maintaining our own block list and continuing to develop Better Blocker.
We will continue our work on alternatives by building on what we learned with Heartbeat and Indienet to start experimenting with a new personal network system based on DAT called Hypha. My goal is to issue small, frequent updates of my experiments and progress.

Expect baby steps.

If there’s one thing I’ve learned in the last five years, it is that we cannot start out by building the iPad Pro of the peer-to-peer, free and open Internet. We must build the scrappy Apple computers in the suitcases first. And yet, all the while, we must never forget what our ultimate goal is.

Finally, while I acknowledge that the countless talks I’ve given have probably had some impact, I’ve decided that I can best raise awareness about this issue at this point with a book. It will be called Peerocracy and I’ll be writing it and sharing it publicly, alongside my work on Better and Hypha.

We can have beautiful, seamless defaults and layer the seams. While proprietary technology can do the former, only we can do the latter with ethical technology that is free as in freedom. ↩︎
See Mastodon. See Elementary OS. See Gnome. See MNT. See Purism. ↩︎

GDMR: this one simple regulation could end surveillance capitalism in the EU

mail@ar.al (Aral Balkan) — Thu, 29 Nov 2018 11:18:19 +0000

GDMR: The regulation EU citizens deserve.

No, you didn’t misread it and, no, it’s not a typo. GDMR – the General Data Minimisation Regulation – can end surveillance capitalism in the EU.

The problem is that no such regulation exists.

So, let’s change that, starting now.

To be effective, GDMR must be succinct and precise. The essence of it can be expressed in a single article with two paragraphs:

In any digital/networked product, if a certain feature can be built where the algorithms and data are kept exclusively on an individual’s own devices, it must be built in that manner.
In any such system, if an always-on centralised node hosted by the service provider is required for purposes of findability¹ and availability², any private information that is replicated to that centralised node must be end-to-end encrypted and the individual must be the exclusive holder of the private key.

That’s it.

Seriously, that’s it.

I keep hearing people ask ‘what is effective regulation?’ Effective regulation is not legislation that encourages the Facebooks and Googles to become arbiters of truth, filters of reality, and master censors.³ It’s this. Effective regulation is regulation that directly prevents the toxic core of the business model of people farming⁴ while incentivising the creation of ethical alternatives.⁵

Implement the GDMR as regulation in the EU today and wake up tomorrow to witness the death of surveillance capitalism and the birth of a European alternative that puts human rights and democracy, not corporate profits, first.

For example, so that the service can be discovered/accessed via an easy-to-type domain name. ↩︎
That is, so that the service is available even if all the person’s other physical devices are offline. ↩︎
All things that they already are and do to a great extent. ↩︎
Tracking, data collection, profiling, and the use of the resulting intimate insight into our lives to manipulate our behaviour to satisfy the profit and political motives of surveillance capitalists. ↩︎
Ethical alternatives are free (as in freedom), decentralised/peer-to-peer, and interoperable. They will not be funded by the venture capitalists who gave us surveillance capitalism and make their billions from it. This alternative ethical technical infrastructure is essential for safeguarding the future of our human rights and democracy. We must fund it from the commons for the common good and it must be owned and controlled by individuals, not the state. And we must ensure that if the independent organisations building these ethical alternatives are successful, they cannot be bought out by Silicon Valley surveillance capitalists or traditional commercial interests so that our efforts don’t end up amounting to a fancy euphemism for privatisation and so that the EU stops acting like an unpaid (European taxpayer-funded) research and development department for Silicon Valley. ↩︎

Gnomit 1.0.6

mail@ar.al (Aral Balkan) — Thu, 08 Nov 2018 16:23:54 +0100

I pushed the 1.0.6 release of Gnomit, my little git commit message editor for Linux, to the Flathub GitHub repo about two weeks ago but I’m only writing about it now as there was a delay with the update appearing on Flathub.

What’s new?

In addition to git commit messages and tag messages, Gnomit now also supports git add -p messages and rebase -i messages. A big thank-you to Philip Chimento for reporting those issues.

Get Gnomit

You can install Gnomit via Gnome Software and Flathub:

flatpak install flathub ind.ie.Gnomit

For detailed installation instructions, please see the readme.

The source code is available from source.ind.ie and there’s also a GitHub mirror where you can open issues and submit pull requests.

Better Blocker 2018.2 release for macOS and iOS

mail@ar.al (Aral Balkan) — Mon, 05 Nov 2018 15:34:28 +0000

Better running on my iPhone and Mac.

Laura and I are happy to announce that Better Blocker version 2018.2 is now available for purchase for iPhone, iPad, and Mac on the App Store and Mac App Store.

As always, you can also grab the source code and build it for your devices yourself as Better Blocker is freedom software and licensed under GPLv3+. (If you do want to see Better continue to exist, please consider funding Ind.ie, our two-person-and-one-husky not-for-profit.)

What’s new?

Even simpler interface, consistent across the macOS and iOS versions.

Displays version and last update information for blocking rules.

Exceptions is now called the Do Not Block list.

macOS: Fixes El Capitan crash.

Comes bundled with the latest blocking rules.

With this new release, we are also tweaking the App Store pricing slightly. Better Blocker, prior to the 2018.1 release, used to be priced at Tier 5 ($4.99/£4.99/€5.49) on macOS and iOS.

With the 2018.1 release, we dropped the price to the lowest possible tier on both platforms. While this has resulted in unit sales increasing, it hasn’t translated into higher proceeds for our tiny not-for-profit. After seeking feedback from the community, we’ve decided to move the pricing from Tier 1 to Alternate Tier 1 on iOS ($0.99/£0.99/€1.99) and from Tier 1 to Alternate Tier 2 on macOS ($1.99/£1.99/€2.99). While this is a small change on the individual level and should not negatively impact the number of units sold, it should hopefully mean that the proceeds will increase and that we can afford to keep working on Better in a more sustainable manner.

On that last note, Laura updated the blocking rules again on November 3rd. The latest rules come bundled with the apps and you can always check for and update to the latest rules from within the apps themselves.

Please tell your friends

If you are enjoying Better, please do tell your friends. We don’t have a marketing budget so word of mouth is the only way people have of finding out about it.

Finally, a big thank-you, once again, to those of you who leave us wonderful reviews on the App Store and send us lovely messages via email and on Mastodon and elsewhere. It means a lot and really brightens up our days.

Gnomit 1.0.5

mail@ar.al (Aral Balkan) — Fri, 26 Oct 2018 19:29:00 +0200

Gnomit 1.0.5, wearing Adwaita-dark

I just released a minor update to Gnomit, my little commit message editor for Linux, that adds dark theme support. Previously, Gnomit did not update the background colour that it uses to highlight the commit message overflow area for dark themes, rendering the overflow area difficult to read.

Now, it uses a darker shade of the light theme colour when it detects a change to the theme.

You can install Gnomit via Gnome Software and Flathub:

flatpak install flathub ind.ie.Gnomit

For detailed installation instructions, please see the readme.

The source code is available from source.ind.ie and there’s also a GitHub mirror where you can open issues and submit pull requests.

With this amazing trick, you can stay an extra night in Paris but only if you use Linux!

mail@ar.al (Aral Balkan) — Fri, 26 Oct 2018 16:27:15 +0100

Give or take an hour.

I was supposed to be back in Cork right now but instead I’m staying an extra night in Paris and it’s all because I was using my Linux machine instead of my Mac at the airport. If you too want to take advantage of this nifty little trick, here’s how it works:

Make sure you fly into France from a country that’s in a different time zone and connect through Paris on your way back.¹ In my case, I flew into France from Ireland, which is one hour behind.
Ensure that both Automatic Date & Time and Automatic Time Zone are set to ON in the Settings app of your Linux laptop². Also, make sure you understand that for those settings to work you must connect to the Internet and thus make a point of connecting your Linux machine to the Internet so that you can safely assume that the time zone and time are automatically updated as promised.³

Linux can automatically update your time and timezone… sometimes… maybe.
Start working on something on your Linux computer⁴ but make sure to keep your eye on the clock on your desktop so that you don’t miss your flight.
With about 20 minutes to go to your boarding time, close the lid of your computer and start to make your way to the gate so that you can get there in plenty of time for your flight.
On the way, glance at your iPhone and notice that it is actually one hour later in Paris than the time that was showing on your Linux machine.
Realise that you’ve missed your flight.
Call your girlfriend and tell her you won’t be home tonight as planned.
Watch as your poor, loving girlfriend scrambles to book you a flight for the next day and a hotel for the night. End up paying €394.47 for the combo (€277.99 for the flight and €116.48 for the hotel room).

That’s it, there is no Step 9! You’ve now got an extra night in Paris. And it’s all thanks to Linux!

It does rather feel like in our relentless drive for MORE! FASTER! SHINIER!, we often neglect the essentials.⁵

And this isn’t even the first time something like this has happened to me. The only other time I’ve ever missed a flight was again due to technology and timezones. But that time it was Apple’s fault.⁶

Here’s hoping that this is yet another small reminder for those of us that build the new everyday things that when those things don’t work as they should it has consequences for people’s lives.

Update (a few hours later)

So, after writing this, I went searching to isolate the bug that led to the situation. I realised that the issue was due to a bug where if Location Services is off under the Privacy section of Settings, it disables the ability to automatically set the time and time zone. However, the interface does not reflect this in any way and allows you to enable the automatic time and time zone settings, thereby giving you a false impression that your time and timezone will be updated automatically.

Thanks to Mathieu, we now know that this was a known issue in Gnome for over a year because the Gnome developers themselves were effected by it when travelling for their developer conference. Although the issue was reported in October, 2017, it took about a year for it to be addressed. And, even then, it was decided to delay its release for 3.30 (which I’m running) and instead wait for 3.32 due to a string freeze.

It’s clear that something seriously went wrong here where a core feature that affects the reality presented to the person using the computer took over a year to be fixed and then wasn’t seen as important enough to include in a major release even though it was ready.

I’d love to have a post-mortem about this with the Gnome folks at some point to see what we can do to improve the triage process so that such core issues can be identified as such and handled with priority. This bug cost me €394.47 and a day away from home. I’m sure I’m not the only who was effected in the last year. How can we improve the bug triage procedure at Gnome to make sure that we can catch such issues earlier and give them the priority they deserve in the future?

This trick will actually work in any country as long as you fly in from a different time zone. ↩︎
Mine’s running Pop!_OS 18.10 so your milage may vary but given that Pop!_OS is based on Ubuntu, there’s a chance that this trick will work for you if you have a recent Ubuntu derivative. ↩︎
In my case, I was connected to the Internet for several hours at the airport. ↩︎
It’s essential here that you focus and only use your Linux machine. If you glance at your iPhone or start working on your Mac, this trick will not work for you. ↩︎
As I write this, connected to the Internet, the time zone and displayed time still hasn’t updated. I just tried manually flipping the Automatic Date & Time and Automatic Time Zone switches off and on again and that didn’t do anything either. Either in Pop!_OS 18.10 or in Ubuntu 18.10 or maybe even somewhere further upstream, that feature seems to be entirely broken. (I was finally able to get the timezone to update by setting it manually.) ↩︎
Many years ago, the calendar app used to have a bug where it would recalculate the times of events based on the timezone they were entered in when you changed timezones. So if you were in Timezone A when you entered the time of a flight back from Timezone B, when you travelled to Timezone B the calendar would offset the time of the flight based on the time difference between Timezone A and B, thereby presenting your flight back at the wrong local time. Apple has long since fixed this issue. ↩︎

version: display Linux version information

mail@ar.al (Aral Balkan) — Fri, 26 Oct 2018 12:55:59 +0100

The dark art of displaying Linux version information.

One thing I find myself always having to look up is how to display version information in Linux. That’s because the commands for displaying version information are in no way intuitive or memorable.

Specifically, if you want to display information about your distribution, including its version, you have to use the lsb_release command¹. And to see the Linux kernel version, you have to remember to enter uname -r.

Why? Because fuck you, that’s why.

I guess it was just too damn hard to create a version command or script. Thankfully, using my uber niche koding skillz, I was able to come up with one after literally seconds of hard work:

Create the following script in your text editor of choice and save it in a file called version somewhere that’s on your system’s execution path (I put it in /usr/local/bin):
```
#!/bin/sh

echo "\nOperating system:\n"
lsb_release -i
lsb_release -d
lsb_release -r
lsb_release -c

echo "\nLinux kernel:\n"
uname -r

echo ""
```
Make the script executable:
```
sudo chmod +x /usr/local/bin/version
```
In Terminal, run version.

You should see output similar to the following that displays information about your distribution and kernel versions:

Operating system:

Distributor ID:	Ubuntu
Description:	Pop!_OS 18.10
Release:	18.10
Codename:	cosmic

Linux kernel:

4.18.0-10-generic

LSB stands for ‘Linux Standard Base’. Don’t you feel smarter now? ↩︎

Gnomit 1.0.4

mail@ar.al (Aral Balkan) — Fri, 19 Oct 2018 21:43:08 +0100

Gnomit’s new look

There’s a new version of Gnomit, my little commit message editor for Linux, thanks to the initiative of Sergey Bugaev who sent me a patch a few weeks ago.¹ Sergey’s patch updates Gnomit to use a standard window instead of the dialog window I was using² and moves the Cancel and Commit buttons to the new window’s header bar.

The new look actually makes it mirror its inspiration, the Komet app on macOS, almost exactly.

Thank you for the patch, Sergey :)

You can install Gnomit via Gnome Software and Flathub:

flatpak install flathub ind.ie.Gnomit

For detailed installation instructions, please see the readme.

The source code is available from source.ind.ie and there’s also a GitHub mirror where you can open issues and submit pull requests.

Sergey actually sent me the patch quite a few weeks ago but first the new release of Better and then my XPS 13’s battery dying meant I only got round to taking a look at it now. ↩︎
Using a regular window has the welcome side-effect of removing the annoying GTK warning in the terminal complaining about the use of a dialogue window as the main window of the application. ↩︎

Updating firmware on Dell XPS 13 With Pop!_OS 18.04

mail@ar.al (Aral Balkan) — Tue, 16 Oct 2018 22:59:30 +0100

Firmware upgrades were failing for me on my Dell XPS 13. Running sudo fwupdmgr update would give me the following error:

UEFI firmware update failed: {error #0} libfwup.c:1501 get_fd_and_media_path(): failed to make /boot/efi/EFI/ubuntu/fw: No such file or directory

A little online research led to me to a page on Debugging UEFI Capsule updates, which in turn suggested that I try the latest fwupdate from master. My inability to RTFM¹ (coupled with being spoiled by package managers because it’s 2018), led me to open an issue on the fwpdate issue tracker when compilation failed.

Thankfully, I also wrote a comment explaining the core problem I was trying to solve and the author, Mario Limonciello, responded immediately with a solution: don’t use fwupdate, install fwupd via snap instead. (Thank you, Mario!)

That worked a charm and I am now running on the latest firmware.

I’m reproducing his instructions, below:

sudo apt purge fwupd
sudo snap install --candidate --classic fwupd
fwupdmgr refresh
fwupdmgr update

Read The Fucking Manual ↩︎

Setting up Hiawatha web server on Ubuntu 16.04

mail@ar.al (Aral Balkan) — Thu, 04 Oct 2018 17:09:00 +0200

The unfortunately-named Hiawatha web server with its problematic logo¹ is an independent, non-commercial, free and open web server built by Dutchman Hugo Leisink as a hobby project for the last 15 years or so. It’s primarily focused on security and the author appears to have a no-nonsense approach to its development.

Installing Hiawatha on Ubuntu

You can install Hiawatha on various platforms, including Linux, macOS, and Windows².

To test it out, I first installed it from source on an Ubuntu 18.04 machine I commissioned from my regular host, CloudScale. While that worked, it was rather involved. If you are going to install it, I would highly recommed using the Hiawatha apt package. Sadly, there isn’t one for 18.04 yet so you’ll have to use 16.04.

Installing via apt

To begin, add the apt repository to your system, update your packages, and install the Hiawatha apt package:

sudo add-apt-repository ppa:octavhendra/hiawatha
sudo apt update
sudo apt upgrade
sudo apt install hiawatha

That will install the server but it will not run it. To do that, you need to use systemd:

sudo systemctl enable hiawatha
sudo systemctl start hiawatha

That will set up Hiawatha to be started automatically on restarts and start it up for the first time.

If you go to your domain/IP at this point, you should see the default Hiawatha welcome page.

Configuration

One of the goals of Hiawatha is to be easy to configure and to provide sane, safe defaults. To that end, once I learned the basics, I did find it much easier to understand than my regular web server, nginx.

The following is my setup³.

Highlights:

Binds to port 80 (insecure; we will set up TLS later)
Binds to both IPv4 and IPv6

Directory details:

Hiawatha’s general logs: /var/log/hiawatha
The default web site (this should be an empty page to thwart IP scanners as recommedended in the default configuation file comments): /var/www/default
My web site’s root: /var/www/equinodal.org/public

Configuration file:

#
# General settings.
#
ConnectionsTotal = 1000
ConnectionsPerIP = 25
SystemLogfile = /var/log/hiawatha/system.log
GarbageLogfile = /var/log/hiawatha/garbage.log

#
# Insecure servers (IPv4 and IPv6).
#
Binding {
  Port = 80
}

Binding {
  Port = 80
  Interface = 2a06:c01:1:1104::9349:73
}

#
# Default site.
#
# The default website is just a blank webpage. This is to thwart automated
# scanners as recommended in the default configuration.
Hostname = 127.0.0.1
WebsiteRoot = /var/www/default
StartFile = index.html
AccessLogfile = /var/log/hiawatha/access.log
ErrorLogfile = /var/log/hiawatha/error.log

#
# Virtual hosts.
#
VirtualHost {
  Hostname = equinodal.org, www.equinodal.org
  WebsiteRoot = /var/www/equinodal.org/public
  AccessLogfile = /var/www/equinodal.org/log/access.log
  ErrorLogfile = /var/www/equinodal.org/log/error.log
}

To activate the current configuration:

sudo systemctl restart hiawatha

You should now be able to access your site from your domain over HTTP (port 80).

TLS

Now that your site is up and running, it’s time to implement TLS. The days of HTTP are behind us, long live HTTPS.

Download PHP CLI:

Hiawatha has Let’s Encrypt support in the form of a script you can download but it is written in PHP and so you will need a bit of setup:

sudo apt install php7-cli

Dowload and install the Hiawatha Let’s Encrypt script:

Once you have the PHP CLI installed, download the Let’s Encrypt script (get the ACME v2 version) and uncompress it:

cd /usr/local/etc
wget https://www.hiawatha-webserver.org/files/letsencrypt-2.0.tar.gz
tar -zxf letsencrypt-2.0.tar.gz
rm letsencrypt-2.0.tar.gz
cd letsencrypt

Configure the Hiawatha Let’s Encrypt script:

In the letsencrypt folder, you’ll find a configuration file named letsencrypt.conf that you need to customise for your own needs.

The only changes I made were to add my email address for the ACCOUNT_EMAIL_ADDRESS field and to change the HIAWATHA_RESTART_COMMAND field to use systemd:

HIAWATHA_RESTART_COMMAND = sudo systemctl hiawatha restart

You also have to create the tls folder as specified by the HIAWATHA_CONFIG_DIR and HIAWATHA_CERT_DIR settings or the script will fail (this feels like a bug):

mkdir /etc/hiawatha/tls

Request the TLS certificate from the staging server:

Once you’ve configured the script, you need to actually request the certificate. Initially, you will be using the Let’s Encrypt staging server to make sure that everything works and then you will switch to using the production server.

With the current script, requesting a certificate is a two-step process (this should really be simplified to a single step) that involves registering and requesting.

From the folder that contains the letsencrypt script:

./letsencrypt register
./letsencrypt request equinodal.org

Request the certificate from the production server:

Unless you changed the configuration file, the first attempt will use the Let’s Encrypt staging server. Once that runs correctly, you can change the configuration file to use the production server and actually get your certificates:

# Let's Encrypt API settings
LE_CA_HOSTNAME = acme-v02.api.letsencrypt.org		# Production
LE_ISSUERS = Let's Encrypt Authority X3 \
             Let's Encrypt Authority X4

Update your Hiawatha configuration to use TLS:

Now it’s time to update your configuration to implement TLS. Here’s what my configuration looks like with TLS enabled and required.

Highlights:

Forwards all connections to TLS
Gets an A on the SSLLabs tests (with the default settings)

Directories:

The TLS certificate (you will need to create this directory manually): /etc/hiawatha/tls

#
# General settings.
#
ConnectionsTotal = 1000
ConnectionsPerIP = 25
SystemLogfile = /var/log/hiawatha/system.log
GarbageLogfile = /var/log/hiawatha/garbage.log

#
# Insecure servers (IPv4 and IPv6).
#
Binding {
  Port = 80
}

Binding {
  Port = 80
  Interface = 2a06:c01:1:1104::9349:73
}

#
# Secure servers (IPv4 and IPv6).
#
Binding {
  Port = 443
  TLScertFile = /etc/hiawatha/tls/equinodal.org.pem
}

Binding {
  Port = 443
  Interface = 2a06:c01:1:1104::9349:73
  TLScertFile = /etc/hiawatha/tls/equinodal.org.pem
}

#
# Default site.
#
# The default website is just a blank webpage. This is to thwart automated
# scanners as recommended in the default configuration.
Hostname = 127.0.0.1
WebsiteRoot = /var/www/default
StartFile = index.html
AccessLogfile = /var/log/hiawatha/access.log
ErrorLogfile = /var/log/hiawatha/error.log

#
# Virtual hosts.
#
VirtualHost {
  Hostname = equinodal.org, www.equinodal.org
  WebsiteRoot = /var/www/equinodal.org/public
  RequireTLS = yes
  AccessLogfile = /var/www/equinodal.org/log/access.log
  ErrorLogfile = /var/www/equinodal.org/log/error.log
}

The only changes are the addition of secure IPv4 and IPv6 binding sections for TLS that reference the Let’s Encrypt certificate and the addition of the RequireTLS = yes setting to the virtual host.

Restart your server and you should be up and running over HTTPS:

sudo systemctl restart hiawatha

Node.js with a reverse proxy

The above is all you need if you want to host a static site with Hiawatha but what if you want to host your Node.js site? For that, we need to configure a reverse proxy and it is stupidly simple to do.

First, install Node.js on your server if you haven’t already:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs

Then, create a project to hold a very simple test server:

mkdir ~/node-server && cd ~/node-server
npm init -y
nano index.js

In index.js:

const http = require('http')

const server = http.createServer((request, response) => {
  response.statusCode = 200
  response.end('Hello from Node.js!\n')
})

server.listen(3000)

Run the Node server⁴:

node index.js

In another ssh session (or terminal window if local), test that the Node server works:

curl http://localhost:3000

You should see something like:

ubuntu@equinodal:~/spike$ curl http://localhost:3000
Hello from Node.js!

So, now, all we need to do is to update our Hiawatha configuration to use a reverse proxy that forwards all requests to our Node server. You do this in the virtual host section:

VirtualHost {
  Hostname = equinodal.org, www.equinodal.org
  ReverseProxy .* http://localhost:3000/
  WebsiteRoot = /var/www/equinodal.org/public
  RequireTLS = yes
  AccessLogfile = /var/www/equinodal.org/log/access.log
  ErrorLogfile = /var/www/equinodal.org/log/error.log
}

The only change is the addition of the ReverseProxy setting. (For some reason, the virtual host still needs the WebsiteRoot setting even though it is overriden by the reverse proxy setting.)

Restart the Hiawatha server and make sure that your Node server is running and you should be able to access it over TLS via your domain.

Conclusion

Hiawatha feels like a great alternative to nginx and other heavier web servers for personal projects and I look forward to using it in some of mine. I do hope that Hugo will do something to update the problematic name and iconography for 2018 so that I don’t have any reservations when I recommend it to others. If you do give it a shot, let me know how you get on via Mastodon.

Cultural appropriation is never fun. Hiawatha is #NotYourMascot. I do hope that Hugo will reconsider the naming and iconography of an otherwise excellent project. ↩︎
Note the following caveat for Windows: “I’ve never fully tested the impact that Cygwin or Windows has on Hiawatha, so don’t use the Windows version for production websites unless you’ve tested it yourself.” ↩︎
Don’t try and access the site itself, I’m just using that domain to test with so it will most likely be offline. ↩︎
Normally, you would run the server with a script like forever. ↩︎

Better Blocker: thank you for our best week yet!

mail@ar.al (Aral Balkan) — Tue, 25 Sep 2018 10:03:40 +0100

Better Blocker’s sales last week topped our original launch, two years ago.

Laura and I want to thank those of you who purchased Better Blocker last week. Thanks to you, we had our best week ever and sold over 2,500 units. That’s quite something given that we have sold just over 20,000 units in total over the last two years.

Better Blocker: sales are up over 999%.

The Better Blocker macOS app became the #1 paid app overall on the Netherlands app store and in Guatalama¹, entered the top ten in 14 countries, and was in the top 100 in 27 countries.

The newly-redesigned Better Blocker on macOS Mojave.

The statistics cover the week from the launch of the new design of the iOS app to yesterday, when the new macOS app had been live for just a day or two and we hadn’t even properly announced it yet as we were waiting for yesterday’s Mojave launch to do so.²

The uptick seems to be driven in large part by sales of the macOS app, combined with an upturn in interest in the iOS 12 app since the launch of the new design. The rise in macOS purchases also no doubt corresponds with some popular legacy “ad block” Safari Extensions becoming unusable under Safari 12.

It’s too soon to tell what the longer-term impact of lowering the price on the iOS and macOS apps to the lowest tier ($0.99/£0.99/€1.09/etc.) will be on purchases. I hope that it will at least mean that price will not be a stumbling block in your being protected by Better.³ The more people are protected from trackers, the better our herd immunity against surveillance capitalism.⁴

Laura and I are so happy to see the renewed interest in Better. Thank you for enabling us to keep working on Better and thank you for telling your friends about Better.

You can purchase Better Blocker for iOS and Better Blocker for macOS on the App Store.

We were also 2nd in Estonia, 3rd in Germany and 4th in Poland and rose to #18 in the US app store. ↩︎
We don’t have a marketing budget and we don’t use online ads (guess why) so by “properly announce” I mean toot about it on Mastodon and tweet about it and hope that people tell their friends via word of mouth. So do tell your friends about Better Blocker, it’s the main way people hear about it. ↩︎
You can also download the source code for the Better apps for free and build them yourself for macOS and iOS. Better is “free as in freedom” software and released under a GPLv3 license. There is also an EasyList version of the Better tracker blocking rules that you can use in uBlock Origin on other platforms and browsers. If you’re on Linux, see these instructions for using the Better Blocker tracking rules with Gnome Web. ↩︎
See Better Blocker: two year review and thoughts on the future for further discussion on the direction we’re heading in with Better. ↩︎

Better Blocker for macOS Mojave

mail@ar.al (Aral Balkan) — Mon, 24 Sep 2018 12:02:54 +0100

Better Blocker for macOS Mojave: simpler, more affordable, and just as effective.

A little under a month ago, in Better Blocker: two-year review and thoughts on the future, I mentioned I wanted to radically simplify the Better Blocker iOS and macOS apps.

I went on to say:

I’m proud of what we’ve achieved with Better so far. However, I’m unhappy with the reach Better Blocker has and we must figure out a way of increasing that reach while not cutting off a source of revenue without which we could not afford to pay the rent.

To that end, on September 17th, after several weeks of hard work and in tandem with the launch of iOS 12, we launched the redesigned Better Blocker iOS app (see Better, simpler, and more affordable).

With the launch of the new version, we also dropped the price to the lowest tier on the App Store ($0.99/£0.99/€1.09/etc.) in line with our goal to get as many people as possible protected with Better while hopefully not destroying our ability to pay the rent in the process.

Today, we complete the roll out of the new design with the launch of the new Better Blocker for macOS. In line with the iOS app, we have also dropped the price to the lowest tier for the Mac app.

Better Blocker is now a status bar app on macOS.

Even though the new apps are radical redesigns, we decided not to release a new app. If you already bought Better Blocker for iOS and/or Better Blocker for macOS, you can simply upgrade to the new apps via the App Store. Again, this is because we want to see Better in as many hands as possible. So we’d appreciate it if you could spread the word and tell your friends who don’t have Better yet.

What’s new

The design of the macOS app mirrors the iOS app as much as possible. Given how much of the interface was removed in the process, it made sense to remove the main window altogether and make the app into a status bar app.

Whether you choose the Light Side or the Dark Side in Mojave, Better Blocker will protect you from the Dark Side of the Web. (I missed a career in marketing, I know!)

The only part of the app that has a window is now the Exceptions section, where you can specify sites that you don’t want Better to block trackers on. If you’re doing this because a site breaks Better, please use the Report a Site feature to let us know so we can fix it for everyone.

As part of the redesign, we also wanted to support the new Dark Mode in Mojave. Which, as you can see in the screenshot above, we did. I’m not a huge fan of Mojave’s Dark Mode (I find it too dark) but I have to admit that Better does look pretty cool in Dark Mode.

Better does look pretty cool in Dark Mode on macOS Mojave.

No rest for the wicked

The 2018.1 releases of both apps are just the first step in a new direction. You can read more about where we intend to take them in my previous post.

This week, we are working on the next minor release, 2018.2.

If you want to support our work, please tell your friends about Better Blocker for iOS and Better Blocker for macOS.

I hope you enjoy the new Better Blocker. I’m excited to see where the second chapter of our little independent adventure in making the Web a safer place takes us.

Better Blocker for iOS 12

mail@ar.al (Aral Balkan) — Mon, 17 Sep 2018 17:58:22 +0100

Better for iOS 12 on the App Store. Is that a 4.8-star rating I see?

Better Blocker for iOS 12 is now available on the App Store.

Better Blocker is our content blocker for iOS and macOS that protects you from privacy-eroding surveillance-based behavioural advertising (adtech) and other trackers. It also removes ads that negatively impact your Web experience.¹

This release is a major redesign. We’ve worked hard to radically simplify the app.

See Better, simpler, and more affordable and Better Blocker: two year review and thoughts on the future for more details on the release itself and our roadmap for Better in general.

I hope you enjoy the new iOS app²

You can purchase Better Blocker for iOS as well as Better Blocker for macOS (now with macOS Mojave support) on the App Store.

Our blocking rules are based on the principles of the Ethical Design Manifesto. ↩︎
As this is not a separate app but an update, if you previously bought Better on iOS, all you need to do is to update the app from the App Store. Please also tell your friends about Better and help support our work at Ind.ie. ↩︎

Workaround for unclickable app menu bug with window.makeKeyAndOrderFront and NSApp.activate on macOS

mail@ar.al (Aral Balkan) — Mon, 17 Sep 2018 11:08:47 +0100

To open a window and make it appear on top of other windows on macOS, you can do the following:¹

myWindow.makeKeyAndOrderFront(nil)
NSApp.setActivationPolicy(.regular)
NSApp.activate(ignoringOtherApps: true)

This works to show the window, make it topmost and also display the app menu. The problem is that the app menu is not clickable until you tab away from the application and back again.

I couldn’t find this issue addressed with some cursory search engine trawling but the symptoms led to me think that it was most likely a timing problem. Which, it turns out, it probably is.

After my initial workaround² exhibited random failures, I dug deeper into the issue and found a more comprehensive workaround:

myWindow.makeKeyAndOrderFront(nil)
NSApp.setActivationPolicy(.regular)
NSApp.activate(ignoringOtherApps: true)

// Workaround for window activation issues:
// toggle focus away from the app and back.
if (NSRunningApplication.runningApplications(withBundleIdentifier: "com.apple.dock").first?.activate(options: []))!
{
    let deadlineTime = DispatchTime.now() + .milliseconds(200)
    DispatchQueue.main.asyncAfter(deadline: deadlineTime)
    {
        NSApp.setActivationPolicy(.regular)
        NSApp.activate(ignoringOtherApps: true)
    }
}

This second workaround seems to be solid so far in my testing, even though it does cause a slight flash of the window, especially the first time it is opened. It’s what I’m currently using in Better Blocker for macOS.

NB. There does seem to be a random element to this bug as, interestingly, as I was testing the app while writing this update, I coud not reproduce the bug even when testing without the workaround, on the latest macOS Mojave. I am keeping the workaround in for the time being in any case it is a failsafe. When I get some time I will try and see if Apple is aware of this issue and follow up on its resolution.

The NSApp.setActivationPolicy(.regular) should not be necessary unless you are trying this from a status bar application like I was. The activation policy was originally set to .accessory so we have to set it back for it behave like a regular application (with a toolbar icon, etc.). ↩︎
My initial solution was to invoke the NSApp.activate call on the next stack frame using my little delay library like this:
```
myWindow.makeKeyAndOrderFront(nil)
_ = delay(0) {
    NSApp.activate(ignoringOtherApps: true)
}
```
↩︎

Responsive design got my app rejected

mail@ar.al (Aral Balkan) — Sun, 16 Sep 2018 13:52:05 +0100

Rejection: a love story.

The iOS 12 version of Better is currently in review with a status of Metadata Rejected and I blame responsive design.

Let me explain.

In previous versions, I was using Fastlane (now owned by surveillance capitalist Google – spit!) to automatically generate screenshots for all possible screen resolutions and uploading them via the App Store API.

With the new version, I decided to take and upload the screenshots manually and to limit them to the only two required screen sizes: 5.5-inch (e.g., iPhone 6 Plus) and 12.9-inch display (i.e., iPad Pro) that I saw listed on the App Store Connect page.

So I used the App Store Connect site to delete the existing screenshots and upload the new ones and, along with an updated description and binary, submitted the app for review on the 14th. I woke up on the 15th to find that it had been Metadata Rejected. The note read:

We noticed that your screenshots do not sufficiently reflect your app in use. Specifically, your 9.7-inch iPad screenshots display older IOS version in the status bar.

Wait a minute! 9.7-inch iPad screenshots? I didn’t upload any 9.7-inch iPad screenshots!

Oh, but previous versions of Fastlane had and so I must have forgotten to delete them before submitting the app.

I was so sure I had checked the interface to try and find all screen sizes but back I went to the App Store Connect site and, lo and behold, there was a faint blue-on-grey link titled “View All Sizes in Media Manager” under App Previews and Screenshots that I had apparently somehow missed while preparing the submission.

The link to View All Sizes in Media Manager (emphasis mine)… how had I missed it?

Cussing my scatterbrained effort, I deleted all of the old screenshots and responded to the comment in the resolution centre.

However, something didn’t sit right with me. I mean, goodness knows I can be terribly aloof but I was certain I had searched for that link earlier and it wasn’t there. Going back to the site and resizing the window to half screen on my 4K 21" monitor, I realised what the problem was: the link disappears on viewports that are smaller than 1154px wide.

Exhibit A: the missing link.

So I wasn’t losing my mind. I had been thwarted by a CSS breakpoint. When I was preparing my submission, the link simply wasn’t there.

And that’s how responsive design got my app rejected.

Epilogue

The app is still in review but in the meanwhile, I reported the issue to Apple (radar 44499555) so hopefully it will be fixed soon.

One way to fix it would be to forgo the use of a link altogether and add another tab to the interface, as below:

Better, simpler, and more affordable

mail@ar.al (Aral Balkan) — Fri, 14 Sep 2018 19:34:56 +0100

The next version of Better embraces the essentials.

Two weeks ago, in Better Blocker: two year review and thoughts on the future, I wrote that I was going to “radically simplify” the Better apps.

Today, I submitted a radical redesign of Better Blocker for iOS to the App Store for approval in time for the launch of iOS 12.

The app has a minimal new design and a beautiful new icon that Laura and I designed together.

Better’s new icon.

Minimal look, minimal price

Alongside the minimal new look, Better is also getting a minimal new price. We want more people protected by Better and we want to increase the impact of our tracker blocking. So, come this release, we will drop the price of Better to the lowest tier on the App Store. The Mac app will follow suit with its Mojave release.

Just as effective

The crucial functionality stays the same: Better still blocks trackers using our unique, hand-curated tracker blocker database. What’s different is that it only includes the content blocker rules instead of the full human-readable encyclopaedic entries, informational articles, etc. These will still be available on the Better web site.

The new design decouples the app from the web site and back-end. This is going to make it easier for us to port the app to more platforms and to try something new with the back-end.

Better for macOS Mojave: work in progress.

This is the first iteration of a new chapter in Better’s evolution.

We are now working hard on getting the Mac version ready to submit in time for the macOS Mojave launch on September 24th.

Laura and I hope you’ll enjoy the minimal new design, not to mention the minimal new price.

(Do tell your friends.) ;)

Extended Codice Interview With Rai 1

mail@ar.al (Aral Balkan) — Wed, 29 Aug 2018 23:29:21 +0100

An extended cut of a recent interview I gave on cyborg rights (human rights in the digital/networked age), surveillance capitalism, and ethical design.

I was recently featured in an episode¹ of a technology show called Codice on Italian national television channel Rai 1. I just stumbled on an extended cut they released on surveillance-based video site YouTube. You can watch the segment here without being tracked.

Registration is required to watch the original show (it is in Italian and my segments are dubbed). ↩︎

Better Blocker: two year review and thoughts on the future

mail@ar.al (Aral Balkan) — Mon, 27 Aug 2018 11:33:37 +0100

The Better Blocker web site.

Better Blocker is a no-bullshit tracker blocker Laura and I built because we wanted an easy-to-use tracker blocker that wasn’t funded by or working for the adtech industry.

We released the iOS version of Better Blocker on June 3rd, 2016.

Since then, we’ve:

Released a macOS version alongside the iOS version.
Maintained a 4.7-star rating the iOS App Store and a 4.2-star rating on the macOS App Store.
Issued lots of content updates.
Re-written the Inspector to use Headless Chrome instead of Electron-HAR (yes, it’s pretty sweet that we‘re using the tools of surveillance capitalism against its interests).
Released an EasyList version of our block rules for use in other blockers on other platforms via tracker blockers like the excellent uBlock Origin.
Been featured in a number of best of lists and mainstream articles to raise awareness about surveillance capitalism in general.

By all accounts, Better does what it says on the tin and is loved by those who use it. But, sadly, given its niche audience, it’s not as effective as it could be against the scurge of Web surveillance.

Limited reach

The challenge: increasing our reach from thousands of people to millions.

I just checked our stats on Apple’s App Store Connect and it looks like in the two years that we’ve been going, Better has:

Sold ~17,200 copies (~12.5K on iOS, 4.76K on macOS).
Made us roughly $60,700 (€52,200) in proceeds (~€26,100/year).

While those proceeds have helped us to pay the rent, they are not enough to hire a full-time developer to work on the project.

Also, due to its limited platform support and the relatively high price (tier five; the equivalent of ~£4.99 in various currencies), our reach in two years has been under 20,000 people.

Our proceeds from Better over its lifetime to date.

While selling close to 20,000 copies of anything is nothing to scoff at for a tiny self-funded two-person not-for-profit with no marketing budget, it is a far cry from the sort of reach we could have if we weren’t worried about being able to sustain ourselves.

If two hundred thousand or two million people were using our blocking rules instead of under twenty thousand, we would be making a much greater dent in the tracking capabilities and bottom lines of surveillance capitalists. Especially considering that we cannot be bought no matter how many millions of dollars they might throw our way to be unblocked.

Getting from here to there

So, to re-cap, I’m proud of what we’ve achieved with Better so far. However, I’m unhappy with the reach Better Blocker has and we must figure out a way of increasing that reach while not cutting off a source of revenue without which we could not afford to pay the rent.

Here are some of my thoughts about the future, on how we can optimise for the right things to address these concerns:

Native apps on iOS and macOS

What matters is to effectively block trackers and to keep those blocking rules up to date. What is nice to have but not essential is having the full database of trackers, etc., on every copy of the app. So let’s optimise for the former.

The current architecture of Better is a reflection of our philosophy towards decentralised design (every app has a full copy of the tracker database, uses Git to download data, etc.). It’s pretty neat, my inner geek loves it, and I’m rather proud of how well it has worked for us so far but I’ve come to think that this design is overkill for the purposes of the apps.

To that end, I am going to radically simplify the iOS and macOS apps to remove all but the most essential functionality. This should also make it much easier for us to maintain those apps on Apple’s rapidly-changing platforms and to port them to other platforms.

Simplify the back-end and reduce costs

The back-end architecture reflects the decoupled and decentralised design philosophy and thus relies on several servers, including two Git servers running GitLab. We host them with an excellent European host called CloudScale.ch but it costs us about ~€250/month to do so. I am going to look into consolidating the Inspector, Builder, and Drafts servers and also the web, data, and archive servers.

Continuous inspections and blocking rule updates

Currently, the Better Inspector is designed to run on demand. I want it to become an engine that runs continuously and feeds into a simple, online mechanism for updating the blocking rules that members of the community can contribute to also.

Multi-platform support

Since we already have an EasyList version of our rules being automatically generated, I would love to see us create a version of Better that runs on other platforms via a fork of uBlock Origin.

That should go a long way in helping to increase our reach and effectiveness.

You can already use the Better blocking rules via uBlock Origin by including the EasyList version. You can also enable Better in Gnome Web. In fact, there is an open issue debating whether to make Better the default blocker in Gnome Web.

Possibly moving to a patronage or subscription-based model on iOS and macOS

This is the hardest one to decide on. If we made Better Blocker free on iOS and macOS, I have no doubt that usage would skyrocket. But that would also mean greater strain on our servers, greater costs, and more support requests while simultaneously cutting off our revenue stream from sales.

We could compensate by encouraging patronage from within the app using subscriptions. The gamble is whether or not this would replace the revenue we lose from sales and cover the increase in costs.

Here’s to a better future

Regardless of what we end up doing, Laura and I are committed to developing and maintaining Better Blocker¹ alongside our ongoing work in helping create a decentralised platform for the next web.

We’d love to hear your thoughts and feedback via the fediverse. You can reach Better, Ind.ie, Laura, and yours truly via our Mastodon instances.

For as long as there is a need for it, that is. Better is just technological regulation. Its goal is to protect you from being abused by surveillance capitalists. There is nothing we would like more than to be able to retire Better Blocker because it is no longer needed as that would mean that we’ve been successful in transitioning the web from the sewer of surveillance capitalism that it is today to become the decentralised, free and open, and interoperable infrastructure we need to protect the personhood, rights, and freedoms of humankind and foster the sort of commons and public sphere that is a prerequisite for democracy in the digital/networked age. ↩︎

Get Unicode-aware length of string in JavaScript

mail@ar.al (Aral Balkan) — Sun, 26 Aug 2018 21:10:59 +0100

stringInstance.length in JavaScript is not Unicode aware.

That means, for example, that the following code:

"🤓".length

Will return 2, not 1.

If you want a Unicode-aware string length, use this function:

function unicodeLength(str) {
  return [...str].length
}

Using that function, the following code:

unicodeLength("🤓")

Will return 1.

References

The function is based on the method by daxim in his response to the question Getting a string length that contains unicode character exceeding 0xffff on Stack Overflow.

Gnomit Flatpak bundle

mail@ar.al (Aral Balkan) — Mon, 20 Aug 2018 16:06:10 +0100

The Gnomit icon

Gnomit, my simple Git commit message editor for Gnome,¹ is now available as a Flatpak bundle.

To get started, see the installation instructions.

Please note that the Flatpak version is currently slower to launch than running the GJS script directly. For more information, see known issues.

I hope to make the installation process simpler by submitting Gnomit to Flathub and I eventually want to host a Flatpak repository at Ind.ie.

If you notice any issues, please let me know.

Inspired by the excellent Komet app for macOS. ↩︎

Introducing Gnomit: a simple Git commit message editor for Gnome

mail@ar.al (Aral Balkan) — Wed, 15 Aug 2018 17:16:31 +0100

Gnomit helps me write better commit messages.

Gnomit is a simple Git commit message editor for Gnome, inspired by the excellent Komet app by Mayur Pawashe that I was using on macOS¹.

I started working on Gnomit this weekend and I’m currently happily using it as my default Git editor. It’s a simple first project and a way for me to get acquainted with GTK+, GJS, and Vala – some of the core elements of Gnome.

I do need to properly package it up and complete the Vala version. If you try it out, please do let me know if you have any trouble with the Gspell spell checker dependency. I had all sorts of issues with the import and I have a suspicion that you might also.²

I hope to have a proper package (via Flatpak, perhaps) ready soon³ but, if you want to play with it in the meanwhile and don’t mind getting your hands dirty with a possible Gspell dependency, the GJS version is entirely usable and available on source.ind.ie.

Before I switched to Pop!_OS. ↩︎
I have a lot to learn on this new platform, especially concerning dependency management and packaging-related matters. ↩︎
16 Aug 2018: I now have a working Flatpak version built using Gnome Builder. It launches considerably slower than the plain script, however, so I’m going to hold off merging it into master. ↩︎

How to install the Vala Reference Manual into Devhelp using the apt package manager

mail@ar.al (Aral Balkan) — Mon, 13 Aug 2018 10:59:49 +0100

The Vala Refence Manual in Devhelp.

The Vala Documentation states that the Vala Reference Manual is available in HTML and PDF versions and “is also available for your installed version of Vala as DevHelp from your distribution, such as Fedora or Ubuntu.”

Sadly, though, it doesn’t tell you the package names or link to the packages.

For Debian’s apt-based systems like Pop!_OS and Ubuntu, the package you’re looking for is called vala-0.40-doc.

To install the Vala Reference Manual into DevHelp, run the following command:

sudo apt install vala-0.40-doc

In case you’re reading this far into the future and there is a different version of Vala and a different version of the package, you can search for the latest version with the following command:

apt search 'vala-[0-9]+\.[0-9]+-doc'

New Philosopher: How power corrupts

mail@ar.al (Aral Balkan) — Tue, 07 Aug 2018 10:24:10 +0000

New Philosopher: How power corrupts

Reading the latest issue of New Philosopher magazine, theme: how power corrupts, I was reminded of this quote by Michel Foucault that I love and which beautifully explain how I feel about what I do also:

All my books… are – if you like – little toolboxes. If you want to open them, and use this or that sentence, this or that idea or analysis as a screwdriver or wrench to short-circuit, dismantle, or explode the systems of power… all right, all the better.

Off to Denmark for Smukfest

mail@ar.al (Aral Balkan) — Tue, 07 Aug 2018 05:14:34 +0100

Laura and me, in a car at silly o’clock.

4AM

I’m in a car at silly o’clock, being driven to the airport by the ever-wonderful Laura.

6AM

Me, on a plane at also silly o’clock.

I’m on a plane at also silly o’clock, en route to Denmark via London to speak at Smukfest tomorrow near Skanderborg.

9AM

Magazines and muffins FTW!

Landed at London Stansted airport and preparing to wait a couple of hours with three magazines (Linux Magazine, New Scientist, and New Philosopher)¹, a lovely organic cappuccino, a lemon, lime, and ginger concoction of some persuasion, and an avocado and halloumi muffin.

TFW you realise you just summarised yourself with a stack of magazines 🙄 ↩︎

Getting your iCloud contacts on GNU/Linux

mail@ar.al (Aral Balkan) — Sun, 05 Aug 2018 22:43:21 +0100

I’m my own best contact.

Just like you can use iCloud calendars on GNU/Linux, you can also synchronise your contacts as iCloud uses an open standard called CardDAV.

No photos please, we’re Gnomish!

The caveat is that contacts with photos will be missing the photos. And you cannot create a contact with a photo either.

I’ve opened two issues for this in the Gnome Contacts source code repository. If this is something you would like to see fixed, please show them some love:

#100: Photos do not sync with iCloud via CardDAV
#101: Error when creating a contact that has a photo when using CardDAV with iCloud

Not having photos is not a show stopper but it does make the experience less than ideal. Needless to say, this is of course something that will be fixed in time. If anyone from Apple is reading this and would like to lend the Gnome Contacts team a friendly hand with this, that would also be very much appreciated.

Instructions

The instructions for setting up contacts are similar to those for setting up calendars.

To use your iCloud contacts under GNU/Linux¹:

A. Install the required software.

Install the following apps if you don’t already have them:

Install Evolution:
```
sudo apt install evolution
```
Install Gnome Contacts:
```
sudo apt install gnome-contacts
```

We will be using Evolution to set up the iCloud accounts but you will most likely want to use Gnome Calendar as your daily calendar as it offers a minimal, beautiful experience.

B. Set up an app-specific password on iCloud.

(If you’ve already set-up an app-specific password for your calendars, you can skip this step and use the same password.)

Sign into your Apple account at https://appleid.apple.com/
Scroll down to the App-Specific Passwords area in the Security section and select the Generate Password… link.
In the resulting pop-over, enter a descriptive name for this password.
Copy the password onto the clipboard.

C. Set up your iCloud address book in Evolution

Select the Contacts section in the main navigation.
Open the drop-down menu next to the New button and select Address Book.
In the resulting New Address Book window, select CardDAV from the Type drop-down.
In the URL field, enter https://contacts.icloud.com
In the User field, enter your Apple ID
Press the Find Address Books button.
In the resulting password entry pop-up, paste the app-specific password you copied onto the clipboard in the last section.
In the resulting Choose an Address Book window, select the address book shown and press the OK button. When I did this, there was only one entry and it was called card.
Back in the New Address Book window, check the options you want to set for your address book and press the OK button. I checked all of the options except for Avoid IfMatch – mostly because I don’t know what an IfMatch is and nor would I recognise one if were to stumble upon it in a dark alley.

This should be easier…

Once you’ve set up your address book, launch Gnome Contacts and you should see your contacts show up, sans any photos you may have had for them.

Any entries you make in Gnome Contacts will sync to iCloud and, from there, to all of your Apple toys, and vice-versa. Just remember not to include photos in your contacts or things will get borked.

Ideally, CardDAV setup should be a seamless process that’s built into Gnome Contacts².

References

How to integrate iCloud contact, calendar, or email accounts on the BlackBerry 10 smartphone

The examples here are tested to work on Pop!_OS 18.04 running Gnome 3.28.2. ↩︎
I opened an issue for this at the Gnome Contacts source code repository: Feature request: integrate iCloud CardDAV setup. ↩︎

Using iCloud calendars on GNU/Linux

mail@ar.al (Aral Balkan) — Sun, 05 Aug 2018 19:25:22 +0100

My week, courtesy of iCloud.

iCloud isn’t just for your Apple toys.

Since iCloud uses an open standard called CalDAV, you can synchronise your calendars to your other devices on other operating systems like GNU/Linux.¹

To use your iCloud calendars under GNU/Linux²:

A. Install the required software.

Install the following apps if you don’t already have them:

Install Evolution:
```
sudo apt install evolution
```
Install Gnome Calendar:
```
sudo apt install gnome-contacts
```

We will be using Evolution to set up the iCloud accounts but you will most likely want to use Gnome Calendar as your daily calendar as it offers a minimal, beautiful experience.

B. Set up an app-specific password on iCloud.

Sign into your Apple account at https://appleid.apple.com/
Scroll down to the App-Specific Passwords area in the Security section and select the Generate Password… link.
In the resulting pop-over, enter a descriptive name for this password.
Copy the password onto the clipboard.

C. Set up your calendar(s) in Evolution.

Select the Calendar section in the main navigation.
Open the drop-down menu next to the New button and select Calendar³.
In the resulting New Calendar window, select CalDAV from the Type drop-down.
In the URL field, enter https://caldav.icloud.com
In the User field, enter your Apple ID
Press the Find Calendars button.
In the resulting password entry pop-up, paste the app-specific password you copied onto the clipboard in the last section.
In the resulting Choose a Calendar window, select the calendar you want to set up⁴.
Back in the New Calendar window, choose a colour to match the one you use on iCloud.
Set your options: I select Copy calendar contents locally for offline operation, as I want to be able to access the calendar even if I don’t have an Internet connection, and Server handles meeting invitations.⁵
If you want this to be your default calendar, also check Mark as default calendar.
Set the Refresh every setting⁶ to decide how frequently your calendars should synchronise.
Press the OK button to create the calendar when you’re happy with your choices.

This should be easier…

That’s it! If all goes well, you should see your calendar entries begin to pop up in Evolution. If you want to set up additional calendars, rinse and repeat the instructions in this section.

Once you’ve set up your accounts, fire up Gnome Calendar and enjoy your synchronised calendars in a beautifully minimal interface.

Any entries you make in Gnome Calendar will sync to iCloud and, from there, to all of your Apple toys, and vice-versa.

Ideally, this should be a seamless process that’s built into Gnome Calendar⁷.

Issues

As I discover issues, I will document them here.

You cannot move an event between calendars

Issue #304: Moving an event from one iCloud CalDAV calendar to another, results in a 403 (Forbidden) error. The exact error is “Failed to put data: HTTP error code 403 (Forbidden): Found component … with same UID in a different collection.”

References

That said, the process is not well-documented at Apple or elsewhere. There is a clear need for better documentation as well as a seamless process for using CalDAV-based calendars in GNU/Linux. This post aims to contribute to the former. ↩︎
The examples here are tested to work on Pop!_OS 18.04 running Gnome 3.28.2. ↩︎
I would have taken a screenshot of the menu itself but the system-wide screenshot hot-key stops responding in Evolution when any drop-down is open. The same thing happens on the New Calendar window with the Type field. I’ve noticed similar issues in modal states in some other apps also. Not sure if this is a bug with the apps, with Gnome, or even Xorg. ↩︎
You will have to set up your calendars one by one. ↩︎
Because iCloud reportedly supports scheduling as described in RFC 6638. ↩︎
I have mine set to every ten minutes at the moment but I might tweak that as – no pun intended – time goes by. ↩︎
I opened an issue for this on the Gnome Calendar source code repository: Add support for iCloud CalDAV. ↩︎

Multi-writer Dat could power the next Web

mail@ar.al (Aral Balkan) — Sat, 04 Aug 2018 23:18:16 +0100

A demonstration of multi-writer Dat from the sample app by Jim Pick.

Dat is an exciting new technology that enables you to synchronise data privately and in a peer-to-peer fashion. It uses the same underlying concepts as blockchain¹, sans global consensus², and it’s run by the not-for-profit Code for Science & Society. The community has top-notch people like Mathias Buus, Tara Vancil, Karissa McKelvey, and Jim Pick.

Dat is currently implemented in Node.js³ and made up of numerous smaller components including HyperDB, a distributed scalable database, and Hyperdrive. The Dat commandline app and Dat desktop app both rely on Dat Node, which in turn uses Hyperdrive.

There is also Beaker Browser, which is a peer Web browser you can use to both view and fork/edit peer Web sites like mine. (To view this site on the peer Web, open the following URL in Beaker Browser: dat://ar.al.)

Multi-writer Dat

Dat is currently single-writer, meaning that although anyone with the archive’s key⁴ can read from the archive, only the original author can write to it. However, work is already underway to implement multi-writer support in Dat. In fact, it’s already in HyperDB and in the HyperDB back-end branch of Hyperdrive. And that is what Jim Pick used to create the Collaborative Dat Shopping List application that I demonstrate in the video, above.

Multi-writer Dat is hugely exciting. It opens up a plethora of potential use cases – anything from a peer-to-peer iCloud to peer-to-peer secure messaging. We’re witnessing the birth of the foundations of the next Web; the peer Web. We will see this implemented at the operating system level of future free-and-open mobile and desktop devices.

I cannot stress enough how groundbreaking Dat is or how important it is to creating a free and open, decentralised, and interoperable world. It’s rare to encounter a technology that you feel could have this potential and rarer still when the people behind it are doing it for entirely the right reasons. I say rare, but what I really mean is unprecedented.

You can find out more about Jim’s Dat Shopping list by reading his blog post and learn more about Dat (and try it out) on the Dat homepage.

The protocol is based on Directed Acyclic Graphs/Merkle trees, public-key cryptography, and a mechanism for sparse replication. You can read up about it in depth in the DAT whitepaper. ↩︎
So there’s no proof of work and it’s not meant for creating cryptocurrencies because it is not a right-libertarian get rich quick scheme. ↩︎
Although there are efforts underway to implement it on other platforms, like Rust. ↩︎
More precisely, the archive’s public key, which – confusingly – should not be made public as it is used to encrypt the archive. While this makes perfect sense when you understand how Dat’s cryptography works, it can be confusing to explain and that’s why I’ve started referring to the public key as just “the key” as that feels safer even if it’s less accurate technically. Revealing the public key gives anyone who has it read-only access to the archive addressed by that key. For this reason, when Dat does discovery, it doesn’t broadcast the public key. It broadcasts a cryptographically-secure hash of the key instead. In a nutshell, anyone who has your private key can write to your Dat archive and anyone who has your public key can read it. To ensure the privacy of your archives, you should share neither. ↩︎

How to enable the Browse Files setting in GSConnect

mail@ar.al (Aral Balkan) — Fri, 03 Aug 2018 15:47:01 +0100

Browse Files? Not so fast…

GSConnect is a beautiful shell extension for Gnome Shell that integrates mobile devices that can run KDE Connect (like phones and tablets that run LineageOS) with desktop computers that run Gnome on GNU/Linux (like my notebook running Pop!_OS).

One of the features that you might have trouble with is Browse Files as it is broken by default. It fails unless you have a separate app installed.

When you flip the relevant switch in the Mobile Settings for your device to turn it on, you are greeted with an exclamation mark icon and the setting stays off. So you have a good idea that something went wrong but you don’t know what, exactly¹. By turning on the debugging option², I was able to see the actual error messages in one of the developer consoles and realised that it had to do with a missing dependency; a package called SSHFS that mounts a remote filesystem locally using SFTP.

To fix this error, you have to install SSHFS separately:

sudo apt install sshfs

Once that’s installed, you can successfully flip the switch and start browsing the file system of your phone on your desktop.

Press the third button from left (highlighted) to see the Browse Files options.

Needless to say, and quite strikingly so given how polished the rest of the extension is, this is not a great experience.

As I mentioned in my bug report, GSConnect should ideally seamlessly install and use SSHFS or, if there is some reason for which that is impossible, detect whether it is installed, disable the setting for turning the Browse Files on if the dependency is missing, and provide instructions³ for how to install it.

Photos from my phone on my desktop, courtesy of GSConnect.

While developers may understand that many apps in Linux make use of specialised commandline packages behind the scenes⁴, that’s an implementation detail as far as the people using these tools are concerned.

If a feature in an app doesn’t work, it’s because the app is broken, not because I haven’t installed a dependency. Installing an app should install everything that the app needs to work. No ifs, no buts⁵.

This is not the first time I’ve encountered apps that are broken by default since switching my main development machine to GNU/Linux. A principle such as the following should be included in the design principles section of the Gnome Human Interface Guidelines to avoid this antipattern:

Your application should work by default

Installing an application should install all dependencies necessary for running the application. If this is not possible (for example, for legal reasons), then the application should detect missing dependencies and both inform and guide people on how to install them. It should not be possible to access or attempt to use features that depend on missing dependencies.

If you hover over the icon you do get a tooltip with the error message informing you that SSHFS has not been installed but, of course, that is a hidden gesture and we cannot expect people to intuit it. ↩︎
Mobile Devices → (My device name) → Mobile Settings → About → Debug Mode ↩︎
Or a link to instructions. ↩︎
Unix philosophy ↩︎
And if it cannot for reasons beyond its control (e.g., for legal reasons), it should be very humble and ashamed about this and do whatever it can to make the person aware of this limitation, apologise for it (even if it’s not its fault) and help the person to fix the problem that it created for them (even if it was for reasons outside of its control). ↩︎

Crafting a continuous-client desktop/mobile experience on Linux with GSConnect

mail@ar.al (Aral Balkan) — Thu, 02 Aug 2018 23:57:29 +0100

A continuous free and open experience between desktop and mobile.

A cornerstone of Apple’s approach to seamless design is reflected in a feature they call Continuity. Continuity aims to provide a “seamless experience” between your various devices. This is essentially what Joshua Topolsky called the continuous client back in 2010 and a concept that I wrote at length about in my chapter titled Mobile Considerations In User Experience Design: “Web or Native?” in Smashing Magazine’s Redesign The Web book back in 2012.

Well, tonight, I discovered that some amazing folks in the free and open source community have been busy implementing similar functionality that works on my desktop running Pop!_OS and my phone running LineageOS. Those folks are the team behind KDEConnect and Andy Holmes, who implemented it for Gnome Shell 3.24+ in a Gnome Extension called GSConnect.

With GSConnect installed, my desktop and phone become best buddies and can share all sorts of things, including notifications and files. You can also send SMS messages (although you should really be using end-to-end encrypted messaging via Wire or Signal instead) and locate your phone if you’ve misplaced it.

KDE Connect on mobile.

On the phone, you will need the KDE Connect app which you can get from the F-Droid catalogue.

It’s quite amazing how well it works. And by that I mean that I’ve found sending files between my devices easier to use and more reliable than AirDrop. And that’s saying something given that AirDrop was created by a multibillion-dollar corporation that touts design as its unique selling point.

How to install DAT on mobile under Termux

mail@ar.al (Aral Balkan) — Tue, 31 Jul 2018 17:21:29 +0100

The instructions below document how to install DAT¹ under Termux and have been tested on LineageOS 15.1 running on an S9+²:

Install dependencies³

pkg install libtool autoconf automake python2 nodejs

Patch node-gyp

There is a bug in node-gyp that prevents the installation of node projects that use native libraries. To fix it, you need to apply the changes shown in the following patch to the common.gypi file in your Node installation:

--- a/common.gypi
+++ b/common.gypi
@@ -90,8 +90,8 @@
             'ldflags': [ '-Wl,-bbigtoc' ],
           }],
           ['OS == "android"', {
-            'cflags': [ '-fPIE' ],
-            'ldflags': [ '-fPIE', '-pie' ]
+            'cflags': [ '-fPIC' ],
+            'ldflags': [ '-fPIC' ]
           }],
           ['node_shared=="true"', {
             'msvs_settings': {
@@ -144,8 +144,8 @@
             'cflags': [ '-fno-omit-frame-pointer' ],
           }],
           ['OS == "android"', {
-            'cflags': [ '-fPIE' ],
-            'ldflags': [ '-fPIE', '-pie' ]
+            'cflags': [ '-fPIC' ],
+            'ldflags': [ '-fPIC' ]
           }],
           ['node_shared=="true"', {
             'msvs_settings': {

You can either do this manually by editing the file by hand, or, if you’re on the same version as me (Node 8.11.3), you can use the following command to automatically apply the patch I generated:⁴

curl https://ar.al/2018/07/31/how-to-install-dat-on-mobile-under-termux/android.patch -o android.patch && patch ~/.node-gyp/8.11.3/include/node/common.gypi android.patch

Install DAT

Now that your environment is ready, you should be able to install DAT in the regular way:

npm install -g dat

References

Building Node-Sass || LibSass-Python Natively on Android 6 & 7

DAT is one of the technologies currently vying to be the fundamental protocol of the peer-to-peer Web. In my opinion, it is also the most promising. It’s what powers the peer web elements of this web site, it’s a core component of Web+, and it’s what we’re basing our current and future work on at Ind.ie. ↩︎
I’m currently running Termux on two devices: on my Nexus 5 under LineageOS 14.1 and on my S9+ under LineageOS 15.1. The instructions in this post have been tested and work on the S9+, which is a 64-bit device. I was able to get DAT to compile but not run on the Nexus 5. Whether that was due to its 32-bit architecture or due to LineageOS 14.1, I cannot say for sure at the moment. ↩︎
You might want to do a pkg upgrade beforehand to make sure you have the latest and greatest of your currently-installed packages. If you already have the required dependencies this command should not hurt your setup, although it may update them to the latest versions. ↩︎
android.patch (809 bytes, MD5: 2f0b9b25e4fa12f9d9db16e2f6616f7c). To verify, download the patch and compare the result you get from running md5sum android.patch in Terminal with the MD5 hash presented here. ↩︎

Workaround for npm install error on Lineageos 15.1

mail@ar.al (Aral Balkan) — Mon, 30 Jul 2018 21:46:04 +0100

The fix.

npm install fails in Termux on LineageOS 15.1¹ with the following error:

NPM ERR! Cannot read property 'length' of undefined

The issue originates from a bug in node. The affected module has implemented a workaround that you can manually apply to your npm installation to avoid the issue until the upstreams fix it properly.

To implement the workaround:

Open the file you need to patch in your editor²:

$EDITOR $PREFIX/lib/node_modules/npm/node_modules/worker-farm/lib/farm.js

Update line 5:

From:

, maxConcurrentWorkers : require('os').cpus().length

To:

, maxConcurrentWorkers : (require('os').cpus() || { length: 1 }).length

Replace the 1 in the code snippet above with the number of cores that your phone has and save the file. (For my Samsung S9+, I used 8 as it has an octa-core processor.)

That should fix the problem and you should be able to use npm again.

I did not encounter this error on my Nexus 5 with LineageOS 14.1. The phone exhibiting the error is running LineageOS 15.1, Termux version 0.64, Node.js version 8.11.3 and npm version 5.6.0. ↩︎
If you get a command not found error at this step, it’s most likely because you haven’t specified a default editor to use in your shell configuration. To fix that and use a simple editor called nano as your default, execute the following command: echo "export EDITOR='nano'" >> ~/.bashrc && source ~/.bashrc. Replace .bashrc with .zshrc if you’re using zsh instead of bash as your shell. ↩︎

Web development on a phone with Hugo and Termux

mail@ar.al (Aral Balkan) — Mon, 30 Jul 2018 20:44:12 +0100

Web development in 2018: all you need is a phone.

Hugo is an excellent static site generator and website framework.

You can build a static web site using your phone by running Hugo on LineageOS under Termux.

Here’s how:

(The instructions below have been tested with LineageOS 15.1 and Termux 0.64.)

Install Termux from the F-Droid catalogue

Install Go (substitute .bashrc with .zshrc if you use zsh):

# Install go.
pkg install golang

# Update your shell configuration.
echo "export GOPATH=$HOME/go\nexport PATH=$PATH:$GOROOT/bin:$GOPATH/bin" >> ~/.bashrc

# Reload your shell configuration.
source ~/.bashrc

Compile Hugo from source:¹

# Install Mage (a build tool for go).
go get github.com/magefile/mage

# Retrieve, compile, and install Hugo.
go get -d github.com/gohugoio/hugo
cd ${GOPATH:-$HOME/go}/src/github.com/gohugoio/hugo
env DEPNOLOCK=1 mage vendor
mage install

That’s it!

You should now have a working Hugo setup. Enjoy web development from your phone and, if you want an easy way to deploy from your phone and/or to join the peer web, check out my other posts on Web+².

Written on my LineageOS phone using Termux, Emacs and Hugo. Deployed via rsync and available on the peer web at dat://ar.al/2018/07/30/compiling-hugo-for-mobile-phones-under-termux/. To view peer web sites, use Beaker Browser.

The official Hugo compilation instructions hang under Termux at the mage vendor command. The instructions here differ from those by setting an environment variable that avoids the hang. ↩︎
Web+ and Web+ on a phone ↩︎

Out of the frying pan and into the fire

mail@ar.al (Aral Balkan) — Mon, 30 Jul 2018 16:33:00 +0100

You cannot base a public good on a normalised systemic violation of human rights.

Mariana Mazzucato¹ has an article in MIT Technology Review titled Let’s make private data into a public good.

Let’s not.

While Mariana’s criticisms of surveillance capitalism are spot on, her proposed remedy is as far from the mark as it possibly could be.

Yes, surveillance capitalism is bad

Mariana starts off by making the case, and rightly so, that surveillance capitalists² like Google or Facebook “are making huge profits from technologies originally created with taxpayer money.”

Google’s algorithm was developed with funding from the National Science Foundation, and the internet came from DARPA funding. The same is true for touch-screen displays, GPS, and Siri. From this the tech giants have created de facto monopolies while evading the type of regulation that would rein in monopolies in any other industry. And their business model is built on taking advantage of the habits and private information of the taxpayers who funded the technologies in the first place.

There’s nothing to argue with here. It’s a succinct summary of the tragedy of the commons that lies at the heart of surveillance capitalism and, indeed, that of neoliberalism itself.

Mariana also accurately describes the business model of these companies, albeit without focusing on the actual mechanism by which the data is gathered to begin with³:

Facebook’s and Google’s business models are built on the commodification of personal data, transforming our friendships, interests, beliefs, and preferences into sellable propositions. … The so-called sharing economy is based on the same idea.

So far, so good.

But then, things quickly take a very wrong turn:

There is indeed no reason why the public’s data should not be owned by a public repository that sells the data to the tech giants, rather than vice versa.

There is every reason why we shouldn’t do this.

Mariana’s analysis is fundamentally flawed in two respects: First, it ignores a core injustice in surveillance capitalism – violation of privacy – that her proposed recommendation would have the effect of normalising. Second, it perpetuates a fundamental false dichotomy – that there is no other way to design technology than the way Silicon Valley and surveillance capitalists design technology – which then means that there is no mention of the true alternatives: free and open, decentralised, interoperable ethical technologies.

No, we must not normalise violation of privacy

The core injustice that Mariana’s piece ignores is that the business model of surveillance capitalists like Google and Facebook is based on the violation of a fundamental human right. When she says “let’s not forget that a large part of the technology and necessary data was created by all of us” it sounds like we voluntarily got together to create a dataset for the common good by revealing the most intimate details of our lives through having our behaviour tracked and aggregated. In truth, we did no such thing.

We were farmed.

We might have resigned ourselves to being farmed by the likes of Google and Facebook because we have no other choice but that’s not a healthy definition of consent by any standard. If 99.99999% of all investment goes into funding surveillance-based technology (and it does), then people have neither a true choice nor can they be expected to give any meaningful consent to being tracked and profiled. Surveillance capitalism is the norm today. It is mainstream technology. It’s what we funded and what we built.

It is also fundamentally unjust.

There is a very important reason why the public’s data should not be owned by a public repository that sells the data to the tech giants because it’s not the public’s data, it is personal data and it should never have been collected by a third party to begin with. You might hear the same argument from people who say that we must nationalise Google or Facebook.

No, no, no, no, no, no, no! The answer to the violation of personhood by corporations isn’t violation of personhood by government, it’s not violating personhood to begin with.

That’s not to say that we cannot have a data commons. In fact, we must. But we must learn to make a core distinction between data about people and data about the world around us.

Data about people ≠ data about rocks

Our fundamental error when talking about data is that we use a single term when referring to both information about people as well as information about things. And yet, there is a world of difference between data about a rock and data about a human being. I cannot deprive a rock of its freedom or its life, I cannot emotionally or physically hurt a rock, and yet I can do all those things to people. When we posit what is permissible to do with data, if we are not specific in whether we are talking about rocks or people, one of those two groups is going to get the short end of the stick and it’s not going to be the rocks.

Here is a simple rule of thumb:

Data about individuals must belong to the individuals themselves. Data about the commons must belong to the commons.

I implore anyone working in this area – especially professors writing books and looking to shape public policy – to understand and learn this core distinction.

There is an alternative

I mentioned above that the second fundamental flaw in Mariana’s article is that it perpetuates a false dichotomy. That false dichotomy is that the Silicon Valley/surveillance capitalist model of building modern/digital/networked technology is the only possible way to build modern/digital/networked technology and that we must accept it as a given.

This is patently false.

It’s true that all modern technology works by gathering data. That’s not the problem. The core question is “who owns and controls that data and the technology by which it is gathered?” The answer to that question today is “corporations do.” Corporations like Google and Facebook own and control our data not because of some inevitable characteristic of modern technology but because of how they designed their technology in line with the needs of their business model.

Specifically, surveillance capitalists like Google and Facebook design proprietary and centralised technologies to addict people and lock them in. In such systems, your data originates in a place you do not own. On “other people’s computers,” as the Free Software Foundation calls it. Or on “the cloud” as we colloquially reference it.

The crucial point here, however, is that this toxic way of building modern technology is not the only way to design and build modern technology.

We know how to build free and open, decentralised, and interoperable systems where your data originates in a place that you – as an individual – own and control.

In other words, we know how to build technology where the algorithms remain on your own devices and where you are not farmed for personal information to begin with.

To say that we must take as given that some third party will gather our personal data is to capitulate to surveillance capitalism. It is to accept the false dichotomy that either we have surveillance-based technology or we forego modern technology.

This is neither true, nor necessary, nor acceptable.

We can and we must build ethical technology instead.

Regulate and replace

As I’m increasingly hearing these defeatist arguments that inherently accept surveillance as a foregone conclusion of modern technology, I want to reiterate what a true solution looks like.

There are two things we must do to create an ethical alternative to surveillance capitalism:

Regulate the shit out of surveillance capitalists.

The goal here is to limit their abuses and harm. This includes limiting their ability to gather, process, and retain data, as well as fining them meaningful amounts and even breaking them up.⁴
Fund and build ethical alternatives.

In other words, replace them with ethical alternatives.

Ethical alternatives do exist today but they do so mainly thanks to the extraordinary personal efforts of disjointed bands of so-called DIY rebels.

Whether they are the punk rockers of the tech world or its ragamuffins – and perhaps a little bit of both – what is certain is that they lead a precarious existence on the fringes of mainstream technology. They rely on anything from personal finances to selling the things they make, to crowdfunding and donations – and usually combinations thereof – to etch out an existence that both challenges and hopes to alter the shape of mainstream technology (and thus society) to make it fairer, kinder, and more just.

While they build everything from computers and phones (Puri.sm) to federated social networks (Mastodon) and decentralised alternatives to the centralised Web (DAT), they do so usually with little or no funding whatsoever. And many are a single personal tragedy away from not existing at all.

Meanwhile, we use taxpayer money in the EU to fund surveillance-based startups. Startups, which, if they succeed will most likely be bought by larger US-based surveillance capitalists like Google and Facebook. If they fail, on the other hand, the European taxpayer foots the bill. Europe, bamboozled by and living under the digital imperialism of Silicon Valley, has become its unpaid research and development department.

This must change.

Ethical technology does not grow on trees. Venture capitalists will not fund it. Silicon Valley will not build it.

A meaningful counterpoint to surveillance capitalism that protects human rights and democracy will not come from China. If we fail to create one in Europe then I’m afraid that humankind is destined for centuries of feudal strife. If it survives the unsustainable trajectory that this social system has set it upon, that is.

If we want ethical technological infrastructure – and we should, because the future of our human rights, democracy, and quite possibly that of the species depends on it – then we must fund and build it.

The answer to surveillance capitalism isn’t to better distribute the rewards of its injustices or to normalise its practices at the state level.

The answer to surveillance capitalism is a socio-techno-economic system that is just at its core. To create the technological infrastructure for such a system, we must fund independent organisations from the common purse to work for the common good to build ethical technology to protect individual sovereignty and nurture a healthy commons.

According to the bio in the article: “Mariana Mazzucato is a professor in the economics of innovation and public value at University College London, where she directs the Institute for Innovation and Public Purpose.” The article I’m referencing is an edited excerpt from her new book The Value of Everything: Making and Taking in the Global Economy. ↩︎
Although she never explicitly uses that term in the article. ↩︎
Centralised architectures based on surveillance. ↩︎
Break them up, by all means. But don’t do anything silly like nationalising them (for all the reasons I mention in this post). Nationalising a surveillance-based corporation would simply shift the surveillance to the state. We must embrace the third alternative: funding and building technology that isn’t based on surveillance to begin with. In other words, free and open, decentralised, interoperable technology. ↩︎

Broadcasting your phone’s screen to web browsers using Screen Stream

mail@ar.al (Aral Balkan) — Sun, 29 Jul 2018 17:58:16 +0100

My phone’s screen, streaming on my desktop in Gnome Web.

Screen Stream by Dmitriy Krivoruchko is an app for streaming your phone’s screen over HTTP so other people on the same network can view it using a web browser. It is free and open source, available from the F-Droid catalogue, and works beautifully on LineageOS.

In the screenshot above, you can see the screen of my phone streaming into my browser on my GNU/Linux desktop. The app being streamed is called ObscuraCam, a free and open source tool for automatically anonymising faces in photos¹.

Screen Stream is free and open source and available via F-Droid.

Using Screen Stream couldn’t be simpler:

Copy your phone’s IP address and port by pressing the copy button next to your device’s address.

The Screen Stream interface is simple and informative.
Press the Start Stream button.

Screen Stream will notify you when it starts streaming your screen.
Send the address you copied earlier to the people you want to share your screen with and ask them to enter it into their web browsers’ address bars.

Anyone with a web browser can then see your screen. Here’s the Screen Stream traffic graph, streaming on my desktop via Screen Stream.

Fixing the empty black screen in browser error

If you see an empty black screen in the browser while attempting to view the stream, try this:

Go back to the Start Stream application on your phone and swipe from the left edge of the screen to show the app menu and select Settings.
Check the Disable MJPEG check setting.

Check the “Disable MJPEG check” setting for greater browser compatibility.

It seems that the code that checks for MJPEG support has a tendency to give false negatives and that stops the stream from working in browsers that otherwise do support the stream. Screen Stream might not work in every browser² but disabling this check will increase the number of browsers it does work in.

Fixing the “no address found” error

Sometimes after I stop a stream, the Screen Stream app will report “no address found” under Device addreses. In these situations, merely closing the app from the task launcher and relaunching it doesn’t appear to have an effect.

To fix this, select Settings → Apps & Notifications → (See All Apps, if you cannot see Screen Stream) → Screen Stream → Force Stop and then restart the app and you should have an address again.

A lovely, useful, and ethical app

Screen Stream is a beautiful ethical app for easily sharing your screen with others on the same network³. It’s also a testament to the wonderful and inspiring free and open source app ecosystem on non-proprietary platforms today.

Thank you, Dmitriy, for making it and sharing it with the world.

Did they miss a trick by not calling it Camera Obscura? ↩︎
In our brief tests, Laura was not able to access the stream from Safari on macOS but was able to using Chrome (by surveillance capitalist Google/Alphabet, Inc.) I was able to get the stream to work under Gnome Web, Firefox (by surveillance capitalist Mozilla), and Chromium (the open source project by surveillance capitalist Google/Alphabet, Inc.) ↩︎
Which, if you set up a proxy or expose your IP, could be the whole world. ↩︎

Fix for “No Bluetooth Found” error after wake from sleep

mail@ar.al (Aral Balkan) — Sun, 29 Jul 2018 16:05:55 +0100

Bluetooth: still asleep, apparently.

After waking from sleep, my computer¹ would stop detecting the built-in bluetooth interface. The exact error I was getting in the Settings panel was:

No Bluetooth Found. Plug in a dongle to use Bluetooth.

The fix was to update BlueZ, the official Linux Bluetooth stack:

sudo add-apt-repository ppa:bluetooth/bluez
sudo apt install bluez

I no longer appear to have the issue with BlueZ version 5.50.

To check the version of your bluetooth driver, you can query the BlueZ daemon:

bluetoothd -v

I’m running Pop!_OS 18.04 but this issue may affect you if you’re on Ubuntu 18.04 or some other distribution also as it appears to be related to a bug in an earlier version of the official Linux bluetooth stack. ↩︎

Getting started with WireGuard on Linux using AzireVPN

mail@ar.al (Aral Balkan) — Fri, 27 Jul 2018 11:05:32 +0100

A Virtual Private Network (VPN) can be a practical tool that reinforces your security and privacy when using an Internet-connected device. It is not a panacea, however. You should be aware of both your own threat model and the security and privacy traits of VPNs so you can use one responsibly and ensure that you don’t succumb to a false sense of security or privacy.

What is a VPN?

A VPN is basically an encrypted (private) connection to some other network through which all your traffic – web browsing, email, chats, etc. – is routed. Since the connection is encrypted, the network you’re currently connected to cannot eavesdrop on you. All your Internet Service Provider (ISP) or the open WiFi network you just connected to at your local cafe can see is that you’ve made an encrypted connection to your VPN provider but they cannot see any of the traffic that flows through that connection.

Similarly, as far as any sites you connect to are concerned, your traffic originates from the servers of your VPN provider and not your own machine. This is why VPNs are commonly used to thwart region-locking schemes by the likes of Netflix and their cohorts in the copyright lobby¹.

In a nutshell, if you live in a somewhat free society and have an everyday threat model, a VPN limits the number of parties that can trivially eavesdrop on your Internet activity to just your VPN provider.

On trust

Never trust anything that can think for itself if you can’t see where it keeps its brain.

– J.K. Rowling, Harry Potter and the Chamber of Secrets

Given that you have to trust your VPN provider, the choice of VPN provider is what determines the level of security and privacy you get by using a VPN.

Nothing about the design of VPN implies that you should in any way trust VPN providers.

It is trivial for your VPN provider (or your web host, if you’re hosting your VPN server yourself) to violate your privacy. Your connection is encrypted between you and the private network you’re connecting to but it is not end-to-end encrypted between your computer and the parties you’re communicating with.

This is why you should never trust free (as in cost) VPN providers. How are they making their money if not by you paying them? What are you paying them with? (Remember, they can see all of your Internet activity.)

Even with a VPN provider that you’ve chosen to trust, you should connect to web sites only using Transport Layer Security (HTTPS) and use end-to-end-encrypted communication apps like Wire and Signal. If you need greater levels of anonymity (e.g., if you are sharing information that might result in your assasination by an autocratic regime), consider using Tor.

It might seem like I am rambling and not getting to the point of the post but I cannot stress how important it is that you understand the security and threat models of the tools that you use. If nothing else, remember that a VPN is only as trustworthy as the people who run it.

I get this question a lot. So before I get to the point of the post (oh, ffs!), I want to cover this too.

On iOS and macOS, I use Encrypt.me (née Cloak). Why? Because my friend Dave Peck set it up and I trust Dave. Dave’s since sold the company but it still seems to be run by good folks.

When I recently switched my main phone from an iPhone to one running LineageOS and asked them for an Android application package (APK) – as I didn’t want to use the Google Play store – they responded immediately² and sent it to me.³ In addition to iOS, macOS, and Android, Encrypt.me is also available on Windows.

Encrypt.me: a beautiful VPN experience.

On usability

The main reason I’ve been using Encrypt.me is its ease of use. You install and launch it and it just works. Also, it automatically blocks network traffic before the VPN connection has been established. This should be – but sadly isn’t – standard functionality in VPN clients to prevent data leaks between Wi-Fi connect and VPN launch.

Since I switched my main machine to GNU/Linux and Encrypt.me is currently not available for it, I signed up for AirVPN, which describes itself as “a VPN based on OpenVPN and operated by activists and hacktivists in defence of net neutrality, privacy and against censorship.” AirVPN has setup instructions for nearly every platform on the face of the planet and an app called Eddie for GNU/Linux that implements an automatic network block feature that protects against data leaks in the time between you connect to a WiFi network and before a VPN connection can be established.

Eddie: an (inter)face only a mother could love.

Unlike Encrypt.me, however, you cannot mark certain networks as trusted and so it’s currently an all or nothing setting. Also, on my machine at least, it’s possible to lose the ability to access the app’s interface while it remains running in the background. And the kindest thing you can say about the interface itself is that it’s what happened when what-the-fuck had a lovechild with oh-hell-no!⁴

So those are the VPNs I currently use. If you were to ask me which VPN I’d unconditionally trust, however, I would say iPredator.

Why?

Because it’s run by my friend Peter and I trust Peter unconditionally. Also, I’ve been in the cool little bunker in Sweden where everything’s hosted and I know that Peter runs it on his own machines that only he and his crew have access to.

The reason I don’t use iPredator myself is because Encrypt.me and AirVPN have apps that make them easier to setup and use. So convenience wins yet again. (I almost feel like there’s a lesson to be learned here for those of us that make the new everyday things for everyday people but I just can’t seem to put my finger on it… 🤔)

And that also brings us, finally, to the subject of this post: WireGuard and how to set it up on Linux.

WireGuard

WireGuard describes itself as a “fast, modern, secure VPN tunnel.”

Part of the reason VPNs can be hassle to host, install, and use is because they are – relatively speaking – based on ancient, bloated technology. The two main protocols in use today, IPsec and OpenVPN date from the Nineties and the Noughties and are behemoths weighing in at roughly 400,000 and 100,000 lines of code, respectively. In comparison, WireGuard is a recent protocol that uses modern cryptographic algorithms and weighs in at ~4,000 lines of code.

In programming, like in most things, less is usually better. As long as it doesn’t come at the expense of obfuscating intent, fewer lines of code means fewer places things can go wrong and less to audit.

That WireGuard is new, however, is both a blessing and a curse. It is currently labelled as a “work in progress” by its authors.

WireGuard is not yet complete. You should not rely on this code. It has not undergone proper degrees of security auditing and the protocol is still subject to change. We’re working toward a stable 1.0 release, but that time has not yet come. – WireGuard

If you can live with that and want to try out the future of VPNs today, read on as there are already VPN hosts beta testing the service. One of those is AzireVPN, by a Swedish company called Netbouncer AB based in Stockholm. According to their about page, they’ve been running a WireGuard beta programme since September 2017.

WireGuard on AzireVPN

AzireVPN makes its money selling VPN services. but, for the duration of the WireGuard beta, you can use WireGuard for “free” (in other words, in exchange for helping them test it. Remember, nothing is ever truly free as in cost unless it is also free as in freedom.)⁵

You can set things up on Android using the WireGuard app (which is available on the F-Droid catalogue) but installation on Linux currently requires use of the command line. You will need to install some packages that the installation script doesn’t automatically install, including WireGuard itself.

Installing

The AzireVPN WireGuard installation is carried out through the terminal.

Install WireGuard. On Debian/Ubuntu-based distributions (like Pop!_OS):

sudo add-apt-repository ppa:wireguard/wireguard
sudo apt install wireguard

Install curl and jq (if you already have them installed, the following commands will not hurt your system. The installer should really do this, and the above step, for you):
```
sudo apt install curl
sudo apt install jq
```

Run the installer script⁶:

curl -LO https://www.azirevpn.com/dl/azirevpn-wg.sh && chmod +x ./azirevpn-wg.sh && ./azirevpn-wg.sh

Using

You activate and deactivate the WireGuard VPN connection using the terminal commands wg-quick up and wg-quick down with the name of the server you want to connect to as the second argument.

To connect to AzireVPN’s server in Stockholm, Sweden, for example, you would type:

wg-quick up azirevpn-se1

Similarly, to disconnect from that server:

wg-quick down azirevpn-se1

At the time I installed it, the servers available to me were:

Location	Connection Name
Stockholm, Sweden	`azirevpn-se1`
London, UK	`azirevpn-uk1`
Malaga, Spain	`azirevpn-es1`
Miami, US	`azirevpn-us1`
Toronto, Canada	`azirevpn-ca1`

Once you’ve activated your VPN connection, you can check that it is working by visiting AzireVPN’s security check page.

The AzireVPN Security Check page.

These are the bastards that want to destroy how the Internet works. ↩︎
Thank you, folks! I haven’t had a chance to try it yet as I’ve been busy, but I will. ↩︎
1password, on the other hand, told me I could pound sand when I asked them the same thing. The cost of being 1password’s customer, if you are running an Android Open Source Project (AOSP)-based mobile operating system like LineageOS, is to be tracked and profiled by Google (Alphabet, Inc.) Needless to say, I am actively considering alternatives to 1password as we speak. ↩︎
That said, the app is free and open source so anyone (hint, hint, maybe me?), can help make it better. And that’s just one reason why free and open source rocks. ↩︎
It looks like the free beta period is over. I noticed that my connection wasn’t working and, when I logged in, I got a prompt to refill my account with credit (which I did). AzireVPN’s prices are entirely reasonable and their service is solid. I’m connected via Wireguard over Sweden as I write this in an airport lounge in Paris. ↩︎
Before you download and run any script, you should really download the script and check out what it does first. That script also downloads a list of locations, so you might want to make sure that what’s in that file is kosher too before running it. ↩︎

Pop!_OS 18.04: the state of the art in GNU/Linux on desktop

mail@ar.al (Aral Balkan) — Thu, 26 Jul 2018 12:02:42 +0100

Pop!_OS is beautiful, thanks in no small part to a consistent minimalist visual style and Kate Hazen’s beautiful space-themed illustrations.

Pop!_OS 18.04 is a GNU/Linux distribution curated by US-based computer maker System76. It is the state of the art in usability and experience when it comes to desktop Linux today¹.

The genius of the System76 team was in realising that all the components for a usable, convenient, and delightful GNU/Linux desktop experience are already here, they’re just not tastefully curated. They took the best bits of the Linux ecosystem, added some of their own special sauce, and ended up creating a minimal, coherent, consistent and – at times – delightful experience.

Take note, because this is a inflection point in desktop Linux.

If you had asked me a few years ago, these are not words I thought I would ever be using to describe a Linux distribution. And they’re not just words. I recently switched my main development machine from a Macbook to a notebook running Pop!_OS after falling in love with it in a virtual machine.

The ingredients of success

A number of core elements constitute the Pop!_OS experience.

The main ones are Ubuntu 18.04 LTS, Gnome 3 with Gnome Shell, a unified theme that uses material design, consistent keyboard shortcuts, an adapted version of the elementaryOS AppCenter, and a minimal set of default apps.

You combine these elements with a formal approach focussed on creating a simple experience for a specific target audience and you get Pop!_OS.

We’re building an OS for the software developer, maker, and computer science professional who uses their computer as a tool to discover and create. – System76

I will be exploring the various aspects of the design in detail in future posts. Suffice to say for now that Pop!_OS offers a more minimalist – and definitely less corporate – experience than macOS without attempting to recreate it. A huge amount of the kudos for that goes to Gnome 3 and Gnome Shell.

Gnome Shell 3: A masterpiece in interface design

Gnome 3 and Gnome Shell should be case studies in great interaction design².

Instead of copying Windows or macOS (an affliction that affects other alternatives and is inexplicably seen as a feature in those communities), Gnome 3 has its own culture and consistencies. These clearly derive from a first-principles approach to design and are refreshing to behold.

Much of the simplicity of Gnome 3 comes from the conceptual unity provided by the Activities view. A single key (the super key), takes you to this view where you can see all of your current apps as well as search for anything on the system or the store from a single place.

The Activities view in Gnome 3 beats Windows and macOS on simplicity.

Unlike Windows, there is no ungainly multi-level start menu from hell. Unlike macOS, the task of finding things is encapsulated in a single place instead of being split among two different features (Mission Control and Spotlight). If you are looking for something, there is one place to go. I know that I can get to anything I want using the Activities button or pressing the super key. This is as low as you can get in terms of cognitive load and is an absolute joy to use. It’s the state of the art in operating system navigation interfaces bar none. It’s as good as it gets.

Another brilliant design decision in Gnome 3 is to have a separate, always on App Menu in the menu bar. This creates what we call a landmark in interface design. Whereas I’m constantly lost on Windows, on Gnome 3, I can always tell exactly where I am. Which is why I was very sad to learn that there is work to reverse this beautiful feature. I cannot stress enough how much of a backwards step this is. It breaks the conceptual integrity of the design. I would strongly urge the Gnome team to stop and reconsider this pending – yet still avoidable – regression.

Am I in Visual Studio Code or Web? Web, of course. How do I know? Because the App Menu tells me so. It’s a crucial landmark.

Three steps forwards, two steps back?

The reasons given for removing the App Menu simply do not make sense to me as an interaction designer. Having the App Menu separate from app windows is the conceptually-correct, properly-encapsulated grouping of elements and serves as a crucial landmark to designate the currently-active application and the actions that can be performed on it.

Given how important I feel this decision is to the future of the platform that I now use myself, I want to take a moment to refute each one of the justifications given for removing the App Menu from Gnome 3:

It’s disconnected from the app window

Yes, of course.

That’s because it is the App Menu, not a window menu. The actions within the App Menu affect the app, not the windows (notice the plural form) that may exist within the app. If desktop apps always had only a single window, then the statement would be true. As things stand, it is not.

It doesn’t make sense with multiple monitors

Yes, it does.

Having multiple monitors doesn’t remove the need to know which app you’re in or to perform actions specific to that app. Just like the Activities button, the App Menu resides at a known location on your primary monitor and serves a crucial role as a landmark.

No other platform has this pattern

This is correct as long as you don’t consider macOS a platform.

This is only true for apps that do not properly conform to the Gnome Human Interface Guidelines (GHIG³)

There is no murkiness in the conceptual separation between an app and its windows. That’s a basic concept in desktop interface design. An app contains one or more windows. Actions that affect the app reside in the App Menu and actions that affect specific windows reside in the window menus of those windows.

It’s only accessible for the app which currently has focus

Yes, that’s the whole idea!

Unless you’re a dual-mouse wielding multitasking wizard (in which case, an operating system hasn’t been written for you yet), you can only interact with one app at a time. The reason the App Menu is a landmark is because it shows you which app currently has focus and encapsulates the actions you can perform on that app. This is not a problem, it is one of the main reasons why the feature exists in the first place.

Our own apps use it inconsistently

This is a great reason to get our own apps to use it consistently, not to formalise the inconsistency by breaking an otherwise consistent conceptual model. There’s a difference between paving the cow paths and framing cow shit.

Third-party apps have not widely adopted it, so it mostly sits there empty

Same point as above and my response is the same: we should expend effort on getting third parties to adhere to the human interface guidelines (HIG) instead of breaking the conceptual integrity of the platform to cater to nonconforming applications. If the HIG is not clear enough, let’s update the HIG.

Some new users don’t seem to find it, and e.g., assume that there are no preferences

It is normal for people who start interacting with a new thing to not know how the thing works. The better the affordances of that thing, the easier they will learn. This is also where landmarks play an important role. The problem isn’t that the App Menu isn’t a strong landmark or that it doesn’t provide a strong affordance but that it is inconsistently applied by apps.

The rest of the proposal goes on to conflate a desktop interface model (multiple windows) with a mobile interface model (single screen). A single menu makes sense on the mobile interface model because that is screen based, not window based. In the screen-based model, the app menu is the screen menu. It does not follow, however, that this then also makes sense in a window-based model where we have a clear separation of an app and its windows.

The only real problem with the App Menu is that some apps don’t use it in conformance with the human interface guidelines. So the only real problem with the App Menu isn’t a problem with the App Menu at all but with developers violating the human interface guidelines and a failing on part of the platform itself to attempt to adequately enforce its human interface guidelines.

Likewise, the only intellectually-honest reason I can see for removing the App Menu from Gnome 3 is “we want it to work more like Windows.” That’s the worst-possible reason I can think of for breaking the conceptual unity of an otherwise beautiful design.

Video of a talk I gave on experience design at the Thinking Digital conference.

A little background: my approach to design.

On consistency

There is a very real problem in the GNU/Linux ecosystem but it’s not the App Menu in Gnome 3. The problem is lack of consistency. Or maybe, more precisely, a culture that celebrates lack of consistency as a feature, confusing it with “choice”.

Beautiful, consistent defaults are not mutually exclusive with choice. Choice is about having the option of diverging from the defaults, not whether or not those defaults mandate a certain cultural cohesion or consistency.

Gnome should not be ashamed of its culture. On the contrary, it should be working to implement it as consistently as possible in its ecosystem. So let’s get Gnome apps to use the App Menu consistently instead of removing it. Let’s improve the consistency of the platform instead of destroying a valuable landmark and violating the conceptual integrity of the design.

Video of a talk I gave on experience design at the Thinking Digital conference.

A short demonstration: great design should empower, amuse, and delight.

The lack of a strong, coherent culture for applications is the last thing holding desktop Linux back.

As evidenced by Pop!_OS, it’s entirely possible to build a Linux distribution that can challenge the flagships of multibillion-dollar behemoths like Apple and Microsoft. The bit where the experience falls flat is the moment you install an app that doesn’t respect the culture of the platform. Using a KDE app on a Gnome desktop is like following a recipe for a cake in imperial measurements when your country uses the metric system. At best, you increase the cognitive load of baking a cake and at worst you end up with an inedible mess.

This is the biggest challenge both for innovative organisations like Purism and System76 who are making progress towards controlling the whole free and open stack, and thus the whole experience. They are competing with the meticulously-crafted and (like it or not) consistent cultures of platforms like macOS.

Imagine if every macOS app looked and behaved differently. Imagine if some third-party could arbitrarily decide to make menus in macOS work more like the menus in Windows 10 in the next release. Apple wouldn’t be where it is today. And yet that’s exactly the situation Purism and System76 find themselves in today because they both use Gnome 3.

If GNU/Linux is going to compete with the likes of Apple, we must start to value consistency, not despise it. And we must not be afraid to create and enforce strong, consistent cultures for our platforms.

Video of a talk I gave on design vs. decoration.

Our greatest advantage: we practice design; surveillance capitalists practice decoration.

Instead of merely going on my gut instinct and over three decades of experience in living and breathing technology, I wanted to make sure I tried a wide gamut of what’s currently available so, before writing this I tried the following distributions: Ubuntu 18.04, elementaryOS, Kubuntu, Mint, Nitrux, Solus, Deepin, Feren, and PureOS. The distribution that comes closest to Pop!_OS right now is PureOS. I also have notebooks running the latest Windows 10 and macOS. ↩︎
That’s not to say that it is perfect (nothing is), but that there is tremendous potential here. As I write this footnote, for example, my external trackpad (an Apple Magic Trackpad) isn’t working because the bluetooth adapter didn’t survive waking from sleep. The other day it was the WiFi. On the same trackpad, when I click, the mouse cursor jumps, giving me the electronic equivalent shaky hands. These are the sorts of issues that we must triage to high priority. They’re not as easy to fix or sexy to work on as moving things around in an interface. But they constitute the core of an experience and make all the difference in empowering people when they work well and enfeebling them when they don’t. For a great example of work in this area, see (and support, if you can) William B. Harding’s effort to improve the experience of touchpads under Linux. ↩︎
Which I’ll henceforth be pronouncing as “gig”, because why not? ↩︎

Camera fix for Samsung S9 (LineageOS 15.1)

mail@ar.al (Aral Balkan) — Sun, 22 Jul 2018 13:30:37 +0100

The first photo from my Samsung S9+ is of the solution that got the camera working.

The camera app on LineageOS 15.1 is broken for Samsung S9 and S9+ phones. The app launches successfully but hangs shortly thereafter with a “camera not responding” error while attempting to activate the camera. This seems to be a common problem, and the common advice given is to flash the latest stock Samsung firmware.

Sadly, that didn’t work for me.

What did work, after flashing the latest stock ROM and reinstalling LineageOS 15.1, was flashing the vendor.img file provided by Jesse Chan.

These instructions are specific to my unique setup. You might have to adapt them to yours. Flashing firmware carries the risk of potentially bricking your device. Needless to say, proceed at your own risk.

To flash the vendor image, follow the instructions in my previous post to set up Heimdall, etc., and, once you’ve got LineageOS 15.1 running successfully on the phone:

Reboot into Download Mode¹
Download and flash the vendor.img file provided by Jesse:
```
heimfall flash --VENDOR vendor.img
```

The phone should reboot and your camera app should now work.

How this is different from the vendor.img file in the latest stock ROM I flashed, I do not know. I do hope that the LineageOS folks will document this issue at the very least and hopefully incorporate a seamless fix for it in the future.

Hold down the power, volume down, and Bixby buttons - that’s the button on the top-right, along with the second and third buttons from the top on the left. When you see the Download Mode splash screen, press the Volume Up button as instructed to enter Download Mode. ↩︎

Upgrade Little Snitch Before Upgrading to Mojave Beta

mail@ar.al (Aral Balkan) — Sat, 21 Jul 2018 08:40:56 +0100

Little Snitch: use the nightly build on the macOS Mojave beta.

If you have the current release version of Little Snitch (4.1.2) installed under High Sierra, upgrade it to the latest nightly build before upgrading to the macOS Mojave beta or your system will lose network access.

If, like me, you did upgrade and lose network access, the workaround is simple: uninstall the release version of Little Snitch (make sure you keep your rules and settings) reinstall the nightly build.

To uninstall Little Snitch, run the uninstaller app and follow the on-screen instructions.

You can run the uninstaller from Terminal with the following command:

/Library/Little\ Snitch/Little\ Snitch\ Uninstaller.app

References

iOTA 360

mail@ar.al (Aral Balkan) — Thu, 19 Jul 2018 22:49:33 +0100

Do you give an iota about a £99 computer?

I’m writing this on a £99 computer.

Well, it was a £99 computer over at Amazon’s Prime Day sale this week. Today, it’s apparently a £171.32 computer.

While I hate contributing to Jeff Bezoz’s Fuck Off To Mars Fund, I wanted to see what a £99 notebook could do so I ordered one. If nothing else, I’d have a low-end device for testing sites on IE and Edge instead of firing up a VM.

The iOTA 360 shows attention to detail and polish you wouldn’t expect at this price point.

So what kind of notebook does £99 get you these days? One with a 1.96Ghz quad core Atom processor and 2GB of RAM which aren’t going to win any performance awards but haven’t stopped me from doing anything I’ve wanted to do so far, 32GB of eMMC storage that’s ample for web development, and Windows 10, which is what you get when you combine the elegance of old men who wear socks in sandals with the Chinese government’s respect for privacy and Martin Shkreli’s sense of ethics.

The keyboard does flex while typing but isn’t uncomfortable.

It also has a passable 11.6" screen running at 1366 x 768 that is best viewed at 125% scale. (It’s even nicer at 150% but some system dialogs get cut off and cannot be scrolled, leaving certain buttons unreachable.)

The iOTA 360 quite happily runs my blogging setup, including node.js, Hugo, and Visual Studio Code.

Web development is definitely possible with this machine.

Could this democratise computing? Could you use this thing for web design? Is it a surveillance device?

Yes to all.

I look forward to elaborating on those points in future posts.

Join me next time when I turn the iOTA 360 on and step into the panopticon cum shopping mall that is Windows 10…

Typographical typing habits for Linux

mail@ar.al (Aral Balkan) — Wed, 18 Jul 2018 20:02:37 +0100

Many of our bad habits date back to the limitations of typewriters.

Typography lends us the voice that the written word takes away from us. Good typography makes your message accessible and comprehensible and strengthens its intent. Bad typography can have the opposite effect: it can cloud your meaning and give you a raspy voice.

I first started making an effort to write using semantic typographical habits about six years ago. This involved, among other things, memorising the various keyboard shortcuts on macOS for printing typographical quotes instead of typewriter quotes and using en dashes, em dashes, and the minus sign for their allotted purposes instead of hyphen-dashes for everything like a caveman that thinks everything is a nail because all he has is a tiny little club¹.

Switching to Linux means that I have to learn new habits².

Some environments provide automatic conversion of such things (i.e, “smart quotes” functionality) and some even do it well. I strongly support such efforts and believe they should be the norm. However, it’s still a good habit to get into to write using semantic typography to begin with. Not least because smart quotes is not a universal feature and even the best implementations won’t necessarily cover all typographical elements.

Alt Graph and Compose keys

When seeking to develop good typographical typing habits on Linux, you have two friends you can count on: the Alt Graph key (AltGr) and the Compose key.

More than likely, if you’re not on a touch-based device, you can see the AltGr key on the keyboard in front of you. You might be wondering about the other one, however. That’s because the Compose doesn’t exist on modern keyboard. You can, however, easily map an existing key to be your compose key under Gnome. We’ll see how after the next section.

The Alt Graph Key

Let’s start with the simpler of the two methods; the one that works right out of the box.

The AltGr key modifier, combined with the Shift key modifier, exposes two separate layers of characters on the keyboard. Using the AltGr key, you can easily enter a number of useful everyday characters including the following³:

Key(s)	Character	Description
`v`	“	Open double curly quotes
`b`	”	Close double curly quotes
`Shift` `v`	‘	Open single curly quote
`Shift` `b`	’	Open single curly quote
`Shift` `3`	#	Octothorpe
`,`	─	Minus
`Shift` `,`	×	Multiplication
`.`	·	Middle dot
`Shift` `.`	÷	Division
`Shift` `c`	©	Copyright
`y`	←	Left arrow
`i`	→	Right arrow
`u`	↓	Down arrow
`Shift` `u`	↑	Up arrow

So those are glyphs that you can start to use right away via the AltGr key. For the second method, you’re going to have to install a Gnome app.

The Compose Key

A cheeky tweak gets you the coveted Compose key.

The Compose key, introduced in 1983, never became a standard but it is hugely useful and rather delightful to use nevertheless. It’s based on the concept of using a mnemonic sequence of easy-to-type characters to specify glyphs that are not readily available on your keyboard.

To set up and use your Compose key:

Install and launch Gnome Tweaks
Navigate to Keyboard & Mouse and choose a Compose Key.

Turn it on and choose a key.

Initially, I set my Compose key to Scroll Lock while using my external keyboard⁴, only to discover that my laptop didn’t have one. So now I use Caps Lock, an otherwise rather useless key when you think about it.

To enter extended characters with this method, you press the Compose key followed by a key combination. The key combinations are delightfully clever and make a huge amount of sense.

Here are some examples of the most useful ones I’ve discovered so far:

Keys	Keys (spelled out)	Character	Description
–.	dash-dash-dot	–	en-dash
—	dash-dash-dash	—	em-dash
..	dot-dot	…	ellipsis
.-	dot-dash	·	middle dot
.=	dot-equals	•	filled bullet
oc	o-c	©	copyright
tm	t-m	™	trademark
c=	c-equals	€	Euro
c/	c–forward slash	¢	Cent
l-	l-dash	£	Pound
->	dash–greater than	→	Right arrow
-<	less than-dash	←	Left Arrow
xx	x-x	×	Multiplication
-:	dash-colon	÷	Division
=/	equals–forward slash	≠	Not equal to
12	one-two	½	Half
34	three-four	¾	Three quarters
^0	caret-zero	⁰	Superscript zero
^9	caret-nine	⁹	Superscript nine

If the built-in ones do not meet your needs, you can also add custom Compose key sequences.

I hope you find this useful and that it helps you express yourself better while typing on a Linux machine. If you have any favourite glyphs that I’ve missed, let me know via Mastodon.

References

Credits

Old typewriter by Peter Heeling.

Yakub Marian has a beautifully-written and comprehensive article titled Hyphen, minus, en-dash, and em-dash: difference and usage in English that I strongly urge you to read. ↩︎
And that’s part of the fun. You always learn the most when you force yourself outside of your comfort zone. In the last two weeks alone I’ve learned heaps about how operating systems boot (due to an issue I had while trying to install first Ubuntu 18.04 and then Pop!_OS on my XPS 13) and about flashing ROMs onto mobile phones. (I’m also switching my main phone to LineageOS and I had an issue with the camera not working on my S9+ so I had to revert back to the stock ROM.) ↩︎
The key sequences listed in the Alt Graph Key section are from my keyboard layout, which is English (UK, Macintosh). The actual physical keyboard layout I use is ISO. The exact combinations may vary depending on specific setup. ↩︎
The excellent Filco Majestouch 2 Ninja Tenkeyless. ↩︎

Open from terminal in Linux

mail@ar.al (Aral Balkan) — Tue, 17 Jul 2018 22:38:23 +0100

On macOS, if you want to open a file in its default viewer/editor from Terminal, you use the open command like this:

open name-of.file

On X-based Linux systems, you can do the same thing using a command called xdg-open. However, it doesn’t really have the same ring to it, does it?

If you want to speak human instead of xdg, add an alias to your shell configuration file (for bash, ~/.bashrc, for zsh, ~/.zshrc). In my case, all I had to do was:

# Add the alias to your shell configuration. 
echo "alias open='xdg-open'" >> ~/.zshrc

# Load your updated configuration.
source ~/.zshrc

And now you can open directories in Files using:

open .

And you can open images in Image Viewer and any other type of file in its default app using open <name of file>.

Enjoy!

Enabling Better Blocker in GNOME Web

mail@ar.al (Aral Balkan) — Tue, 17 Jul 2018 17:40:50 +0100

GNOME Web: what ethical design looks like.

Update: (2020-03-14) GNOME Web now supports WebKit Content Blocker rules but sadly still comes with AdBlock rules that do not protect your privacy (they block some ads) by default. I’ve updated this instructions here for replacing them with Better Blocker’s rules instead if you care about protecting your privacy.

TL;DR: Enter the following line in a terminal and you’re done:

gsettings set org.gnome.Epiphany content-filters "['https://better.fyi/blockerList.json']"

~~GNOME Web is an ethical, simple, elegant, and beautiful web browser. And, since I switched my main development machine to one running GNOME Shell, it is now my default browser.~~

~~Unlike products like Chrome and Firefox from surveillance capitalists like Google and Mozilla, GNOME Web has safe, private defaults that respect your human rights. As~~ their privacy policy states:

GNOME Web receives no funding from advertisers, and as such is able to offer privacy features like built-in adblocking and tracking query removal that are relegated to extensions by other browsers. We aim to offer the best out-of-the-box privacy settings of any general purpose web browser.

How refreshing. How just as it should be.

GNOME Web is private by default. That’s the only definition of privacy that matters. Contrast that to ‘tracked and profiled by default but with the choice of turning the surveillance off’ in Mozilla Firefox.

Tracker and adtech blocking

By default, GNOME Web comes with ad ~~and tracker~~ blocking based on AdBlock Plus . To see the default lists, you can use the following command in a terminal window:

gsettings get org.gnome.Epiphany content-filters

The result should be an array of URLs to blocking lists in EasyList format:

['https://easylist-downloads.adblockplus.org/easylist_min_content_blocker.json']

~~This is good but I feel~~ we can make it… uhum… Better! 🤓

Why’s Better any better?

Better Blocker: it’s not just for Apple’s platforms anymore.

Laura and I built Better Blocker for three reasons:

We were hugely frustrated with the bullshit of so-called ‘ad blockers’ like AdBlock Plus by Eyeo who actually get paid millions of dollars by surveillance capitalists like Google not to block adtech and trackers.
We wanted to have a curated list of blocking rules that we understood. Looking at an EasyList block list, I have no idea why any of the rules were included. Reading through the entries of our hand-curated database of trackers, on the other hand, I know exactly why.
We wanted a tracker blocker that didn’t require any technical knowledge to install and use on the platforms that were the primary platforms for both of us at the time (iOS and macOS).

So we built Better as a free and open tracker blocker that is also available for sale on the iOS and macOS app stores.

Better is entirely independent of the adtech industry and surveillance capitalism and uses our ethical design manifesto as its guiding principle in deciding what to block and what not to block. Sales of Better don’t pay our bills entirely but they do contribute towards our ability to do so when combined with our professional speaking fees and the proceeds from our patrons.

How Better works

With Better, we have a straightforward and transparent process for deciding which trackers to block:

We crawl the most popular domains in the world to find the third-party trackers on them.
We organise the trackers by popularity.
We manually go through these top trackers, document what they do, and block them.

This works very well because the Achilles' heel of centralisation can be summed up by the old adage “the bigger they are, the harder they fall.” Centralised topologies are, by their very nature, both very powerful and extremely fragile.¹

Cleaning up the Web

You do not need to block every tracker ever made to choke the income stream of surveillance capitalism, you just need to block the centres; the Goliaths – the biggest players, who among themselves account for 99% of all tracking on the Web today.

Google, for example, has eyes on 70-80%² of the web. While that’s horrible, it also means that once we block their trackers, we’ve cleared a huge swath of surveillance from the Web in one go. Rinse and repeat for the top-however-many-thousand trackers and you’ve done a pretty good job of rinsing the web of surveillance devices.

A happy side-effect of blocking trackers and adtech is that you also greatly improve the experience of the Web. Adtech and tracking scripts amount for a huge portion of the physical size and visual complexity of web sites. So much so that a few years ago, the worst site we could find was costing people in the US over $4 million every month in mobile bandwidth costs just to load their trackers.

So blocking adtech and trackers makes sites load faster, stops them from interrupting your experience with interstitial modals, reduces visual complexity and cognitive load, and generally gets the Web looking more like it would if it hadn’t been poisoned by the surveillance-based business model of Silicon Valley.

The hand-curated nature of what we do also means that we can try and respond to those cases where things break as we control over and a good understanding of what the totality of the block rules actually do.

However, because Better was originally created as a native WebKit content blocker for iOS and macOS, the only people who could take advantage of it were people on Apple’s platforms. I was never happy with this but it was what we needed to do in order to keep the lights on.

~~So, when I got the opportunity last year, I updated the build process to also output the Better blocking rules in EasyList format.~~

A Better GNOME

As Better’s blocking rules are now available in EasyList format, it means that you can use them on other platforms. For example, you can use them in Chromium and Firefox across all platforms supported by those browsers via uBlock Origin – another no-bullshit tracker and adtech blocker.

It also means that you can use Better in GNOME Web. Which brings me, finally, to the great reveal: how?

~~It’s rather anti-climactic how easy it is.~~

Update: (2020-03-14) GNOME Web now supports WebKit content blocking rules, the same as Safari on iOS and macOS. I’ve updated the instructions below accordingly.

To get GNOME Web to use Better’s blocking rules, all you need to do is to enter the following command in terminal:

gsettings set org.gnome.Epiphany content-filters "['https://better.fyi/blockerList.json']"

That’s it! 🎉

That will replace the default filters with Better’s list.

As long as you install it using the exact method above and our list is the only one you include, we will also be happy to offer you support³. You can use our issue tracker and forums to report issues and keep in touch.

GNOME Web is a beautiful example of ethical design that makes me smile every time I use it. Now that I have Better on it, I have another reason to smile. Here’s hoping it puts a smile on some of your faces too.

References

Blocking advertisements in GNOME Web (GNOME Web Wiki)

This is a lesson that autocratic governments know all too well. It is also why they fear their own people more than anyone or anything else and try to control them through fear, mass surveillance, and violence. ↩︎
The 70% statistic is from a Princeton study. The 80% statistic is from our own web crawls using Better Inspector. ↩︎
Laura and I can’t offer support when more than one blocking list is active as we simply do not have the capacity to debug what could be going wrong with third-party lists. It’s hard enough trying to fix the Web when you have full control over the blocking rules. It’s next to impossible when you have multiple lists that might be interacting with each other in wild and wonderful ways. ↩︎

Changes

mail@ar.al (Aral Balkan) — Mon, 16 Jul 2018 16:42:29 +0100

My current setup: work in progress.

For the last 12 years, my main development machine has been a Mac. As of last week, it’s a Dell XPS 13 running Pop!_OS 18.04.

From the day the first iPhone was released, my main phone has been an iPhone. As of this week, it will be a Samsung S9+ running LineageOS. (I had it running LineageOS last week but there was an issue with the camera that meant that I had to reinstall the stock surveillance-ridden Samsung ROM. I plan to rectify that this week.)

Why the change?

There are two reasons I have been using and developing on Apple’s platforms for the last decade. Here they are in the order in which I was aware of them:

Apple’s focus on experience design. This is not a cosmetic consideration. Experiences are literally all we have in life.
Apple has an absolute competitive advantage in privacy over surveillance capitalists like Google, Facebook, etc. This is because they inherited their business model of selling products (not people) from the personal computer era.

So, all in all, Apple does exceedingly well on the top two layers of the ethical design manifesto (respect for human effort and respect for human experience) and is the least of possible evils when it comes to multibillion-dollar multinational corporations that make proprietary mainstream technology in relation to the core layer (respect for human rights).

Ethical design respects human rights, human effort, and human experience.

That said, I am severely disappointed with how Apple employs one set of ethics when it comes to “the West” and another set when it comes to countries like China¹.

Apple has shown that they are willing to be complicit to government surveillance and censorship in authoritarian markets like China. Just because I have the privilege of living in a Western country where - for the time being - it is in Apple’s economic interests to protect my privacy doesn’t give me any trust in them for the future given that we live in a world of Trump and Brexit. I also wonder which of my friends in Turkey (where my family is from) will be sold out by Apple to Erdoğan once the latter starts imposing China-like demands (or maybe the market isn’t large enough so Apple will rediscover its backbone in Turkey). Needless to say, no multibillion-dollar multinational company is your friend.

Why now, not earlier?

Where proprietary systems and the people farming devices of surveillance capitalists fail at the core layer of the hierarchy of ethical design by violating human rights, Free and Open Source alternatives have a rich history of succeeding radically at the core while failing at respecting human effort and experience.

If freedom technologies have any chance at competing with the products of surveillance capitalism, they must embrace design. If we expect people to use our wares because of our superior ideological arguments, we will be waiting a long time. Thankfully, there is a huge momentum of development groups that understand this and aren’t waiting around. Instead, they’re working hard to make convenient, usable, and - dare I say, delightful - tools and experiences that happen to protect your rights and freedoms.

This brings us to the answer to the “why now?” question. Because I felt I could without negatively impacting my day to day experience.

And it’s all thanks to the work that went into Gnome Shell, Ubuntu 18.04, elementary OS and the curation of all three of these by System76 that resulted in the state of the art of GNU/Linux distribution that is Pop!_OS 18.04.

Take note: this is the first operating system I’ve used that is simpler, more elegant, and does certain things better than macOS.²

I never thought I’d ever write those words about GNU/Linux and it gives me a lot of hope for the future.

See Apple’s capitulation to China’s VPN crack-down will return to haunt it at home, Apple’s China-friendly censorship caused an iPhone-crashing bug, Apple is under fire for moving iCloud data to China. ↩︎
I’ll explore some of these in detail in future posts. ↩︎

Adding “command not found” apt package suggestions to zsh

mail@ar.al (Aral Balkan) — Mon, 16 Jul 2018 12:49:05 +0100

How apt: package suggestions under zsh.

Under the GNU Bash shell on a Debian-based system, if you type a command that is unrecognised, in addition to an error message telling you that the command is not found, you also get package suggestions for the apt package manager.

This is a wonderful pattern in interface design: instead of just telling you that something went wrong and leaving you to figure out what it was, we give you helpful suggestions to try and lead you in the right direction. In this specific use case, it also means that you get to discover new app packages that you might otherwise not have.

My shell of choice, zsh¹, doesn’t have this feature built-in but, thankfully, someone created a package called command-not-found that you can easily install to enable it:

sudo apt update
sudo apt install command-not-found
echo 'source /etc/zsh_command_not_found' >> ~/.zshrc
source ~/.zshrc

Now, when you try to execute a command that doesn’t exist, zsh will helpfully suggest a package or two that you might want to install that might have the command you were thinking of.

I use zsh with the excellent Oh My ZSH! configuration with the agnoster theme with powerline fonts. ↩︎

Flashing stock firmware onto a Samsung Galaxy S9+ (SM-G965F) on Ubuntu 18.04 using Heimdall

mail@ar.al (Aral Balkan) — Sun, 15 Jul 2018 20:06:45 +0100

Samsung stock Android.

On the way back from London last week, I picked up an unlocked, EU region Samsung Galaxy S9+ to install LineageOS on.

When I got back, I proceeded to install the July 3rd nightly build and everything was good until I tried the camera app and it kept hanging. Apparently, this is a known issue and a “won’t fix” as far as the LineageOS folks are concerned as it can be resolved by updating the stock firmware. Which, of course, means that you have to install said firmware again.

(Lesson learned: before installing a different operating system, make sure you update your phone to the latest official firmware to potentially save yourself some trouble.)

Adventures in firmware

To complicate things, Samsung UK doesn’t make the firmware for its phones available on the S9+ support site. When I reached them for help via DM on Twitter, they very nicely told me where I could stick my new €1,000 phone because I had had the audacity to install a different operating system on it than the dark-pattern-ridden spyware/bloatware that it came with. Thanks, folks, great customer service! (Every day that passes, I’m thankful that Todd and the lovely folks at Purism are working on full-stack freedom with their computers and upcoming phone.)

Samsung’s reluctance to offer system restoration firmware for download means that a cottage industry, most of it rather shady, has popped up to fill the demand. Of the more reputable sites for information is XDA Developers and that’s where I found the firmware for my S9+.

(Again, Samsung, you are failing your customers and potentially opening them up to security issues by not making the firmware available on your site. Please reconsider. The folks on XDA and SamMobile get it from you somehow and you know this is happening so why not “legalise” it so that people know they’re getting the real deal and don’t end up installing trojans and backdoors on their devices and end up making both themselves and the rest of us less safe.)

That all said, finding the firmware was only the beginning of the journey.

Here are the other steps I had to follow to install said firmware.

Flashing with Heimdall

Heimdallr brings forth the gift of the gods to mankind (1907) by Nils Asplund

These instructions are for the Samsung S9+ (SM-G965F), EU region, unlocked, using the stock firmware linked to from the “Official Stock Firmware Thread for S9+ (SM-G965F) on XDA. Furthermore, they assume that you are carrying out the flashing using Heimdall (son of Odin), on a computer running some variant of Debian so you can use the excellent apt package manager. I’m currently running Pop!_OS 18.04 which is based on elementary OS/Ubuntu.

Install Heimdall:
```
sudo apt install heimdall-flash
```
Install lz4 (you will need this later to uncompress files):
```
sudo apt install liblz4-tool
```
Unzip all the files from the firmware file (which, in my case, is called G965FXXU1BRF8.zip. This is a little like unpacking Russian dolls given that there’s a zip that contains tar.md5 files which contain lz4 files. The following commands will extract everything (this may take some time):
```
unzip G965FXXU1BRF8.zip -d firmware && cd firmware
for f in *.tar; do tar xf $f; done
lz4 -dm *.lz4
```
You should now have a bunch of files. The important ones end in .img and .bin. These are what we’re going to flash onto the partitions on your phone using Heimdall. If you want to jump right into doing that without understanding how we know how to map the files to the partitions, you can safely jump to Step 9 now. Otherwise, read on and learn…
To find out where we need to flash the various files we have, we need to ask Heimdall to inspect our phone and dump a Partition Information Table (PIT) file for us. To do this, first connect your phone to your computer via USB and boot the phone into Download Mode (hold down the power, volume down, and Bixby buttons - that’s the button on the top-right, along with the second and third buttons from the top on the left). When you see the Download Mode splash screen, press the Volume Up button as instructed to enter Download Mode.
Test the connection:
```
heimdall detect
```
Dump the PIT file using Heimdall. (Note: your phone will reboot after this. Enter Download Mode again using the technique you learned in Step 5):
```
heimdall print-pit > phone.pit
```
Open up phone.pit in a text editor and search for the names of the extracted files in Step 3 that end with .img and .bin. Note the corresponding Partition Name values as those are what you will be using in the next step as the names of the flags to the heimdall flash command.

Partition name to image file name mapping.

Flash the firmware using Heimdall (your partition name -> flash filename mappings may vary. I’d highly recommend not skipping Steps 4-8 above and confirming that the mappings in your PIT file match before executing the following command):

sudo heimdall flash --BOOT boot.img --CACHE cache.img --CM cm.bin --DQMDBG dqmdbg.img --HIDDEN hidden.img --KEYSTORAGE keystorage.bin --RADIO modem.bin --CP_DEBUG modem_debug.bin --ODM odm.img --OMR omr.img --PARAM param.bin --RECOVERY recovery.img --BOOTLOADER sboot.bin --SYSTEM system.img --UP_PARAM up_param.bin --USERDATA userdata.img --VENDOR vendor.img

That should do it! (Whew!)

Your phone should restart and you may be dumped into a recovery screen (I was). Just unplug your USB cable and choose to reboot your phone and you should see the stock Samsung firmware boot up following a pulsating Samsung logo (this might take a little while). You should eventually be greeted with the “Hello!” screen and prompted to set up your phone.

This process is much more convoluted than it should be (due, in no small part, to Samsung’s lack of cooperation). I hope that this post makes it a bit easier to grasp and carry out.

A huge thank-you to everyone who documented their own experiences (see links above and references, below) and to Benjamin Dobell and Glass Echidna for Heimdall without which none of this would be possible.

Next up: I’ll be installing LineageOS again and hopefully have the camera work this time.

References

Initial git configuration

mail@ar.al (Aral Balkan) — Fri, 13 Jul 2018 20:00:22 +0100

I’m setting up my new XPS 13 running Pop!_OS and one of the things I always have to do on any new machine is to configure git.

So, both for my own reference and in case it helps anyone else, here’s a list of things I do to set up git:

Set up my identity (used in commits, tags, etc.)

git config --global user.name "Aral Balkan"
git config --global user.email mail@ar.al

Set up automatic signing of my commits

I sign my work and so should you (it’s about security, not vanity).

Either generate GPG keys if you don’t already have a pair or export/import your existing GPG keys.
Configure git to sign all your work:
```
git config --global commit.gpgsign true
```

You can check that it’s working by making a commit and then viewing at it in the log with the --show-signature flag:

git log -1 --show-signature

If you get a warning along the lines of WARNING: This key is not certified with a trusted signature! There is no indication that the signature belongs to the owner after importing your GPG key to a new machine, do the following (substituting your email address for mine):

gpg --edit-key aral@ind.ie

Then, in the gpg shell, use the trust command and select 5 = I trust ultimately.

If you don’t trust yourself, who are you going to trust?

Create ssh keys for the machine and import them into hosted git remotes like GitLab

In order to clone and push to git remotes using ssh, you have to create an ssh keypair and then upload it to your git remote (in my case, we use our own GitLab instance running on source.ind.ie).

Generate your ssh keys (substitute your email address for mine):
```
ssh-keygen -t rsa -C "aral@ind.ie" -b 4096
```
Copy it onto the system clipboard.
```
cat ~/.ssh/id_rsa.pub | pbcopy
```

Note: On Linux, you can simulate the hugely useful pbcopy and pbpaste commands on macOS by adding the following to your .zshrc or .bashrc files:

alias pbcopy='xsel --clipboard --input'
alias pbpaste='xsel --clipboard --output'

Disable paging

Set cat as the pager so that git commands do not use paging (I’d much rather scroll than switch modes.)

git config --global core.pager cat

That’s all I can think off at the moment. If I notice that I’ve left anything off, I’ll update the post accordingly.

If you have any initial configuration steps that you cannot live without, please let me know on Mastodon.

References

Casino Data

mail@ar.al (Aral Balkan) — Fri, 06 Jul 2018 12:31:49 +0000

Wouldn’t it be nice if every spider clearly labelled the entrance of its web?

Welcome to Casino Data, where you pay with your data. No, I’m not talking about just another day in our present dystopia under surveillance capitalism.

Unlike our world, this installation at FutureFest makes it very clear that what you’re gambling with is aspects of your self.

Gambling with our data… an analogy for life today.

Whither ethics, New Scientist?

mail@ar.al (Aral Balkan) — Fri, 06 Jul 2018 06:49:36 +0100

New Scientist: what’s science sans ethics?

I’m on a flight from Cork to London to speak at Nesta’s FutureFest this weekend and I’m reading New Scientist. A full-spread ad for their conference in September, New Scientist Live catches my eye. Hmm, might be interesting. And then I see the list of sponsors. Wow.

When attempting to explain why a privacy or human rights conference having Google (Alphabet, Inc.) or Facebook, Inc., as sponsors is problematic, I often use the analogy that no one would trust a healthcare conference spnsored by Philip Morris or a conference on the environment sponsored by Exxon Mobil. Well, guess what?

New Scientist’s “Earth Zone Sponsor” for their conference is (drumroll) Shell.

Greenwashing. (And whatever it is you call legimitising weapons manufacturers.)

Take a bow, New Scientist. I thought we lacked ethics in the technology world, but you are exploring new frontiers in corruption. How do the folks who work at New Scientist sleep at night knowing that their publication is using whatever legitimacy it has in the field of science to greenwash a petrolium company?

And it doesn’t stop there.

Their “technology zone sponsor” is BAE systems. You know, the company that makes technology that kills people.

Seriously, New Scientist, it’s time you took a long, hard look at the ethics of your choices.

Rsync Permissions and Termux

mail@ar.al (Aral Balkan) — Thu, 05 Jul 2018 18:08:48 +0000

The way my blogging setup works in live mode is that I have a watcher that uses rsync to deploy changes to my server.

The first time I started live mode while blogging on my phone, I ended up with a 403 Forbidden error on my site.

Connecting to the server via ssh, a quick ls -la on the web root¹ showed me that the permissions were more restrictive than they should have been. The immediate fix was easy:

chmod -R 755 <web root>

However, the problem would resurface the next time rsync ran from my phone. So I looked at the permissions for the local folder on my phone and saw that they matched the restrictive ones on the server. Thinking that the permissions were simply getting copied over, I tried setting the local folder’s permissions to 755 and trying rsync again. No go. The permissions reset again to the restrictive ones.

What did end up working was to add the --chmod=755 flag to the rsync command (I was already passing the --archive flag, which includes the --perm flag, without which this might not have worked.

Since this isn’t an issue when running rsync from my Mac, I’m assuming it’s an oddity related to how Termux manages directory permissions.

If you forget where you set your web root to under nginx, as I sometimes do, look for it in your server definition in the relevant server configuration file that’s symlinked from /etc/nginx/sites-available. ↩︎

Web+ on a Phone

mail@ar.al (Aral Balkan) — Thu, 05 Jul 2018 12:47:04 +0100

E.T. blog phone.

I’m blogging this from a phone.

OK, so your mind isn’t blown. I understand. After all, anyone can blog from a phone using a native or web app that speaks to a server somewhere. The difference is I have an entire copy of my site on my phone and I’m blogging using the exact same Web+ setup I described earlier on my Mac.

To recap: there’s a Node.js-based watcher running which, if I’m online, synchronises my local site with the site on my server using rsync. And I’m using a Go-based static site generator, Hugo, to generate the site every time I save. And all this is running on the old Nexus 5 I had lying around thanks to a little miracle of an app called Termux. And, of course, on my server, I’m running DAT so that updates to my site automatically get shared on the peer web. The only thing that’s different is that I’m writing this in GNU Emacs instead of in VSCode.

Just like on my main machine, I have full local/offline storage, local and (if online and running in live mode) remote live reload, etc.

Video of static site generation with live reload on my phone.

Live reload on a phone.

Setup

Here are some brief notes on how I got this working in case any other developers want to play with a similar setup at any point.

I installed LineageOS on my old Nexus 5 using the instructions on their Wiki. They have detailed installation instructions for a plethora of devices there. I also compiled a list of phones that can run the latest version of LineageOS (15.1). My Nexus 5 can only run 14.1, which runs just fine.
I installed the F-Droid free software catalogue. This is a freedom technology alternative for people who don’t want Google’s surveillance services on their phones.
Using F-Droid, I installed Termux. This is where the magic happens. Termux gives you a lightweight Linux environment on your Android phone and doesn’t even need root access on your device to do so. I could have installed it on a stock Android phone (but then I’d have a Google surveillance device in my pocket… so no thanks).
As Termux comes with the apt package manager via the pkg command (this is amazing), I was able to install git, clone my Web+ setup repositories, install Node.js, Go¹, and Hugo (which I had to compile from source).

Following those steps, I was up and running with essentially the same setup that I have on my Macbook.

The experience

In a nutshell: it’s rather amazing.

I am using a Logitech hardware keyboard that I had for my iPad, without which this would not be a viable tool for coding or blogging. With it… well, my mind is blown by how well this four-year-old phone runs all this stuff.

There is absolutely no lag! Hugo’s rebuilds are instant as is the responsiveness of the rsync watcher. As far as I can tell, everything runs as well as it does on my Macbook.

Challenges and processes

That’s not to say that it’s perfect. This is not the most ergonomic setup ever, even with the excellent Logitech keyboard. Using an external monitor via Miracast mirroring the screen does help make it better but also adds a slight lag to the display. I have to find a slimport adapter and see what it’s like using a wired connection. From a demonstration video I watched on the web, it looks like it has a perfectly smooth mirroring experience.

Also, I initially battled with my lazy fallback editor of choice on Linux terminals, GNU nano. I ended up somehow pressing some keyboard combination, I guess, that got nano to hard-wrap the lines of my post. While shaving that particular yak, I ended up giving GNU Emacs a shot. After finally working my way through its tutorial some 30-odd years after it was written, I have to say, I really like Emacs. (I know, I’m shocked too… here, have some tea.)

There is nothing intuitive about Emacs but it is the product of another time and there is method behind the madness. And, for the moment at least, it suits my coding and blogging needs just fine. Running it in split screen with my browser also works really well (nano had trouble recovering from being resized).

Using a combination of split screen and task switching, I can easily browse the web to find and copy links.

Adding images and screenshots taken on the phone itself is slightly more convoluted due to the lack of a visual process (e.g., drag and drop). Instead I gave Termux storage access and I use plain old cp to copy images in from ~/storage/pictures and ~/storage/dcim/Camera. I also use the built-in Gallery app to browse images and the termux-open command from the Termux API add-on to trigger image previews from the commandline.

As Termux supports multiple sessions, I can have my watchers running in one session and Emacs in another. When that, combined with Emacs’s support for easy suspend and resume (Ctrl x Ctrl z to suspend, fg on the commandline to resume) isn’t enough, I can also use tmux – a terminal multiplexer – to split my screen with multiple terminal sessions. On a mobile phone. I know! 😱

Communicating with the outside world

Another issue is how to get things on this phone from my other devices, all of which are locked into Apple’s nursery. This is where cross-platform freedom apps become an invaluable lifeline.

First off, Signal and Wire. Freedom from the surveillance silos without becoming hermits requires that we have end-to-end encrypted means of communication between devices, platforms, and people. Both Signal and Wire solve this problem beautifully and in a manner that can be conveniently enjoyed by people without the necessary technical knowledge or time.

I initially found it difficult to sign into services I was using as all my passwords are in 1Password. 1Password does have an Android app but it’s only available from the Google Play Store. I will not be installing any of Google’s surveillance services on this phone. This creates a problem. (One that I also ran into with the VPN I’m using on Apple’s platforms, EncryptMe I’ve raised the issue with both companies and I hope that we can make some progress there. Otherwise, I’ll be looking for alternatives that enable people to use them without subjecting themselves to Google’s surveillance machine.

To get around the problem of sharing information between my nursery machines (Apple devices) and my grown-up tools, I ended up just using Signal with disappearing messages. It’s not a hassle to send a password, end-to-end encrypted from 1Password on my Mac/iPhone to my freedom phone. I also use the same system to send photos I’ve taken on my iPhone to my freedom phone. For videos, I upload them to Vimeo (Ind.ie has a Pro account with Vimeo so that we can embed the video files directly without the Google surveillance that comes bundled with the default Vimeo player).

In the middle-term, I plan to try out my old friend SyncThing for cross-platform secure file transfers. In the longer term, I will be looking into KeePassXC as an alternative to 1Password and my own DAT-based solutions for private/secure file sync, messaging, and publishing.

It’s a great time for ethical technology

I have to say that I haven’t been this excited about technology in a long while because I can finally see the pieces coming together for ethical alternatives to start making inroads into the mainstream. In a lot of ways, this process has already begun. Mastodon, DAT, Signal, Wire, Matrix, Purism, Eelo, Gnome, Pop!_OS, LineageOS, Termux… these are just some of the products and projects off the top of my head that give me immense hope for the future of ethical technology.

That’s not to say that the setup I’ve described here and that I’m using is one that I expect people without technical knowledge and the time to spare to play with today. But it is giving me a myriad of ideas and this is stuff that developers (and not just I-think-in-binary-and-speak-fluent-hex developers) definitely can (and should) toy with. The challenge is exploring and then exposing what’s possible with such freedom-respecting and freedom-preserving setups conveniently so that people who don’t have the technical knowledge or perhaps just the time or resources can take advantage of the freedoms that they offer. For those of us that care about ethical technology, our job is to build the convenient bridges between the centralised, surveillance-based silos of today and the ethical, interoperable, and open freedom technologies of tomorrow.

Let’s create beautiful defaults and layer the seams.

I did discover a bug with Go under Termux: it doesn’t get the correct timezone. A symptom of this, for Hugo, is that the dates of posts have the wrong timezone. I’ve opened an issue for it. ↩︎

Spellcheck With Emacs on Termux

mail@ar.al (Aral Balkan) — Thu, 05 Jul 2018 11:12:08 +0000

I missspell sometimes.

I’m using Emacs on Termux to blog with on my phone and realised that, by default, spellcheck doesn’t work. M-x ispell (Altx ispell) fails with the error “Searching for program: no such file or directory, ispell”.

After investigating a bit, it turns out that neither ispell or aspell are available as packages for Termux. Instead, they’re concentrating their efforts on supporting hunspell.

To get spellcheck working under Emacs with hunspell:

Install the hunspell package
```
pkg install hunspell
```

Add the following to your Emacs configuration:

(setq ispell-program-name (executable-find "hunspell"))

Restart Emacs.

That should get you a spell checker in Emacs.

Nginx: Remember to remove the default site

mail@ar.al (Aral Balkan) — Thu, 05 Jul 2018 08:10:11 +0100

What a minute, where did my site go?

I realised yesterday that people accessing my new site for the first time via HTTP (not HTTPS), were seeing the default page for my nginx web server. I only realised it because I was trying the site out on a Samsung S9+ that I was playing with at PC World. (It’s always a good idea to test out your sites on the range of devices they have at such places. They don’t mind - in fact, you might find yourself having pleasant conversations about the state of surveillance capitalism and ethical technology with the folks who work there, like I did with Paul yesterday - and you might discover things you missed on your own setup. PC World is basically a large, free device lab.)

Anyway, a quick search on DuckDuckGo revealed the issue: I hadn’t removed the default site configuration from /etc/nginx/sites-enabled/ and it was competing with the HTTP → HTTPS redirect that I’d instructed Let’s Encrypt to set up for me.

Deleting that symbolic link fixes the problem.

Phones LineageOS 15.1 is officially supported on

mail@ar.al (Aral Balkan) — Tue, 03 Jul 2018 11:18:13 +0100

It’s GNU Linux. On a phone.

LineageOS is the successor to Cyanogenmod – a Google-free, freedom technology alternative to stock Android that isn’t infected with Google Play Services (Google’s surveillance services).

The LineageOS logo.

I have a Nexus 5 (hammerhead) from several years ago that I installed LineageOS 14.1 on and it works flawlessly. In fact, thanks to its freedom and the mind-blowingly awesome Termux, I now have a basic Linux setup on it running Node.js, Go, and my Web+ blogging setup that uses Hugo, rsync, and DAT. It’s awesome that LineageOS enables people to keep using their older handsets beyond the couple of years that some manufacturers deem an acceptable period for supporting their products with updates (hey, landfills are cheap and the Earth is boundless, right?… Right?!)

Anyway… I do want to try LineageOS on a modern 64-bit device not least because I have a feeling the segmentation faults I’m getting while trying to run DAT on it might be due to it being 32-bit. So I found myself sifting through the list of downloads and installation instructions for all officially-supported handsets as it’s not grouped by version. I had to go through the whole thing, but there’s no reason anyone else should have to as I compiled the list of handsets that LineageOS 15.1 is currently supported on and you can find it, below.

If you already have an older handset that no longer receives operating system updates, you should seriously consider extending its life, ditching Google’s spyware, and improving your security using LineageOS (and the F-Droid freedom software catalogue). Version 14.1 is stable and runs flawlessly on older hardware. If you’re on the market for a new handset to run LineageOS on, however, I hope the following list helps you identify potential candidates easily.

The list should be considered current as of the time of this post but I will not be maintaining it. (Pull requests are welcome.)

Discounting the Google devices (because even though the bar on ethics is set pretty darn low in the industry, I’ll be damned if I give that particular surveillance capitalist any more money), the newest/best-equipped models appear to be the Samsung Galaxy S9 and S9 plus, the OnePlus 5T, and the LeEco LePro 3. I’ll either pick one of those three up or wait a little to see how things shape up with LineageOS support on the new Project Treble-based phones.

Manufacturer	Supported Models
Asus	✗
BQ	✗
Fairphone	✗
Google	Nexus 6P (angler), Nexus 6 (shamu), Nexus 5X (bullhead), (note: not Nexus 5), Nexus 4 (mako), Pixel C (dragon), Nexus Player (fugu)
HTC	One A9 (hiae)
Huawei	✗
LeEco	Le Max 2 (x2), LePro 3
LG	✗
Motorola	Moto Z (griffin), Moto Z2 Force (nash)¹
Nextbit	✗
Nubia	✗
Nvidia	Shield Android TV (foster)
OnePlus	One (bacon), 5 (cheeseburger), 5T (dumpling), 2 (oneplus2), 3 / 3T (oneplus3)
OPPO	Find 7a (find7)
Samsung	Galaxy Tab S2 9.7 2016 Wi-Fi (gts210vewifi), Galaxy Tab S2 8.0 2016 Wi-Fi (gts28vewifi), Galaxy S5 Plus (kccat6), Galaxy S5 LTE-A (lentislte), Galaxy S9+ (star2lte), Galaxy S9 (starlte)
Sony	Xperia XA2 (pioneer)¹
Wileyfox	✗
Xiaomi	Mi 5s (capricorn), Mi MIX 2 (chiron), Mi 5 (gemini), Mi MIX (lithium), Redmi Note 4 (mido), Mi 5s Plus (natrium), Mi 6 (sagit), Mi Note 2 (scorpio)
YU	✗
ZTE	✗
Zuk	✗

Experimental. ↩︎

Refining the RSS

mail@ar.al (Aral Balkan) — Sun, 01 Jul 2018 01:37:19 +0100

I can’t optimise my feed for every reader out there but I can for the one I’m using.

I realised today¹ that my site’s feed wasn’t displaying well on the RSS reader I’m using (Leaf on macOS). Some other feeds had image previews for items as well as an icon for the site itself while mine didn’t.

How utterly unacceptable!

Sweat the small stuff (it’s all small stuff)

Given I’m writing this at almost 2AM on a Saturday night, you can probably deduce that there was a level of – ahem – trial and error involved in getting things to work.

Initially, I thought the icon for my site wasn’t showing up because I hadn’t added an <image> tag to my RSS feed. So I added that to my RSS template in Hugo:

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    …
    <image>
      <url>{{ "/apple-touch-icon-144x144.png" | absURL }}</url>
      <title>{{ if eq  .Title  .Site.Title }}{{ .Site.Title }}{{ else }}{{ with .Title }}{{.}} on {{ end }}{{ .Site.Title }}{{ end }}</title>
      <link>{{ .Permalink }}</link>
      <width>144</width>
      <height>144</height>
    </image>
    …
  </channel>
</rss>

Sadly, Leaf didn’t bite.

Images were working for Laura’s site so I took a look at her feed and found a clue: her images had absolute URLs while mine were relative.

Hmm… 🤔

Relatively speaking

I can’t use absolute URLs on this site as it’s a Web+ site. Since its accessible over both old school HTTPS and on the peer web, I can’t hardcode a protocol like https:// as that would break links for the dat:// protocol.

That limitation doesn’t really apply to the site’s RSS feed, however.

The W3C Feed Validation Service actually warns you not to use relative URLs in your feeds. (This is a limitation I feel we should remove, going forward. Not least of all because peer replication of RSS feeds over DAT might have some interesting use cases for push/streaming syndication.)

So to cut to the chase, I decided to use a regular expression to massage the URLs in my RSS feed template in Hugo to convert relative URLs to absolute ones:

<item>
  …
  <description>{{ replaceRE "img src=\"(.*?)\"" (printf "%s%s%s" "img src=\"" .Permalink "$1\"") .Content | html }}</description>
</item>

Initially, I also had a negative lookahead in there to ensure I wouldn’t rewrite any URLs that might have been absolute URLs to begin:

img src="(((?!http?).)*?)"

But, sadly, Go’s regular expression engine does not support negative lookaheads. This isn’t a big deal, really, since it’s not good form to link to external images directly anyway.

With that change, image previews started working in Leaf.

Favicon blues

But my site’s icon still wasn’t showing up.

It was one-something-AM and instead of going to bed like a sensible person, I decided that this needed fixing right now (the sadder part is that I’m writing this sentence while editing this post at 2:30AM).

Anyway, looking at an icon for site that was displaying properly, I noticed that the image was rather high resolution. So it wasn’t just grabbing a favicon.ico. I already had the ridiculous number of icons one needs for a respectable web site in 2018 generated using Favic-O-Matic. In the end, after more trial and error, I found that Leaf was looking for an apple-touch-icon.png file in the root of the site.

Adding that fixed the icon issue also.

My kingdom for standards-compliant RSS readers

It would be ridiculous, of course, to try and support the idiosyncrasies of every RSS reader out there but I can at least make my feed look good on the one(s) I use.

That said, it would also be great if more RSS readers supported standard elements when available.

Leaf, for example, should make use of the image tag if one exists in a feed. Similarly, RSS readers should present items in enclosure elements (images, audio, and video) when possible. While not part of the standard, it also wouldn’t take much work to resolve relative URLs properly in feeds either.

i.e., “Became unable to ignore any longer…” 😁 ↩︎

Reclaiming RSS

mail@ar.al (Aral Balkan) — Fri, 29 Jun 2018 11:33:13 +0100

RSS like it’s 1999.

English | French

Before Twitter, before algorithmic timelines filtered our reality for us, before surveillance capitalism, there was RSS: Really Simple Syndication.

RS-what now?

For those of you born into the siloed world of the centralised web, RSS is an ancient technology from Web 1.0 (“the naïve Web?”). Like most things back then, it does what it says on the tin: it enables you to easily syndicate the content of your site. People interested in following your posts subscribe to your feed and receive updates using their RSS readers. There is no Twitter or Facebook in the middle to algorithmically ~~censor~~ … ahem … “curate” your posts.

RSS is stupidly simple to implement (it’s just an XML file). You could hand-roll it manually if you wanted to (although I wouldn’t recommend it).

Here’s an excerpt of this site’s RSS feed, showing some of the fields of the current entry for this post:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Aral Balkan</title>
    <link>https://ar.al/</link>
    <description>Recent content on Aral Balkan</description>
    <lastBuildDate>Fri, 29 Jun 2018 11:33:13 +0100</lastBuildDate>
    …
    <item>
      <title>Rediscovering RSS</title>
      <link>https://ar.al/2018/06/29/rediscovering-rss/</link>
      <pubDate>Fri, 29 Jun 2018 11:33:13 +0100</pubDate>
      <author>mail@ar.al (Aral Balkan)</author>
      <description>(The content of this post goes here.)</description>
    </item>
    …
  </channel>
</rss>

It is also almost universally implemented.

Chances are, if you have a web site, you already have an RSS feed whether you know it or not. If you use Hugo to generate your site, for example (like I do), your RSS feed is at /index.xml.

Other generators might put it at /rss, /feed, /feed.xml, etc.

Where’s the RSS?

The Noun Project has a great selection of RSS icons you can use.

Time was, you couldn’t browse the web without seeing RSS icons of all persuasions gracing the façades of Web 1.0’s finest. This was before they were mercilessly devoured by the ~~tracking devices~~ … ahem … “social sharing buttons” of people farmers like Google and Facebook.

There was also once a push for browsers to auto-detect and expose RSS feeds. Currently, none of the major browsers appears to do so.

It’s time to push back against this and demand first-class support for RSS as part of the move to re-decentralise the Web.

But you don’t have to wait for browser vendors (some of which – like Google – are surveillance capitalists themselves, and others, like Mozilla, get all their money from surveillance capitalists). You can start making RSS more visible again today by finding the URL for your own RSS feed and exposing it visibly on your site.

It’s not complicated: just a link in the head of your page¹ and a link in the body with an RSS icon and Bob’s your decentralised Uncle.

Here’s the link in the head:

<link 
  rel="alternate"
  type="application/rss+xml"
  href="https://ar.al/index.xml"
/>

And here’s the header in the body that links to the RSS feed visually using an icon.

<a 
  rel='alternate'
  type='application/rss+xml'
  href='/index.xml'
>
  <img 
    class='rss' 
    src='/icons/rss.svg'
    alt='RSS feed icon'
    title='Subscribe to my RSS feed'
  >
</a>

Check out The Noun Project for a set of RSS icons you can use under Creative Commons licenses.

Full-fat vs skinny RSS

The Leaf RSS reader displays full HTML content perfectly.

When generating an RSS feed for your site, you have the option to include only summaries of your posts or the full content. I modified my Hugo configuration and the default RSS template using these instructions by Brian Wisti to include the full content feed and I recommend that you do the same.

Six years ago, I was arguing the opposite, stating that “full fat RSS is duplicate content by another name”. I was wrong. I was too obsessed with maintaining a formalistic stranglehold over my designs and thus failed to correctly weigh the decision using ethical design criteria.

The Newsbar RSS Reader doesn’t display images or styles correctly in content previews.

The more ways people have of consuming your content, the more resilient that content becomes and the more freedom people have.

Duplicate content? Yes, please. The more the better! Heck, on the peer-web version of this site, the goal is to ideally have the content duplicated as many times as there are people consuming it.

Yes, your content may not display the same in certain RSS readers that are not standards-compliant but that’s their problem, not yours. In my limited testing, the Leaf RSS reader for macOS displayed my full-fat RSS perfectly while the NewsBar app didn’t. That’s OK. (And I hope NewsBar will take note and improve its rendering in its next update. No app is born perfect.)

As we move away from the centralised web to the peer web, it’s time to rediscover, re-embrace, and reclaim RSS.

Everything old is new again.

RSS was an essential part of Web 1.0 before surveillance capitalism (Web 2.0) took over.

It will be a cherished part of Web+ and beyond.

Thanks to Ed Summers for pointing out via Mastodon that I’d forgotten to add the RSS feed URL to the heads of my pages. I also noticed while looking into it on MDN that there are additional semantics you can add to the links you use in the body. ↩︎

Are You Logging IPs Without Even Knowing?

mail@ar.al (Aral Balkan) — Thu, 28 Jun 2018 18:04:39 +0100

Log off.

My blog is served both over the centralised web via HTTPS by nginx and over the peer web via DAT.

nginx has an access log that is enabled by default that logs IP addresses. Needless to say, I don’t want this information.

Thankfully, turning it off is easy:

server {
  access_log off;
}

That’s it!

(You might still be logging IP addresses in the error log so it’s a good idea to clear those out also on a regular basis.)

Remember that clearing out existing logs in nginx is as easy as:

rm /var/log/nginx/*

If you want to keep logs (e.g., to calculate unique views, etc.), you can still do that in a privacy-respecting manner by storing a hash of the IP address in your logs instead of the IP address itself. There’s an nginx module called ipscrub you can use for that purpose.

I do wish that servers like nginx came with privacy-respecting (and secure) settings by default.

The Real News: Turkish Elections

mail@ar.al (Aral Balkan) — Wed, 27 Jun 2018 19:13:44 +0100

A short one-minute teaser of my interview on the Turkish elections with Ben Norton.

Yesterday, I had a lovely chat with Ben Norton for The Real News.

We talked about two things: this week’s Turkish elections and the effect of technology on human rights and democracy.

The first part of our chat is now live.

Watch it here: Turkey’s Erdogan Expands Authoritarian Powers with Fascist Coalition Election Victory.

Kyriarchy

mail@ar.al (Aral Balkan) — Wed, 27 Jun 2018 11:21:15 +0100

Kyriarchy: a many-headed monster. Illustration by Josette Souza.

I learned a new word today: Kyriarchy.

I found it in Diana’s hugely inspirational initial post on her new blog and immediately realised it was the word I was struggling for all those times that “patriarchy” just wasn’t cutting the intersectional mustard in conversation.

According to Wikipedia, Kyriarchy is:

a social system or set of connecting social systems built around domination, oppression, and submission. The word was coined by Elisabeth Schüssler Fiorenza in 1992 to describe her theory of interconnected, interacting, and self-extending systems of domination and submission, in which a single individual might be oppressed in some relationships and privileged in others. It is an intersectional extension of the idea of patriarchy beyond gender.

Everyday Feminism magazine has a comic by Josette Souza that explains it visually.

And in Diana’s words:

I am more resolved than ever that we will end the kyriarchy. No one needs to starve or freeze or die. We can all live without fear.

Here’s to that.

Web+

mail@ar.al (Aral Balkan) — Tue, 26 Jun 2018 16:59:55 +0100

Web+ is like training wheels for the Peer Web. It creates a bridge between the centralised Web and the Peer Web to introduce the former to the latter.

My personal blog is now a Web+ site.

A what?

Well, it’s a web site but also a little more than a regular web site. It’s available on the regular web at https://ar.al, but it is also available on the peer-to-peer Web (henceforth “Peer Web”) via the DAT protocol at dat://ar.al.

To load the second link, you will need a DAT-capable browser. Currently, the only browser with native support for the DAT protocol is Beaker Browser.

Web + ?

So this site is a Web+ DAT site. It is accessible to the mainstream via the centralised Web and it progressively enhances and helps decentralise the centralised Web by layering on peer-to-peer functionality via a peer-to-peer channel.

A Web+ site doesn’t have to use DAT to be a Web+ site. There are other federation and decentralisation protocols and systems that can and should be explored. These include:

If I’ve forgotten any (that aren’t blockchain-based – spit! – or venture-capital-funded – spit!), please let me know.

How do I Web+?

Well, right now, the honest answer is: not easily. Not if you’re not a developer.

If you are a developer, it’s not difficult but the experience can (and will) still be made easier.

I was able to set up the first version of this site in an hour, including provisioning, configuring, and deploying the server (all while live tooting it). I wrote a little more about my setup earlier and I plan to streamline the setup further in the future.

You can also look through the source code to see how things work.

TL; DR: I use Hugo on my local machine to write my posts (in VSCode). If I’m in live mode, there’s a file system watcher that rsyncs updates to my server on every save. There, I have DAT share running as a service and it automatically shares the updated DAT site on its peer-to-peer swarm.

Of course, the ultimate goal isn’t just to create yet another fun toy for the technically-privileged. This is one step towards implementing a Web+ experience for everyone. That’s the project I’m working on and I will be evolving it to use DAT.

Advantages

So why go to the trouble of creating a Web+ experience?

Here are three advantages, off the top of my head:

Censorship-circumvention
Streaming web site updates
Scalability without additional resources

While #2 might be the coolest feature for everyday use, #1 has the most potential to change the world for the better so we’ll start there.

Censorship-circumvention

While you can access this site via a friendly DAT URL (using DAT DNS), the actual long-form DAT address that it maps to is a cryptographic public key that is unique to the mutable torrent-like archive³ of this site:

dat://d82084b53fce5f07696c2acf98a363601ec9410d06c633513269979786464eb3/

So, let’s say I’m in a country with an autocratic government and that I post something that the government wants to block. They can easily block my domain. But what if three or four people have already accessed my site over DAT? Well, now the site exists in three or four other places. And they can serve it to maybe another thirty or forty. And as long as they share the long-form DAT address, the government will have to block everyone that has a copy to block the content.

Right now, the long-form address is a bit hidden on this site but I will be exploring ways of exposing it more explicitly. One way to share the addresses might be with QR-Codes. (Might there actually finally be a good use case for QR-Codes after all?)

Streaming web site updates

We can already do server push on the centralised Web using either long-polling (like a caveman) or WebSockets. The problem is that it’s not trivial. And the more popular your content gets, the more resources you will need in order to meet the demand.

What if streaming updates to all clients just came for free? And you had to do nothing to support it? And what if the load on your server didn’t increase as your site got more popular.

Well, that’s how it is with DAT.

If you view this site in Beaker Browser and you turn on the Live Reload feature, you will get streaming updates on changes. I’ve already live blogged two events: the Public Stack Summit and the Next Generation Cities session at We Make the City in Amsterdam.

Scalability without additional resources

I touched upon this in the previous point also: one of the big advantages of peer-to-peer systems is that the network absorbs the load. As your site becomes more popular, you do not need to expand additional resources as your load is distributed on the network. Also, the more popular your site becomes, the more resilient it becomes as a growing number of other nodes begin to mirror your content.

So basically, it’s win-win.

And since we’re progressively enhancing the centralised Web, we are not leaving anyone behind. If people want to view the site over HTTPS, they can go right ahead. And if they want to download Beaker Browser out of curiosity (or because they’re staging a democratic revolution, I guess) then it could possibly be their introduction to the freedom of the Peer Web.

OpenNIC might seem like the odd one out in the list but it isn’t.

The centralised and capitalist nature of the domain name system is a huge design problem for those of us working to build bridges between the centralised Web and the Peer Webs as the current system makes it very difficult to match the onboarding experience of centralised platforms. (You can sign up to a centralised social network in 30 seconds… can you register your own domain name, have your server and app set up, and be up and running with your decentralised one in that time? You could. But not with our current system.) Web+ must also move beyond the greed-led shortsightedness of ICAAN to embrace a world of zero time-to-live domain propagations and a domain name commons.) ↩︎
Yes, good old RSS. ↩︎
My original description said that the address was “unique to the content” which was ambiguous as it could easily be interpreted to mean to an immutable hash of the content.

An advantage of DAT is that while the hash of the public key is used to discover content on the swarm, the content itself is mutable and can be updated by the owner of the private key. Multi-writer DAT – which is being developed as we speak and is what I’m most excited about – extends this to allow multiple authorised writers. I’m grateful to Diana for pointing this out via Mastodon. ↩︎

Demonstrating Web + Dat

mail@ar.al (Aral Balkan) — Mon, 25 Jun 2018 21:29:27 +0100

This is a short demonstration to show live reload and the peer-to-peer features of my new blog using Web + DAT.

21:41:16

The above content was created as part of a screen recording for my talk at Nesta’s FutureFest at the start of July. As part of the talk I will be showing this blog which progressively enhances the Web with peer-to-peer functionality using DAT.

In the recording (which I’ve just finished and haven’t even exported from Screenflow yet), I show how changes to my posts are almost immediately reflected in Beaker Browser when live reload is on whereas, with a plain web browser like Safari, you have to refresh manually.

Also, while a growing number of people manually refreshing a site would create increasing load for the site, on the peer-to-peer web, the load is distributed. Unlike the old web, the more popular a site gets, the better it performs, without requiring additional resources from the site itself.

In the talk I will also touch upon the importance of content-addressing and the peer-to-peer nature of DAT in relation to censorship prevention and network resilience in general.

We Make the City: Next Generation Cities

mail@ar.al (Aral Balkan) — Thu, 21 Jun 2018 10:11:03 +0200

This post was live blogged. Entries are in reverse chronological order.

11:30:03

Next up: Dan Hill (Associate Director, Arup). He’s a visiting professor at UCL Bartlett “Institute of Innovation and Public Purpose” and “Mayor of London Design Advocate”.

(So this is going to be a follow-up of Theo’s presentation to give details of the City of London’s ~~Smart City~~ Panopticon efforts.)

I won’t be live blogging this, see the previous talk notes for my thoughts on the matter.

11:27:08

I just called him out in a question for following neoliberal/Silicon Valley talking points and asked him “what would you say to those who might say that your goal is to make London into an ever better panopticon.” Also asked him how he feels partnering with Google Deepmind and other surveillance capitalists and exposing the citizens of London to surveillance by these factory farmers for human beings.

The response was essentially that he doesn’t see a problem with surveillance capitalism.

11:03:14

Next up: Theo Blackwell, CTO London.

“We need to set the foundations for innovation.” (There’s that word again.)

Theo Blackwell outlining his plans for making London an even better panopticon.

Theo is basically parroting neoliberal/Silicon Valley talking points.

London has… “Hundreds and hundreds of systems that collect data on people.” (Oh, yay!)

Essentially, the City of London’s strategy from what I’m hearing is that they want London to become an even better panopticon.

(I’m done live blogging this session. It’s clear what it is.)

11:01:28

The end of the panel: with links to GNU.

Richard: “We’ve got to prohibit face recognition in the streets.”

10:58:10

Marleen: “We don’t want backdoors in our cities… what do we do?”

Richard: “use free software. Get rid of the Internet of Stings … they are designed to spy on people … Any product that works by talking to its manufacturer, treat it like an enemy … but the city can impose these things on people.”

Francesca: “this is part of GDPR…”

Richard: ”I’m afraid they won’t interpret that this way.”

10:54:01

Francesca: “public code, public money”

Richard: “non-free software is full of malicious functionality like backdoors. Microsoft has a universal backdoor on their operating systems.”

10:49:07

Richard, Marleen, and Francesca.

Richard: “With free software you have a free market for support.”

Richard suggests that Munich changed its mind about free software because Microsoft moved its headquarters there.

Richard: “It’s illegal to bribe an individual politician but it is not a crime to bribe an entire government.”

Richard: “The worship of the invisible hand is an absurd religion. You see all the suffering that religion has caused in the world. Neoliberalism … ‘trade treaties’ … they give foreign companies more rights than their own citizens.”

Richard: “There is a powerful force that says ‘business is important, freedom is not’. Most of the governments that talk about software stay away from any talk of ethics.”

10:44:05

Richard: “Whenever they say ‘smart’ in relation to computing, think ‘spy’”

Richard: “All I’ve done in working about Barcelona is to give some advice. I discuss the ethical questions. What is it legitimate to do? What is it wrong to do? You see, Barcelona’s City Hall is a government … evertthing they do is supposed to be for the people or it is not justified at all and that includes all computing they do … it must never allow that computing to fall into private hands.”

Richard: “If you let some company do your computing activity, that company controls your computing activity.”

(Richard’s asked if Marleen’s opinion that switching fully to free software in Barcelona will take 25 years… “is that too long?”)

Richard: “The point is not to save one year. The point is: don’t make it take a century.”

Francesca: “We are enshrining Richard’s values in municipal law.”

10:37:58

Richard: “Either the users control the program or the program controls the users. If users control the program, that’s called free software.”

(He’s now taking the audience through the four freedoms.)

“Non-free software should not exist. 100% of the money Barcelona spends should go to free software. People should not work on making non-free software. Making non-free software works to subjugate people.”

10:32:16

Richard Stallman takes the stage to talk about freedom in technology.

Richard’s on stage now.

Francesca: “we’re working with Richard in Barcelona … not many people know that without the work Richard has done we couldn’t be doing what we’re doing to rebuild the city and give control back to people. We are starting to use free software in public administration.”

Richard: “forget the term sharing economy … it’s mistreatment of workers – I call it the sweatshop economy. And don’t ever use Uber!”

Richard: “‘free’ in English is ambiguous. It means free as in cost and free as in freedom. When I use the word, I only mean ‘freedom’”

10:27:27

Marleen and Francesca talking about the work of the City of Barcelona in creating a commons-based city.

Following Marleen’s session, we are transitioning to a panel with Richard Stallman and Francesca Bria (CTO Barcelona).

Said hi to Richard before the event and he was very kind: “I read about what you’re doing in print.” (Richard’s not on stage yet.)

Francesca: “giving back sovereignty to citizens is a fundamental concern… we must move away from surveillance capitalism.” (I agree: individual sovereignty is a prerequisite for a healthy commons.)

10:15:41

Marleen Stikker from Waag presents her opening keynote on the public stack.

Opening keynote: Marleen Stikker from Waag. (I’m here as Marleen’s guest and she’s awesome.)

“Who owns the city?”

“Do we want to be smart or do we want shared cities?”

Marleen presenting the public stack we put together on the boat two days ago.

(I really want us to start talking about incentivising smart citizens not smart cities. The question to ask: “who’s getting smarter about whom?” If it’s corporations, cities, governments getting smarter about people, we are building a panopticon. If it’s individuals getting smarter about individuals then we have a system that is compatible with human rights, democracy, and a healthy commons.)

“It’s not just open… we need freedom in our technology.” (Stallman will be on stage soon.)

“In the last 25 years we lost the Internet to the market and it might take us another 25 years to reclaim it for the commons.” (Do we have 25 years? I feel we must do this much more quickly.)

10:11:24

The deputy mayor of Amsterdam kicking off the day with her introduction.

I’m at the opening of the We Make The City event (Next Generation Cities). Right now, the deputy mayor is talking about where they see the “digital city” going forward. Introduces the “Amsterdam Innovation Tour” app.

A good time to remind the kind reader that “innovation” is a Silicon Valley/neoliberal mantra.

We Make the City: Opening Night

mail@ar.al (Aral Balkan) — Thu, 21 Jun 2018 08:15:57 +0200

Last night, I was mostly positively surprised by the opening night of the We Make The City festival in Amsterdam. I was expecting a wholly neoliberal affair and it didn’t turn out that way.

Finally got to see Kate Raworth, she of doughnut fame, and it did not disappoint.

After speaking at at least two conferences together, I finally got to see Kate Raworth present on Doughnut Economics (for the city). It was great to see her directly address the need to move from centralised to distributed/decentralised systems.

We must from centralised to distributed/decentralised systems.

We must move from an extractive to a generative mindset.

In a refreshing all-female line-up, the evening included mostly progressive discourse.

It was a refreshingly progressive affair for the most part.

Inexplicably, in the middle of this, we were presented by the CTO of Dubai in what I can only describe as a major whitewashing PR win for Dubai. Needless to say, there is nothing progressive about Dubai or the UAE. On stage, when asked about concerns about Big Brother-like surveillance, the CTO of Dubai responded with “We don’t call it Big Brother, we call it Fatherhood” to the sound of gasps from the audience.

Today, I am going to be taking part in the Next Generation Cities event as well as an evening event before returning to Ireland tomorrow.

Digital Masquerade

mail@ar.al (Aral Balkan) — Wed, 20 Jun 2018 17:10:39 +0200

“Today, we are all cyborgs. This is not to say that we implant ourselves with technology but that we extend our biological capabilities using technology. We are sharded beings; with parts of our selves spread across and augmented by our everyday things.” – The Nature of the Self in the Digital Age

Louisa preparing the camera for our shoot.

Louisa Treichmann is preparing an exhibition titled Digital Masquerade as part of the Manifesting Futures series. The event will take place next week in the Theatrum Anatomicum at De Waag. Today, I had the pleasure of chatting at length with Louisa on the topic of identity and to record a direct-to-camera piece on the cyborg nature of the self that will be projected on the wall during the event.

Visit the Digital Masquerade web site to learn more about the event and to sign up.

Working From De Waag

mail@ar.al (Aral Balkan) — Wed, 20 Jun 2018 16:42:26 +0200

De Waag: a historic building in the heart of Amsterdam

This evening, Laura and I are attending the opening of We Make The City as guests of Waag. And today, we were working out of Waag’s beautiful home, De Waag, in central Amsterdam.

Our office for the day which we shared with Sophie Bloemen from the Commons Network.

Laura caught using the stairs at De Waag.

Laura appreciating the Rembrandt painting “The Anatomy Lesson of Dr. Nicolaes Tulp” which was set in this very room.

Override BaseURL in Hugo Server

mail@ar.al (Aral Balkan) — Wed, 20 Jun 2018 15:33:28 +0200

Hugo has a baseURL setting in its configuration file (config.toml) that is used when creating absolute URLs. Sadly, this setting is ignored if you’re running Hugo with:

hugo server

I use relative URLs in all of my posts and templates but I got bitten by this with the generated RSS feed. All the URLs were being written with localhost:1313 prefixes.

To work around this issue, start Hugo using the --baseURL and --appendPort flags to override the default settings. e.g. I start a live session on this blog with:

hugo server -D --renderToDisk --baseURL=https://ar.al --appendPort=false

Little Saigon

mail@ar.al (Aral Balkan) — Wed, 20 Jun 2018 13:20:39 +0200

Having lunch at Little Saigon with Laura.

Little Saigon: view from the outside

Laura, salad rolls, and coconut juice.

The menu

Close-up of Laura’s salad roll

Location

View Larger Map

Public Stack Summit

mail@ar.al (Aral Balkan) — Tue, 19 Jun 2018 10:38:02 +0200

This post was live blogged. Entries are in reverse chronological order.

14:22:43

Katja, Laura, Todd and myself had a lively conversation about what constitutes a public stack and presented it. Core principles: individual ownership/control and convenience. A main challenge is protecting personhood in the digital age. Personhood is a prerequisite to the social contract. Individual sovereignty is a necessary (but not sufficient) prerequisite for a healthy commons.

Other teams raised points including: keeping values at the core of the stack (see Ethical Design Manifesto), ethics in sourcing and manufacture (see Fairphone).

11:28:29

I’m going to take a little break now so I can participate more directly. Will update if and when I have something interesting to share.

11:25:23

This is how I’m live blogging with Hugo, rsync, and DAT, by the way.

11:17:57

Katja from Nesta introducing herself. Laura and I will be at the FutureFest conference they’re organising in London soon.

11:01:28

Introductions: Purism, Ind.ie, Yoti, Commons Network and others.

10:43:18

We are on a boat so my connectivity may be limited.

10:42:36

We’re being asked to draw each other an ice-breaker.

10:38:59

Public Stack Summit kicking off. The boat is leaving.

Giddy about CSS grid

mail@ar.al (Aral Balkan) — Sun, 17 Jun 2018 10:26:17 +0100

The list of posts in the index of this blog are grouped by year, month, and day. Within the days, multiple posts are grouped by hour.

The initial index page during the one-hour hack on Friday that resulted in this site simply listed the post titles (do the Simplest Thing That Could Possibly Work (STTCPW)).

Version 1: two days ago.

Yesterday, I refined the site and implemented the chronological grouping. However, again going with STTCPW, I used a monospace script font to fake three-column layout of the list items:

Yesterday’s site: fake it till you make it!

As you can see, the date and day name are being repeated for every entry in a day. Less than ideal. Removing the redundancy in the red date stamp was easy. I simply tweaked the template code to improve the grouping so it was only displayed once and then floated it left. Hiding the redundant day names was a bit more convoluted: I added class names using a counter to the day name spans and used the following CSS rule to hide all but the first:

.day:not(.item-0) { color: white; }

The remaining problem was that since I was faking columns, if the title of a post was too wide (or if you were viewing the site on a narrow viewport), the title would wrap to the start of the list instead of staying within its “column”.

CSS grids to the rescue!

Laura waxes lyrical about CSS grids every chance she gets so I thought I’d give them a shot. How difficult could they be? (A question one learns early on never to ask when the topic is CSS.)

Needless to say, judging my the title of this post, I was pleasantly surprised.

A quick glance at the Defining a Grid example on Rachel Andrew’s excellent Grid by Example site gave me all I needed to go on.

I quickly spiked it with a simple three column, two row grid using an unordered list, saw that it worked as intended, and implemented it on the actual site. Here’s the source of the spike:

<style>
ul { list-style-type: none; }
li {
  display: grid;
  grid-template-columns: 100px 100px 1fr;
  grid-gap: 10px;
  margin-top: 10px;
}
.first { background-color: lightblue; }
.second { background-color: lightcoral; }
.third { background-color: khaki; }
</style>

<ul>
  <li>
    <span class='first'>Hello</span>
    <span class='second'>there</span>
    <span class='third'>how are you doing?</span>
  </li>
  <li>
    <span class='first'>And</span>
    <span class='second'>another</span>
    <span class='third'>row.</span>
  </li>
</ul>

Pop that into an index.html file, run http-server (http-server -c-1) and you’ll see:

A quick and dirty grid

(Notice the HTML isn’t even valid. That’s OK. It’s a spike. Think of it as a back-of-the-napkin sketch. It’s meant to be pragmatic.)

So, with that, two days after its birth, the site is beginning to shape up with an index that has a proper four-column layout with a neat and logical chronological grouping.

Formatting an ISO 8601 date stamp in Hugo

mail@ar.al (Aral Balkan) — Sun, 17 Jun 2018 09:14:55 +0100

Hugo has a plethora of functions that you can use in your templates, including one for formatting dates called dateFormat.

The index of this blog has a list of the posts in chronological order and uses a <time> tag. I wanted to include the machine-readable datetime attribute but the standard serialisation of Hugo’s .Date property doesn’t return an ISO 8601 timestamp. This led me to look up the esoteric syntax of Hugo’s dateFormat function and use the following format string to create an ISO 8601 string:

{{ $post.Date.Format "2006-01-02T15:04:05Z0700" }}

And here it is in use in the block that renders the posts for a given day (full source):

<ul> 
  {{ range $index, $post := .Pages }}
    <li>
      <time datetime="{{ $post.Date.Format "2006-01-02T15:04:05Z0700" }}">
        <span class='postdate day item-{{ $index }}'>{{ $post.Date.Format "Mon" }}</span>
      </time>
      <span class='postdate'>{{ $post.Date.Format "15:04" }}</span>
      <a href="{{ .RelPermalink }}">{{ $post.Title }}</a>
    </li>
  {{ end }}
</ul>

Refining the blog

mail@ar.al (Aral Balkan) — Sat, 16 Jun 2018 17:04:54 +0100

Yes, I blog in VSCode. What was the question again?

Yesterday, in a one-hour session that I live-tooted, I set up and deployed this blog by hacking together plain HTML/CSS, Rsync Watch, and DAT.

Maintaining a blog entirely by hand isn’t fun though.

Automate all some of the things

There are two things you must automate to make a usable web site authoring system. These are:

Indices: Index pages listing posts, articles, etc.
Partials: content like headers, footer, etc. that are included on multiple pages.

Then, there are nice-to-haves (especially for blogging) that are hard to live without in this day and age:

Markdown authoring: Markdown lets you keep your focus on the content rather than having it bogged down with formatting.
Live reload: Live reload shows you your local changes immediately. It’s like having a WYSIWYG preview.¹

In the previous version of my blog, I had hand-rolled a static site generator in Node.js to do these things for me. Today, there’s no reason to do that. We’re spoiled for choice when it comes to static site generators and one of the best around is called Hugo.

Hugo checks all the boxes above, does a million other things that I will probably never use, and is fast.

So I spent a few hours today setting up the blog to use Hugo. As I want the site to be as minimal as possible, I didn’t use an existing template. I’m hand-rolling the layouts and styles by hand.

Right now, I have indices being generated, and partials for the head, header, and footer.

I also set up a convention where content is stored chronologically (e.g., 2018/06/16/refining-the-blog) and I wrote a small script to make it easy for me to create new posts fitting that path structure:

So my workflow right now for updating this blog is:

Start up the system (Hugo server and rsync watcher):
```
./connect
```
Create a post:
```
./new my-latest-amazing-post
```
Edit the post using VSCode.

Every time I save, my changes are automatically synchronised with my server using rsync and, on the server, automatically shared on the peer-to-peer web via DAT.

I’ve been wanting to get back into blogging more frequently and I hope this new system will enable me to do just that.

Of course, on this blog, with DAT, the site itself becomes a streaming site and updates automatically with every change for people viewing it in Beaker Browser with Live Reload turned on. ↩︎

Revision history

mail@ar.al (Aral Balkan) — Fri, 15 Jun 2018 15:44:00 +0100

Beaker-ception Part II: a screenshot of this site in Beaker Browser with the revisions pane showing.

Another cool thing about DAT archives is that they’re based on append-only logs and thus maintain a full revision history. When you’re viewing this site via Beaker Browser, you can go back and see how the site evolved over time.

Hello Peer-to-Peer Web

mail@ar.al (Aral Balkan) — Fri, 15 Jun 2018 15:00:00 +0100

Beaker-ception: a screenshot of version of this page in Beaker browser with live reload on.

While documenting my deep dive into multiwriter hyperdb, I realised I was doing so on our forums, running Discourse, when I’d just demonstrated the day before how you can live blog on the peer-to-peer Web.

So I went on a little tangent and set up this “live” blog. Live as in, every time I save what I’m doing, the changes get pushed out to anyone viewing the site with live reload turned on in Beaker Browser. If you’re viewing the site on a regular web browser, you’ll just see a regular site and you won’t get the streaming updates. It’s an example of how we can progressively enhance the centralised web with the peer-to-peer web using the DAT protocol.

Biography

mail@ar.al (Aral Balkan) — Mon, 01 Jan 0001 00:00:00 +0000

Aral’s been making things with computers since he was 7 years old.

Along the way, he dabbled in Flash, did some consulting, taught some workshops, gave a few talks, organised a smattering of conferences, made some apps, helped launch an after-school coding initiative for kids, got rather enraged by Silicon Valley’s bullshit, and co-founded Small Technology Foundation to do something about it.

He’s currently working on developing the Small Web with Kitten and Domain.

Headshot

Please credit Cristina von Poser when you use my headshot.

Talks

mail@ar.al (Aral Balkan) — Mon, 01 Jan 0001 00:00:00 +0000

I’ve given hundreds of talks over the last two decades and beyond. Here are a few of my favourites in chronological order so you can see how my focus and thoughts have evolved through the years.

I hope you enjoy them.

PS. If you want me to speak about the Small Web at your event, feel free to send a short message to mail@ar.al with the details.

PPS. You can see Laura and me live every third Thursday of the month on Small is Beautiful, which we stream from our own Owncast server.

Superheroes and Villains in Design

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download the video, and watch it with your favourite video player.

Superheroes & Villains in Design at Thinking Digital 2013 in Newcastle, UK.

I believe that good design is crucial for what we do. Why? Watch this talk to find out or, if you’re short on time, read this.

Free is a Lie

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download the video, and watch it with your favourite video player.

Free is a Lie at Thinking Digital 2014 in Newcastle, UK.

This was one of the first talks I gave when I started speaking out against Silicon Valley’s toxic business model (what I now call “people farming”). I also gave a version of this as an RSA Talk.

Decentralise Everything

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download the video, and watch it with your favourite video player.

Decentralise Everything at The Conference 2015 in Malmö, Sweden.

In this talk, I give the first public demonstration of Heartbeat, a native macOS social media app that never left alpha and was the spiritual predecessor of my work on the Small Web. You can also watch just the Heartbeat reveal.

Beyond the Clouds

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download the video, and watch it with your favourite video player.

Beyond the Clouds at See Conference 2016 in Wiesbaden, Germany.

Excuse Me, Your Unicorn Keeps Shitting In My Back Yard, Can He Please Not?

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download the video, and watch it with your favourite video player.

Excuse Me, Your Unicorn Keeps Shitting In My Back Yard, Can He Please Not?at Doku:Tech 2016 in Prishtina, Republic of Kosovo.

In this talk, alongside raging against the machine, I talk about our tracker blocker, Better, which we retired at the end of 2021, and some of the insights we gained using it.

Don’t you just love the title, though? I know I do.

Design or Decoration?

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download the video, and watch it with your favourite video player.

Design or Decoration at IxDA Berlin 2017 in Germany.

In which I ask a room full of designers whether they are practicing design or decoration. A question we should all be asking ourselves. The answer to which might prove important in deciding whether we live in a democracy or an autocracy in the future.

Kitten (Small is Beautiful)

Sorry, your browser doesn't support embedded videos. But that doesn’t mean you can’t watch it! You can download the video, and watch it with your favourite video player.

Kitten demo at Small is Beautiful, December 2022 (online; live stream).

A demonstration of authentication and public-key cryptography in Kitten from our Small is Beautiful live stream with Laura.