Ceci n'est pas une -EPIPE

Using PKCS#11 on GNU/Linux

2016-08-07T20:17:00-04:00

PKCS#11 is a standard API to interface with HSMs, Smart Cards, or other types of random hardware backed crypto. On my travel laptop, I use a few Yubikeys in PKCS#11 mode using OpenSC to handle system login. libpam-pkcs11 is a pretty easy to use module that will let you log into your system locally using a PKCS#11 token locally.

One of the least documented things, though, was how to use an OpenSC PKCS#11 token in Chrome. First, close all web browsers you have open.

sudo apt-get install libnss3-tools

certutil -U -d sql:$HOME/.pki/nssdb
modutil -add "OpenSC" -libfile /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -dbdir sql:$HOME/.pki/nssdb
modutil -list "OpenSC" -dbdir sql:$HOME/.pki/nssdb 
modutil -enable "OpenSC" -dbdir sql:$HOME/.pki/nssdb

Now, we'll have the PKCS#11 module ready for nss to use, so let's double check that the tokens are registered:

certutil -U -d sql:$HOME/.pki/nssdb
certutil -L -h "OpenSC" -d sql:$HOME/.pki/nssdb

If this winds up causing issues, you can remove it using the following command:

modutil -delete "OpenSC" -dbdir sql:$HOME/.pki/nssdb

Hacking a Projector in Hy

2016-07-31T12:02:00-04:00

About a year ago, I bought a Projector after I finally admitted that I could actually use a TV in my apartment. I settled on buying a ViewSonic PJD5132. It was a really great value, and has been nothing short of a delight to own.

I was always a bit curious about the DB9 connector on the back of the unit, so I dug into the user manual, and found some hex code strings in there. So, last year, between my last gig at the Sunlight Foundtion and USDS, I spent some time wandering around the US, hitting up DebConf, and exploring Washington DC. Between trips, I set out to figure out exactly what was going on with my Projector, and see if I could make it do anything fun.

So, I started off with basics, and tried to work out how these command codes were structured. I had a few working codes, but to write clean code, I'd be better off understanding why the codes looked like they do. Let's look at the "Power On" code.

0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D

Some were 10 bytes, other were 11, and most started with similar looking things. The first byte was usually a 0x06 or 0x07, followed by two bytes 0x14 0x00, and either a 0x04 or 0x05. Since the first few bytes were similarly structured, I assumed the first octet (either 0x06 or 0x07) was actually a length, since the first 4 octets seemed always present.

So, my best guess is that we have a Length byte at index 0, followed by two bytes for the Protocol, a flag for if you're Reading or Writing (best guess on that one), and opaque data following that. Sometimes it's a const of sorts, and sometimes an octet (either little or big endian, confusingly).

Length
 |         Read / Write
 |              |
 |   Protocol   |            Data
 |    |----|    |    |------------------------|
0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D

Right. OK. So, let's get to work. In the spirit of code is data, data is code, I hacked up some of the projector codes into a s-expression we can use later. The structure of this is boring, but it'll let us both store the command code to issue, as well as define the handler to read the data back.

(setv *commands*
  ;  function                       type family         control
  '((power-on                         nil nil            (0x06  0x14 0x00  0x04  0x00 0x34 0x11 0x00 0x00 0x5D))
    (power-off                        nil nil            (0x06  0x14 0x00  0x04  0x00 0x34 0x11 0x01 0x00 0x5E))
    (power-status                   const power          (0x07  0x14 0x00  0x05  0x00 0x34 0x00 0x00 0x11 0x00 0x5E))
    (reset                            nil nil            (0x06  0x14 0x00  0x04  0x00 0x34 0x11 0x02 0x00 0x5F))
    ...

As well as defining some of the const responses that come back from the Projector itself. These are pretty boring, but it's helpful to put a name to the int that falls out.

(setv *consts*
  '((power        ((on           (0x00 0x00 0x01))
                   (off          (0x00 0x00 0x00))))

    (freeze       ((on           (0x00 0x00 0x01))
                   (off          (0x00 0x00 0x00))))
    ...

After defining a few simple functions to write the byte arrays to the serial port as well as reading and understanding responses from the projector, I could start elaborating on some higher order functions that can talk projector. So the first thing I wrote was to make a function that converts the command entry into a native Hy function.

(defn make-api-function [function type family data]
  `(defn ~function [serial]
      (import [PJD5132.dsl [interpret-response]]
              [PJD5132.serial [read-response/raw]])
      (serial.write (bytearray [~@data]))
      (interpret-response ~(str type) ~(str family) (read-response/raw serial))))

Fun. Fun! Now, we can invoke it to create a Hy & Python importable API wrapper in a few lines!

(import [PJD5132.commands [*commands*]]
        [PJD5132.dsl [make-api-function]])

(list (map (fn [(, function type family command)]
               (make-api-function function type family command)) *commands*)))

Cool. So, now we can import things like power-on from *commands* which takes a single argument (serial) for the serial port, and it'll send a command, and return the response. The best part about all this is you only have to define the data once in a list, and the rest comes for free.

Finally, I do want to be able to turn my projector on and off over the network so I went ahead and make a Flask "API" on top of this. First, let's define a macro to define Flask routes:

(defmacro defroute [name root &rest methods]
  (import os.path)

  (defn generate-method [path method status]

    `(with-decorator (app.route ~path) (fn []
       (import [PJD5132.api [~method ~(if status status method)]])

       (try (do (setv ret (~method serial-line))
               ~(if status `(setv ret (~status serial-line)))
                (json.dumps ret))
       (except [e ValueError]
          (setv response (make-response (.format "Fatal Error: ValueError: {}" (str e))))
          (setv response.status-code 500)
          response)))))

  (setv path (.format "/projector/{}" name))
  (setv actions (dict methods))
  `(do ~(generate-method path root nil)
       ~@(list-comp (generate-method (os.path.join path method-path) method root)
                    [(, method-path method) methods])))

Now, we can define how we want our API to look, so let's define the power route, which will expand out into the Flask route code above.

(defroute power
  power-status
  ("on"  power-on)
  ("off" power-off))

And now, let's play with it!

$ curl http://192.168.1.50/projector/power
"off"
$ curl http://192.168.1.50/projector/power/on
"on"
$ curl http://192.168.1.50/projector/power
"on"

Or, the volume!

$ curl 192.168.1.50/projector/volume
10
$ curl 192.168.1.50/projector/volume/decrease
9
$ curl 192.168.1.50/projector/volume/decrease
8
$ curl 192.168.1.50/projector/volume/decrease
7
$ curl 192.168.1.50/projector/volume/increase
8
$ curl 192.168.1.50/projector/volume/increase
9
$ curl 192.168.1.50/projector/volume/increase
10

Check out the full source over at github.com/paultag/PJD5132!

The Open Source License API

2016-07-16T15:30:00-04:00

Around a year ago, I started hacking together a machine readable version of the OSI approved licenses list, and casually picking parts up until it was ready to launch. A few weeks ago, we officially announced the osi license api, which is now live at api.opensource.org.

I also took a whack at writing a few API bindings, in Python, Ruby, and using the models from the API implementation itself in Go. In the following few weeks, Clint wrote one in Haskell, Eriol wrote one in Rust, and Oliver wrote one in R.

The data is sourced from a repo on GitHub, the licenses repo under OpenSourceOrg. Pull Requests against that repo are wildly encouraged! Additional data ideas, cleanup or more hand collected data would be wonderful!

In the meantime, use-cases for using this API range from language package managers pulling OSI approval of a licence programmatically to using a license identifier as defined in one dataset (SPDX, for example), and using that to find the identifier as it exists in another system (DEP5, Wikipedia, TL;DR Legal).

Patches are hugely welcome, as are bug reports or ideas! I'd also love more API wrappers for other languages!

Hello, InfluxDB

2016-07-02T13:13:00-04:00

Last week, I posted about python-sense, and API wrapper for the internal Sense API. I wrote this so that I could pull data about myself into my own databases, allowing me to use that information for myself.

One way I'm doing this is by pulling my room data into an InfluxDB database, letting me run time series queries against my environmental data.

#!/usr/bin/env python

from influxdb import InfluxDBClient

import json
import datetime as dt
from sense.service import Sense

api = Sense()

data = api.room_sensors(quantity=20)

def items(data):
    for flavor, series in data.items():
        for datum in reversed(series):
            value = datum['value']
            if value == -1:
                continue

            timezone = dt.timezone(dt.timedelta(
                seconds=datum['offset_millis'] / 1000,
            ))

            when = dt.datetime.fromtimestamp(
                datum['datetime'] / 1000,
            ).replace(tzinfo=timezone)

            yield flavor, when, value


client = InfluxDBClient(
    'url.to.host.here',
    443,
    'username',
    'password',
    'sense',
    ssl=True,
)


def series(data):
    for flavor, when, value in items(data):
        yield {
            "measurement": "{}".format(flavor),
            "tags": {
                "user": "paultag"
            },
            "time": when.isoformat(),
            "fields": {
                "value": value,
            }
        }


client.write_points(list(series(data)))

I'm able to run this on a cron, automatically loading data from the Sense API into my Influx database. I can then use that with something like Grafana, to check out what my room looks like over time.

Hello, Sense!

2016-06-26T21:42:00-04:00

A while back, I saw a Kickstarter for one of the most well designed and pretty sleep trackers on the market. I fell in love with it, and it has stuck with me since.

A few months ago, I finally got my hands on one and started to track my data. Naturally, I now want to store this new data with the rest of the data I have on myself in my own databases.

I went in search of an API, but I found that the Sense API hasn't been published yet, and is being worked on by the team. Here's hoping it'll land soon!

After some subdomain guessing, I hit on api.hello.is. So, naturally, I went to take a quick look at their Android app and network traffic, lo and behold, there was a pretty nicely designed API.

This API is clearly an internal API, and as such, it's something that should not be considered stable. However, I'm OK with a fragile API, so I've published a quick and dirty API wrapper for the Sense API to my GitHub..

I've published it because I've found it useful, but I can't promise the world, (since I'm not a member of the Sense team at Hello!), so here are a few ground rules of this wrapper:

I make no claims to the stability or completeness.
I have no documentation or assurances.
I will not provide the client secret and ID. You'll have to find them on your own.
This may stop working without any notice, and there may even be really nasty bugs that result in your alarm going off at 4 AM.
Send PRs! This is a side-project for me.

This module is currently Python 3 only. If someone really needs Python 2 support, I'm open to minimally invasive patches to the codebase using six to support Python 2.7.

Working with the API:

First, let's go ahead and log in using python -m sense.

$ python -m sense
Sense OAuth Client ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense OAuth Client Secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense email: paultag@gmail.com
Sense password: 
Attempting to log into Sense's API
Success!
Attempting to query the Sense API
The humidity is **just right**.
The air quality is **just right**.
The light level is **just right**.
It's **pretty hot** in here.
The noise level is **just right**.
Success!

Now, let's see if we can pull up information on my Sense:

>>> from sense import Sense
>>> sense = Sense()
>>> sense.devices()
{'senses': [{'id': 'xxxxxxxxxxxxxxxx', 'firmware_version': '11a1', 'last_updated': 1466991060000, 'state': 'NORMAL', 'wifi_info': {'rssi': 0, 'ssid': 'Pretty Fly for a WiFi (2.4 GhZ)', 'condition': 'GOOD', 'last_updated': 1462927722000}, 'color': 'BLACK'}], 'pills': [{'id': 'xxxxxxxxxxxxxxxx', 'firmware_version': '2', 'last_updated': 1466990339000, 'battery_level': 87, 'color': 'BLUE', 'state': 'NORMAL'}]}

Neat! Pretty cool. Look, you can even see my WiFi AP! Let's try some more and pull some trends out.

>>> values = [x.get("value") for x in sense.room_sensors()["humidity"]][:10]
>>> min(values)
45.73904
>>> max(values)
45.985928
>>>

I plan to keep maintaining it as long as it's needed, so I welcome co-maintainers, and I'd love to see what people build with it! So far, I'm using it to dump my room data into InfluxDB, pulling information on my room into Grafana. Hopefully more to come!

Happy hacking!

Go Debian!

2016-06-19T12:30:00-04:00

As some of the world knows full well by now, I've been noodling with Go for a few years, working through its pros, its cons, and thinking a lot about how humans use code to express thoughts and ideas. Go's got a lot of neat use cases, suited to particular problems, and used in the right place, you can see some clear massive wins.

I've started writing Debian tooling in Go, because it's a pretty natural fit. Go's fairly tight, and overhead shouldn't be taken up by your operating system. After a while, I wound up hitting the usual blockers, and started to build up abstractions. They became pretty darn useful, so, this blog post is announcing (a still incomplete, year old and perhaps API changing) Debian package for Go. The Go importable name is pault.ag/go/debian. This contains a lot of utilities for dealing with Debian packages, and will become an edited down "toolbelt" for working with or on Debian packages.

Module Overview

Currently, the package contains 4 major sub packages. They're a changelog parser, a control file parser, deb file format parser, dependency parser and a version parser. Together, these are a set of powerful building blocks which can be used together to create higher order systems with reliable understandings of the world.

changelog

The first (and perhaps most incomplete and least tested) is a changelog file parser.. This provides the programmer with the ability to pull out the suite being targeted in the changelog, when each upload was, and the version for each. For example, let's look at how we can pull when all the uploads of Docker to sid took place:

func main() {
    resp, err := http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")
    if err != nil {
        panic(err)
    }
    allEntries, err := changelog.Parse(resp.Body)
    if err != nil {
        panic(err)
    }
    for _, entry := range allEntries {
        fmt.Printf("Version %s was uploaded on %s\n", entry.Version, entry.When)
    }
}

The output of which looks like:

Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST

control

Next is one of the most complex, and one of the oldest parts of go-debian, which is the control file parser (otherwise sometimes known as deb822). This module was inspired by the way that the json module works in Go, allowing for files to be defined in code with a struct. This tends to be a bit more declarative, but also winds up putting logic into struct tags, which can be a nasty anti-pattern if used too much.

The first primitive in this module is the concept of a Paragraph, a struct containing two values, the order of keys seen, and a map of string to string. All higher order functions dealing with control files will go through this type, which is a helpful interchange format to be aware of. All parsing of meaning from the Control file happens when the Paragraph is unpacked into a struct using reflection.

The idea behind this strategy that you define your struct, and let the Control parser handle unpacking the data from the IO into your container, letting you maintain type safety, since you never have to read and cast, the conversion will handle this, and return an Unmarshaling error in the event of failure.

Additionally, Structs that define an anonymous member of control.Paragraph will have the raw Paragraph struct of the underlying file, allowing the programmer to handle dynamic tags (such as X-Foo), or at least, letting them survive the round-trip through go.

The default decoder contains an argument, the ability to verify the input control file using an OpenPGP keyring, which is exposed to the programmer through the (*Decoder).Signer() function. If the passed argument is nil, it will not check the input file signature (at all!), and if it has been passed, any signed data must be found or an error will fall out of the NewDecoder call. On the way out, the opposite happens, where the struct is introspected, turned into a control.Paragraph, and then written out to the io.Writer.

Here's a quick (and VERY dirty) example showing the basics of reading and writing Debian Control files with go-debian.

package main

import (
    "fmt"
    "io"
    "net/http"
    "strings"

    "pault.ag/go/debian/control"
)

type AllowedPackage struct {
    Package     string
    Fingerprint string
}

func (a *AllowedPackage) UnmarshalControl(in string) error {
    in = strings.TrimSpace(in)
    chunks := strings.SplitN(in, " ", 2)
    if len(chunks) != 2 {
        return fmt.Errorf("Syntax sucks: '%s'", in)
    }
    a.Package = chunks[0]
    a.Fingerprint = chunks[1][1 : len(chunks[1])-1]

    return nil
}

type DMUA struct {
    Fingerprint     string
    Uid             string
    AllowedPackages []AllowedPackage `control:"Allow" delim:","`
}

func main() {
    resp, err := http.Get("http://metadata.ftp-master.debian.org/dm.txt")
    if err != nil {
        panic(err)
    }

    decoder, err := control.NewDecoder(resp.Body, nil)
    if err != nil {
        panic(err)
    }

    for {
        dmua := DMUA{}
        if err := decoder.Decode(&dmua); err != nil {
            if err == io.EOF {
                break
            }
            panic(err)
        }
        fmt.Printf("The DM %s is allowed to upload:\n", dmua.Uid)
        for _, allowedPackage := range dmua.AllowedPackages {
            fmt.Printf("   %s [granted by %s]\n", allowedPackage.Package, allowedPackage.Fingerprint)
        }
    }
}

Output (truncated!) looks a bit like:

...
The DM Allison Randal <allison@lohutok.net> is allowed to upload:
   parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
...
The DM Benjamin Barenblat <bbaren@mit.edu> is allowed to upload:
   boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
   dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
   transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
   urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
...
The DM أحمد المحمودي <aelmahmoudy@sabily.org> is allowed to upload:
   covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
   dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
   drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
   fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
   ...
...

deb

Next up, we've got the deb module. This contains code to handle reading Debian 2.0 .deb files. It contains a wrapper that will parse the control member, and provide the data member through the archive/tar interface.

Here's an example of how to read a .deb file, access some metadata, and iterate over the tar archive, and print the filenames of each of the entries.

func main() {
    path := "/tmp/fluxbox_1.3.5-2+b1_amd64.deb"
    fd, err := os.Open(path)
    if err != nil {
        panic(err)
    }
    defer fd.Close()

    debFile, err := deb.Load(fd, path)
    if err != nil {
        panic(err)
    }

    version := debFile.Control.Version
    fmt.Printf(
        "Epoch: %d, Version: %s, Revision: %s\n",
        version.Epoch, version.Version, version.Revision,
    )

    for {
        hdr, err := debFile.Data.Next()
        if err == io.EOF {
            break
        }
        if err != nil {
            panic(err)
        }
        fmt.Printf("  -> %s\n", hdr.Name)
    }
}

Boringly, the output looks like:

Epoch: 0, Version: 1.3.5, Revision: 2+b1
  -> ./
  -> ./etc/
  -> ./etc/menu-methods/
  -> ./etc/menu-methods/fluxbox
  -> ./etc/X11/
  -> ./etc/X11/fluxbox/
  -> ./etc/X11/fluxbox/window.menu
  -> ./etc/X11/fluxbox/fluxbox.menu-user
  -> ./etc/X11/fluxbox/keys
  -> ./etc/X11/fluxbox/init
  -> ./etc/X11/fluxbox/system.fluxbox-menu
  -> ./etc/X11/fluxbox/overlay
  -> ./etc/X11/fluxbox/apps
  -> ./usr/
  -> ./usr/share/
  -> ./usr/share/man/
  -> ./usr/share/man/man5/
  -> ./usr/share/man/man5/fluxbox-style.5.gz
  -> ./usr/share/man/man5/fluxbox-menu.5.gz
  -> ./usr/share/man/man5/fluxbox-apps.5.gz
  -> ./usr/share/man/man5/fluxbox-keys.5.gz
  -> ./usr/share/man/man1/
  -> ./usr/share/man/man1/startfluxbox.1.gz
...

dependency

The dependency package provides an interface to parse and compute dependencies. This package is a bit odd in that, well, there's no other library that does this. The issue is that there are actually two different parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev) and another in C (in dpkg).

To date, this has resulted in me filing three different bugs. I also found a broken package in the archive, which actually resulted in another bug being (totally accidentally) already fixed. I hope to continue to run the archive through my parser in hopes of finding more bugs! This package is a bit complex, but it basically just returns what amounts to be an AST for our Dependency lines. I'm positive there are bugs, so file them!

func main() {
    dep, err := dependency.Parse("foo | bar, baz, foobar [amd64] | bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")
    if err != nil {
        panic(err)
    }

    anySparc, err := dependency.ParseArch("sparc")
    if err != nil {
        panic(err)
    }

    for _, possi := range dep.GetPossibilities(*anySparc) {
        fmt.Printf("%s (%s)\n", possi.Name, possi.Arch)
    }
}

Gives the output:

foo (<nil>)
baz (<nil>)
fnord (armhf)

version

Right off the bat, I'd like to thank Michael Stapelberg for letting me graft this out of dcs and into the go-debian package. This was nearly entirely his work (with a one or two line function I added later), and was amazingly helpful to have. Thank you!

This module implements Debian version comparisons and parsing, allowing for sorting in lists, checking to see if it's native or not, and letting the programmer to implement smart(er!) logic based on upstream (or Debian) version numbers.

This module is extremely easy to use and very straightforward, and not worth writing an example for.

Final thoughts

This is more of a "Yeah, OK, this has been useful enough to me at this point that I'm going to support this" rather than a "It's stable!" or even "It's alive!" post. Hopefully folks can report bugs and help iterate on this module until we have some really clean building blocks to build solid higher level systems on top of. Being able to have multiple libraries interoperate by relying on go-debian will be a massive ease. I'm in need of more documentation, and to finalize some parts of the older sub package APIs, but I'm hoping to be at a "1.0" real soon now.

It's all relative

2016-06-10T23:45:00-04:00

As nearly anyone who's worked with me will attest to, I've long since touted nedbat's talk Pragmatic Unicode, or, How do I stop the pain? as one of the most foundational talks, and required watching for all programmers.

The reason is because netbat hits on something bigger - something more fundamental than how to handle Unicode -- it's how to handle data which is relative.

For those who want the TL;DR, the argument is as follows:

Facts of Life:

Computers work with Bytes. Bytes go in, Bytes go out.
The world needs more than 256 symbols.
You need both Bytes and Unicode
You cannot infer the encoding of bytes.
Declared encodings can be Wrong

Now, to fix it, the following protips:

Unicode sandwich
Know what you have
TEST

Relative Data

I've started to think more about why we do the things we do when we write code, and one thing that continues to be a source of morbid schadenfreude is watching code break by failing to handle Unicode right. It's hard! However, watching what breaks lets you gain a bit of insight into how the author thinks, and what assumptions they make.

When you send someone Unicode, there are a lot of assumptions that have to be made. Your computer has to trust what you (yes, you!) entered into your web browser, your web browser has to pass that on over the network (most of the time without encoding information), to a server which reads that bytestream, and makes a wild guess at what it should be. That server might save it to a database, and interpolate it into an HTML template in a different encoding (called Mojibake), resulting in a bad time for everyone involved.

Everything's awful, and the fact our computers can continue to display text to us is a goddamn miracle. Never forget that.

When it comes down to it, when I see a byte sitting on a page, I don't know (and can't know!) if it's Windows-1252, UTF-8, Latin-1, or EBCDIC. What's a poem to me is terminal garbage to you.

Over the years, hacks have evolved. We have magic numbers, and plain ole' hacks to just guess based on the content. Of course, like all good computer programs, this has lead to its fair share of hilarious bugs, and there's nothing stopping files from (validly!) being multiple things at the same time.

Like many things, it's all in the eye of the beholder.

Timezones

Just like Unicode, this is a word that can put your friendly neighborhood programmer into a series of profanity laden tirades. Go find one in the wild, and ask them about what they think about timezone handling bugs they've seen. I'll wait. Go ahead.

Rants are funny things. They're fun to watch. Hilarious to give. Sometimes just getting it all out can help. They can tell you a lot about the true nature of problems.

It's funny to consider the isomorphic nature of Unicode rants and Timezone rants.

I don't think this is an accident.

U̶n̶i̶c̶o̶d̶e̶ timezone Sandwich

Ned's Unicode Sandwich applies -- As early as we can, in the lowest level we can (reading from the database, filesystem, wherever!), all datetimes must be timezone qualified with their correct timezone. Always. If you mean UTC, say it's in UTC.

Treat any unqualified datetimes as "bytes". They're not to be trusted. Never, never, never trust 'em. Don't process any datetimes until you're sure they're in the right timezone.

This lets the delicious inside of your datetime sandwich handle timezones with grace, and finally, as late as you can, turn it back into bytes (if at all!). Treat locations as tzdb entries, and qualify datetime objects into their absolute timezone (EST, EDT, PST, PDT)

It's not until you want to show the datetime to the user again should you consider how to re-encode your datetime to bytes. You should think about what flavor of bytes, what encoding -- what timezone -- should I be encoding into?

TEST

Just like Unicode, testing that your code works with datetimes is important. Every time I think about how to go about doing this, I think about that one time that mjg59 couldn't book a flight starting Tuesday from AKL, landing in HNL on Monday night, because United couldn't book the last leg to SFO. Do you ever assume dates only go forward as time goes on? Remember timezones.

Construct test data, make sure someone in New Zealand's +13:45 can correctly talk with their friends in Baker Island's -12:00, and that the events sort right.

Just because it's Noon on New Years Eve in England doesn't mean it's not 1 AM the next year in New Zealand. Places a few miles apart may go on Daylight savings different days. Indian Standard Time is not even aligned on the hour to GMT (+05:30)!

Test early, and test often. Memorize a few timezones, and challenge your assumptions when writing code that has to do with time. Don't use wall clocks to mean monotonic time. Remember there's a whole world out there, and we only deal with part of it.

It's also worth remembering, as Andrew Pendleton pointed out to me, that it's possible that a datetime isn't even unique for a place, since you can never know if 2016-11-06 01:00:00 in America/New_York (in the tzdb) is the first one, or second one. Storing EST or EDT along with your datetime may help, though!

Pitfalls

Improper handling of timezones can lead to some interesting things, and failing to be explicit (or at least, very rigid) in what you expect will lead to an unholy class of bugs we've all come to hate. At best, you have confused users doing math, at worst, someone misses a critical event, or our security code fails.

I recently found what I regard to be a pretty bad bug in apt (which David has prepared a fix for and is pending upload, yay! Thank you!), which boiled down to documentation and code expecting datetimes in a timezone, but accepting any timezone, and silently treating it as UTC.

The solution is to hard-fail, which is an interesting choice to me (as a vocal fan of timezone aware code), but at the least it won't fail by misunderstanding what the server is trying to communicate, and I do understand and empathize with the situation the apt maintainers are in.

Final Thoughts

Overall, my main point is although most modern developers know how to deal with Unicode pain, I think there is a more general lesson to learn -- namely, you should always know what data you have, and always remember what it is. Understand assumptions as early as you can, and always store them with the data.

Docker PostgreSQL Foreign Data Wrapper

2014-09-18T21:49:00-04:00

For the tl;dr: Docker FDW is a thing. Star it, hack it, try it out. File bugs, be happy. If you want to see what it's like to read, there's some example SQL down below.

This post was edited on Sep 21st to add information about the DELETE and INSERT operators

The question is first, what the heck is a PostgreSQL Foreign Data Wrapper? PostgreSQL Foreign Data Wrappers are plugins that allow C libraries to provide an adaptor for PostgreSQL to talk to an external database.

Some folks have used this to wrap stuff like MongoDB, which I always found to be hilarous (and an epic hack).

Enter Multicorn

During my time at PyGotham, I saw a talk from Wes Chow about something called Multicorn. He was showing off some really neat plugins, such as the git revision history of CPython, and parsed logfiles from some stuff over at Chartbeat. This basically blew my mind.

All throughout the talk I was coming up with all sorts of things that I wanted to do -- this whole library is basically exactly what I've been dreaming about for years. I've always wanted to provide a SQL-like interface into querying API data, joining data cross-API using common crosswalks, such as using Capitol Words to query for Legislators, and use the bioguide ids to JOIN against the congress api to get their Twitter account names.

My first shot was to Multicorn the new Open Civic Data API I was working on, chuckled and put it aside as a really awesome hack.

Enter Docker

It wasn't until tianon connected the dots for me and suggested a Docker FDW did I get really excited. Cue a few hours of hacking, and I'm proud to say -- here's Docker FDW.

This lets us ask all sorts of really interesting questions out of the API, and might even help folks writing webapps avoid adding too much Docker-aware logic. Abstractions can be fun!

Setting it up

I'm going to assume you have a working Multicorn, PostgreSQL and Docker setup (including adding the postgres user to the docker group)

So, now let's pop open a psql session. Create a database (I called mine dockerfdw, but it can be anything), and let's create some tables.

Before we create the tables, we need to let PostgreSQL know where our objects are. This takes a name for the server, and the Python importable path to our FDW.

CREATE SERVER docker_containers FOREIGN DATA WRAPPER multicorn options (
    wrapper 'dockerfdw.wrappers.containers.ContainerFdw');

CREATE SERVER docker_image FOREIGN DATA WRAPPER multicorn options (
    wrapper 'dockerfdw.wrappers.images.ImageFdw');

Now that we have the server in place, we can tell PostgreSQL to create a table backed by the FDW by creating a foreign table. I won't go too much into the syntax here, but you might also note that we pass in some options - these are passed to the constructor of the FDW, letting us set stuff like the Docker host.

CREATE foreign table docker_containers (
    "id"          TEXT,
    "image"       TEXT,
    "name"        TEXT,
    "names"       TEXT[],
    "privileged"  BOOLEAN,
    "ip"          TEXT,
    "bridge"      TEXT,
    "running"     BOOLEAN,
    "pid"         INT,
    "exit_code"   INT,
    "command"     TEXT[]
) server docker_containers options (
    host 'unix:///run/docker.sock'
);


CREATE foreign table docker_images (
    "id"              TEXT,
    "architecture"    TEXT,
    "author"          TEXT,
    "comment"         TEXT,
    "parent"          TEXT,
    "tags"            TEXT[]
) server docker_image options (
    host 'unix:///run/docker.sock'
);

And, now that we have tables in place, we can try to learn something about the Docker containers. Let's start with something fun - a join from containers to images, showing all image tag names, the container names and the ip of the container (if it has one!).

SELECT docker_containers.ip, docker_containers.names, docker_images.tags
  FROM docker_containers
  RIGHT JOIN docker_images
  ON docker_containers.image=docker_images.id;

     ip      |            names            |                  tags                   
-------------+-----------------------------+-----------------------------------------
             |                             | {ruby:latest}
             |                             | {paultag/vcs-mirror:latest}
             | {/de-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
             | {/ny-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
             | {/ar-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
 172.17.0.47 | {/ms-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
 172.17.0.46 | {/nc-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
             | {/ia-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
             | {/az-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
             | {/oh-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
             | {/va-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
 172.17.0.41 | {/wa-openstates-to-ocd}     | {sunlightlabs/scrapers-us-state:latest}
             | {/jovial_poincare}          | {<none>:<none>}
             | {/jolly_goldstine}          | {<none>:<none>}
             | {/cranky_torvalds}          | {<none>:<none>}
             | {/backstabbing_wilson}      | {<none>:<none>}
             | {/desperate_hoover}         | {<none>:<none>}
             | {/backstabbing_ardinghelli} | {<none>:<none>}
             | {/cocky_feynman}            | {<none>:<none>}
             |                             | {paultag/postgres:latest}
             |                             | {debian:testing}
             |                             | {paultag/crank:latest}
             |                             | {<none>:<none>}
             |                             | {<none>:<none>}
             | {/stupefied_fermat}         | {hackerschool/doorbot:latest}
             | {/focused_euclid}           | {debian:unstable}
             | {/focused_babbage}          | {debian:unstable}
             | {/clever_torvalds}          | {debian:unstable}
             | {/stoic_tesla}              | {debian:unstable}
             | {/evil_torvalds}            | {debian:unstable}
             | {/foo}                      | {debian:unstable}
(31 rows)

OK, let's see if we can bring this to the next level now. I finally got around to implementing INSERT and DELETE operations, which turned out to be pretty simple to do. Check this out:

DELETE FROM docker_containers;

DELETE 1

This will do a stop + kill after a 10 second hang behind the scenes. It's actually a lot of fun to spawn up a container and terminate it from PostgreSQL.

INSERT INTO docker_containers (name, image) VALUES ('hello', 'debian:unstable') RETURNING id;

                                id                                
------------------------------------------------------------------
 0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05

INSERT 0 1
(1 row)

Spawning containers works too - this is still very immature and not super practical, but I figure while I'm showing off, I might as well go all the way.

SELECT ip FROM docker_containers WHERE id='0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05';

     ip      
-------------
 172.17.0.12
(1 row)

Success! This is just a taste of what's to come, so please feel free to hack on Docker FDW, tweet me @paultag, file bugs / feature requests. It's currently a bit of a hack, and it's something that I think has long-term potential after some work goes into making sure that this is a rock solid interface to the Docker API.

Linode pv-grub chaining

2014-06-14T21:40:00-04:00

I've been using Linode since 2010, and many of my friends have heard me talk about how big a fan I am of linode. I've used Debian unstable on all my Linodes, since I often use them as a remote shell for general purpose Debian development. I've found my linodes to be indispensable, and I really love Linode.

The Problem

Recently, because of my work on Docker, I was forced to stop using the Linode kernel in favor of the stock Debian kernel, since the stock Linode kernel has no aufs support, and the default LVM-based devicemapper backend can be quite a pain.

I tried loading in btrfs support, and using that to host the Docker instance backed with btrfs, but it was throwing errors as well. Stuck with unstable backends, I wanted to use the aufs backend, which, despite problems in aufs internally, is quite stable with Docker (and in general).

I started to run through the Linode Library's guide on PV-Grub, but that resulted in a cryptic error with xen not understanding the compression of the kernel. I checked for recent changes to the compression, and lo, the Debian kernel has been switched to use xz compression in sid. Awesome news, really. XZ compression is awesome, and I've been super impressed with how universally we've adopted it in Debian. Keep it up! However, it appears only a newer pv-grub than the Linode hosts have installed will fix this.

After contacting the (ever friendly) Linode support, they were unable to give me a timeline on adding xz support, which would entail upgrading pv-grub. It was quite disappointing news, to be honest. Workarounds were suggested, but I'm not quite happy with them as proper solutions.

After asking in #debian-kernel, waldi was able to give me a few pointers, and the following is very inspired by him, the only thing that changed much was config tweaking, which was easy enough. Thanks, Bastian!

The Constraints

I wanted to maintain a 100% stock configuration from the kernel up. When I upgraded my kernel, I wanted to just work. I didn't want to unpack and repack the kernel, and I didn't want to install software outside main on my system. It had to be 100% Debian and unmodified.

The Solution

Left unable to run my own kernel directly in the Linode interface, the tact here was to use Linode's old pv-grub to chain-load grub-xen, which loaded a modern kernel. Turns out this works great.

Let's start by creating a config for Linode's pv-grub to read and use.

sudo mkdir -p /boot/grub/

Now, since pv-grub is legacy grub, we can write out the following config to chain-load in grub-xen (which is just Grub 2.0, as far as I can tell) to /boot/grub/menu.lst. And to think, I almost forgot all about menu.lst. Almost.

default 1

timeout 3

title grub-xen shim
root (hd0)
kernel /boot/xen-shim
boot

Just like riding a bike! Now, let's install and set up grub-xen to work for us.

sudo apt-get install grub-xen
sudo update-grub

And, let's set the config for the GRUB image we'll create in the next step in the /boot/load.cf file:

configfile (xen/xvda)/boot/grub/grub.cfg

Now, lastly, let's generate the /boot/xen-shim file that we need to boot to:

grub-mkimage --prefix '(xen/xvda)/boot/grub' -c /boot/load.cf -O x86_64-xen /usr/lib/grub/x86_64-xen/*.mod > /boot/xen-shim

Next, change your boot configuration to use pv-grub, and give the machine a kick. Should work great! If you run into issues, use the lish shell to debug it, and let me know what else I should include in this post!

Hack on!

Hy at PyCon 2014

2014-04-18T20:13:00-04:00

I gave a talk this year at PyCon 2014, about one of my favorite subjects: Hy. Many of my regular readers will have no doubt explored Hy's thriving GitHub org, played with try-hy, or even installed it locally by pip installing it. I was lucky enough to be able to attend PyCon on behalf of Sunlight, with a solid contingent of my colleagues. We put together a writeup on the Sunlight blog if anyone was interested in our favorite talks.

Tons of really amazing questions, and such an amazingly warm reception from so many of my peers throughout this year's PyCon. Thank you so much to everyone that attended the talk. As always, you should Fork Hy on GitHub, follow @hylang on the twitters, and send in any bugs you find!

Hopefully I'll be able to put my talk up in blog-post form soon, but until then feel free to look over the slides or just watch the talk.

An extra shout-out to @akaptur for hacking on Hy during the sprints, and giving the exception system quite the workthrough. Thanks, Allison!

Musings about Debian and Python

2013-09-21T22:49:00-04:00

On a regular basis, I find myself the odd-man-out when it comes to talking about how to work with Python on Debian systems. I'm going to write this and post it so that I might be able to point people at my thoughts without having to write the same email in response to each thread that pops up.

Turns out I don't fit in with the Debian hardliners (which is to say, the mindset that pip sucks and shouldn't exist), nor do I fit in with the Python hardliners (which is to say apt and dpkg are out of date, and neither have a place on a Development machine).

I think our discourse on this topic has become petty and stupid in general. Let's all try to step back and drop a bit of the attitude.

`pip` doesn't suck, and neither does `apt`.

The truth is, both sides are wrong. As with any subject, the real answer here is much more nuanced than either side presents it. I'm going to try and present my opinion on this, in the way that both my Pythonista self and my Debianite self see the issue. Hopefully I can keep this short, to the point, and caked with logic.

The case for `dpkg` (the Debianite in me)

In defense of dpkg and apt, imagine having to install python-gnome2 on all your systems when you install. It'd be hell on earth. Imagine having a user try to do this. It's insane to assume that end-users will be using pip for this purpose.

pip is fun and all, but it's also installing 100% untrusted code to your system (perhaps as root, if you're using pip with sudo for some reason), and hasn't been reviewed for software freeness, which is something Debian (and Debian users) take seriously. This isn't even to mention the hell that pip wreaks on dpkg controlled files / packages.

Try to remember how much of your system running (yes, right now) is running because of Python or Python modules. Try to imagine how much of a pain in the ass it'd be if you couldn't boot into GNOME to use nm-applet to connect to wifi to pip install something. I'm sure even the most extreme pip'er understands the need for Operating System level package management.

Debian also has a bigger problem scope - we're not maintaining a library in Debian for kicks, we're maintaining it so that end user applications may use the library. When we update something like Django, we have to make sure that we don't break anything using it (although, to be honest, the fact that we package webapps is an entire rant for later) before we get to update it to the newest release.

Hell, with a few coffees, I could automate the process of releasing a .deb with a new upstream release, 100% unattended. I won't, however, since this is an insane idea. Let's go over a brief list of things I do before uploading a new package:

Review the entire codebase for simple mistakes.
Review the entire codebase for license issues.
Review the entire codebase for files without source, and track down (and include source for) any sourceless files (such as pickle files, etc).
Get to know the upstream, get to know open bugs, write something using the lib, in case I need to debug later.
Install the package.
Test the package.
Work out any Debian package issues (this is easy).

Now, a brief list of things I do before I update a package:

Review the changes between the last uploaded version (in diff format, if it's sane, otherwise get the VCS and review), ensure all the above are still OK.
Review for Debian-local issues (such as how it will upgrade, using piuparts, and adequate, etc).
Check to make sure it won't break any reverse dependencies.
Review for bugfixes that I might need to bring back to the stable release.
Figure out if I should (or even can) backport the package, if API is stable.
Review for bugs (upstream or in Debian) that I need to mention in the debian/changelog.

Clearly, this isn't a quick-and-dirty task. It's not a matter of getting a package updated (technically), it's a much more detailed process than that. This is also why Debian is so highly regarded for its technical virtuosity, and why the ISS decided to deploy Debian in space, despite other commercial distros such as Red Hat, or Ubuntu, and community distros, such as Fedora or Arch.

It's also not Debian's job to package the world in the archive. This is an insane task, and it's not Debian's place to do it. We introduce libraries as things need them, not because we wrote some new library that someone may find slightly useful at some point in the future. maybe.

Upstream developers and language communities (not only Python here) tend to lose sight of why we're doing this in the first place, which is our users. This isn't some sort of technical pissing contest to see who can distribute the software in the best way. Debian-folk always keep end users as our highest priority.

I quote the Debian Social Contract, when I say that Our priorities are our users and free software. No one's trying to get developers to use dpkg to create software. In fact, as you'll see below, I actively discourage using system modules for development.

The case for `pip` (the Pythonista in me)

In defense of pip, the idea that Debian will keep the latest versions of packages is insane. The idea that we can keep pace with upstream releases is nuts, and the idea that every upstream release on pypi is ready to ship is bananas. b-a-n-a-n-a-n-a-s. As a developer, I don't want to support every release, and I surely don't want other people depending on some random snapshot.

Often times, I'll put stuff up on pypi as a preview, or to release often, and solicit feedback without having to give out instructions on using a git checkout (it's also easier to have them try a version from pypi so I can cross-ref the git tag to reproduce issues when they file them)

Even Debian tools I write, like python-schroot are released to pypi first, and I treat that as the upstream location when packaging it in Debian.

pypi is easy, ubiquitous and works regardless of the platform, which means less of my development time is spent packaging stuff up for platforms I don't really care about (Arch, Fedora, OSX, Windows), even though I value feedback from users on those systems. The effort it takes to release something is limited to python setup.py sdist upload, and it's in a place (and in a shape) that anyone can use it without having 10 sets of platform-local instructions.

Even ignoring all the above, when I'm writing a new app or bit of code, I want to be sure I'm targeting the latest version of the code I depend on, so that future changes to API won't hit me as hard. By following along with my dependencies' development, I can be sure that my code breaks early, and breaks in development, not production. Upstreams also tend to not like bug reports against old branches, so ensuring I have the latest code from pypi means I can properly file bugs.

Lastly, I prefer virtualenv based setups for development, since I'm usually working on many things at once. This often means version mismatches in libraries, which brings in API changes (another whole rant here as well). I don't want to keep installing and uninstalling packages to switch between the two projects, and using a chroot(8) means a lot of overhead and that it's disconnected from my development environment / filesystem, so I resort to virtualenv to isolate my Development environment.

Final notes

I don't want to keep arguing about this. Just accept that the world's a big place and that there exist use-cases that both apt and pip need to exist and work in the way they're working now. At the very least, try and understand there exist smart people on both sides, and no one is trying to screw anyone over or keep their own little private club to themselves. Hopefully, going forward, we can make sure that the integration between these two tools gets better, not worse.

Help make this dream a reality. Contribute to a productive tone, not a destructive one. In short:

Use pip without sudo always. Don't tell people to use sudo.
Use apt or dpkg when deploying system-wide.
Understand people are going to package, and they will be more concerned about software using your library then keeping your library up to date.
Understand Debian Developers and package maintainers have to do a lot of work when updating or sponsoring an upload.
Understand upstream developers can't be bothered to fix every issue with every release (release early, release often) with some snapshot you introduced into unstable.
Use pip and virtualenv in development setups, so we can upgrade your app when we upgrade the lib.

Hy: The survival guide

2013-08-02T23:19:00-04:00

One of my new favorite languages is a peppy little lisp called hy. I like it a lot since it's a result of a hilarious idea I had while talking with some coworkers over Mexican food. Since I'm the most experienced Hypster on the planet, I figured I should write a survival guide. This will go a lot easier if you already know Lisp, but you can get away with quite a bit of Python.

The Tao of Hy

We don't have many rules (yet), but we do have quite a bit of philosophy. The collective Hyve Mind has spent quite a bit of time working out Hy's internals, and we do spend a bit of time looking at how the language “feels”. The following is a brief list of some of the design decisions we've picked out.

Look like a lisp, DTRT with it (e.g. dashes turn to underscores, earmuffs turn to all-caps.)
We're still Python. Most of the internals translate 1:1 to Python internals.
Use unicode everywhere.
Tests or it doesn't exist.
Fix the bad decisions in Python 2 when we can (see true_division)
When in doubt, defer to Python.
If you're still unsure, defer to Clojure
If you're even more unsure, defer to Common Lisp
Keep in mind we're not Clojure. We're not Common Lisp. We're Homoiconic Python, with extra bits that make sense.

Naturally, this doesn't cover everything, but if you can drop into that mindset, things start to make quite a bit of sense.

The Style of Hy

Although I am perhaps the least qualified person to do so (I still don't write idiomatic Lisp all the time), I'm going to set up a few ground-rules when it comes to idiomatic Hy code. We borrow quite a bit of syntax from Common Lisp and Clojure, so again, feel free to defer to either if you're not working on Hy internals. I prefer the Clojure Style Guidelines myself. As such, these are what we will defer to in the case that the Hy style is undefined.

Clojure-isms

Hy has quite a few Clojure-isms that I rather prefer, such as the threading macro, and dot-notation (for accessing methods on an Object), which I would rather see used throughout the hylands.

;; good:
(with [fd (open "/etc/passwd")]
    (print (.readlines fd)))

;; bad:
(with [fd (open "/etc/passwd")]
    (print (fd.readlines)))

Some other hy devs very much disagree, and there's nothing syntactically invalid about the latter, and it will continue to be supported (in fact, it makes some things easier!), but it will not be considered for Hy internal code.

We also very much encourage use of the threading macro throughout code where it makes sense.

;; good:
(import [sh [cat grep]])
(-> (cat "/usr/share/dict/words") (grep "-E" "tag$"))

;; bad:
(import [sh [cat grep]])
(grep (cat "/usr/share/dict/words") "-E" "tag$")

However, do use it when it helps aid in clarity, like all things, there are cases where it makes a mess out of something that ought to not be futzed with.

Python-isms

In addition to stealing quite a bit of syntax from Clojure, I'm going to take a few Python rules from PEP8 that apply to Hy as well. These are taken because PEP8 is a really great set of rules, and Hy code ends up pretty, well, Pythonic. The following are a collection of Pythonic rules that explicitly apply to Hy code.

Trailing spaces is a huge one. Never ever ever shall it be OK to have trailing spaces on internal Hy code. For they suck.

As with Python, you shall always double-space module-level definitions if separated with a newline.

All public functions must always contain docstrings.

Inline comments shall be two spaces from the end of the code, if they are inline comments. They must always have a space between the comment character and the start of the comment.

Hy-isms

Indentation shall be two spaces, except where matching the indentation of the previous line.

;; good (and preferred):
(defn fib [n]
  (if (<= n 2)
      n
      (+ (fib (- n 1)) (fib (- n 2)))))

;; still OK:
(defn fib [n]
  (if (<= n 2) n (+ (fib (- n 1)) (fib (- n 2)))))

;; still OK:
(defn fib [n]
  (if (<= n 2)
    n
    (+ (fib (- n 1)) (fib (- n 2)))))

;; Stupid as hell
(defn fib [n]
    (if (<= n 2)
            n
      (+ (fib (- n 1)) (fib (- n 2)))))

Parens must never be alone, sad, all by their lonesome on their own line.

;; good (and preferred):
(defn fib [n]
  (if (<= n 2)
      n
      (+ (fib (- n 1)) (fib (- n 2)))))

;; Stupid as hell
(defn fib [n] 
    (if (<= n 2)
      n
      (+ (fib (- n 1)) (fib (- n 2)))
    )
)  ; GAH, BURN IT WITH FIRE

Don't use S-Expression syntax where vector syntax is really required. For instance, the fact that:

;; bad (and evil)
(defn foo (x) (print x))
(foo 1)

works is just because the compiler isn't overly strict. In reality, the correct syntax in places such as this is:

;; good (and preferred):
(defn foo [x] (print x))
(foo 1)

Notice

This guide is, above all, a guide. This is also only truly binding for working on Hy code internally. This post is also super subject to change in the future, whenever I can be bothered to ensure that we have more of the de facto rules written down.

Automatically lint your packages with debuild.me

2013-06-09T17:43:00-04:00

Over my time working with Debian packages, I've always been concerned that I have been missing catchable mistakes by not running all the static checking tools I could run. As a result, I've been interested in writing some code that automates this process, a place where I can push a package and come back a few hours later to check on the results. This is great, since it provides a slightly less scary interface to new packagers, and helps avoid thinking they've just been “told off” by a Developer.

I've spent the time to actually write this code, and I've called it debuild.me. The code itself is in its fourth iteration, and is built up from a few core components. The client / server code (lucy and ethel) are quite interconnected, but firehose works great on its own, and is a single, unified (and sane!) spec that is easy to hack with (or even on!). Hopefully, this means that our wrappers will be usable outside of debuild.me, which is a win for everyone.

Backend Design

The backend (lucy) was the first part I wanted to design. I made the decision (very early on) that everything was going to be 100% Python 3.3+. This lets me use some of the (frankly, sweet) tools in the stdlib. Since I've written this type of thing before (I've tried to write this tool many, many, many, many times before), so I had a rough sense of how I wanted to design the backend. Past iterations had suffered from an overly complex server half, so I decided to go ultra minimal with the design of debuild.me.

The backend watches a directory (using a simple inotify script) and processes .changes files as they come in. If the package is a source package, a set of jobs are triggered (such as lintian, build and desktop-file-validate), as well as a different set for binary packages (such as lintian, piuparts and adequite). Only people may upload source packages (without any debs) and only builders can upload binary packages (without source).

The client and server talk using XML-RPC with BASIC HTTP auth. I'm going to (eventually) SSL secure the transport layer, but for now, this will work as a proof of concept.

Since I tend to like to keep my codebase simple and straightforward, I've used MongoDB as Lucy's DB. This lets me move between documents in Mongo to Python objects without any trouble. In addition, I evaluated some of the queue code out there (ZMQ, etc), and they all seemed like overkill for my problem, and had a hard time keeping track of jobs that (must never!) get lost. As a result, I wrote my own (very simple) job queue in Mongo, which has no sense of scheduling (at all), but can do its job (and do it well).

Jobs describe what's to be built with a link to the package document that the job relates to, and its arch and suite (don't worry about the rest just yet). Jobs get assigned via natural sort on its UUID based _id, and assigned to the first builder that can process its arch / suite. Source packages are considered arch:all / suite:unstable (so they always get the most up-to-date linters on any arch that comes along).

Lucy also allows for uploads to be given an X-Lucy-Group tag to manage which set of packages they're a part of. This comes in handy for doing partial archive rebuilds, or eventually using it to manage what jobs should be run on which uploads. This will allow me to run much more time-consuming tools for packages I want to review versus rebuilding to ensure packages don't FTBFS or aren't adequite.

Client Design

The buildd client (ethel) talks with lucy via XML-RPC to get assigned new jobs, release old jobs, close finished jobs, and upload package report data. When the etheld requests a new job, it also passes along what suites it knows of, which arches it can build, as well as what types it can run (stuff like lintian, build or cppcheck.) Lucy then assigns the builder to that job (so that we don't allocate the same job twice), and what time it was assigned at.

Ethel then takes the result of the job (in the form of a firehose.model tree) and transmits it over the line back to the Lucy server as a report (which also contains information on if the build failed or not), at which point lucy hands back a location (on the lucy host) that the daemon can write the log to.

If the job was a binary build, the etheld process will dput the package to the server, with a special X-Lucy-Job tag to signal which job that build relates to, so that future lint runs can fetch the deb files that the build produced.

Tooling

Ethel runs a set of static checkers on the source code, which are basically fancy wrappers around the tools we all know and love (like lintian, desktop-file-validate, or piuparts) which output Firehose in place of home-grown stdout. This allows us to programmatically deal with the output of these tools in a normal and consistent way.

Some of the more complex runners are made of 3 parts - a runner, wrapper and command. The server invokes the command routine, which invokes the runner (the command just provides a unified interface to all the runners), who's output gets parsed by the wrapper to turn it into a Firehose model tree.

The goal here is that tons of very quick-running tools get run over a distributed network, and machine-readable reports get filed in a central location to aid in reviewing packages.

Ricky

In addition to the actual code to run builds, I've worked on a few tools to aid with using debuild.me for my DD related life. I have some uncommon use-cases that are nice to support. One such use-case is the ability to rebuild packages from the archive (unmodified) to check that they rebuild OK against the target. This is handy for things like arch:all packages that get uploaded (since they never get rebuilt on the buildd machines, and FTBFSs are sadly common) or packages that have had a Build-Dependency change on them.

Ricky is able to create a .dsc url to your friendly local mirror, and fetch that exact version of the package. Ricky can then also use the .dsc (in a monumental hack) to forge a package_version_source.changes file, and sign it with an autobuild key and upload it to the debuild.me instance. Since it can also modify the .changes's target distribution, you can also use this to test if a package will build on stable or testing, unmodified.

Fred

Fred is a wrapper around Ricky, to help with fetching packages that may not exist yet. Fred also contains an email scraper to read off such lists as debian-devel-changes, and add an entry to fetch that upload when it becomes available on the local mirror, pass it to ricky, and allow debuild.me to rebuild new packages that match a set of criteria.

I'm currently playing around with the idea of rebuilding all incoming Python packages to ensure they don't FTBFS in a clean chroot.

Loofah

Loofah is also another wrapper around Ricky, but for use manually. Loofah is able to sync down the apt Sources list, and place it in Mongo for fast queries. This than allows me to manually run rebuilds on any Source package that fits a set of criteria (written in the form of a Mongo query), which get pulled and uploaded by Ricky.

An example script to rebuild any packages that Build-Depend on python3-all-dev in Debian unstable / main would look like:

[
    { "version": "unstable", "suite": "main" },
    { "Build-Depends": "python3-all-dev" }
]

Or, a script to rebuild any package that depends on CDBS:

[
    {},
    {"$or": [{"Build-Depends": "cdbs"},
             {"Build-Depends-Indep": "cdbs"}]}
]

You can use anything that exists in the Sources.gz file to query off of ( including Maintainer!)

Future Work

The future work on debuild.me will be centered around making it easier for buildd nodes to be added to the network, with more and more automation in that process (likely in the form of debs). I also want to add better control over the jobs, so that packages I upload only go to my personal servers.

I'd also very much like to get better EC2 / Virtualization support integrated into the network, so that the buildd count grows with the queue size. This is a slightly hard problem that I'm keen to fix.

I'm also considering moving the log parsing code out of the workers, so that the parsing code can be fixed without upgrading all the workers. This would also drop the Firehose dep on the client code, which would be nice.

Migration from a debuild.me build into a local reprepro repo is something that would be fairly easy to do as well, likely to be done remotely via the XML-RPC interface, which calls a couple of reprepro commands (such as includedsc and includedeb) and publishes it to the user's repo. This is a nice use of the debs that get built, and could also allow debuild.me to be used like a PPA system, but this allows the user to not migrate packages that may contain piuparts issues.

A primer on apt's mirror:// protocol

2013-02-23T21:04:00-05:00

It's sometimes helpful to keep your machines using a list of apt archives to use, rather then a single mirror, because redundancy is good. Rather then using (the great) services like http.debian.net or ftp.us.debian.org, you can set your own mirror lists using apt's mirror:// protocol.

While initially hacking this through, Micah ended up filing a bug on mirror://, more information in the bts. I've since been able to get it to work for me, but beware!

All of this is ultra unstable, so be a bit careful when using this. I've been using mirror:// for a few months now, and it seems fine (even have my servers using it), but it was a bit of a pain to set up. It gets slightly confused if you point it at something bad, and it's a mild pain to debug. Hopefully more people will see the value in mirror://, and contribute code to it's development.

Why bother?

If you have a local network mirror, it's helpful to have your machines default to a local mirror, if you're the sort to keep an archive mirror on the LAN, and fall back to your nearest friendly mirror otherwise. In addition, this lets you hand-define where apt searches for mirrors, which is great, since you can control the subset of servers you ping a bit more closely.

Practical Bits / quickstart

The following block covers the quick and dirty details on how to set up mirror:// for use on your machine (today!). This is very basic, and details are very sparse, but hopefully there's enough here to help folks use this on their local system. Basically, you've got three core things to do:

Pick your mirrors (this one's a bit of a duh)
Put them in a public place you can always get to, regardless of where you are in cyberspace (I use static.pault.ag) - remember, this is the one thing all your machines need to always get to, no matter where they are.
Configure your sources.list to use the mirror.txt file by pointing to the text file with the mirror:// protocol.

Turns out mirror://'s protocol handler will segfault if you give it something bad, so don't be afraid if you see apt-get update segfault - it just means you've likely not pointed it at a valid text file. The format of the text file should be a simple text file of mirrors it can try, in order of priority. Mine looks a bit like:

http://127.0.0.1:3142/debian.lcs.mit.edu/debian/
http://debian.lcs.mit.edu/debian/
# http://http.debian.net/debian/

Finally, your sources.list entry should look a bit like:

deb mirror://static.pault.ag/debian/mirrors.txt unstable main
deb mirror://static.pault.ag/debian/mirrors.txt experimental main
deb-src mirror://static.pault.ag/debian/mirrors.txt unstable main
deb-src mirror://static.pault.ag/debian/mirrors.txt experimental main

Problems

With the good comes the bad. Not everything fully supports this, and most tools that parse sources.list break in a really silly way.

command-not-found

update-command-not-found will blow up like:

W: Don't know how to handle mirror
W: Don't know how to handle mirror
W: Don't know how to handle mirror
W: Don't know how to handle mirror
W: Don't know how to handle mirror
W: Don't know how to handle mirror

Using env(1) in the shebang

2013-01-15T20:02:00-05:00

Some of you out there may have tried to pass flags to a script that was being invoked via /usr/bin/env in the shebang (#!), such as python. You might recall an error such as:

/usr/bin/env: python -d: No such file or directory

This error is super annoying, so I went about trying to figure out how I can pass arguments to python (or even things like ipython or bpython).

The idea is we can abuse the concept of a polygot to shim in some things we care about.

Implementation

Let's take a look at a quick script I hacked up to use bpython with a pre-made script that drops into interactive work.

#!/bin/sh
"""":
exec /usr/bin/env bpython -i $0 $@
"""
import hy
print "Hython is now importable!"

Let's step through this slowly. First, the bits the bash sees:

#!/bin/sh
"""":
exec /usr/bin/env bpython -i $0 $@

Which will cause bpython to reload the file, which looks like the following to Python:

#!/bin/sh
"""":
exec /usr/bin/env bpython -i $0 $@
"""
import hy
print "Hython is now importable!"

Where Python can now ignore the docstring. Magic!