Code Words

Data driven literary analysis

Tue, 11 Oct 2016 12:00:00 +0000

Why am I doing this?

When I was in high school my studies were heavily skewed towards foreign languages and literature in general. This means that between ages 14 and 18 I wrote a whole lot of literary analysis essays. The idea behind this exercise is that if you analyze and describe the techniques that make a piece of literature effective, you’re more likely to understand how and why that piece has been written.

Some 10 years later I became interested in Natural Language Processing (NLP). What I found most fascinating is the idea of machines understanding human languages, a task that humans often fail at. As a result, I spent most of my time at the Recurse Center teaching myself about NLP and playing with Shakespeare’s body of work in the attempt to automate the process of writing a literary analysis essay.

This article is a summary of what I learned and the method I came up with.

What is natural language processing?

Natural Language Processing is a field of computer science studying all interactions between natural and artificial languages. Natural languages are those used in every day communications between humans, and are the result of constant evolution through time and usage. Artificial languages are constructed for a specific purpose (think mathematical symbols or programming languages).

In a broad sense, NLP includes all computer manipulations of human languages, from simple word counts to understanding human utterances. By enabling computers to understand as well as generate natural, human language, they can be used for a wide range of everyday applications, like machine translation and spam filtering.

Some of these methods can easily be applied to literary texts to extract structure and meaning.

Genre classification

The first step in the analysis of a piece of literature is identifying what category it belongs to. This task is essentialy Document Classification, the process of assigning one or more labels to a document. This is generally a supervised problem, where labels are predifined and assigned to the input documents prior to training the model.

The two stages of supervised document classification: During training, each document is passed through a feature extractor and labelled before being processed by the training algorithm. The input pairs are then used to define a function (classifier) from the feature space to the labels space. In prediction phase, the classifier takes features extracted from a new, unlabelled document and uses them to return an appropriate label.

The issue with this approach is that it requires a large collection of labelled documents to train the prediction model. When a labeled set is not available someone has to manually tag the documents in the corpus prior to the training stage. This can be time consuming.

The alternative is an unsupervised approach. Groups are created with the goal of capturing, or extracting, the natural structure in the data. When this is done correctly it is possible to inspect only a subset of each group and attach a label to each of them.

In unsupervised document classification the input documents are not pre-labelled. The features extracted are used to group the inputs based on similarity, then labels are assigned to describe the inner structure of the groups. New documents are assigned to the group they're closest to, by computing the same similarity metric used to create the partition.

Shakespeare analysis: classifying plays

Shakespeare is best known for his plays, most of which were produced between the end of the 16^th century and the beginning of the 17^th century. There is some controversy regarding whether all the plays were written by the same author, but scholars generally agree on dividing them into three categories: comedies, tragedies, and histories. I’m going to leave the histories out of the analysis because they’re much harder to define, and focus on the differences between the other two categories, comedies and tragedies.

Comedies tend to focus on situations rather than characters. Multiple plot lines that see characters separated and reunited, use of puns, identity confusion, family conflicts, and young love are all common traits of Shakespeare’s comedies.

Tragedies focus on characters over plot and emphasize their honesty or lack thereof. The plot is generally more linear and follows a classic definition of tragedy in which a hero of noble birth is brought to ruin by some tragic flaw.

Shakespeare wrote 17 comedies and 10 tragedies for a total of 27 plays and a vocabulary (excluding stop words¹) of over 11,000 words (the features). This influences the analysis in two ways:

The number of features: in high dimensional spaces, all objects appear sparse and dissimilar, which makes it really hard to efficiently organize them in groups based on similarities. I use content-based feature extraction to represent the document in a lower number of dimensions.
The size of the training set: the number of documents is simply too small for a model to learn the differences between classes. Instead of learning pre-defined categories I use a clustering algorithm to find the best two-way split of the set and compare it to the traditional classification.

Step 1: Content-based feature extraction

Feature extraction is a general term for methods that convert raw data into a set of derived values, called features, intended to be non-redundant and informative. When the initial set of inputs is too large or redundant, feature extraction facilitates learning by feeding the classification algorithm with a reduced representation of the data containing the information relevant to the task, rather than the initial complete set. This is very domain specific and depends on the available data.

In the example, it’s important to extract the semantic information in the text so that it can easily be interpreted by a machine. A simple way of doing this is using tf-idf, a statistical score for the importance of words in both the document and the corpus. Tf-idf, however, doesn’t reduce the number of features. For that, we’ll use topic modeling. Topic modeling allows us to extract features that describe the content of the corresponding document, representing it in the form of word distributions, revealing hidden semantic structures in the text.

Latent Dirichlet Allocation

Latent Dirichlet Allocation is a topic modeling technique that describes the probabilistic procedure used to assign words to documents.

In this context:

a word w is an item from a vocabulary indexed $\{1, ..., V\}$
a document is a sequence of N words denoted by $\mathbf{w} = (w_1, ..., w_N)$ where $w_n$ is the nth word in the sequence.
a corpus is a collection of M documents denoted by $D = \{\mathbf{w}_1, .., \mathbf{w}_M\}$

Given a collection of documents, and K topics, LDA finds the formal relationship between the observed frequencies of the words in the documents and the latent topics, where a topic is characterized by a distribution over words. For example, assume our vocabulary is made up of the words panda, broccoli, cat, eat and apple. A topic about cute animals will assign higher probabilities to panda and cat (e.g. P(panda) = 0.3, P(cat) = 0.3, P(eat) = 0.15, P(apple) = 0.125, P(broccoli) = 0.125), whereas a topic about food would assign them lower probabilities (e.g. P(panda) = 0.05, P(cat) = 0.05, P(eat) = 0.3, P(apple) = 0.3, P(broccoli) = 0.3).

The process that describes documents generation has 3 steps.

For each document:

Select the number of words
Draw a distribution of topics $\theta ~ Dir(\alpha)$
For each word $w_n$ $w n $ in the document:
- Draw a specific topic $z_n ~ Multinomial(\theta)$
- Choose a word $w_n$ from $p(w_n \vert z_n, \beta)$ , a multinomial probability conditioned on the topic $z_n$

What is most interesting about LDA is the use of the Dirichlet distribution, which is particularly useful for modeling the randomness of probability mass functions (pmf). A pmf gives the probability of a discrete random variable to take a specific value. A simple example is a 6-sided die, where in order to select a random value we roll the die and produce a number from 1 to 6. In theory, each number has the same probability of being selected, but in real life, due to the laws of physics and manufacturing, each die is slightly different and represents a different pmf. If we have a bag full of dice and we randomly extract one of them we are effectively drawing from a random pmf. This can be modeled with a Dirichlet of parameter $\alpha$ .

In the context of text analysis, a document is a pmf of length V (the number of words in the dictionary) and a collection of documents produces a collection of pmfs. In LDA topics are latent multinomials so we can use the Dirichlet distribution to describe the probability of each topic in each document. Ultimately, this means that LDA can be used to extract topics in form of words distributions and then represent each document as a mixture of topics².

Running LDA on the collection of plays attributed to Shakespeare, we obtain the probability distribution of the words in the dictionary for each topic. By looking at the words that have highest probability in the topic it is possible to assign a label to each of them.

Probability distribution of the top 10 words in topic 2, 3, 5, and 6. Each bar represents the probability of a word to be selected when drawing terms from the specific topic, e.g. when selecting from the topic common words the probability of using the word shall is ~0.04 while the probability of using like is just below 0.02.

The figure shows the probabilities for the top ten words in four of the topics extracted from the plays. The most likely words to appear in topic 2 are terms that are commonly used in a conversation without adding too much meaning to it, but are generally not included in the list of stop words. It’s not ideal, but it makes sense that they cluster together into a single topic. The top distributions of topics 2, 5, and 6 are more promising and can be easily associated with three of the main themes in Shakespeare’s plays, respectively death, love and the hero.

The other output of LDA is the distribution of topics in the documents.

Plays as mixtures of topics. Topics 3 and 9, which assign higher probabilities to words frequently used in conversation, have relatively high probabilities of appearing across all plays. Other topics present more variance across the set and are represented with darker colors, matching plays that are more likely to touch specific themes. For example, topic 5, related to love, has a higher probability/darker blue box for Midsummer Night's Dream and lower probability/light blue box for Macbeth.

In the figure above, topics are on the x-axis and each line represents a document. Boxes are colored according to the probability of topics in each play: darker blues indicate higher probability of appearing in a specific document and lighter colors indicate lower probabilities. Unsurprisingly, the boxes in the columns of the topics associated with common words (3 and 9) are consistently darker. Since the clustering algorithm aims for a partition that maximizes the differences between groups, these topics will not be very useful in grouping the plays. However, there is enough variability to pass these representations of the documents as input to a clustering algorithm for the grouping.

Step 2: K-Means clustering

K-Means is a simple unsupervised algorithm for solving clustering problems. The goal is to create a partition of n elements into k groups so that each element is assigned to the cluster with the nearest mean. Given an initial set of k means the algorithm assigns each observation to the nearest cluster in terms of Euclidean distance, then recalculates new means to be the averages of the observations in each cluster and reiterates until the assignment no longer changes. At the end of the procedure, elements in the same group are more similar to each other than to elements in different groups.

The topics’ probabilities are the features used as inputs of K-Means. The result is a vector of labels that divides the plays in two groups. In the table, tragedies are highlighted in bold.

Group 1	Group 2
Twelfth night, The Merchant of Venice, Love’s Labour’s Lost, Much Ado About Nothing, Taming of the Shrew, As You Like it, Merry Wives of Windsor, Midsummer Night’s Dream, Romeo and Juliet, Comedy of Errors, Two Gentlemen of Verona	Titus Andronicus, All’s Well What Ends Well, Macbeth, Hamlet, Antony and Cleopatra, King Lear, Julius Caesar, Tempest, The Winter’s Tale, Timon of Athens, Coriolanus, Troilus and Cressida, Measure for Measure, Cymbeline, Othello, Pericle Prince of Tyre

The first group is entirely composed by comedies, except for Romeo and Juliet, while the second group is evenly split between the two categories. This is not a great result. In order to understand the grouping, it is useful to look at the average distributions of the groups. As expected, these show that when a topic is more likely to appear in a group of documents, its probability is lower in the other.

Average distribution of topics by group. Topics that have higher variance determine the split of the two groups, e.g. topic 5 has, on average, higher probability for the plays in group 1, and topic 6 has higher probability for the plays in group 2. Topics that have low variance across the plays (6 and 9), also have similar group-averages.

Explaining the results

Content doesn’t seem enough to explain the differences between comedies and tragedies, but it can explain the evolution of Shakespeare’s personal writing style.

When looking at the distribution of the year the plays were first performed the two groups show some overlap around 1600, but it seems clear that the plays in one group were staged significantly earlier than the others.

Box plot of the year each play was first staged by group. The box represents the central 50% of the distribution, with the lower, middle and top lines representing respectively the minimum, the median and the maximum of the distribution exluding outliers (single dots). Only one of the plays in group 2 was staged before 1599, while all plays in group 1 were staged before 1601.

At the time of this analysis I had no idea what this could possibly mean, but the internet is a magical place and has a good explanation for this. The traditional classification of comedies, tragedies, and histories follows the categories of the First Folio, the first complete collection of Shakespeare’s works, published in 1623. Modern critics don’t necessarily agree with it and have introduced different terms and categories. One of the categorizations follows the evolution of Shakespeare’s style over time, identifying an Elizabethan Shakespeare, who is younger and more influenced by the classics, as opposed to a Jacobean one, more mature and writing for a different king. It is not possible to pinpoint exactly when the change happens, but scholars indicate the end of Queen Elizabeth’s reign (1603) and first years of the reign of King James as the time he reaches his maturity, shifting not just the subject of his work but also his style.

Epilogue

It turns out my teachers were right and text analysis does lead to a better understanding of how and why a piece was written. This was a fun exercise (far more fun than writing the essays) and led to some unexpected insight. The thought of machines understanding human languages still blows my mind, but I think I have a better understanding of how NLP works. At least good enough to come up with my own analysis and get some useful information out of a corpus, and that’s a win.

All the analysis is done in Python using the Natural Language Toolkit for text manipulation and scikit-learn and lda packages for the modeling. If you’re interested in the code, there is a repo here: it includes all the inputs, papers and a not-so-messy notebook with all the code used for this article.

The term stop words refers to the list of most common words in a language. There is no unique list of stop words used by all natural language processing tools, and not all tools have such a list. It generally includes short function words such as the, is, which etc. ↩
For more details on LDA see Blei et al. ↩

A tour of random forests

Tue, 11 Oct 2016 12:00:00 +0000

Random forests are an excellent “out of the box” tool for machine learning with many of the same advantages that have made neural nets so popular. They are able to capture non-linear and non-monotonic functions, are invariant to the scale of input data, are robust to missing values, and do “automatic” feature extraction.

Additionally, they have other benefits that neural nets do not. What follows is a look into how random forests work, how they may be usefully applied, and a discussion of some situations in which they may be preferable to neural networks.

Forest Internals

So how do random forests work? The random forest is an ensemble machine learning model, a composite of many simpler models called decision trees. The individual trees are formed by branching on each of the included features and the random forest is an aggregation of these trees.

Decision Trees

Decision trees branch on each of the included features in order to partition the data. For each bin in the partition (i.e. leaf of the tree), we assign an output value to inputs contained in that bin.

The simple case is a classification problem in which the output is categorical. Suppose, for example, we were attempting to model the effects of cooperation in the classic “prisoner’s dilemma” game. Our decision tree might look something like this (features in circles, outputs are leaves):

Our features are the behavior of each of the prisoners (our outputs are the corresponding prison terms). From the combination of these features we can partition the outcome space into each of the four possible cases.

Slightly more complicated is the case of continuous output. The leaves of the tree will still form a partition of the output space but there are many possible values within each bin. How do we assign a single value? We take the mean among values assigned to that bin.

For example, suppose we have a sample for which the input values are uniformly distributed accross the interval $(-10, 10)$ . For each input value, $x$ , the corresponding output is $\lvert x \rvert$ . Our decision tree might look something like this:

Given our features, we are able to partition the data into four bins. The output for a given bin is the mean output of the observations contained within it.

There are a few things worth noting here. First, the fact that we can branch on ranges of a continuous value allows us to model non-linear / non-monotonic behaviour. Additionally, it’s straightforward to see that with more features, we can construct a tree with more leaves (a finer partition of the data) and more closely approximate continuous valued output.

Entropy & “Guess Who?”

It’s clear that the order of branches in a decision tree matters. Less transparent is how the path to each leaf is constructed.

This happens according to an iterative approach. For each level of a predetermined depth, we branch on the feature which maximizes the information gain. Information gain is defined in terms of entropy, which is defined as $H(S) = -\sum_{x \in S} p(x)\cdot log_2 p(x)$ . The information gain of branching on a feature, $a$ , is $IG(S, a) = H(S) - H(S \lvert a)$ . Thus, at each iteration, we branch on the feature which minimizes the conditional entropy.

Unless you’re reasonably familiar with information theory, the intuition behind this procedure may not be obvious.

A more intuitive way to understand this is the game “Guess Who?”. What questions should we ask to most quickly identify the opposing player’s character (or more formally, given a feature space of all binary classifications, how do we partition the output space with a minimum depth tree)?

It’s typical to ask questions like whether the character “wears glasses” or “has white hair”. But these questions represent features with low information gain; more likely than not, the answer will be no and we will have eliminated only a few possibilities.

A higher information gain feature will partition the data as evenly as possible. Consider a different strategy. “Is your character Connor, or James, or Nick, or Sarah, or Justin, or Tyler, or Ashley, or Kyle, or Joshua, or Megan, or Andy, or Joseph?”

This question represents a feature which perfectly bisects the data; we are guaranteed to eliminate exactly half of the possibilities. And we can repeat this in subsequent iterations. In the general case (of $n$ characters), we can identify the target character with only $log_2(n)$ operations.

Trees To Forests

If the unnatural features in our “Guess Who?” example make you nervous about overfitting, you’re not wrong. Overfitting is a well-known pitfall for decision trees. If we add an additional 1000 characters onto the board, asking about the 12 characters from before is a bad question and will almost certainly underperform “white hair”.

This is the purpose of random forests. To avoid overfitting with a single tree, we build an ensemble model through a procedure called bagging.

For some number of trees, $T$ , and predetermined depth, $D$ , select a random subset of the data (convention is roughly $2/3$ with replacement) and train a decision tree on that data (as discussed above). To train the tree, use a subset of the available features (roughly $\sqrt{M}$ by convention, where $M$ is the total number of features).

Obviously, these parameters can be tuned to fit the needs of the application. A model with more trees / data can take longer to train but may have greater accuracy. More depth / features increases the likelihood of overfitting but may be appropriate if features have complex interactions.

In the case of a classification problem, we use the mode of the trees’ output to classify each value. For regression problems, we use the mean of the output trees. Note that the aggregation process is independent of the internal workings of the individual decision trees. Because each tree can make a prediction using the available features (or a subset thereof), they can be polled to form an aggregate prediction.

It’s difficult to overfit with only a subset of the available information. By building the random forest model as an aggregation of weaker models (weak in that the trees are trained on a subset of the available information) we are able to build a strongly predictive model while avoiding the pitfalls of overfitting.

Some Illustrative Examples

What makes random forests such an effective tool is their robustness to different types of data (e.g. non-linear / non-monotonic functions, un-scaled data, data with missing values, data with poorly chosen features). This makes them an excellent “out of the box” tool for general machine learning problems which do not immediately suggest themselves to a specific alternative.

Take, for example, a function like $y = x \cdot sin(x) + U$ , where $U$ is a random value uniformly distributed in the interval $(0, 1)$ . It’s a “simple” function but it is both non-monotonic and non-linear. A technique like simple regression is a non-starter without significant feature extraction. However, it is a simple task for a random forest.

from sklearn.ensemble import RandomForestRegressor
import numpy as np
import math

def generate_data():
        X, Y = [], []
        f = lambda x : x * math.sin(x) + np.random.uniform()
        for x in np.arange(0, 10, 0.05):
                X.append([x])
                Y.append(f(x))
        return (X, Y)

X, Y = generate_data()
model = RandomForestRegressor()
model = model.fit(X, Y)

This code gives us a model which looks like this.

The actual creation of the trained model requires no configuration or feature extraction at all. Said another way, it doesn’t require the practitioner to understand anything about the (complex) shape of the function we’re modeling.

It’s worth noting that other algorithms within “classical” machine learning can accomplish this as well. You could get similar results with k-nearest neighbors or support vector machines. But such approaches fall apart in other common cases such as data with unscaled features (intuitively speaking, models which depend on “distance” will be dominated by features with larger scale).

Consider the following example. We will seek to classify points as being within four quadrants: “NE” ( $x > 0$ and $y > 0$ ), “NW” ( $x < 0$ and $y > 0$ ), “SE” ( $x > 0$ and $y < 0$ ), and “SW” ( $x < 0$ and $y < 0$ ). A straightforward example, except that our $x$ values will cover the interval $(-0.1, 1)$ while our $y$ values will cover the interval $(-5000, 5000)$ .

import numpy as np
from sklearn.ensemble import RandomForestClassifier
import math

def quadrant(coords):
        EW = 'W' if coords[0] < 0 else 'E'
        NS = 'S' if coords[1] < 0 else 'N'
        return NS + EW

N = 10000
x = -0.1 + np.random.sample(N) * 1.1
y = -5000 + np.random.sample(N) * 10000

xy_coords = list(zip(x, y))
quadrant = list(map(quadrant, xy_coords))

model = RandomForestClassifier()
model = model.fit(xy_coords, quadrant)

This code gives us a classification model which handles the data exactly as intended.

Try to solve the same classification problem with k-nearest neighbors (use a KNeighborsClassifier in place of the RandomForestClassifier) and you will end up with a model like this (support vector machines have similarly dissapointing results).

What About Neural Nets?

In a time when neural networks are as popular as they are, it’s tempting to ask why any of this matters. Why not just use neural nets?

There are several advantages that random forests have over neural networks.

Parameter tuning tends to be simpler; there are well established conventions for choosing parameters in random forests but how to determine network layer structure is fairly opaque.
There is a more robust body of academic literature around them which makes the internal workings (arguably) easier to understand.
Generally simpler to implement. Popular implementations (e.g. scikit-learn) allow users to train sensible random forests with as little as a single line of code. Configuring network layer architecture generally involves more set up.
Random forests are extremely well supported for distributed deployment. Through MLlib, random forests are included in Apache Spark and are therefore easily scalable.

With all that said, neural networks do have distinct advtanges. They tend to be even better at automatic extraction of features than forests. They can also be more efficient to train. With nets, there is a global loss function to which gradient descent can be applied; forests must be trained by layers.

The clearest benefit is around predictive performance. The best performing models in many well publicized machine learning challenges tend to be deep neural nets. If incremental gains in model performance are important, this concern may reasonably trump all others.

Ultimately, the choice is application specific. Depending on the details of the problem, one or both of these approaches may be appropriate.

In any case, random forests’ robustness to various types of challenging data and amenability to scaling make them an excellent choice for a large range of applications and a very reasonable first choice for problems that don’t clearly suggest a specific alternative.

A history of storage media

Tue, 11 Oct 2016 12:00:00 +0000

Storage is an essential part of every computer architecture, from the hypothetical paper tape in a Turing machine to the registers and memory unit of the von Neumann architecture. Non-volatile storage media and volatile memory devices have both gone through several technological evolutions over the last century, including some delightfully strange mechanisms. This article is a very selective history, focusing on a handful of the earliest technologies that I find especially interesting:

Punch cards, one of the earliest forms of computer data, used for weaving patterns and census data!
Williams-Kilburn tubes, the first electronic memory, where bits were stored as dots on the screen!
Mercury delay lines, two-foot-long tubes of hot mercury, which stored bits as pulses of sound!

For a more comprehensive resource, the Computer History Museum has a detailed timeline, featuring lovely photographs. Storage and memory have a rich history. While CPU architectures are quite diverse, they tend to use the same basic elements – switches, in the form of transistors or vacuum tubes, arranged in different configurations. In comparison, the basic components of memory mechanisms have been incredibly varied; each iteration essentially reinvents the wheel.

1. Punch cards

Punch cards aren’t really a strange storage medium, but they are among the earliest, and have had some fascinating historical applications.

1.1. Jacquard looms

Many early ancestors of computers were controlled by punch cards. The ancestor of these machines, in turn, was the Jacqard loom, which automated the production of elaborately woven textiles. It was the first machine to use punch cards as an automated instruction sequence.

First – some background information about weaving patterns into fabric.

Most fabric is woven with 2 sets of yarn that are perpendicular to each other. The first set of threads, the warp, is stretched out over a loom, and the second set, the weft, is passed either over or under the threads of the warp to form the fabric. This over–under operation is easily represented by binary information. Changing the order of the over–under produces different kinds of fabrics, from satins to twills to brocades.

In the following example of a woven design, the warp is white, and the weft is blue. To form a twill weave, which is what most denims are, the loom raises the first two threads, drops the second two, raises the next two, and so on, across the warp. The weft is then passed between the raised and lowered sets of threads, to weave in the first row. The loom repeats the process, raising alternating sets of warp threads, then passing the weft through. After a few rows¹, the characteristic zigzag pattern of a twill appears.

Although weaving can be very intricate, this particular kind has two simple operations, repeated according to very specific patterns. Computers excel at simple and repetitive work!

Before the Jacquard loom, this process of raising and lowering threads was done by hand on a drawloom. A warp might be about 2000 threads wide, so weaving a detailed pattern in fabric would require making decisions about raising or lowering thousands of threads, thousands of times. Depending on the complexity of the design, an experienced weaving team could weave a few rows every minute, so an inch of fabric would take a day to produce. For context, a ball gown might take up to four yards of decorated fabric, which represents nearly four months of weaving.

A portrait of Jacquard woven on a loom.

Around 1803, Joseph-Marie Jacquard started building a prototype for a loom that automated the production of decorated cloth. The Jacquard loom wove elaborate silk patterns by reading patterns off a series of replaceable punch cards. The cards that controlled the mechanism could be chained together, so complex patterns of any length could be programmed in.

Twill weaves, like the one above, are fairly easy and repeatable patterns, and they are a trivial example of what an automated loom can produce. The special power of the Jacquard loom comes from its ability to independently control almost every warp end.² This is pretty amazing! A Jacquard mechanism can produce incredibly detailed images. The portrait of Joseph-Marie Jacquard is a stunning demonstration of the intricacy of the loom; the silk textile was woven using tens of thousands of punch cards.

Image courtesy the McCarl Gallery.

The ability to change the pattern of the loom’s weave simply by changing cards was a conceptual precursor to programmable machines. Instead of a looped, unchanging pattern, any arbitrary design could be woven into the fabric, and the same machine could be used for an infinite set of patterns.

This mechanism inspired many other programmable automatons. Charles Babbage’s Analytical Engine borrowed the idea. In fact, Babbage himself was rumored to have the woven portrait of Jacquard in his house! Player pianos use punch cards (or punched drums) to produce music. That said, most of the early uses of punch cards were for basic, repetitive control of the machines —- simple encodings of music or weaves for specialty use cases. The control languages of the early automatons were limited and not very expressive. The full expressive power of punch cards wasn’t realized until almost 100 years later, when they became the tool for all large-scale information processing.

1.2. The 1890 United States Census

In the late 19th century, the U.S. Census Bureau found itself collecting more information than it could manually process. The 1880 census took over 7 years to process, and it was estimated that the 1890 census would take almost twice as long.³ Spending a decade processing census information meant that the information was obsolete almost immediately after it was produced!

To cope with the growing population, the census bureau held a competition for a more efficient method to count and process the data. Herman Hollerith came up with the idea to represent census information on punched cards, and produced a tabulating machine that could sort and sum the data. His design won the competition, and was used to count the U.S. Census in 1890. The tabulator enabled much faster processing, and provided many more statistics than were available for earlier years.

After the success of the tabulator in the 1890 census, other countries began adopting the technology. Soon, Hollerith’s Tabulating Machine Company produced machines for many other industries. After his death, the Computer Tabulating Recording Company renamed itself to the International Business Machines Corporation, or IBM, leading us into a new era of computing.

The Hollerith Tabulator.

Soon after the 1890 census, almost all information processing was done via punch card. By the end of World War I, the army used punch cards to store medical and troop data, insurance companies stored actuarial tables on punch cards, and railroads used them to track operating expenses. Punch cards also became well-established in corporate information processing: throughout the 1970s, punch cards were used in scientific computing, HR departments of large companies, and every use case in between.

Though the once-ubiquitous punch cards have been replaced by other data formats, their echoes are still with us today —- the suggested 80-character line limit for code comes from the IBM 80-column punch card format.

2. Williams-Kilburn Tubes

In the era of punch cards, the cards were mostly used for data while control programs were inputted through complicated system of patch cords. Reprogramming these machines was a multi-person, multi-day undertaking! To speed up this process, researchers proposed using easily replaceable storage mechanisms for storing both programs and data.

Two women wire a portion of ENIAC with a new program. U.S. Army Photo.

While it’s possible to build computers using mechanical memory, most mechanical memory systems are slow and tediously intricate. Developing electronic memory for stored-program computing was the next big frontier in computer design.

In the late 1940s, researchers at Manchester University developed the first electronic computer memory, essentially by cobbling together surplus World War II radar parts. By making a few ingenious modifications to a cathode ray tube, Frederic Williams’ and Tom Kilburn’s lab at Manchester built the first stored-program computer.

During the Second World War, cathode ray tubes (CRTs) became standard in radar systems, which jump-started research into more advanced CRT technology. Researchers at Bell Labs took advantage of a few secondary effects of cathode ray tubes to use the tubes themselves to store the location of past images. Williams and Kilburn took the Bell Labs work a step further, adapting the CRTs for use as digital memory.

2.1. Secondary effects: principles of operation

The Williams-Kilburn tube turned spare parts from radar research and a few side effects of CRT tubes into the first digital memory.

A conventional CRT displays an image by firing an electron beam at a phosphorescent screen. Electrostatic plates or magnetic coils⁴ steer the beam, scanning across the entire screen. The electron beam switches on and off to draw out images on the screen.

Depending on the type of phosphor on the screen and the energy of the electron beam, the bright spot lasts from microseconds to several seconds. During normal operation, once written, the bright spot on the screen can’t be detected again electronically.

However, if the energy of the electron beam is above a specific threshold, the beam knocks a few electrons out of the phosphor, an effect called secondary emission. The electrons fall a short distance away from the bright spot, leaving a charged bullseye that persists for a little while.

Thus, to write data, Williams-Kilburn tubes use a high-energy electron beam to charge spots on the phosphor screen. Bits of storage are laid out in a grid across the face of the tube, much like pixels on a screen. To store data, the beam sweeps across the face of the tube, turning on and off to represent the binary data. The charged areas across the screen are essentially little charged capacitors.

When the higher energy electron beam hits the screen, the secondary emission of electrons induces a small voltage in any conductors near it. If a thin metal sheet is placed in front of the CRT’s phosphor screen, the electron beam knocks a few electrons out of the phosphor screen, inducing a voltage change in the metal sheet, as well.

Thus, to read data, the electron beam again sweeps across the face of the tube, but stays on at a lower energy. If there were a ‘1’ written there already, the point of positive charge on the screen gets neutralized, discharging the capacitor. The pickup screen then sends a pulse of current. If there were a ‘0’, no discharge occurs, and the pickup plate sees no pulse. By noting the pattern of current that comes through the pickup plate, the tube can determine what bit was stored in the register.⁵ The Williams tube was true random access memory – the electron beam could scan to any point in the screen to access data near-instantly.

Data stored on a Williams-Kilburn tube, transferred to a display CRT. Image courtesy the Computer History Museum.

The charge of areas leak away over time, so a refresh/rewrite process is necessary. Modern DRAM has a similar memory refresh procedure, too! Since the read is destructive, every read is followed by a write to refresh the data.

The data in a Williams-Kilburn tube could be transferred to a regular CRT display tube for scrutiny, which was great for debugging. Kilburn also reported that it was quite mesmerizing to watch the tube flicker as these early machines computed away.

2.2. The Manchester Baby

After the team at Manchester had a working memory tube that stored 1024 bits, they wanted to test the system in a proof-of-concept computer. They had a tube that would store data written to it manually, at slow speeds, but they wanted to ensure that the whole system would still work at electronic speeds under a heavy write load. They built the Small Scale Experimental Machine (a.k.a. the Manchester Baby) as a test bed. This would be the world’s first stored-program computer!

The Baby had four cathode ray tubes:

one for storing 32 32-bit words of RAM,
one as a register for an accumulator,
one for storing the program counter and current instruction,
one for displaying the output or the contents of any of the other tubes.

Williams-Kilburn tubes are an unusually introspectable data storage device.

Programs were entered one 32-bit word at a time, where each word was either an instruction or data to be manipulated. These words were inputted by manually setting a set of 32 switches to on or off! The first program that was run on the Manchester Baby calculated factors for large numbers. Later, Turing wrote a program that did long division, since the hardware could only do subtraction and negation.

2.3. Later history

Parts from the Manchester Baby were repurposed for the Manchester Mark 1, which was a larger and more functional stored-program computer. The Mark 1 developed into the Ferranti Mark 1, which was the first commercially available general-purpose computer.

The Williams-Kilburn tube was used as RAM and storage in a number of other early computers. The MANIAC computer, which did hydrogen bomb design calculations at Los Alamos, used forty Williams-Kilburn tubes to store 1024 40-bit numbers.

Though the tubes played a large part in early computer history, they were quite difficult to maintain and run. They frequently had to be tuned by hand, and were eventually phased out in favor of core memory.

3. Mercury Delay Lines

In addition to Williams-Kilburn tubes, radar research yielded another memory mechanism for early computers – delay line memory. Defensive radar systems in the 1940s used primitive delay lines to remember and filter out non-moving objects at ground level, like buildings and telephone poles. This way the radar systems would only show new, moving objects.

Delay lines are a form of sequential access memory, where the data can only be read in a particular order. A vertical drainpipe can be a simple delay line – you push a ball with data written on it in one end, let it fall through the pipe, read it, then throw it back in the other end. Eventually, you can juggle many such bits of data through the pipe. To read a particular bit, you let the balls fall and cycle through until you reach the bit you want. Sequential access!

The most common form of delay lines in early computers was the mercury delay line. This was essentially a two-foot-long pipe filled with mercury, with a speaker on one end, and a microphone on the other (in practice, these were identical piezoelectric crystals). To write a bit, the speaker would send a pulse through the tube. The pulse would travel down the tube in half a millisecond, where it would be read by the microphone. To keep the bits stored, the the speaker would then retransmit the bits just read back through the tube. Like the drainpipe example, to read a particular bit, the circuitry had to wait for the pulse it wanted to cycle through the system.

Mercury delay lines of UNIVAC.

Mercury was chosen because its acoustic impedance at 40ºC (104ºF) closely matched the piezoelectric crystals used as transducers. This reduces echoes that might interfere with the data signals. At this temperature, the speed of sound in mercury is about four times that in air, so a bit passes through a 2-foot-long tube in about 420 microseconds. Since the computer’s clock needs to be kept exactly in line with the memory cycles, keeping the tube at exactly 40ºC was critical.

The first battery of mercury delay lines in EDSAC, with Maurice Wilkes for scale.

While the Manchester Baby was the first stored-program computer, it was only ever meant as a proof-of-concept. EDVAC, built for the U.S. Army, was the first practically used stored-program computer. John von Neumann’s report on EDVAC inspired the design many other stored program computers.

So, mercury delay lines were giant tubes of liquid mercury, kept in ovens at 40ºC, cycling bits of memory through as sound waves! Despite their unwieldiness, mercury delay lines were used in a number of other early computers. EDSAC, which was inspired by von Neumann’s report, was the first computer used for computational biology, and was also the computer on which the first video game was developed (a version of tic-tac-toe with a CRT display). UNIVAC I, which was the first commercially available computer, also used mercury delay lines. The first UNIVAC was installed at the U.S. Census Bureau in 1951, where it replaced an IBM punch card machine. It was eventually retired in 1963.

And more!

This is just the story of the first few memory mechanisms that came on the market; other unique mechanisms have been developed in the decades since delay lines. Soon after, core memory – ferrite donuts woven into mats of copper wire – became ubiquitous. A read-only version of core memory was on the Apollo Guidance Computer, which took astronauts to the moon. Until a few years ago, nearly every computer had several spinning platters coated in a thin layer of iron, carefully magnetized by a needle hovering over a cushion of air. For years, we passed data and software around on plastic disks covered in little pits, read by bouncing a laser off the surface. Fancier disks, which can store up to 25 GB of data, use blue lasers, instead.

Computers are incredible Rube Goldberg machines – each layer is made of fractal complexity, carefully hidden away behind layers of abstraction. Examining the history of the technology gives us some insight into why the interfaces look the way they do now, and why we use the terminology we do. Early computing history produced a bestiary of fascinating and complicated memory devices. Each system is an incredibly well-engineered piece of machinery that leaves its imprint on computers to come.

Parts of this article originally appeared on Kiran’s blog.

An accuracy apology: The weft on the end should be looped over, making the selvage, but drawing convincing curved shapes isn’t Google Drawing’s forte. ↩
A nitpick: not all automated looms allow independent control of every warp end. Dobby looms, which appeared about 40 years after the Jacquard device, control groups of threads attached to shafts. Jacquard devices have more shafts than Dobby looms, but don’t always allow for control of every thread. This explanation differentiates between Dobby looms and Jacquard looms, and includes pretty gifs! ↩
The U.S. Census Bureau describes the tabulation and processing of the 1880s census ↩
Oscilloscopes and Williams tubes use electrostatic plates, which operate well at high frequencies. However, when the beam deflection angle is large, electrostatic plates can obstruct and distort the beam, making them a poor choice for large screens and short tube lengths. Televisions use magnetic coils, which don’t have distortion problems at high deflection angles. These coils can’t be driven at higher frequencies because they present an inductive load, but the refresh rate required for video is low enough for this not to matter for TVs. ↩
More information about the tube’s operation from a Naval training manual on digital computers! ↩

Immutability is not enough

Tue, 29 Mar 2016 10:00:00 +0000

Immutable data structures – also known as persistent or purely functional data structures – are a key part of modern functional programming. Once created, immutable data structures cannot be modified. Instead, they support operations which efficiently produce a new copy of the structure with the desired changes applied.

As I’ve gotten more experience working with immutable data structures, I’ve found that they can be beneficial in many ways. However, most programs that interact with the outside world must model state changes somehow. In these cases, I’ve noticed that I still need to be careful about state updates – even though I’m using immutable data structures. Surprisingly, many of the state update bugs that are rife in imperative programs can also occur in purely functional code.

To make it easier to discuss some of these issues, let’s use a concrete example. I’ve implemented a simple retro-style platform game¹ in JavaScript where you control a tiny carpenter named Manuel:

Here’s the code for an initial implementation which uses mutable state updates:

var canvasEl = document.querySelector('canvas');
var ctx = canvasEl.getContext('2d');

var state = {
  pos: {x: 200, y: 220}  // Manuel's position.
};

window.requestAnimationFrame(function render() {
  var pos = state.pos;

  // Clear the canvas
  canvasEl.width = 500;
  canvasEl.height = 300;

  // Draw the background
  ctx.fillStyle = '#008A00';
  ctx.fillRect(0, 250, 500, 50);

  // Process input
  if (dpad.right) {
    pos.x += 3;
  } else if (dpad.left) {
    pos.x -= 3;
  }

  // Draw Manuel
  ctx.fillStyle = '#AF8452';
  ctx.fillRect(pos.x + 5, pos.y - 10, 10, 10);
  ctx.fillStyle = '#C91B13';
  ctx.fillRect(pos.x, pos.y, 20, 20);
  ctx.fillRect(pos.x, pos.y + 20, 8, 10);
  ctx.fillRect(pos.x + 12, pos.y + 20, 8, 10);

  window.requestAnimationFrame(render);
});

For simplicity, I’ve left out the event handling implementation – it’s enough to know that dpad.left and dpad.right are boolean properties that indicate whether the left and right arrow keys are currently pressed.

As you can see, this code is not functional: the input handling code is updating the state object’s pos.x property in-place. You’ll also see that there are a few other places in the code where we refer to pos. At first glance, it’s hard to see all the spots where we’re reading and writing the state object. We can improve this a bit by factoring out the different concerns into separate functions:

var state = {
  pos: {x: 200, y: 220}  // Manuel's position.
};

window.requestAnimationFrame(function render() {
  clearCanvas();
  drawBackground(state);
  processInput(state);
  drawManuel(state);
  window.requestAnimationFrame(render);
});

That’s better. Now we can see which functions are state-dependent. However, we still don’t know what they do with the state. Do they modify it, or do they only read it? And are there ordering dependencies between the two operations?

Going functional

A purely functional approach will help us address some these problems. First, let’s make the state object immutable, and rename the variable to initialState. The root of the state becomes an immutable map with a single entry named ‘pos’, whose value is another map:

var initialState = Immutable.Map({
  pos: Immutable.Map({x: 200, y: 220})  // Manuel's position.
});

Since JavaScript doesn’t support immutable data structures by default, I’m using Immutable.js, a library which provides efficient, purely-functional versions of commonly-used data structures. The root of the state becomes an immutable map with a single entry named ‘pos’, whose value is another map.

Rather than updating the state directly (which is now impossible), the simplest change we can make to the helper functions is to have them return a new state value. This makes them side-effect free, and moves the burden of maintaining state into the higher-level code. For example, here’s what processInput looks like now:

function processInput(state) {
  var currX = state.getIn(['pos', 'x']);
  if (dpad.right) {
    return state.setIn(['pos', 'x'], currX + 3);
  } else if (dpad.left) {
    return state.setIn(['pos', 'x'], currX - 3);
  }
  return state;
}

Note that state.setIn(['pos', 'x'], <value>) is analogous to calling state.pos.x = <value>, except that it doesn’t modify the object. Instead, it returns a new copy of the state object with the pos.x field set to the desired value.

The next step is to compose the functions into a pipeline, so that state value returned from one function will be passed as an argument to the next:

var newState =
    drawManuel(
    drawBackground(
    processInput(state)));

The final step is to actually do something with the new state object. We can use a technique similar to tail recursion by adding a state parameter to the render function:

window.requestAnimationFrame(function render(state) {
...
});

…and using Function.prototype.bind to ensure that the state is passed as an argument when render is invoked on each animation frame:

window.requestAnimationFrame(function render(state) {
  var newState = ...;
  window.requestAnimationFrame(render.bind(null, newState));
}.bind(null, initialState));

The new, purely-functional render loop now looks like this:

window.requestAnimationFrame(function render(state) {
  clearCanvas();

  var newState =
      drawManuel(
      drawBackground(
      processInput(state)));

  window.requestAnimationFrame(render.bind(null, newState));
}.bind(null, initialState));

This is probably the most straightforward way to transform stateful code into a purely functional style. The old (mutable) state object is now an immutable object called initialState. On each invocation of render, the current state is passed through a function pipeline – each function takes the state as an argument, and returns a new state object to be passed to the next function. A function that only needs to read the state (like drawManuel) can simply return the same object it was passed. A function that needs to update the state can do so by returning a new object with the updated values.

Let’s take a moment to appreciate a few of the things we’ve gained by rewriting this code in a functional style:

At a glance, we can now see exactly which functions can read and write the state: only the ones in the pipeline that returns newState. A function like clearCanvas becomes easier to reason about, because we know that it can’t affect (or be affected by) the state.
The main components of the game now have a clearly-defined interface: the state object. This makes the components easier to test.

You can try the purely functional version here.

Adding collisions

Now, let’s make things a bit more interesting: let’s give Manuel an obstacle, and do some simple collision detection to prevent him from running into it. Here’s what the new world looks like with a boulder in it:

To prevent Manuel from colliding with the boulder, let’s write a simple collision detection function:

function handleCollisions(state) {
  var pos = state.get('pos');
  var boulderPos = state.get('boulderPos');
  // If Manuel is overlapping the boulder, move him.
  if (pos.get('x') + 20 > boulderPos.get('x')) {
    return state.setIn(['pos', 'x'], boulderPos.get('x') - 20);
  }
  return state;
}

…and add it to the pipeline:

var newState =
    drawManuel(
    drawBackground(
    processInput(
    handleCollisions(state))));

A bug

If you try this out, it appears to work. But when we write tests for it, we’ll discover that it’s still possible to render a frame where Manuel is occupying the same space as the boulder. Uh oh, what’s going on?

After a bit of debugging, we’ll probably figure out the problem: we’re doing collision detection first, and then moving Manuel into his new position based on the user’s input. It should be the other way around. By changing the order of the functions in the pipeline, we can ensure that collision detection runs after input processing, thus preventing the bug.

It turns out that our purely functional rendering code is sensitive to ordering in non-obvious ways. The first time I encountered this kind of bug, it felt strangely familiar – it’s something that often occurs in imperative programs. This is exactly the kind of problem that functional programming was supposed to help us avoid!

You could argue that the dependency between input handling and collision detection should be obvious. Yes, someone who has developed games before will probably catch this bug before it’s even written. But this is only a simple example – it’s easy to avoid state problems in small imperative examples too. The real problem comes when the codebase grows large enough that there are too many different dependencies to keep track of in any one programmer’s head. And as this example shows, the same problem can happen in purely functional programs as well.

Functional Design, Take 2

A savvy functional programmer will point out that there’s another way to write the rendering loop that would prevent this kind of ordering issue. Instead of each component passing along a new state value to the next component in the sequence, we could instead invoke every component with the same state value as an argument, as if they are all executing simultaneously. It would look something like this:

var updatedStates = [
  processInput(state),
  handleCollisions(state),
  drawBackground(state),
  drawManuel(state)
];
var newState = ...;

With this approach, we’ll quickly run into a problem. Both processInput and handleCollisions can modify Manuel’s position. If they each return a new state object, we’ll have to merge the two states somehow. It’s difficult to figure out how we could do that in a general way, because at this level, we have no idea what part of the structure each function has changed.

A solution to this is that instead of returning a new state object, each component could return a piece of data representing a desired state change. In terms of types, the functions would have type State -> [StateUpdate] instead of State -> State. Here’s what handleCollisions would look like:

function handleCollisions(state) {
  var pos = state.get('pos');
  var boulderPos = state.get('boulderPos');
  // If Manuel is overlapping the boulder, move him.
  if (pos.get('x') + 20 > boulderPos.get('x')) {
    return [{op: 'setIn', path: ['pos', 'x'], value: boulderPos.get('x') - 20}];
  }
  return [];  // No update.
}

Then, at the end of the render function, we can process the list of updates to produce the new state object to be passed to the next render call:

var updates = []
    .concat(processInput(state))
    .concat(handleCollisions(state))
    .concat(drawBackground(state))
    .concat(drawManuel(state));
var newState = applyUpdates(state, updates);

In applyUpdates, we simply need to go through the list of updates, and apply each one in turn to produce a new state object:

function applyUpdates(state, updates) {
  return updates.reduce(function(prevState, update) {
    switch (update.op) {
      case 'setIn':
        return prevState.setIn(update.path, update.value);
      default:
        throw new Error("Unknown op '" + update.op + "'");
    }
  }, state);
}

More bugs

Unfortunately, this solution has some other subtle bugs. Since all of the components are now seeing a consistent state value, it means they can’t observe each other’s changes until the next frame. This means if processInput puts Manuel into the same space as the boulder, it won’t be seen by handleCollisions until the next frame, after we’ve rendered an inconsistent state. Try this version out and you’ll see what I mean.

Additionally, this code is once again sensitive to subtle ordering issues. If two components will modify the same part of the state (e.g., Manuel’s position), then the final result will depend on the order in which we process the changes. Whichever component is processed last will overwrite any previous updates.

This is known as the “lost update” problem. It’s another issue that’s commonly encountered in imperative code with mutable state updates – and once again, we see that it does not automatically go away when we use immutable data structures.

What’s going on?

When I first encountered these kinds of bugs, I was confused – isn’t functional programming supposed to prevent these kinds of errors? How could I have state update bugs in a program without mutable state? After thinking about it for a while, I realized what was happening.

Programming is largely about building models – abstractions that have certain properties and obey a particular set of laws. Immutable data structures are one such abstraction. We use this abstraction (and others) to model the higher-level concepts which make up Manuel’s world.

Each level of the model has its own set of properties, and the properties at one level do not automatically extend to other levels. Our higher level model includes identity: there is a stable, logical entity that we refer to as Manuel, whose position changes from frame to frame. In a sense, we are modelling a mutable object. Implementing that model with immutable data structures does not eliminate the difficulties of dealing with state updates – it just shifts them to a different conceptual level.

As another example, imagine implementing a simple accounting ledger using a grow-only set. A grow-only set is an unordered set to which elements can be added, but never removed. This has the nice property that the order of operations doesn’t matter – if we have a $500 asset and a $200 liability, the balance is the same no matter what order the entries are added.

At this point, both the low-level model (the grow-only set) and the higher-level model (assets and liabilities) share the property that operations are commutative. But what happens if we decide to model asset depreciation? Naively, we might try just adding a new “depreciation” entry to the ledger once a month – but this would not be correct, because the meaning of the depreciation operation is dependent on the state of the ledger when the depreciation occurs. If we add a $1000 asset, then 12 months of depreciation, the final balance should be different than if the asset was acquired yesterday. The operations in the higher-level model are no longer commutative – it doesn’t matter that it’s implemented using a grow-only set.

What now?

After experiencing the kinds of bugs I described above, it became clear to me that – despite their merits – immutable data structures are not a silver bullet. Many programs still need additional support to safely and easily model state updates.

In the first version of the code, we had a bug that was caused by incorrect ordering between two of our components. In that case, all of our components were run in sequence, even when there was no ordering dependency between them. This is actually very similar to the problem with imperative languages – since everything runs sequentially, it’s hard to see what parts of the code have a true sequential dependency (as with our collision detection and input handling) and which parts do not.

In the second approach, we eliminated the sequential dependencies between components. But as we saw, this opened us a new class of errors. Some components do need to be run in sequence, and we had no easy way to detect that. Also, since each component saw the same version of the state, it was as if they were running in parallel. This made the code vulnerable to “lost update” errors, which (again) we had no easy way to detect.

A powerful idea: effect systems

Despite its downsides, the second solution has one big thing going for it: by representing our side effects as plain data, we have more flexibility in how to apply them. For example, we could detect lost updates by raising an error if there is more than one update for a particular location. This would show that there is an ordering dependency between two components that should be addressed by the programmer. This very similar to detecting serializability in databases.

Representing side effects as data is the core idea behind effect systems. In much the same way that type systems manage values, effect systems manage side effects. Most of the work on effect systems has been in pure functional languages, but the idea is more widely useful. Just like with types, you can imagine different kinds of effect systems. The system I hinted at in the previous paragraph could be considered a basic dynamic effect system. That is, we can’t tell what the effects are without executing the program. Most existing effect systems are static, in that effects can be determined at compile time. Other kinds of effect systems could be strong or weak, structural or nominal.

Although there’s a lot to be gained from purely functional techniques, state updates still pose a problem. Immutability is not enough: as I discovered, even purely functional programs are susceptible to state-related errors. Experienced programmers know how to avoid these problems, but it’s important enough to deserve a built-in language solution. I think that effect systems offer a promising approach.

Telling stories with data using the grammar of graphics

Wed, 16 Mar 2016 10:00:00 +0000

Different types of graphs may, at first glance, appear completely distinct. But in fact, graphs share many common elements, such as coordinate systems and using geometric shapes to represent data. By making different visual choices (Cartesian or polar coordinates, points or lines or bars to represent data), you can use graphs to highlight different aspects of the same data. For example, here are three ways of displaying the same data:

The pie chart focuses the reader on large percentages, and encourages the reader to think of the total (here, the cut of different diamonds) as a finite quantity that is being apportioned to different groups. The stacked bar plot provides the same information, but makes it easier to accurately gauge how large each category is. The histogram splits the categories horizontally, and draws attention to how the categories are ordered. It encourages the reader to think about the distribution rather than disconnected categories, and provides a sense of scale.

We often talk about types of graphs – bar plots, pie charts, scatterplots – as though they are unrelated, but most graphs share many aspects of their structure. We can think of graphs as visual representations of (possibly transformed) data, along with labels (like axes and legends) that make the meaning clear. Much like the grammar of a language allows you to combine words into meaningful sentences, a grammar of graphics provides a structure to combine graphical elements into figures that display data in a meaningful way. The grammar of graphics was originally introduced by Leland Wilkinson in the late 1990s, and was popularized by Hadley Wickham with ggplot, an R graphical library based on the grammar of graphics. I started using ggplot a few years ago. The syntax felt foreign for a long time, so I decided to learn about the theory behind the grammar, to get an intuition for the concepts underlying the code. Understanding this theory helped me understand ggplot and think more deeply about graphics, and I hope this introduction does the same for others. ggplot is the best-known implementation of the grammar, but thanks to its success, it is being implemented in a variety of languages, including Python, Julia, and D3, but I’ll be using R code for purposes of illustration here.

In the grammar of a language, words have different parts of speech, which perform different roles in the sentence. Analagously, the grammar of graphics separates a graphic into different layers. These are layers in a literal sense – you can think of them as transparency sheets for an overhead projector, each containing a piece of the graphic, which can be arranged and combined in a variety of ways. But what is in a layer? Let’s think about an example. Say we have a dataset with an independent variable, x, and a dependent variable, y. If we perform a simple linear regression, we can also calculate the predicted values for y at specific values of x (we can call these predictions y’). Using these data, we want to make a scatterplot with a line of best fit. What are the elements of this plot?

We have:

The data itself (x, y, and the best fit prediction, y’)
Dots on the scatterplot representing the relationship between x and y
The line representing the relationship between x and y’ (the line of best fit)
The scaling of the data (linear)
The coordinate system (Cartesian)

What if we want to make a histogram of the distribution of x? Then we have:

The data itself (x)
Bars representing the frequency of x at different values of x
The scaling of the data (linear)
The coordinate system (Cartesian)

Clearly there are many similar components between these graphs, and for most graphs, these elements do a pretty good job describing what a plot will look like. These are our “parts of speech”, the pieces of a graphic that we can use to tell a story. So let’s look at each one of them and try to understand what they do.

Data

Before it’s possible to talk about a graphical grammar, it’s important to know the format of the data you’re working with. After all, it contains all of the information you’re trying to convey. The grammar speaks in terms of data as “tidy” rows of individual observations. Here’s a sample of data in this format, taken from ggplot’s sample dataset diamonds.

##   carat       cut color clarity depth table price
## 1  0.23     Ideal     E     SI2  61.5    55   326
## 2  0.21   Premium     E     SI1  59.8    61   326
## 3  0.23      Good     E     VS1  56.9    65   327
## 4  0.29   Premium     I     VS2  62.4    58   334
## 5  0.31      Good     J     SI2  63.3    58   335
## 6  0.24 Very Good     J    VVS2  62.8    57   336

Here, each row represents observations of a single diamond. This seems like an obvious format, but not all datasets have this structure by default. Count data can be stored as a matrix. For example, you might imagine a matrix of locations and the number of birds spotted there:

##        cardinal blue jay chickadee
## site_1        5        0         1
## site_2        4        0         2

Here, each matrix element, rather than each row, represents a single observation. Using the package reshape2, we can transform the matrix into a format that is compatible with the grammar. In this case we transform the matrix into a list of observations and store the value in the new column count.

require(ggplot2)
require(reshape2)
reshape2::melt(birdmat, value.name = "count")

##     Var1      Var2 count
## 1 site_1  cardinal     5
## 2 site_2  cardinal     4
## 3 site_1  blue jay     0
## 4 site_2  blue jay     0
## 5 site_1 chickadee     1
## 6 site_2 chickadee     2

Run this code in r-fiddle

Other representations of data include summary tables, or the storage of different columns in different variables. ggplot is fairly picky about data formatting, but in return it gives us the power to make significant changes to how we are plotting the data without changing the data object itself. To understand the grammar of graphics, it helps to think of the one-observation-one-row dataset as a fixed entity that we can view in different ways.

Geoms

The most obvious part of a graph is the visual display of the data itself. This is often a basic geometric object like a point, line, or bar, so in ggplot, each of these elements is called a “geom”. You can display multiple pieces of information by layering geoms (scatterplot layer + line of best fit), or you can explore the same data by visualizing it with different types of geoms.

## fit a linear regression for the relationship between carat and price
## the "fitted" column here is calculating the predicted price for
## each value of carat.
diamonds$fitted <- lm(price ~ carat, data = diamonds)$fitted
## ggplot can actually fit simple models like a linear regression
## using geom_smooth(). Since this only works for a limited set of
## models, I prefer to do the model fitting outside of the plotting.
g <- ggplot(data = diamonds, aes(x = carat, y = price)) +
    geom_point(alpha=.3) + geom_line(aes(y = fitted)) +
    scale_y_continuous(limits = c(0, 20000))
plot(g)

Run this code in r-fiddle

## three ways of looking at the distribution of diamond price,
## conditional on the quality of the cut
g <- ggplot(diamonds, aes(x = cut, y = price, fill = cut))

## jittering the points helps prevent overplotting
gjitter <- g + geom_jitter()

## box and whiskers plot
gbox <- g + geom_boxplot()

## a fancier version of the boxplot, which shows the whole distribution,
## not just quantiles
gviol <- g + geom_violin()

Run this code in r-fiddle

Scaling

Sometimes it’s useful to transform or rescale data. Our eyes are good at seeing linear relationships, so if a relationship is log-linear, it makes sense to simply change the scale. Similarly, it is common to fit a regression line to log-transformed data, and it makes sense to plot this as a linear relationship, rather than plotting a curved fit on a linear scale. The same logic applies for other transformations. In the example below, the distribution looks close to unimodal (that is, it has a single peak) until we log-tranform it. I like to think of this as a transformation of our view of the data, rather than a transformation of the dataset itself. Thinking about the dataset as a fixed entity, it makes sense to apply transformations while plotting rather than altering the dataset itself.

# define linear histogram
g <- ggplot(data = diamonds, aes(x = price)) +
    geom_histogram()

# apply lograrithmic scale
g2 <- g + scale_x_log10()

Run this code in r-fiddle

Coords

We’re very used to thinking using the Cartesian coordinate system, but sometimes polar coordinates make sense. Pie charts are a common (albeit controversial) use of polar coordinates, and there are other flashy graphics that use them. You may also want to use a map projection, or flip the coordinates for a horizontal bar graph rather than a vertical one.

flip <- ggplot(data = diamonds, aes(x= cut, fill = cut)) +
    geom_histogram() +
    coord_flip()
plot(flip)

Run this code in r-fiddle

Polar coordinates can sometimes be misleading or confusing to the eye, but if your data are fundamentally cyclic rather than linear in nature, it’s a useful option to have.

Facets are a way to split data into subplots based on another factor in the data. In my opinion, facets are one of the most compelling reasons for using the grammar of graphics. Let’s say we’re looking at diamond carat versus diamond price. This is pretty easy to plot in most programming languages.

## base R
plot(diamonds$carat, diamonds$price, type = 'p')

## ggplot
g <- ggplot(diamonds, aes(x = carat, y = price)) + geom_point()
plot(g)

But we also have information about the cut. Bigger, heavier diamonds probably cost more, but not if they’re cut poorly. So we can create different plots for each cut, to tease out this relationship. In base R, you would have to write out separate commands for each cut. Basically, each cut category gets treated as an entirely separate dataset, and each plot as a separate unit. But using the grammar of graphics, the facet is simply another layer to apply to one dataset. Look at how much nicer the ggplot code (and output!) looks:

## base R
par(mfrow = c(2,3)) ## sets up multi-plot window
## plot each facet separately
## also notice that we have to specify the axes limits for x using
## xlim, or the plots would be differently scaled
plot(diamonds$carat[diamonds$cut == 'Fair'],
     diamonds$price[diamonds$cut == 'Fair'], type = 'p',
     xlim = c(0,5))
plot(diamonds$carat[diamonds$cut == 'Good'],
     diamonds$price[diamonds$cut == 'Good'], type = 'p',
     xlim = c(0,5))
plot(diamonds$carat[diamonds$cut == 'Very Good'],
     diamonds$price[diamonds$cut == 'Very Good'], type = 'p',
     xlim = c(0,5))
plot(diamonds$carat[diamonds$cut == 'Premium'],
     diamonds$price[diamonds$cut == 'Premium'], type = 'p',
     xlim = c(0,5))
plot(diamonds$carat[diamonds$cut == 'Ideal'],
     diamonds$price[diamonds$cut == 'Ideal'], type = 'p',
     xlim = c(0,5))

## ggplot
## we can just reuse 'g' from the previous example, and add facetting
## also note that ggplot keeps identical axis limits for all facets,
## because it's treating the facets as parts of a single dataset
g <- g + facet_grid(. ~ cut)
plot(g)

Run this code in r-fiddle

Now let’s say we also want to split the data by clarity. Having each cut-clarity combination get its own separate plot would be a lot of plots, so let’s color the points by the level of clarity. Base R treats each point color as a different set of points, and each subplot in the window as a separate unit. But color is like an adjective in the grammar: an aesthetic element which modifies the other layers.

## base R
## set up subplot window
par(mfrow = c(2,3))

## we can make things more concise by looping
for(i in levels(diamonds$cut)){
    ## color number for points
    colnum <- 1
    for(j in levels(diamonds$clarity)){
        ## for each new plot, the first call need to create the plot
        ## Otherwise we layer on points
        if(colnum == 1){
            plot(diamonds$carat[diamonds$cut == i & diamonds$clarity == j],
                 diamonds$price[diamonds$cut == i & diamonds$clarity == j],
                 type = 'p', col = colnum, xlim = c(0,5))
        } else {
            points(diamonds$carat[diamonds$cut == i & diamonds$clarity == j],
                   diamonds$price[diamonds$cut == i & diamonds$clarity == j],
                   col = colnum)
        }
        colnum = colnum + 1
    }
}

## ggplot
## The only change we need to make is to add an aesthetic in ggplot()
g <- ggplot(diamonds, aes(x = carat, y = price, colour = clarity)) +
    geom_point() + facet_grid(. ~ cut)
plot(g)

Run ggplot code in r-fiddle (the base R code is clunky enough that it doesn’t run well on r-fiddle)

Clearly, the base code is clunky and verbose next to the ggplot version. With a grammar to work with, we can communicate our intentions to ggplot in a clearer, more concise way. To anthropomorphize heavily, Base R graphics happily plots data, but it doesn’t “understand” anything about how data are structured. If you want to split your data into groups by a another variable, you have to tell it specifically what to do with each data chunk; it doesn’t understand what “split by another variable” means. But the grammar of graphics provides a common language between the computer and the user. ggplot understands the concept of a dataset, and how its rows and columns are related. It understands that a pie chart and a stacked bar chart are the same plot with different coordinate systems. It understands the idea of faceted plots as a single visual unit. And most importantly, it makes those ideas and relationships visible to the user, to make it simple to switch between different visual elements to represent the data.

Communication is the core idea at work here. When you have a structured language for graphics, it’s a lot easier to think and talk about them. It’s a great mental framework for when you’re trying to decide how to display data for yourself or for a presentation. If you aren’t sure how you want to display your data, it provides a concise, consistent way to move between different possible representations.

Making graphical choices

Choices of geometry matter. Points suggest that each data point is its own unit, independent of or distinct from the other points. Lines highlight the relationship between data points, rather than the points themselves. Similarly, bars, rectangles, and distributions emphasize that data points are parts of some larger category, and we are using the data to estimate the size or spread of that category.

This is why I find it useful to plot my data multiple ways before I present it to others. What do I learn from each plot? What information is excluded or difficult to see? How much detail am I providing, relative to what my audience needs to understand? What I find so powerful about grammar of graphics-style plotting is that, once my data are properly formatted, it is easy to slice, group, and facet my data in a variety of ways, swapping between geoms and aesthetics to explore my data. It’s not just a syntax but a different way to think about data, and a powerful tool for exploration and understanding.

Image Processing 101

Thu, 10 Mar 2016 09:00:00 +0000

Why this was written

At the Recurse Center, I spent some time teaching myself image processing. When I started, I had no idea what it entailed. I just knew that it could help me recognize text, shapes and patterns and to do interesting things with them.

My sources have mainly been Wikipedia pages, books and publicly available university lecture notes. As I became more familiar with the material, I wished for an ‘Image Processing 101’ article that could give anyone a gentle introduction to the world of image processing.

This is my attempt at writing that article.

Prerequisites

This article is designed for those who are comfortable with Python. No other previous knowledge is required, although some familiarity with numpy and matrix operations will be helpful.

Getting started

We’ll be using OpenCV for Python, Python 2.7¹, and iPython Notebook. For instructions on how to set up OpenCV on MacOS, there are instructions here.

All code and images used are available as a runnable iPython notebook on Github. You can also see the iPython notebook online.

I highly recommend learning image processing work on iPython notebook as it allows you to display images inline, which makes it really easy to get feedback on what your code is actually doing.

What is image processing?

Image processing is the process of manipulating or performing operations on images to achieve a certain effect (making an image grayscale, for example), or of getting some information out of an image with a computer (like counting the number of circles in it).

Image processing is also very closely related to computer vision, and we do blur the line between them a lot. Don’t worry too much about that – you just need to remember that we are going to learn about methods of manipulating images, and how we can use those methods to collect information about them.

In this article, I will go through some basic building blocks of image processing, and share some code and approaches to basic how-tos. All code written is in Python and uses OpenCV, a powerful image processing and computer vision library.

Building blocks

Let’s start off with the imports. We’re using cv2, numpy and a little bit of matplotlib (mostly as a convenient way of displaying images).

import cv2, matplotlib
import numpy as np
import matplotlib.pyplot as plt

Image format

Alright! Let’s get started. Firstly, we’ll need to read in images and to understand the format in which they are represented to us.

In OpenCV, images are represented as 3-dimensional Numpy arrays. An image consists of rows of pixels, and each pixel is represented by an array of values representing its color.

Given an image above, the array representation of it will be:

# read an image
img = cv2.imread('images/noguchi02.jpg')

# show image format (basically a 3-d array of pixel color info, in BGR format)
print(img)

Results:

[
    [[72 99 143]   [76 103 147]  [78 106 147]  ...,   [159 186 207]  [160 187 213]  [157 187 212]]
    [[74 101 145]  [77 104 148]  [77 105 146]  ...,   [160 187 208]  [158 186 210]  [153 183 208]]
    [[76 103 147]  [77 104 148]  [76 104 145]  ...,   [157 181 203]  [160 188 212]  [158 186 210]]
    ...,
    [[39 78 130]  [39 78 130]  [40 79 131]  ...,   [193 210 223]  [195 212 225]  [197 214 227]]
    [[32 71 123]  [32 71 123]  [32 71 123]  ...,   [198 215 228]  [200 217 230]  [200 217 230]]
    [[39 78 130]  [39 78 130]  [39 78 130]  ...,   [199 216 229]  [200 217 230]  [201 218 231]]
]

Where [72 99 143], etc., are the blue, green, and red (BGR) values of that one pixel. Note that OpenCV loads an image in BGR format by default. Matplotlib, however, reads in images as RGB. To display an image in matplotlib, we will need to convert the BGR format to RGB. I’ll let you figure out what happens when you forget to do the conversion before passing the image in to matplotlib.

# convert image to RGB color for matplotlib
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

# show image with matplotlib
plt.imshow(img)

Colors

Wait, hold up, hold up. What’s the reason for all this business with BGR and RGB?

Red, Green and Blue (RGB)

In the digital world, colors are commonly represented using the RGB color model. By this color model, red, green and blue light can be added together in various ways to produce a range of colors on the visible spectrum. Each one of thse colors are referred to as a channel. This works slightly differently than, say, mixing paint colors. In the RGB color model:

Image from Wikipedia ³

red + green = yellow
blue + green = cyan
red + blue = magenta
red + blue + green = white

I won’t go too much into the technicalities, but Wikipedia has a section on how the color combinations work.

On most systems, RGB values are represented as values ranging from 0 to 255, with higher values corresponding to a higher intensity of that color channel. For example, we can guess that [255, 51, 0] is probably a reddish color, because its R channel is the highest, and that [51, 102, 0] is probaby a greenish color, with the G channel being the highest.

Hue, Saturation and Value (HSV)

Another useful color model is the HSV color model. Instead of representing colors by the red, green and blueness of it, we represent them by the hue (where it is on the range of the rainbow), saturation (the ‘colorfulness’ of the color) and value (also known as brightness, or how much perceived light is coming out of it).

Image from Wikipedia ⁴

The HSV color model is especially useful when you want to think of an image’s color in either one of those channels, e.g. looking for parts of an image that fall in the blue hue range.

A variation of the HSV is the HSL color model, which consists of the hue, saturation and lightness. It is similar to HSV, but differs in the definition of saturation and the third channel (value vs lightness)².

Grayscale

We will also be working with grayscale images. Grayscale images only have one color channel on the scale of 0 to 255, representing the brightness of that pixel, with 0 being totally dark (black) and 255 being totally bright (white).

Converting the image we have into grayscale gives us this 2-dimensional array:

# convert image to grayscale
gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

# grayscale image represented as a 2-d array
print(gray_img)

Result:

[[109 113 115 ..., 189 192 191]
 [111 114 114 ..., 190 190 187]
 [113 114 113 ..., 185 192 190]
 ...,
 [ 89  89  90 ..., 212 214 216]
 [ 82  82  82 ..., 217 219 219]
 [ 89  89  89 ..., 218 219 220]]

I recommend playing around with a color picker to get a better idea of how colors vary as you change them around. It’s especially interesting to see how RGB values change as you modify one of the HSV channels.

Exercise:

Knowing this, we can now find the average color of an image! If we averaged each of the R, G and B channels, it will give us an RGB value that is the average pixel color. Here’s an example of how it can be done with np.average().

# find average per row, assuming image is already in the RGB format.
# np.average() takes in an axis argument which finds the average across that axis.
average_color_per_row = np.average(img, axis=0)

# find average across average per row
average_color = np.average(average_color_per_row, axis=0)

# convert back to uint8
average_color = np.uint8(average_color)
print(average_color)

Result:

[179 146 123]

In order to display a color on matplotlb, we need to create a small 100x100 pixel image populated with that RGB value.

# create 100 x 100 pixel image with average color value
average_color_img = np.array([[average_color]*100]*100, np.uint8)

plt.imshow(average_color_img)

What is the average color of the image?

Segmentation

When we’re trying to gather information about an image, we’ll first need to break it up into the features we are interested in. This is called segmentation. Image segmentation is the process representing an image in segments to make it more meaningful for easier to analyze³.

Thresholding

One of the simplest ways of segmenting an image is thresholding. The basic idea of thresholding is to replace each pixel in an image with a white pixel if a channel value of that pixel exceeds a certain threshold, and a black pixel if it doesn’t. We typically convert an image into a binary image, i.e. a single channel image. Grayscale images are examples of single channel images.


# threshold for image, with threshold 60
_, threshold_img = cv2.threshold(gray_img, 60, 255, cv2.THRESH_BINARY)

# show image
threshold_img = cv2.cvtColor(threshold_img, cv2.COLOR_GRAY2RGB)
plt.imshow(threshold_img)

Result:

This makes it easy to single out parts of the image that are of different brightness. This doesn’t only have to apply to grayscale versions of images. We can also segment out parts of images by color channels. Color thresholding works best with HSV. We talked about how HSV has a hue channel, that is, on the scale from red to green to blue to magenta, where does the color of the pixel lie?

Rather than finding the value that is under a threshold, we can find parts of the image with hues that lie within a range with cv2.inRange().

# open new Mondrian Piet painting photo
piet = cv2.imread('images/piet.png')
piet_hsv = cv2.cvtColor(piet, cv2.COLOR_BGR2HSV)

# threshold for hue channel in blue range
blue_min = np.array([100, 100, 100], np.uint8)
blue_max = np.array([140, 255, 255], np.uint8)
threshold_blue_img = cv2.inRange(piet_hsv, blue_min, blue_max)

threshold_blue_img = cv2.cvtColor(threshold_blue_img, cv2.COLOR_GRAY2RGB)

plt.imshow(threshold_blue_img)

Result:

Original image

Blue hue threshold

Exercise: How would we extract the red or yellow parts of this painting? What ranges do we use for reds or yellows if the entire hue of colors is represented on a scale of 0 to 255?

Masking with a binary threshold

Now that we can identify colors, we can do interesting things like using the binary image as a mask. A mask is a matrix of zero and non-zero values used for a bitwise operation. Masks can be used to cut, or ‘mask’, out certain sections of an image. A mask is usually a matrix of zeros (for the parts to exclude) and non-zeros (for the parts we want to keep).

Let’s say I would like a version of an image of an outdoor landscape that excludes the sky. We can first find the pixels that are within range of the blue hue, which will identify the blue-sky parts of an image. To get the parts of the image that are not the sky, we can inverse the values with a bitwise_not, which leaves us the parts that are not blue, giving us our mask. Performing a bitwise_and on that mask with an image will leave only the parts that are not blue.

upstate = cv2.imread('images/upstate-ny.jpg')
upstate_hsv = cv2.cvtColor(upstate, cv2.COLOR_BGR2HSV)
plt.imshow(cv2.cvtColor(upstate_hsv, cv2.COLOR_HSV2RGB))

# get mask of pixels that are in blue range
mask_inverse = cv2.inRange(upstate_hsv, blue_min, blue_max)

# inverse mask to get parts that are not blue
mask = cv2.bitwise_not(mask_inverse)
plt.imshow(cv2.cvtColor(mask, cv2.COLOR_GRAY2RGB))

# convert single channel mask back into 3 channels
mask_rgb = cv2.cvtColor(mask, cv2.COLOR_GRAY2RGB)

# perform bitwise and on mask to obtain cut-out image that is not blue
masked_upstate = cv2.bitwise_and(upstate, mask_rgb)

# replace the cut-out parts with white
masked_replace_white = cv2.addWeighted(masked_upstate, 1, \
                                       cv2.cvtColor(mask_inverse, cv2.COLOR_GRAY2RGB), 1, 0)

plt.imshow(cv2.cvtColor(masked_replace_white, cv2.COLOR_BGR2RGB))

Results :

Original image

Thresholded (mask)

Masked image

Blurring

Photographs can be quite noisy, which means there can be small irregularities that may get in the way of image segmenting. A common way to get rid of the noisy bits is to preprocess the image with a Gaussian blur. You can think of blurring as a way of smoothing out high intensities or drastic changes between pixels.

Gaussian blurs work by applying transformations to each pixel in an image. This is done by convolving an image with an n x n-sized kernel. You can think of a convolution as an act of applying an operation on a pixel depending to the values of the n x n pixels around it. The operation is defined by the kernel. So on a Gaussian blur with a 5 x 5 kernel, for every pixel, the 5 x 5 surrounding pixels are considered, and an averaging calculation is performed that will give us the pixels’ new, blurred color. The bigger the Gaussian kernel size, the more blurred the image will be.

img = cv2.imread('images/oy.jpg')

# gaussian blurring with a 5x5 kernel
img_blur_small = cv2.GaussianBlur(img, (5,5), 0)

Original image (600px by 450px)

Blur with 5x5 kernel

Blur with 15x15 kernel

Gaussian blurring is especially useful when you have a noisy image and would like to smooth over all of those irregularities before performing a thresholding.

Results:

Thresholded original image

Thresholded 5x5 kernel-blurred image

Above is an example of performing a threshold on the unblurred image versus the one blurred by a 5x5 kernel. Blurring gives us cleaner lines on our thresholded portions, which make them easier to work with.

Contours and Bounding Rectangles

Now that we have simplified, binary versions of those images, we can use them to identify features of interest. As an example, let’s look at how we can identify the individual coins in an image.

In this section, we will see how we can use contours and bounding rectangles to segment out features.

Original image

Preprocess image

First, we’ll convert the image to grayscale and will perform a Gaussian blur on it to simplify it and to remove noise. This is a common form of preprocessing, and if often the first step in working with an image.

Then, we’ll perform a binary threshold on the preprocessed image. Since the coins are on a light background, the threshold will pick up the lighter background as the feature of interest. We’ll invert the binary image to pick up the coins.

# get binary image and apply Gaussian blur
coins = cv2.imread('images/coins.jpg')
coins_gray = cv2.cvtColor(coins, cv2.COLOR_BGR2GRAY)
coins_preprocessed = cv2.GaussianBlur(coins_gray, (5, 5), 0)

# get binary image
_, coins_binary = cv2.threshold(coins_preprocessed, 130, 255, cv2.THRESH_BINARY)

# invert image to get coins
coins_binary = cv2.bitwise_not(coins_binary)

Results:

Preprocessed binary image

Find contours

Contours are curves joining all the continuous points that have the same color or intensity along a boundary. They’re useful for object or feature detection as well as shape analysis⁴. Using cv2.findContours(), we’ll find the contour of each coin. Passing in the cv2.RETR_EXTERNAL flag to the function returns only the external contours, so it won’t pick up contours for the smaller details on the coin surface.

From those contours, we’ll find the area of each, and filter out the ones that are too small to be coins. Images from real life are rarely perfect, and checks like this are often necessary to filter out noise and outliers. To obtain the area of the contour, we’re using cv2.contourArea().

# find contours
coins_contours, _ = cv2.findContours(coins_binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

# make copy of image
coins_and_contours = np.copy(coins)

# find contours of large enough area
min_coin_area = 60
large_contours = [cnt for cnt in coins_contours if cv2.contourArea(cnt) > min_coin_area]

# draw contours
cv2.drawContours(coins_and_contours, large_contours, -1, (255,0,0))

# print number of contours
print('number of coins: %d' % len(large_contours))

number of coins: 8

Results:

Contours of coins from binary image

Find bounding rectangles

A bounding rectangle is the smallest rectangle that can contain a contour. We can use them to segment out individual coins in our image. Do note that the cv2.boundingRect() method returns the bounding rectangle as x and y coordinates of the top left corner of the rectangle, and its width and height. We can also use bounding rectangles to crop out the eight individual coins.

# create copy of image to draw bounding boxes
bounding_img = np.copy(coins)

# for each contour find bounding box and draw rectangle
for contour in large_contours:
    x, y, w, h = cv2.boundingRect(contour)
    cv2.rectangle(bounding_img, (x, y), (x + w, y + h), (0, 255, 0), 3)

Results:

Bounding boxes of contours

Edge detection

Sometimes segmenting via color or intensity as we did with binary thresholding isn’t sufficient. What if we needed to segment out a multicolored object? Think about a blue and yellow striped bowl under non-uniform light – its color is not uniform throughout.

In comes edge detection, a way of finding edges in an image. Edges are defined as points in an image where there is a change in brightness or intensity, which usually means a boundary between different objects. Edge detection is a fundamental part of image processing and is often a starting point for detecting and working with features⁵.

There is quite a bit of math behind edge detection, but we won’t go into that here. The basic idea behind edge detection is that we can measure changes in the brightness of areas of an image, which we call the gradient. We can measure both the magnitude (how drastic the change is) and direction of a gradient. If the magnitude of change at a set of points exceeds a given threshold, then it can be considered an edge.

The Canny edge detection algorithm is a popular edge detection algorithm that produces accurate, clean edges. Below is an example of its OpenCV implementation in action, compared to a binary threshold of the same image. Note that the non-uniform lighting on the image makes it impossible to pick out both the bowl and cups with simple thresholding.

Original image

Binary threshold

As standard practice, we’ll preprocess the image with grayscaling and blurring before performing the thresholding. Note how the topmost teacup cannot be captured with simple thresholding. Even if we adjust the threshold value to be much higher to capture the teacup, it will exclude the bowl.

cups = cv2.imread('images/cups.jpg')

# preprocess by blurring and grayscale
cups_preprocessed  = cv2.cvtColor(cv2.GaussianBlur(cups, (7,7), 0), cv2.COLOR_BGR2GRAY)

# find binary image with thresholding
_, cups_thresh = cv2.threshold(cups_preprocessed, 80, 255, cv2.THRESH_BINARY)
plt.imshow(cv2.cvtColor(cups_thresh, cv2.COLOR_GRAY2RGB))

Results:

Binary image with threshold value of 80. Teacup not captured.

Binary image with threshold value of 200. Teacup captured, but not the bowl.

Canny edges

Instead of thresholding, let’s perform Canny edge detection on the image. Note that the cv2.Canny() function takes in two thresholds – the algorithm does what is called double thresholding. If the gradient magnitude is higher than threshold2, it is accepted as a strong edge. If it is lower than threshold2 but higher than threshold1, it will also be considered an edge, albeit a weak one if it is connected to another strong edge.

# find binary image with edges
cups_edges = cv2.Canny(cups_preprocessed, threshold1=90, threshold2=110)
plt.imshow(cv2.cvtColor(cups_edges, cv2.COLOR_GRAY2RGB))
cv2.imwrite('cups-edges.jpg', cups_edges)

Results:

Canny edges detected

Canny edge detection does a much better job at picking out features of an image that are otherwise not detected by a simple binary threshold.

Edges are pretty cool and make for interesting effects, but what can do with them?

Line and shape detection

If our objects of interest are of regular shapes like lines and circles, we can use Hough Transforms to detect them.

Line Detection

The Hough Line Transform works by coming up with a list of probable lines that points can be on, where each line is defined in polar coordinate terms of r and theta as r = x * cos (theta) + y * sin (theta). If a probable line has enough other points on it, then it is considered a line.

Here is an example of the Hough Line Transform in action.

# copy of image to draw lines
cups_lines = np.copy(cups)

# find hough lines
num_pix_threshold = 110 # minimum number of pixels that must be on a line
lines = cv2.HoughLines(cups_edges, 1, np.pi/180, num_pix_threshold)

for rho, theta in lines[0]:
    # convert line equation into start and end points of line
    a = np.cos(theta)
    b = np.sin(theta)
    x0 = a * rho
    y0 = b * rho

    x1 = int(x0 + 1000*(-b))
    y1 = int(y0 + 1000*(a))

    x2 = int(x0 - 1000*(-b))
    y2 = int(y0 - 1000*(a))

    cv2.line(cups_lines, (x1,y1), (x2,y2), (0,0,255), 1)

Results:

Edges

Straight lines detected from edges

Circle Detection

The Circle Hough Transform works similarly, but searches for all a, b, and r values that make for possible circles where circles are defined as ( x - a ) ^ 2 + ( y - b ) ^ 2 = r ^ 2. As they have extremely large search spaces, we should ideally set boundaries for the search space (e.g. setting minimum or maximum radius values).

Here is an example of the Circle Hough Transform in action:

# find hough circles
circles = cv2.HoughCircles(cups_edges, cv2.cv.CV_HOUGH_GRADIENT, dp=1.5, minDist=50, minRadius=20, maxRadius=130)
cups_circles = np.copy(cups)

# if circles are detected, draw them
if circles is not None and len(circles) > 0:
    # note: cv2.HoughCircles returns circles nested in an array.
    # the OpenCV documentation does not explain this return value format
    circles = circles[0]
    for (x, y, r) in circles:
        x, y, r = int(x), int(y), int(r)
        cv2.circle(cups_circles, (x, y), r, (255, 255, 0), 4)
    plt.imshow(cv2.cvtColor(cups_circles, cv2.COLOR_BGR2RGB))

print('number of circles detected: %d' % len(circles[0]))

number of circles detected: 3

Results:

Edges

Circles detected from edges

Note that only one circle is detected for the bowl, as we specified that the minimum distance, minDist between circles must be at least 50 pixels.

What next?

Cool! Now that you know some of the basics, you should hopefully be in a good place to start thinking about your own image processing projects. And even if you don’t have a project in mind, just playing around with the different OpenCV features with different parameters is great way to familiarize yourself with it further.

If you’d like to delve deeper to how these features work (e.g. how does edge detection actually happen?), implementing them in a language of your choice can be a rewarding learning experience. For example, I didn’t fully understand edge detection until I worked on a JavaScript implementation. Don’t let the math scare you! Reading pseudocode will often give you a pretty good understanding of the logic behind the algorithms.

Credits

Thank you to John Workman, a facilitator at the Recurse Center who got me excited about image processing, and to Jesse Gonzalez and Miriam Shiffman, who worked together on Set Solver, from which I learned many image processing techniques.

Photos used in this article are my own, unless stated otherwise.

Support for OpenCV on Python3 is still in beta at the time of this writing ↩
Wikipedia article on the HSL and HSV color model, explaining the difference between HSV and HSL. ↩
Definition taken from Wikipedia entry for Image Segmentation ↩
Definition taken from OpenCV documentation on contours ↩
Wikipedia article on Edge detection ↩

This just isn't functional

Mon, 07 Mar 2016 12:00:00 +0000

Functional is in vogue, especially on the frontend.

It turns out that if one is simply reacting to incoming information from some source of truth, using pure functions to transform that information into a UI is a very natural thing to do. With the increasing popularity of React, lots of programmers are finding themselves having to deal with functional constructs.

When writing functional JavaScript, a person will inevitably think: “Some sort of object-orientedness would be very useful for my program.” Maybe there is some code reuse happening. Maybe, just maybe, you have an inheritance situation on your hands; an honest-to-god taxonomy. If you have gone down this route, you may wind up thinking, “Oh, I now have to use the this keyword.”

Cold sweats. Do you really know what this is? What is it bound to? There are four different ways that this could have been bound. Do you, at this moment, know which applies to your program? The most terrible situation would be a this bound to the global scope. Who knows what has been mutated now?

You might think, “No, my function is well reasoned, and well defined.” Traditionally, in functional programming, bits of state that only exist locally in a function, and are not passed around to other functions, are okay. This is because local state avoids some of the problems associated with stateful code. You might think your constructor is doing just that, but if you are wrong about what kind of this you are dealing with, your local assignment, which was okay by functional programming standards, can suddenly mutate into something outside the local function scope.

Still not scared of this¹? Well, take a gander at THIS piece of code:

function Dog(){
    this.bark = function(){
        return "WOOF!";
    }
}

// Then a very bad person comes along
var cat = { purring: true };
Dog.bind(cat)();

cat.bark(); // WHAT HAVE WE WROUGHT

Chilling.

Those of you who have not yet been burned by this may not understand what is going on. Other than the global object, the previous function scope, or an object that can be passed in by new, the this object in a function can also be explicitly bound this way.

Someone has used the constructor function in a way that was not intended. Now, the little kitty can bark. It is tragic. The problem with using this in your constructor is that you need to trust that your users will be calling your function with new. This constructor is definitely not a pure function. It is possibly side-effecting. If your constructor is is not called with new, it can be mutating.

What alternatives exist?

// Let's try...
function Person(name){
    return {
        name: name,
        greet: function(other){
           return "Hello " + other.name + ", my name is " + name + ".";
        }
    };
}

var karen  = Person("Karen");
var tom = Person("Tom");

tom.greet(karen); // Hello Karen, my name is Tom.

// That's nice, but we can still do this...
tom.name = "Jeff";
tom.greet(karen); // Hello Karen, my name is Tom.
tom.name == "Jeff"; // true
// And now everything is inconsistent!

Our constructor does not use this, so it is safe from one form of side effect, but it still returns a piece of mutable state. To make matters worse, when it creates the greet function it uses a variable it closed over, which does not mutate. Our object is now meaningless.

You might now be thinking, “OK, let’s just use Object.fre–”

Let me stop you there. Object.freeze does not actually address all of our problems. In functional programming, the way one achieves change is not through in place mutation, but in the creation of a new object. This means that functional programming never deletes any old facts, it only creates new facts. When we change the name of our functional object by copying it and refreezing, we will not change the functions inside the object. All those functions will still only have access to the old version of the object through reference. In an immutable world, the coupling of functionality and data is extremely difficult to use because any functionality would have to be rebuilt any time the facts change.

So what do we do?

We use Protocols². A protocol is a namespace that provides you with a defined set of functions that can be called on anything that implements that protocol. A protocol checks an object’s type metadata at runtime, and selects the right behavior based on that type. Furthermore, a protocol is not directly attached to your object. To call a protocol, one must pass the object into the protocol. This means we do not have to update functionality whenever we create a new version of an object.

Now we can build ourselves a new tower of object-orientedness. A tower where nothing ever changes and where we are stuck, forever, immutably. It is the functional dystopia we’ve always wanted.

As a side note; this article uses a quick and dirty implementation of protocols. To follow along, copy the source code and paste it into a Node project. It is already packaged as a Node module. This article will not concern itself with implementation details for purposes of brevity and clarity. If you are interested in implementation details, check the source; it is well commented.

To define a protocol, we call Protocol(protocolName, [fns...]), where protocolName is the name of the protocol, and fns are the names of all the functions that an implementation of a protocol must define.

Consider two protocols: HypeRater and TemperatureRater. Something can be cool in a cool/unpopular way. Something can also be cool in a cool/warm way. We, human beings, can be cool and warm, or unpopular and cool, or cool and cool, or unpopular and warm. Much like someone can be, like, cool cool, someone can be, like, HypeRater.isCool, you know? Someone can be TemperatureRater.isCool, too, and those values can be different.

When we specify what kind of cool we’re talking about, we get a nifty thing called namespacing. Check out the code below.

var HypeRater        = Protocol("HypeRater", ["isCool", "isUnpopular"]);
var TemperatureRater = Protocol("TemperatureRater", ["isCool", "isWarm"]);

// Imagine we had a theSun object, which implemented both protocols
HypeRater.isCool(theSun, []); //=> true
TemperatureRater.isCool(theSun, []); //=> false

Protocols solve the problem of method clashing in object-oriented programming. An object with multiple base classes might wind up having to implement two equally named methods from those two base classes. Protocols also solve the duck typing problem of “Sure, this object has a .run function, but is it talking about the same kind of running that I need?”

Protocols require an Implementation. An implementation provides a function that implements each method specified in the protocol.

function isCool(thiz, args){
    return get(thiz, "temperature") < 15;
}
var tempImpl = Implementation({isCool: isCool, isHot: not(isCool)});

tempImpl is a valid implementation for any protocol with two methods, isCool and isHot. tempImpl implements these two functions by using the function isCool. This function takes two arguments, a record thiz and an array of arguments args. In the call TemperatureRater.isCool(theSun, [1,2,3]), theSun would be bound to thiz and [1,2,3] would be bound to args.

The final step in being able to use our protocols is to create a sort-of-like-a-class thing that is able to implement protocols. We will be using something called a Record³. A record is an immutable object that declares a set of required fields and protocol implementations.

To obtain a record constructor, one should call Record(typeName, fieldDeclarations, implementations). typeName is the name of your record, fieldDeclarations declare the fields all records of that type must define, and implementations is a map of protocol names to their implementations. This corrals a record into a very neat space. The fields which it is guaranteed to have are pretty well defined. The protocols for which it can be used are very well defined. Let us take a look at an example.

var Human = Record("Human", ["name", "temperature", "isTheFonz?"],
                   {TemperatureRater: tempImpl
                    HypeRater:{
                       isCool: isTheFonz,
                       isUnpopular: not(isTheFonz)}});

Our Human constructor is created by the Record call. We tell it that we have a "Human" for the name of the record. A Human has three fields, ["name", "temperature", "theFonz?"]. The Human implements two protocols, HypeRater and TemperatureRater. For the HypeRater, it provides its own implementation. For the TemperatureRater, it reuses the already defined tempImpl.

What are some of the properties of our records and what can we do with them? First things first: to create a new Human, we would call the function returned to us by our Record call, the function we have stored in our Human variable. Human(["Sonali", 22, true]) would create a new human, whose name is Sonali, whose temperature is 22, who is the Fonz (in spirit). How would we access her name?

That depends on implementation⁴. The implementation I have provided with this article creates records using Mori⁵. Mori provides us with persistent data structures. These data structures use something called structural sharing to make copies and updates at a very low cost. If we were using Object.freeze we would be copying the entire object every time we wanted to create an update. With persistent data structures, we share as much of the old data structure as possible.

Our use of persistent data structures mean we cannot use the traditional JavaScript forms of property access, because we are not using traditional JavaScript data structures. Thus, we will use assoc and get to interact with our object.

var mori  = require('mori');
var vec   = mori.vector; // mori vectors are like immutable arrays
var get   = mori.get;    // get allows us to search inside an object
var assoc = mori.assoc;  // assoc allows us to create new versions of our objects

var sonalisName = get(sonali, "name"); // sonalisName === "Sonali"
var sonalisTemp = get(sonali, "name"); // sonalisTemp === 22
var sonalisFonz = get(sonali, "theFonz?"); // sonalisFonz === true

// assoc will create a copy of sonali, where "name" is now equal to "Sal"
var sal = assoc(sonali, "name", "Sal");
var salsName = get(sal, "name"); // salsName === "Sal"
var sonalisName = get(sonali, "name"); // sonalisName === "Sonali"

As you can see, assoc is not mutative. It creates a copy. This means that you can feel free to pass copies of your objects around to anywhere in your program, and not fret about it getting mangled deep inside some callbacks. This property makes working with others a breeze.

One final advantage that records have over vanilla JavaScript objects is that records can have field declarations that are generated programmatically. Let’s try and implement a tic-tac-toe board.

When creating two-dimensional boards, the most common strategy is to create a two-dimensional array. This can make some things confusing. For one, when doing things like mapping across that array, we have to zip up and down in one direction. If we are just iterating across the array, we need to access two things. This means you would have to use two for loops to do stuff. In one, you would access a row, and in the other, you would dig inside the row for the proper cell. As human beings, we do not think about two dimensional boards in terms of “get row, and then dig inside the row.” We think about boards in terms of coordinates.

You might be asking me to wait. We can use a flat representation of a board instead, and by doing that, and utilizing modulos and other tricks, we can flatten a two dimensional array into a single dimension. If we try to use just one array, so that we can use a single for loop, we exacerbate the problem of representation; that is, we make our representation of our board very different from how we think about it in our minds so that we can utilize our old data structure. No one thinks about two dimensional boards as a single list of cells, where one can access things inside of of it using modulo.

var cells = ["top left","top middle","top right",
             "mid left","mid center","mid right",
             "bot left","bot center","bot right"];

var TicTacToe = Record("Tic Tac Toe", cells, {});
var myBoard = TicTacToe([0, 0, 0,
                         0, 0, 0,
                         0, 0, 0]);

// Here we create a new board, where we change the key
// vec("center", "middle") to hold an "X"
var myBoard2 = assoc(myBoard, "mid center", "X")
get(myBoard, "mid center") // "X"

That’s pretty beautiful! As you can see, we can access fields by getting the exact cell that we want. We access things in exactly one get. We can iterate over all the keys in the board my using mori.map(mori.keys(myBoard), ...) where ... represents some sort of function. No two for loops required. Getting a new object with something changed is just as easy. You do it in one fell swoop using assoc.

But that’s relatively simple.

Typing all that stuff out for tic-tac-toe was doable because we only had nine different entries, but what if I want a chess board? We can do it! Here, instead of building that array of cells by hand, we’ll generate it.

var rows = [1,2,3,4,5,6,7,8];
var cols = ["a", "b", "c", "d", "e", "f", "g"];
var cells = rows.map(function(row){
    return cols.map(function(col){
        return row + col;
    });
});

var ChessGame = Record("Chess Game", cells, {});
var myGame = ChessGame([]);
var myGame1 = assoc(myGame, "a1", "knight");
get(myGame1, "a1"); //=> "knight"

Well, I promised you a dystopic tower of well defined functional dispatch, and you’ve got it! While I do not recommend adopting it for any serious codebase (because this library is not tested or optimized), I encourage you to try it out for small projects. Good representation of data is a key aspect of getting lots of power out of functional programming, and without constructs like this, you might find functional programming in JavaScript to be relatively bulky because of all the copy-on-write involved. Feel free to check out the source code. If you like the style, ask library providers to give you these tools so that you can use it in your own codebase!

If not, maybe You Don’t Know JS ↩
Clojure Protocols ↩
Clojure Records ↩
As an alternative to the library we will use, check out Facebook’s Immutable.js ↩
Mori Documentation ↩

How to trick a neural network into thinking a panda is a vulture

Wed, 23 Dec 2015 09:00:00 +0000

Neural networks are magical

When I go to Google Photos and search my photos for ‘skyline’, it finds me this picture of the New York skyline I took in August, without me having labelled it!

When I search for ‘cathedral’, Google’s neural networks find me pictures of cathedrals & churches I’ve seen. It seems magical.

But of course, neural networks aren’t magic–nothing is! I recently read a paper, “Explaining and Harnessing Adversarial Examples”, that helped demystify neural networks a little for me.

The paper explains how to force a neural network to make really egregious mistakes. It does this by exploiting the fact that the network is simpler (more linear!) than you might expect. We’re going to approximate the network with a linear function!

It’s important to understand that this doesn’t explain all (or even most) kinds of mistakes neural networks make. There are a lot of possible mistakes! But it does give us some insight into one specific kind of mistake, which is awesome.

Before reading the paper, I knew three things about neural networks:

they can perform really well on image classification tasks (so I can search for “baby” and find that adorable picture of my friend’s baby)
everyone talks about “deep” neural networks on the internet
they’re composed of layers of simple (often sigmoid) functions, which are often illustrated like this¹:

Mistakes

The fourth and final thing I knew about neural networks was: sometimes they make REALLY RIDICULOUS mistakes. Spoilers from later in this post: these are a couple of images, and how the neural network classifies them. We can convince it that a black image is a paper towel, and that a panda is a vulture!

Now, none is this is too surprising to me–machine learning is my job, and machine learning habitually produces super weird stuff. But if we’re going to fix the super weird errors, we need to understand why they happen! We’re going to learn a little about neural networks, and then I’ll teach you how to trick a neural network into thinking a panda is a vulture.

Making our first prediction

We’re going to load a neural network, make some predictions, and then break those predictions. It’s going to be awesome. But first I needed to actually get a neural network on my computer.

I installed Caffe, which is neural network software written by some people at the Berkeley Vision and Learning Center at Berkeley. I picked it because it was the first one I could figure out, and that I could download a pre-trained network for. You could also try Theano or Tensorflow. Caffe has pretty clear installation instructions, which means it only took 6 hours of cursing before I got it to work.

If you want to install Caffe, I wrote up a procedure that will take you less than 6 hours. Just go to the neural-networks-are-weird repo and follow the instructions. Warning: it downloads approximately 1.5GB of data and needs to compile a bunch of stuff. These are the instructions to build it (3 lines!), which you can also find in the repository’s README:

git clone https://github.com/jvns/neural-nets-are-weird
cd neural-nets-are-weird
docker build -t neural-nets-fun:caffe .
docker run -i -p 9990:8888 -v $PWD:/neural-nets -t neural-nets-fun:caffe /bin/bash -c 'export PYTHONPATH=/opt/caffe/python && cd /neural-nets && ipython notebook --no-browser --ip 0.0.0.0'

This starts up an IPython notebook server on your computer where you can start making neural network predictions in Python. It should be running on port 9990 on localhost. If you don’t want to play along, that’s also totally fine. I included pictures in this article, too!

Once we have the IPython notebook up and running, we can start running code and making predictions! From here on I’m just going to post pretty pictures and small code snippets, but the full code and the gnarly details are in this notebook.

We’re going to use a neural network called GoogLeNet², which won the ILSVRC 2014 competition in several categories. The correct classification was in the network’s top 5 guesses 94% of the time. It’s the network that the paper I read uses. (If you want a cool read, you can see how a human can’t do much better than GoogLeNet. Neural networks really are pretty magical.)

First off, let’s classify an adorable kitten³ using our network:

Here’s the code to classify the kitten.

image = '/tmp/kitten.png'
# preprocess the kitten and resize it to 224x224 pixels
net.blobs['data'].data[...] = transformer.preprocess('data', caffe.io.load_image(image))
# make a prediction from the kitten pixels
out = net.forward()
# extract the most likely prediction
print("Predicted class is #{}.".format(out['prob'][0].argmax()))

That’s all! It’s just 3 lines of code. I also classified a cute dog!

It turns out that this dog is not a corgi, but the colors are very similar. This network already knows dogs better than I do.

What a mistake looks like (featuring the queen)

The most amusing thing that happened while doing this work is that I found out what the neural network thinks the queen of England is wearing on her head.

So now we’ve seen the network do a correct thing, and we’ve seen it make an adorable mistake by accident (the queen is wearing a shower cap ❤). Now… let’s force it to make mistakes on purpose, and get inside its soul.

Making mistakes on purpose

We’re going to need to do some math before we truly understand how this works, but first let’s look at some pictures of a black screen.

This completely black image is classified with probability 27% as velvet, and 4% as a paper towel. There are some other probabilities I haven’t displayed, and they add up to 100%.

I would like to figure out how to make the neural network more confident that this is a paper towel.

To do that, we need to calculate the gradient of the neural network. This is the derivative of the neural network. You can think of this as a direction to take to make the image look more like a paper towel.

To calculate the gradient, we first need to pick an intended outcome to move towards, and set the output probability list to be 0 everywhere, and 1 for paper towel. Backpropagation is an algorithm for calculating the gradient. I originally thought it was totally mystical, but it turns out it’s just an algorithm implementing the chain rule. If you want to know more, this article has a fantastic explanation of backpropagation.

Here’s the code I wrote to do that–it’s actually really simple to do! Backpropagation is one of the most basic neural network operations so it’s easily available in the library.

def compute_gradient(image, intended_outcome):
    # Put the image into the network and make the prediction
    predict(image)
    # Get an empty set of probabilities
    probs = np.zeros_like(net.blobs['prob'].data)
    # Set the probability for our intended outcome to 1
    probs[0][intended_outcome] = 1
    # Do backpropagation to calculate the gradient for that outcome
    # and the image we put in
    gradient = net.backward(prob=probs)
    return gradient['data'].copy()

This basically tells us what kind of shape the neural network is looking for at that point. Since everything we’re working with can be represented as an image, here it is–the output of compute_gradient(black, paper_towel_label), scaled up to be visible.

Now, we can either add or subtract a very light version of that from our black screen, and make the neural net think our image is either more or less like a paper towel. Since the image we’re adding is so light (the pixel values are less than 1/256), the difference is totally invisible. Here’s the result:

Now the neural network thinks our black screen is a paper towel with 16% certainty, instead of 4%! That’s really neat. But, we can do better. Instead of taking one step in the direction of a paper towel, we can take ten tiny steps and become a little more like a paper towel every step. You can see the probabilities evolve over time here. You’ll notice the probabilities are different than before because our step sizes are different (0.1 instead of 0.9).

and the final result:

And here are the pixel values that make up that image! They all started out at exactly 0, and you can see that we’ve mutated them to trick the network into thinking the image is a paper towel.

We can also multiply the image by 50 to get a better sense of what it looks like.

It doesn’t look like a paper towel to me, but maybe it does to you! I like to imagine all the swirls in that image are tricking the neural network into thinking it’s a paper towel roll. So, that’s the basic proof of concept, and some of the math. We’re going to go more into the math in a second, but first we’re going to have fun.

Having fun tricking the neural network

Once I figured out to do this, it was SO FUN. We can change a cat into a bath towel:

a garbage can into a water jug / cocktail shaker:

and a panda into a vulture. What better way to spend a Sunday morning?

This image showing how the probabilities progress from panda to vulture over 100 steps is pretty rad:

You can see the code to make all this work in the IPython notebook. It’s so fun.

Now, it’s time for a little more math.

How it works: logistic regression

First, let’s talk about the simplest way to classify an image–with logistic regression. What’s logistic regression? I’ll try to explain.

Suppose you have a linear function, for classifying whether an image was a raccoon or not. How would we use that linear function? Well, suppose your image had only 5 pixels in it. $(x_1, x_2, x_3, x_4, x_5)$ , between 0 and 255. Our linear function would have weights, for instance $(23,-3,9,2,-5)$ , and then to classify an image, we’d take the inner product of the pixels and the weights:

\text{result} = 23x_1 - 3x_2 + 9x_3 + 2x_4 - 5x_5

Now, suppose the result is 794. Does 794 mean it’s a raccoon or it’s not? Is 794 a probability? 794 IS NOT A PROBABILITY. A probability is a number between 0 and 1. Our result is a number between $-\infty$ and $\infty$ . The way people normally get from a number between $-\infty$ to $\infty$ to a probability is by using a function called the logistic function: $S(t) = \frac{1}{1 + e^{-t}}$

which looks like this:

S(794) is basically 1, so if we got 794 out of our raccoon weights, we’d determine that it’s 100% a raccoon. This model–where we first transform our data with a linear function and then apply the logistic function to get a probability–is called logistic regression, and it’s a super simple and popular machine learning technique.

The “learning” part of machine learning is how to determine the right weights (like $(23,-3,9,2,-5)$ ) given a training dataset so that our probabilities can be as good as possible. The bigger the training set, the better!

Now that we understand what logistic regression is, let’s talk about how to break it!

Breaking logistic regression

There’s a gorgeous blog post called Breaking Linear Classifiers on ImageNet by Andrej Karpathy, which explains how to break a simple linear model beautifully (it’s not quite a logistic regression, but it is a linear model). We’re going to use the same principles later to break neural networks.

Here’s an example (from Karpathy’s article) of some linear classifiers for various foods, flowers, and animals, visualized as images. (click to make it bigger)

You can see that the “Granny Smith” classifier is basically asking “is it green?” (not the worst way to find out!) and the “menu” one has discovered that menus are often white. Karpathy explains this super clearly:

For example, Granny Smith apples are green, so the linear classifier has positive weights in the green color channel and negative weights in blue and red channels, across all spatial positions. It is hence effectively counting the amount of green stuff in the middle.

So, if I wanted to trick the Granny Smith classifier into thinking I was an apple, all I’d have to do is

figure out which pixels it cares about being green the most
tint those green
profit!

So now we know how to trick a linear classifier. But a neural network isn’t linear at all–it’s highly nonlinear! Why is this even relevant?!

How it works: neural networks

I’m going to be totally honest here: I’m not a neural networks expert, and my neural networks explanations are not going to be great. Michael Nielsen has written a book called Neural Networks and Deep Learning which is very well-written. I’m also told that Christopher Olah’s blog is quite good.

What I do know about neural networks is: they’re functions. You put in an image, and you get out a list of probabilities, one for each class. Those are the numbers you’ve been seeing next to the images throughout this article. (is it a dog? nope. A shower cap? nope. A solar dish? YES!!)

So a neural network is, like, 1000 functions (one for each probability). But 1000 functions are really complicated to reason about. So what neural networks people do is, they combine the 1000 probabilities into a single “score”. They call this the “loss function”.

This loss function for each image depends on the actual correct output for the image. So suppose that I have a picture of an ostrich, and the neural network has output probabilities $p_j$ for $j = 1 \ldots 1000$ , but the probabilities I WANT for the ostrich are $y_j$ . Then the loss function is

L = - \sum_j y_j \log p_j

And let’s say the label corresponding to ‘ostrich’ is number 700. Then $y_{700} = 1$ , all the rest of the $y_j$ are 0, and $L = - \log p_{700}$ .

The important thing here is to understand is that a neural network gives you a function from an image (the panda) to a final value of the loss function (a number, like 2). Because it’s a single-valued function, taking the derivative (or gradient) of that function gives you another image. You can then use that image to trick the neural network, using what we’ve talked about in this article!

Breaking neural networks

Here’s how all our words about breaking a linear function / logistic regression relate to neural networks! This is the math you’ve been waiting for! Near our image (an adorable panda), the loss function looks like

L(x + \text{delta}) \approx L(x) + \text{delta} \cdot \text{grad}

where the gradient grad is $\nabla L(x)$ . This is because of calculus. In order to change the loss function by a lot, we want to maximize the dot product of the direction we’re moving delta and the gradient grad. Let’s calculate grad via our compute_gradient() function, and draw it as a picture:

The intuition behind what we want to do is–we want to create a delta which emphasizes the pixels in the picture that the neural network thinks are important. Now, let’s suppose grad was $(-0.01, -0.01, 0.01, 0.02, 0.03)$ .

We could make $\text{grad} \cdot \text{delta}$ big by taking $\text{delta} = (-1, -1, 1, 1, 1)$ , to get 0.08. Let’s try that! In code, that’ll be delta = np.sign(grad). And when we move it by that amount, sure enough – now the panda’s a weasel.

But, why? Let’s think about the loss function. We started out by looking at the probability it was a panda, which was 99.57%. $-\log(0.9957) = 0.0018$ . Pretty small! So adding a multiple of delta should increase our loss function (and make it less like a panda), and subtracting a multiple of delta should decrease our loss function (and make it more like a panda). But actually the opposite happens! I’m still confused about this.

The dog you can’t trick

Now that we know the math, a short story. I also tried to trick the network about this adorable dog from before:

But the dog was highly resistant to being classified as something other than a dog! I spent some time trying to convince it that the dog was a tennis ball but it remained, well, a dog. Other kinds of dogs! But still a dog.

I ran into Jeff Dean (who works on neural nets at Google) at a conference, and asked him about this. He told me that there are a ton of dogs in the training set for this network, way more than there are pandas. So he hypothesized that the network is better-trained to recognize dogs. Seems plausible!

I think this is pretty cool and it makes me feel more hopeful about training networks that are more accurate.

One other funny thing about this conversation–when I tried to trick the network into thinking a panda is a vulture, it spent a while thinking it was an ostrich in the middle. And when I asked Jeff Dean this question about the panda and the dog, he mentioned off-hand “the panda ostrich space” without me bringing up ostriches at all! I thought it was cool that he’s clearly spent enough time with the data & these neural networks to just know offhand that ostriches and pandas are somehow close together.

Less magic

When I started doing this, I didn’t know almost anything about neural networks. Now that I’ve tricked them into thinking a panda is a vulture and seen how it’s smarter at dealing with dogs than pandas, I understand them a tiny bit better. I won’t say that I don’t think what Google is doing is magic any more–I’m still pretty confused by neural nets. There’s a lot to learn! But tricking them like this removes a little of the magic, and now I feel better about them.

And so can you! All the code to reproduce this is in this neural-networks-are-weird repo. It uses Docker so you can get it installed easily, and you don’t need a GPU or a fancy computer. I did all this on my 3-year-old GPU-less laptop.

And if you want to feel even better, read the original paper! (“Explaining and Harnessing Adversarial Examples”). It’s short, well written, and tells you more that I didn’t go into here, including how to use this trick to build better neural networks!

Thanks to Mathieu Guay-Paquet, Kamal Marhubi, and others for helping me with this article!

Neural network image by Glosser.ca - CC BY-SA 3.0 ↩
Pre-trained version of GoogLeNet, amazingly helpfully provided by the Berkeley vision lab ↩
By Riannacone (Own work) CC BY-SA 3.0, wikipedia link ↩

What RESTful actually means

Sat, 19 Dec 2015 09:00:00 +0000

If you do web development, you’ve probably heard of REST. But if you’re like me, you usually just pretend to know what it is, and nod politely when someone asks you if what you’re making is RESTful. I use HTTP, err, that means it’s RESTful right? Recently, I decided to take the plunge and actually find out what this peaceful-sounding buzzword means.

What is REST?

REST stands for Representational State Transfer. It is a set of design principles for making network communication more scalable and flexible. These principles answer a number of questions. What are the components of the system? How should they communicate with each other? How do we ensure we can swap out different parts of the system at any time? How can the system be scaled up to serve billions of users?

Roy T. Fielding first coined the term REST in 2000, in his PhD dissertation Architectural Styles and the Design of Network-based Software Architectures. At the time of the dissertation’s publication, the World Wide Web (or “the Web” for short) was already very popular. Fielding essentially took a step back and analyzed the traits that made the Web more successful than competing Internet protocols. He then envisioned a design framework for making network communication “web-like.” So REST is a generic set of principles not specific to the Web. It can be applied to other kinds of networks such as embedded systems. REST is also not a protocol since it does not prescribe implementation details.

The Fielding Constraints

Fielding’s dissertation outlines a number of architectural constraints that a system must satisfy to be considered RESTful. Below, I’ll give a brief overview of each of these constraints and discuss how the Web satisfies them using its core technologies: HTTP, HTML, and URIs.¹(If you’re unfamiliar with URI, just think “URL”. There is a distinction, but it is not important to our discussion.) Let’s take a look at the Fielding Constraints one by one.

Client-server

The first Fielding Constraint specifies that the network must be made up of clients and servers. A server is a computer that has resources of interest, and a client is a computer that wants to interact with the resources stored on the server. When you browse the Internet, your computer is acting as a client and sends HTTP requests to a server in order to access and manipulate information. A RESTful system has to operate in the client-server model, even if a component sometimes acts like a client and sometimes acts like a server.

A non-RESTful alternative to client-server architecture is event-based integration architecture. In this model, each component continuously broadcasts events while listening for pertinent events from other components. There’s no one-to-one communication, only broadcasting and eavesdropping. REST requires one-to-one communication, so event-based integration architecture would not be RESTful.

Stateless

Stateless does not mean that servers and clients do not have state, it simply means that they do not need to keep track of each other’s state. When a client is not interacting with the server, the server has no idea of its existence. The server also does not keep a record of past requests. Each request is treated as a standalone.

Uniform interface

The “uniform interface” constraint ensures that there is a common language between servers and clients that allows each part to be swapped out or modified without breaking the entire system. This is achieved through 4 sub-constraints: identification of resources, manipulation of resources through representations, self-descriptive messages, and hypermedia.

Interface constraint 1: identification of resources

The first sub-constraint of “uniform interface” affects the way that resources are identified. In REST terminology, a resource could be anything – an HTML document, an image, information about a particular user, etc. Each resource must be uniquely identified by a stable identifier. A “stable” identifier means that it does not change across interactions, and it does not change even when the state of the resource changes. If a resource does get moved to another identifier, the server should give the client a response indicating that the request was bad, and give it a link to the new location of the resource.

The Web uses URI to identify resources, and HTTP as its communication standard. To get a resource stored on a server, a client makes a HTTP GET request to the URI that identifies that resource. Every time you type an address into your browser, your browser makes a GET request to that URI. If it receives a 200 OK response and an HTML document back, then it renders the page in the window so that you can view it.

Interface constraint 2: manipulation of resources through representations

The second sub-constraint of “uniform interface” says that the client manipulates resources through sending representations to the server–usually a JSON object containing the content that it would like to add, delete, or modify. In REST, the server has full control of the resources, and is responsible for making any changes. When a client wishes to make changes to resources, it sends the server a representation of what it would like the resulting resource to look like. The server takes the request as a suggestion, but still has ultimate control.

Let’s think about the case of a blog on the Web. When a user makes a new blog post, their computer wants to tell the server that a new blog post needs to be added. To do this, it sends an HTTP POST or PUT request with the content for the new blog post. The server sends back a response indicating whether the post was created, or if there was a problem. In a non-REST world, the client may literally be sending instructions for operations such as add a new line and make the title of the blog “What RESTful Actually Means”, instead of simply sending a representation of what it would like the final product to look like.

Interface constraint 3: self-descriptive messages

Self-descriptive messages are another constraint that ensures a uniform interface between clients and servers. A self-descriptive message is one that contains all the information that the recipient needs to understand it. There should not be additional information in a separate documentation or in another message.

To understand how this applies to the Web, let’s analyze a set of HTTP requests and responses.

When a user types http://www.example.com in the address bar of their web browser, the browser sends the following HTTP request:

GET / HTTP/1.1
Host: www.example.com

This message is self-descriptive because it told the server what HTTP method was used, and the protocol that was used (HTTP 1.1).

The server may send back a response like this:

HTTP/1.1 200 OK
Content-Type: text/html

<!DOCTYPE html>
<html>
  <head>
    <title>Home Page</title>
  </head>
  </body>
    <div>Hello World!</div>
    <a href= “http://www.recurse.com”> Check out the Recurse Center! </a>
    <img src="awesome-pic.jpg">
  </body>
</html>

This message is self-descriptive because it told the client how it needs to interpret the message body, by indicating that Content-type was text/html. The client has everything it needs in this single message to handle it appropriately.

Imagine an alternative method of communication where the server sent binary code in one response, and then sent a seperate response telling the client how to interpret the binary code–whether it is an image or a code snippet. If the two responses had somehow arrived out of order, or if a third response was inserted between them, then the client may be confused about how to interpret the responses properly.

Interface constraint 4: hypermedia

The last interface constraint is the hypermedia constraint. Hypermedia is a fancy word for data sent from the server to the client that contains information about what the client can do next–in other words, what further requests it can make. In REST, servers should be sending only hypermedia to clients.

HTML is a type of hypermedia. To understand this better, let’s look again at the server response above. * <a href= “http://www.recurse.com”> Check out the Recurse Center! </a> tells the client that it should make a GET request to http://www.recurse.com if the user clicks on the link. * <img src="awesome-pic.jpg"> tells the client to immediately make a GET request to http://www.example.com/awesome-pic.jpg so it can display the image to the user.

When a system has identifiers for each resource, manipulates them through sending representations from the client to the server, and has messages that are self-descriptive and composed of hypermedia, it is said to have a uniform interface. This is perhaps the most important attribute of a RESTful system, as it allows for clients to intelligently adapt to changes. A server can change the underlying implementation without breaking all the clients that interacted with it, because each interaction is self-contained, identifiers do not change when the underlying state or implementation changes, and hypermedia gives clients instructions for state transitions that it can do next. The server does not need to remember anything about the client or do anything special to cater to it, and vice versa.

Other Fielding Constraints: caching, layered system, and code on demand

There are three more Fielding Constraints, which we’ll explore briefly here.

Caching refers to the constraint that server responses should be labelled as either cacheable or non-cacheable. Caching occurs when the client stores previous responses it received from the server, so that when that data is needed again, it can save a round trip over the network by using the cached data. The ability to cache is made possible by the interface constraint of “self-descriptive messages”, since the client knows that all the relevant data about a single resource is being sent in one response. It doesn’t have to worry about accidentally only caching part of the information it needs, and missing other parts.

Layered system refers to the fact that there can be more components than just servers and clients. This means there can be more than one layer in the system. However, each component is constrained to only see and interact with the very next layer. A proxy is an additional component, and it relays HTTP requests to servers or other proxies. Proxies can be useful for load balancing and security checks. A proxy acts like a server to the initial client that sends the request, and then acts like a client when it relays that request. A gateway is another additional component and it translates an HTTP request into another protocol, propagates that request, and then translates the response it receives back into HTTP. A client can simply treat a gateway as a regular server. An example use case for gateways is a system that needs to download a file from an FTP server.

Code on demand is the only optional constraint and refers to the ability for a server to send executable code to the client. This is what happens in HTML’s <script> tag. When the HTML document is loaded, the browser automatically fetches the JavaScript from the server and executes it locally.

In summary, a RESTful system is any network that fulfills the Fielding Constraints. A RESTful system is meant to be flexible for different use cases, scalable to accommodate a large number of users and components, and adaptable over time. We explored how the World Wide Web is RESTful when it comes to interactions involving browsers and human users. Designing RESTful Web APIs for machine-to-machine interactions is a much more complex topic; see the resources listed below for more information. Remember that REST is a theoretical design, and the Web we have today still falls short in some cases.

Next time, you no longer have to pretend to know what RESTful actually means.

Building a web framework from scratch in Elixir

Wed, 16 Dec 2015 09:00:00 +0000

Elixir is a fantastic new functional programming language that targets the Erlang VM, designed to build fault-tolerant distributed systems. There are numerous exciting use-cases, but one that always gets a lot of attention is building web applications. Elixir already has a Rails-like framework called Phoenix, but today, we’ll use a much more simple library, called Plug, to write a Sinatra-like web framework from scratch.

Why you should care

Phoenix is a great framework, and actually uses Plug under the hood. It works well with other Plug-compatible libraries, but tends to put these features behind macros, obfuscating the direct calls to Plug functions. I think that understanding the way the underlying Plug library works is an excellent way to learn both Elixir and Phoenix, both because it helps you understand what Phoenix’s macros are doing under the surface, and also because writing performant code is incredibly simple and easy with just Plug, Elixir, and Erlang’s pattern-matching. We could just browse through Phoenix’s macro definitions, but those are difficult to understand, since there’s a lot of extra code to make the DSLs work properly. Instead, we’ll write straightforward code ourselves, using Plug directly to build our own framework from scratch! Who knows, maybe you’ll even find you prefer the simple router that we build.

Following along

If you are already familiar with the basics of Elixir’s syntax, that’s great, though not required. I’ll do my best to explain the trickier parts in more depth in case you aren’t, but if you get stuck, try taking a look at the Elixir Crash Course. You don’t need to know anything about Phoenix, although I’m assuming you’ve programmed a web app before in some language or framework, and won’t explain simple concepts like headers and query strings.

To follow along with this guide, you’ll need Elixir installed, and you’ll also need to create a new project with mix new helloplug. Then, add Cowboy, Plug, Ecto, and Sqlite.Ecto to your project’s mix.exs file by adding the following lines:

def deps do
  [{:cowboy, "~> 1.0.0"},
   {:plug, "~> 1.0"},
   {:sqlite_ecto, "~> 1.0.0"},
   {:ecto, "~> 1.0"}]
end
def application do
  [applications: [:logger, :sqlite_ecto, :ecto, :cowboy, :plug]]
end

Don’t forget to run mix deps.get when you’re done! Once the Plug is set up properly, you can write any code in the lib/helloplug.ex file, although if you’re comfortable with it, Elixir’s standard directory structure¹ is cleaner.

What is a Plug?

A Plug is a module that responds to web requests. To create one, we just need two functions, init/1² and call/2.

defmodule Helloplug do
  def init(default_opts) do
    IO.puts "starting up Helloplug..."
    default_opts
  end

  def call(conn, _opts) do
    IO.puts "saying hello!"
    Plug.Conn.send_resp(conn, 200, "Hello, world!")
  end
end

init/1 is called once when the server is started, and call/2 is called every time a new request comes in. There are two arguments to call/2.

conn: This is a Plug.Conn, the connection with the client. It contains information about the request. We also use it to send responses.
opts: This is whatever the output of our init/1 function was. It doesn’t change from request to request.

Any options passed to the module are given to init/1. We’ll take a look at that later, but for now, just know that whatever init/1 returns gets passed to every subsequent call/2.

Finally, let’s take a look at send_resp. This is a function that accepts three arguments: the connection, the HTTP status code to send, and the body of the reply to send. You’ll note that this is the last line of call/2, and this is intentional. In Elixir, the last line of a function is implicitly returned. send_resp, as well as the rest of the Plug.Conn functions, returns a mutated copy of conn, and it’s important that call/2 returns this mutated conn so that the outside function calling it knows what has changed.

We can connect Helloplug to cowboy, the web server, by running iex -S mix in the project directory, and typing {:ok, _} = Plug.Adapters.Cowboy.http Helloplug, [] into the resulting Elixir REPL.³ If you browse to http://localhost:4000 a few times, you should see “Hello, world!” in your browser, and this in the terminal:

starting up Helloplug...
saying hello!
saying hello!
saying hello!

The call function is run every time we visit the web page. We can also use the conn object to set headers.

def call(conn, _opts) do
  IO.puts "saying hello!"
  conn |> Plug.Conn.put_resp_header("Server", "Plug") |> Plug.Conn.send_resp(200, "Hello, world!")
end

In case you haven’t seen the |> operator before, A() |> B() |> C() is equivalent to C(B(A)). We could rewrite the line above if we wanted a less idomatic and more verbose version:

conn2 = Plug.Conn.put_resp_header(conn, "Server", "Plug")
Plug.Conn.send_resp(conn2, 200, "Hello, world!")

If there are arguments passed to any of the functions, the output of the previous function is inserted before the specified arguments. So conn |> Plug.Conn.put_resp_header("Server", "Plug") is equivalent to Plug.Conn.put_resp_header(conn, "Server", "Plug"). Also, remember that the last statement of a function in Elixir is implicitly returned.

put_resp_header and send_resp are two examples of functions that manipulate connections. If you’ve done some web programming before, you’ll likely recognize what they do—they add a header to the response Server: Plug and then send the response “Hello, world!” with status code 200 OK. Neither of these functions can modify conn directly, since all variables and values in Elixir are immutable. That’s why we can’t have this:

# This is broken!
Plug.Conn.put_resp_header(conn, "Server", "Plug")
Plug.Conn.send_resp(conn, 200, "Hello, world!")

put_resp_header doesn’t touch the conn passed to it, but it does return a duplicated, modified version of the conn. We need the mutated conn returned from put_resp_header to go into send_resp, which is why we chain the output from one function into the first argument of the next with |> in the first example. With the broken code, send_resp is receiving the original conn with no Server header, which means that no Server header would be sent to the client.

There are a number of other Plug.Conn functions for manipulating connection details. Check out the Plug.Conn documentation for a full list.

The connection object is not just for sending responses. It contains the details of the request, too! We can use Elixir’s powerful pattern matching to match based on conn’s path_info (the path requested, split on / into an array) to call different functions for different pages. We can also match on method so that a POST request calls a different function than a GET request.

def call(conn, _opts) do
  route(conn.method, conn.path_info, conn)
end

def route("GET", ["hello"], conn) do
  # this route is for /hello
  conn |> Plug.Conn.send_resp(200, "Hello, world!")
end

def route("GET", ["users", user_id], conn) do
  # this route is for /users/<user_id>
  conn |> Plug.Conn.send_resp(200, "You requested user #{user_id}")
end

def route(_method, _path, conn) do
  # this route is called if no other routes match
  conn |> Plug.Conn.send_resp(404, "Couldn't find that page, sorry!")
end

The second route shows off my favorite part of using this syntax—we can extract a user_id variable from the URL entirely using Elixir’s pattern matching.

Writing a macro

We may want to have separate routers for different parts of our application, all using this route syntax. Rather than retyping the same identical call in every router, we can move it into a macro, and then just call the macro in each router. Macros are evaluated at compile time, so the Erlang VM will receive the same code, and we get to DRY up our code. We can also reuse this macro in future projects.

defmodule Router do
  defmacro __using__(_opts) do
    quote do
      def init(options) do
        options
      end
      def call(conn, _opts) do
        route(conn.method, conn.path_info, conn)
      end
    end
  end
end

defmodule Helloplug do
  use Router
  def route("GET", ["users", user_id], conn) do
    conn |> Plug.Conn.send_resp(200, "You requested user #{user_id}")
  end
  def route(_method, _path, conn) do
    conn |> Plug.Conn.send_resp(404, "Couldn't find that page, sorry!")
  end
end

Remember that call needs to return the modified connection object. When Plug runs call/2, this is how it knows what has changed about the connection. However, it also makes it composable. We can call plugs within other plugs. For instance, let’s say we want to route all calls to user endpoints to a different Plug:

defmodule UserRouter do
  use Router
  def route("GET", ["users", user_id], conn) do
    # this route is for /users/<user_id>
    conn |> Plug.Conn.send_resp(200, "You requested user #{user_id}")
  end
  def route("POST", ["users"], conn) do
    # do some sort of database insertion here maybe
  end
  def route(_method, _path, conn) do
    conn |> Plug.Conn.send_resp(404, "Couldn't find that user page, sorry!")
  end
end

defmodule WebsiteRouter do
  use Router

  @user_router_options UserRouter.init([])
  def route("GET", ["users" | path], conn) do
    UserRouter.call(conn, @user_router_options)
  end
  def route(_method, _path, conn) do
    conn |> Plug.Conn.send_resp(404, "Couldn't find that page, sorry!")
  end
end

If you haven’t seen the syntax for module attributes used in @user_router_options UserRouter.init([]) before, when WebsiteRouter is compiled, it stores the output from UserRouter.init([]) and inserts it anywhere you see @user_router_options appear. It’s a compile-time constant for the module. We run it once to get the options hash. We don’t use the options hash in UserRouter, but we need to do this in case we’re using a plug that does.

Also, remember when I mentioned init/1 could sometimes be passed options from outside? This is where we could do that! For instance, (to use a somewhat contrived example) maybe we want to treat two different models, User and Admin almost identically. We could create one plug, and initialize it twice, passing a different model into the options each time. Our plug’s init/1 would return the options given to it, and our call/2 function would then be able to see via its second parameter whether it should serve User objects or Admin objects.

A Plug doesn’t need to be a router—we could use this technique to include Plugs that do authentication, DOS protection, logging, and more. For instance, if there was a third-party ApiLogPlug module, we could add it as a dependency, and update our user route to use it:

@user_router_options UserRouter.init([])
@api_logger_options ApiLogPlug.init([])
def route("GET", ["users" | _path], conn)
  conn |> ApiLogPlug.call(@api_logger_options) |> UserRouter.call(@user_router_options)
end

Basic templates

At some point, we’re probably going to want to serve more than just plain text with our framework. We could continue to write HTML strings manually, but that’s not practical for larger webapps. Fortunately, Elixir comes built-in with a handy templating language. We can use it pretty easily:

def route("GET", ["users", user_id], conn) do
  page_contents = EEx.eval_file("templates/show_user.eex", [user_id: user_id])
  conn |> Plug.Conn.put_resp_content_type("text/html") |> Plug.Conn.send_resp(200, page_contents)
end

The first argument to eval_file is the filename of the template, and the second is a map of variables that we’ll be able to use inside the template. We’ll also need to create a template, which we can do in our project directory under templates/show_user.eex. Elixir’s templating language is similar to many other templating languages, especially Ruby’s erb.

<!DOCTYPE html>
<html>
  <body>
    <h1>User Information Page</h1>
    <p>Looks like you've requested information for the user with id <%= user_id %>.</p>
    <p>Also, 1 + 1 = <%= 1 + 1 %></p>
    <%= if user_id == "0" do %>
      <%# we should tell everybody that user 0 is really cool %>
      User zero is really cool!
    <% end %>
  </body>
</html>

As you can see above, you can put any sort of Elixir statement in a <% %> block and it will run. Adding a equal sign like <%= %> will make it print the statement’s result to the page, and <%# %> creates a comment that will not be sent to the client. Note how we have a = before the if statement—everything that prints something to the page needs an =. The if statement returns the contents of the block inside of it, and we need the = to make it print instead of discarding it.

If we visit /users/1092, we should now see a rendered HTML page stating that we requested information on user 1092.

Precompiling templates

Our template code is a little inefficient, though, since every time the route is called, Elixir has to load the template file, parse the template file into an Elixir function, and then run the function. If we can just parse and compile the template once, and then run the function every time the route is called, we’ll save a lot of time, especially for larger, frequently-called templates.

require EEx # you have to require EEx before using its macros outside of functions
EEx.function_from_file :defp, :template_show_user, "templates/show_user.eex", [:user_id]
def route("GET", ["users", user_id], conn) do
  page_contents = template_show_user(user_id)
  conn |> Plug.Conn.put_resp_content_type("text/html") |> Plug.Conn.send_resp(200, page_contents)
end

Ecto models

Phoenix, the Rails-like framework we talked about earlier, has a database model library called Ecto. Fortunately, Ecto is a standalone project, so we can use it too! Let’s update our /users/<user_id> route to actually retrieve user data from a Sqlite database. In your config.exs file, add:

config :helloplug, Helloplug.Repo,
  adapter: Sqlite.Ecto,
  database: "hello_plug.sqlite3"

We’re also going to need to create a new module for our database’s “Repo”, a module that holds functions that query our database.

defmodule Helloplug.Repo do
  use Ecto.Repo,
    otp_app: :helloplug,
    adapter: Sqlite.Ecto
end

We’ll also need a User model.

defmodule User do
  use Ecto.Model

  schema "users" do
    # id field is implicit
    field :first_name, :string
    field :last_name, :string
  end
end

The Ecto schema has way too many options to even start covering here. If you start using Ecto more, you should read the docs.

We’re also going to need this User model to actually exist as a table in the database—let’s do that now. Run mix ecto.gen.migration create_users to generate an empty migration, and then we’ll update the migration’s change function.

def change do
  create table(:users) do
    add :first_name, :string
    add :last_name, :string

    timestamps
  end
end

You can run migrations with mix ecto.migrate.

We’ll need to modify our UserRouter to actually use this User model:

def route("GET", ["users", user_id], conn) do
  case Helloplug.Repo.get(User, user_id) do
    nil ->
      conn |> Plug.Conn.send_resp(404, "User with that ID not found, sorry")
    user ->
      page_contents = EEx.eval_file("templates/show_user.eex", [user: user])
      conn |> Plug.Conn.put_resp_content_type("text/html") |> Plug.Conn.send_resp(200, page_contents)
  end
end

And finally, let’s update our template to list the first and last name of the user.

...
<h1><%= user.first_name] %> <%= user.last_name %>'s page</h1>
<p>Looks like you've requested information for the user with id <%= user.id %>.</p>
<%= if user.first_name == "Fluffums" do %>
  Also, let me just say that Fluffums is awesome.
<% end %>
...

Our database is empty, so let’s fix that! If we jump into an Elixir shell with iex -S mix, we can use Ecto to insert a record.

iex> Helloplug.Repo.start_link
iex> user = %User{id: 1, first_name: "Fluffums", last_name: "the Cat"}
iex> Helloplug.Repo.insert!(user)

The User struct is something we get for free when we create an Ecto model! We can just pass the struct to Repo, and it’ll automatically figure out which table to insert it into in the database.

Now, if we run the web server and visit localhost:4000/users/1, we should see “Fluffums the Cat” appear on our screen! Success!

Testing everything

One of the coolest features of this functional setup is that it’s incredibly simple to test! We don’t need any fancy magic; all we need to do is pass a fake Conn to our router’s call function, and inspect the result we get back. Elixir ships with a unit testing framework called ExUnit that we can use. We won’t go in-depth into to the nuances of testing in Elixir, but here is an example Plug test:

defmodule HelloTest do
  use ExUnit.Case, async: true
  use Plug.Test

  @website_router_opts WebsiteRouter.init([])
  test "returns a user" do
    conn = conn(:get, "/users/1")
    conn = WebsiteRouter.call(conn, @website_router_opts)

    assert conn.state == :sent
    assert conn.status == 200
    assert String.match?(conn.resp_body, ~r/Fluffums/)
  end
end

use Plug.Test simply imports functions from Plug.Test, including the conn function used above that creates fake connections to pass to the router.

Our framework’s performance

The Erlang VM’s pattern matching code has been heavily optimized. Rather than do a linear search through all our route definitions, the VM actually does a binary search through all the possibilities, which makes our route lookups run in O(log n) time instead of O(n). You’d likely only see improvements if you had a truly gargantuan number of routes in a single router, but still, it’s cool to know that we get this faster routing lookup for free, with no extra work on our end.

By using just a very thin layer over Plug and Cowboy, we get all of the performance benefits put into those projects with very little overhead. Cowboy can run multiple requests through our Plug simultaneously, and since all values in Elixir and Erlang are immutable, our framework is thread-safe by default. These libraries are also just generally fast—Matthew Rothenberg has a benchmark table comparing various frameworks. In his benchmark, our web framework would rank roughly near the “Plug” listing (it uses Plug’s Router, which works similarly to ours) at 54948 requests per second, which puts it at a higher throughput than even a Go based solution, Gin, which does around 51483 requests per second! These benchmarks are very simplistic, and don’t take into database fetching or a lot of other common web framework tasks, so it’s best to not give much weight to them. Still, it’s exciting to see that our simple framework is competitive with much bigger ones.

Not too shabby, considering we’ve barely written any code, and haven’t been considering performance until now.

Next steps

One of the cool side-effects of writing our own framework from scratch is that we ~~have~~ get to organize our own directory structure! Right now, everything is put together in the global namespace, which isn’t so great. We could improve this by putting things in sub-modules and sub-directories.⁴ For instance, models could go in a lib/helloplug/models folder, and we could call them Helloplug.Models.User instead of just User.

We could also add some more macros. For instance, the syntax for chaining Plugs together and precompiling templates is clunky. We also have no way of running a Plug before every single route in a router; this could perhaps be an argument passed to __using__ with a list of Plugs that call routes through before calling route/3.

We could also use a library like active to automatically reload modules when they change.

Code Words

Data driven literary analysis

Why am I doing this?

What is natural language processing?

Genre classification

Shakespeare analysis: classifying plays

Step 1: Content-based feature extraction

Latent Dirichlet Allocation

Step 2: K-Means clustering

Explaining the results

Epilogue

A tour of random forests

Forest Internals

Decision Trees

Entropy & “Guess Who?”

Trees To Forests

Some Illustrative Examples

What About Neural Nets?

A history of storage media

1. Punch cards

1.1. Jacquard looms

1.2. The 1890 United States Census

2. Williams-Kilburn Tubes

2.1. Secondary effects: principles of operation

2.2. The Manchester Baby

2.3. Later history

3. Mercury Delay Lines

And more!

Immutability is not enough

Going functional

Adding collisions

A bug

Functional Design, Take 2

More bugs

What’s going on?

What now?

A powerful idea: effect systems

Further reading

Telling stories with data using the grammar of graphics

Data

Geoms

Scaling

Coords

Groups and Facets

Making graphical choices

Further Reading

Image Processing 101

Why this was written

Prerequisites

Getting started

What is image processing?

Building blocks

Image format

Colors

Red, Green and Blue (RGB)

Hue, Saturation and Value (HSV)

Grayscale

Segmentation

Thresholding

Masking with a binary threshold

Further reading

Blurring

Further reading

Contours and Bounding Rectangles

Preprocess image

Find contours

Find bounding rectangles

Further reading

Edge detection

Binary threshold

Canny edges

Line and shape detection

Line Detection

Circle Detection

Further Reading

What next?

Further reading

Credits

This just isn't functional

How to trick a neural network into thinking a panda is a vulture