Last year I wrote up an annotated version of the original jQuery release in which I dumped many of my thoughts regarding the work and influences that went into that very first release.

It’s fascinating to see what place jQuery continues to hold in the world of web development. When I originally created the library I wanted to scratch two personal itches: 1) To provide a simple interface for interacting with the DOM and 2) Reduce the number of cross-browser issues that exist during development. Thankfully we now live in a world which is much rosier than back in 2005, when I was writing jQuery. Most users are on evergreen browsers and most have access to the recent standards-specified technology. It pleases me that there is apparently still a place for simple API design in the world, as jQuery’s continued success attests. jQuery is more popular than ever, surprisingly enough! 77.8% of the top million web site on the web use it, which is astounding.

When I stepped down from the project in 2011, to focus on my transition to working at Khan Academy, a fantastic team stepped up to ensure that the project would continue to run smoothly. Reading Sebastian McKenzie’s recent blog post about the struggles he’s faced in running the popular Babel project struck close to home for me. It made me glad that I had put time and effort into building a team of amazing people who were happy to take over the project once I stepped down. jQuery would not be in the place it is today without them.

I’m thankful that jQuery gave me the opportunity to meet so many new people and work on the things that I’m most passionate about. However I’m most happy that it’s empowered so many people to build things on the web, I’m so glad to have been a part of it all. I can’t wait to see where we’ll be another 10 years from now!

Art history photo archives are an interesting tool used by scholars: they’re a collection of thousands, if not millions, of photos of art. These are not the artworks themselves but photos of the artworks – oftentimes from all over the world – taken by individuals or institutions and aggregated into this single archive. This is an invaluable resource for researchers as oftentimes you can find more information about the history of an artwork (before/after conservation) or even find artworks that are lost (stolen, destroyed, or moved into unknown private collections).

The Frick Photoarchive has 1.2 million photos of art which they are in the process of digitizing and bringing online. They’re part of a larger photo archive consortium, called PHAROS, which has many other photo archives from all over the world. In total they’re likely going to have over 10 million photos of art once all of their collections are pooled.

This poses an interesting problem however as combining this much data from all of these institutions is especially challenging. In the world of digital humanities the most-common tool to utilize is that of Linked Open Data. Making sure all the records have consistent attributes being used, along with consistent data, then it’ll become easy to match up data from multiple institutions. In practice, however, this is quite difficult. Moving massive databases over to another schema can be very challenging, in fact very few institutions have done it. The British Museum is an especially notable example that has made the switch to this format and has served as an example to other institutions around the world.

Madonna and Child, 13th century, La Chiesa di S. Eufrasia, Pisa.

I’ve been proposing an alternative means of merging archives, namely using computer vision techniques to find the missing connections between images of artworks, to avoid the process of having to manually identify images and making sure that their data is pre-aligned. I published a paper on my research looking in to this topic (which has since been re-published in the Journal of Digital Humanities). The results are extremely promising. I’ve been using the MatchEngine service graciously provided by Idee Inc. to great effect. The algorithm they use is able to match duplicate images of the same artwork, even if there are differences in the photo (e.g. color vs. black-and-white, skewed photo angles, etc.). I highly recommend reading my paper for more information on the algorithm’s capabilities.

Building off of that work last year the Frick Art Reference Library and I received a digital resources grant from the Kress Foundation to continue exploring these computer vision techniques. Specifically I looked in to using the technology to merge multiple art history photo archives together. I attempted to merge the Frick Photoarchive’s Anonymous Italian Art collection and the Zeri Foundation’s 15th and 16th century Italian art collections using only the images and no additional metadata.

As it turns out this technique proved to be very successful – so successful that we’re now looking to use this technology as a primary means of merging records between all the archives in the PHAROS consortium. You can learn more about the results of this process in a talk that I gave at last year’s PHAROS (then IDPAC) annual meeting:

Now that we’ve proven that inter-institutional merging of photo archives is possible the next step becomes obvious: building a new database, in many ways like the Japanese print database I built previously, but focused on the digital photo collections of these archives.

I’m happy to announced that the Frick Art Reference Library and I have just received another grant from the Kress foundation to fund the development of this new database. At the start it’ll primarily focus on Italian art, as that’s the majority what has already been digitized by PHAROS archives. However even just focusing on Italian art it’ll immediately become one of the largest and most-important online collections for studying Italian art in the world.

A few key features of this new database:

• The database, images, and data contained within it will all be publicly accessible. This was an extremely important point for me as I wanted to ensure that we were providing a resource for everyone to learn from.
• Computer vision techniques will be used to merge the records from the archives together and will also be provided as a tool for research: you’ll be able to upload an image of an artwork and find other copies of it in the archive. This feature alone will save researchers so much time and energy when attempting to learn about an artwork.
• Metadata about the images will be provided by the institutions and will be made searchable. In the first version of this database we will not be doing any work to rectify the metadata between the institutions (many of them will be in different languages) but that should serve as a good “next step” for the project after launch.
• The interface for the database will be provided in multiple languages. Initially in English, German, and Italian.

More information about this project can be found on the project proposal page. I will be working on this project, outside of my work at Khan Academy, over the next year. I’m extremely excited to have this opportunity to work on this project. Being able to improve art history scholarship is a passion of mine and being able to provide these tools for others is a fantastic opportunity. I’ll be sure to write more about my work as it progresses, especially the interesting technical innovations that come in to play.

I’ve been doing a lot of work exploring different computer vision, and machine learning, algorithms to see how they might apply to the world of art history study. I’m especially interested in finding novel uses of technology that could greatly benefit art historians in their work and also help individuals better appreciate art.

One tool that I came across yesterday is called Waifu2x. It’s a convolutional neural network (CNN) that is designed to optimally “upscale” images (taking small images and generating a larger image). The creator of this tool built it to better upscale poorly-sized Anime images and video. This is an effort that I can massively cheer on – while I’m not a purveyor of Anime, I love applying algorithmic overkill to non-tech hobbies.

Waifu2x also provides a live demo site that you can use to test it on other images. When I saw this I became immediately intrigued. Anime has direct stylistic influences drawn from the “old” world of Japanese woodblock printing, which was popular from the late 1600s to the late 1800s. Maybe the pre-trained upscaler could also work well for photos of Japanese prints? (Naturally I could train a new CNN to do this, but it may not even be necessary!)

Now the first questions that should come up, before even attempting upscale Japanese prints, are simply: Are there enough tiny images of prints that need to be made bigger? Who will benefit from this?

To answer those questions: Unfortunately there are tons of tiny pictures of Japanese prints in the world. To provide one very real example: The Tokyo National Museum has one of the greatest collections of Japanese prints in the world… none of which are (publicly) digitized. If a researcher wants to see if a the TNM has a particular copy of a print they’ll need to use the following 3 volume set of books (which I own):

Inside the book is 3,926 small, black-and-white, scans of every print in their collection:

I plan on digitizing these books and bringing these (not-ideal) prints online as they will be of the utmost use to scholars. However given their tiny size it will be very hard for most researchers to be able to make out what exactly the print is depicting. Thus any technology that is able to upscale the images to make them a bit easier to view would be greatly appreciated.

I began by experimenting with a few different, existing, print images and was really intrigued by the results.

I started with a primitive Ukiyo-e print by Hishikawa Moronobu:

(Source Image)

And here is the image scaled 2x (OSX Preview) and then using Waifu2x using high noise reduction (be sure to click the images to see them full size):

And here is another early actor print by Utagawa Kunisada:

(Source Image)

And here is the image scaled 2x (OSX Preview) and then using Waifu2x with low noise reduction and high noise reduction (be sure to click the images to see them full size):

I also have a few blown-up details comparing the results between these three (OSX Preview 2x, Waifu2x with low noise reduction, Waifu2x with high noise reduction):

It’s immediately apparent that the lines are still quite “crisp” in both of the Waifu2x versions. This is extremely compelling as being able to see those details can be quite important. In fact seeing these upscaled images like this is seriously quite impressive. It definitely reinforces the fact that the algorithms Anime training data has suited this subject matter well!

To my eye it almost looks like they entire image has also become more “smooth”, it seems much more mottled, almost as if someone spilled water on it (Japanese prints are printed with watercolors and thus are quite susceptible to water damage that actually creates similar results as to what’s seen here). This is especially true when using Waifu2x’s high noise reduction. It’s not clear that this would become any better with better training data as the original source image really only has so much data to begin with.

Waifu2x’s high noise reduction also causes the background to become much more tumultuous, as if it was crumpled wrapping paper. And the details in the signature, that make it readable, tend to be lost with that much processing. I suspect that using just the low noise reduction may be a better sweet spot.

Helping researchers be able to better see the lines of a print from a tiny image can be a double-edged sword. They will certainly appreciate being able to have a larger and semi-crisp image. However the lack of precision is a massive problem (not that the original image at 2x would’ve been any better). Researchers rely upon being able to spot minute differences between print impressions in order to be able to understand when a print could’ve been created. I suspect that using this technique will need to be reserved to only extremely-small images and come with a massive caveat warning the viewer as to the nature of the image and its appearance.

Regardless, this technology is quite exciting and it’s extremely serendipitous that the subject matter that was used just so happens to correlate nicely with one of my areas of study. I’ll be very curious to see where else people have success with this particular utility and in what context.

I then took that opportunity to put that code online and I used the new Genius annotation beta to mark it up with a bunch of thoughts and memories around the release of the code.

Annotated jQuery Release

The comments are kind of sprawling, they cover the code itself to the circumstances of the release. I’ll certainly add more as I think of them. Please feel free to ask questions if there are certain things that you want to learn about!

One of the things I was most touched about was just how obviously inspired I was by many great developers who came before me. I’m glad I went out of my way to credit them on the original version of the jQuery site. jQuery would not have existed if it wasn’t for Sam Stephenson creating Prototype, Simon Willison creating getElementsBySelector, Valerio Proietti creating moo.fx, and especially Dean Edwards creating so much: event binding, CSS selector libraries, DOM ready techniques and more. I cover a lot of these influences in the annotations but I want to especially thank them now as without their work I never would’ve made jQuery.

I can’t vouch for the quality of the particular services, only that this is their stated price for renewing a .com domain through their service. I opted to only focus on the cost of renewing the domain as often-times registrars will have a much-lower cost during the first year of registration and then later increase the price (as with any privacy services). Hope this can be useful to others who are researching the price of private domain registration!

Update: After compiling this list I found this master list on Registrarowl.com, which includes many more registrars (along with privacy price information).

There were a few major problems with how I was working on my side projects. I was primarily working on them during the weekends and sometimes in the evenings during the week. This is a strategy that does not work well for me, as it turns out. I was burdened with an incredible amount of stress to try and complete as much high quality work as possible during the weekend (and if I was unable to it felt like a failure). This was a problem as there’s no guarantee that every weekend will be free – nor that I’ll want to program all day for two days (removing any chance of relaxation or doing anything fun).

There’s also the issue that a week between working on some code is a long time, it’s very easy to forget what you were working on or what you left off on (even if you keep notes). Not to mention if you miss a weekend you end up with a two week gap as a result. That massive multi-week context switch can be deadly (I’ve had many side projects die due to attention starvation like that).

Inspired by the incredible work that Jennifer Dewalt completed last year, in which she taught herself programming by building 180 web sites in 180 days, I felt compelled to try a similar tactic: working on my side projects every single day.

I decided to set a couple rules for myself:

1. I must write code every day. I can write docs, or blog posts, or other things but it must be in addition to the code that I write.
2. It must be useful code. No tweaking indentation, no code re-formatting, and if at all possible no refactoring. (All these things are permitted, but not as the exclusive work of the day.)
3. All code must be written before midnight.
4. The code must be Open Source and up on Github.

Some of these rules were arbitrary. The code doesn’t technically need to be written before midnight of the day of but I wanted to avoid staying up too late writing sloppy code. Neither does the code have to be Open Source or up on Github. This just forced me to be more mindful of the code that I was writing (thinking about reusability and deciding to create modules earlier in the process).

Thus far I’ve been very successful, I’m nearing 20 weeks of consecutive work. I wanted to write about it as it’s completely changed how I code and has had a substantial impact upon my life and psyche.

With this in mind a number of interesting things happened as a result of this change in habit:

Minimum viable code. I was forced to write code for no less than 30 minutes a day. (It’s really hard to write meaningful code in less time, especially after remembering where you left off the day before.) Some week days I work a little bit more (usually no more than an hour) and on weekends I’m sometimes able to work a full day.

Code as habit. It’s important to note that that I don’t particularly care about the outward perception of the above Github chart. I think that’s the most important take away from this experiment: this is about a change that you’re making in your life for yourself not a change that you’re making to satisfy someone else’s perception of your work. The same goes for any form of dieting or exercise: if you don’t care about improving yourself then you’ll never actually succeed.

Battling anxiety. Prior to starting this experiment I would frequently feel a high level of anxiety over not having completed “enough” work or made “enough” progress (both of which are relatively unquantifiable as my side projects had no specific deadlines). I realized that the feeling of making progress is just as important as making actual progress. This was an eye-opener. Once I started to make consistent progress every day the anxiety started to melt away. I felt at peace with the amount of work that I was getting done and I no longer had the over-bearing desire to frantically get any work done.

Weekends. Getting work done on weekends use to be absolutely critical towards making forward momentum (as they were, typically, the only time in which I got significant side project coding done). That’s not so much the case now – and that’s a good thing. Building up a weeks-worth of expectations about what I should accomplish during the weekend only ended up leaving me disappointed. I was rarely able to complete all the work that I wanted and it forced me to reject other weekend activities that I enjoyed (eating dim sum, visiting museums, going to the park, spending time with my partner, etc.) in favor of getting more work done. I strongly feel that while side projects are really important they should not be to the exclusion of life in general.

Background processing. An interesting side effect of writing side project code every day is that your current task is frequently running in the back of your mind. Thus when I go for a walk, or take a shower, or any of the other non-brain-using activities I participate in, I’m thinking about what I’m going to be coding later and finding a good way to solve that problem. This did not happen when I was working on the code once a week, or every other week. Instead that time was consumed thinking about some other task or, usually, replaced with anxiety over not getting any side project work done.

Context switch. There’s always going to be a context switch cost when resuming work on a side project. Unfortunately it’s extremely hard to resume thinking about a project after an entire week of working on another task. Daily work has been quite helpful in this regard as the time period between work is much shorter, making it easier to remember what I was working on.

Work balance. One of the most important aspects of this change was in simply learning how to better balance work/life/side project. Knowing that I was going to have to work on the project every single day I had to get better at balancing my time. If I was scheduled to go out in the evening, and not get back until late, then I would need to work on my side project early in the day, before starting my main Khan Academy work. Additionally if I hadn’t finished my work yet, and I was out late, then I’d hurry back home to finish it up (instead of missing a day). I should note that I’ve been finding that I have less time to spend on hobbies (such as woodblock printing) but that’s a reasonable tradeoff that I’ll need to live with.

Outward perception. This has all had the added benefit of communicating this new habit externally. My partner understands that I have to finish this work every day, and thus activities sometimes have to be scheduled around it. It’s of considerable comfort to be able to say “Yes, we can go out/watch a movie/etc. but I have to get my coding in later” and have that be understood and taken into consideration.

How much code was written? I have a hard time believing how much code I’ve written over the past few months. I created a couple new web sites, re-wrote some frameworks, and created a ton of new node modules. I’ve written so much I sometimes forget the things I’ve made – work from even a few weeks prior seem like a distant memory. I’m extremely pleased with the amount of work that I’ve gotten done.

I consider this change in habit to be a massive success and hope to continue it for as long as I can. In the meantime I’ll do all that I can to recommend this tactic to others who wish to get substantial side project work done. Let me know if this technique does, or doesn’t, work for you – I’m very interested in hearing additional anecdotes!

Discuss this post on Hacker News.

Both of these were largely provoked by this tweet by Chris Anderson:

Advice received on hiring software devs: reject anyone who doesn't have a @GitHub profile (the more active the better). Agree?

— Chris Anderson (@chr1sa) October 30, 2013

I had a similar, extremely popular, tweet from 2011:

When it comes to hiring, I'll take a Github commit log over a resume any day.

— John Resig (@jeresig) February 5, 2011

I wanted to take this chance to clarify my position on using Github during the interview process a bit.

To start, the position being posited by Chris is absolutely wrong. No one should ever require a Github profile in favor of a resume (unless you’re applying for a job at Github, perhaps?). Nor should anyone prefer candidates who provide a Github profile over those who don’t.

(And to clarify, some people seem to be getting caught up in the terminology – I see many people using ‘Github’ as a placeholder for ‘any OSS activity’, I hope no one is being that literal.)

When you’re applying for a job it’s generally advised that the more high-quality information that you provide, the better. Anything that emphasizes your work habits, the quality of work that you will perform, and your interests is ideal. Thus the more targeted information you’re able to provide, the better. Github can provide this for some but almost certainly can’t be the final arbiter of this information for all.

We’ve had dev applicants at Khan Academy who’ve applied with traditional resumes, resume + Github profile, resume + personal web site, resume + links to projects they’ve worked on, resume + a custom project made special for the interview.

Out of all of these submissions the ones that’ve had the most impact are the ones where the candidates have gone above-and-beyond and done some custom coding, made a custom web site, or contributed to one of our Open Source projects as part of the application process. A popular project is one where candidates make a custom exercise using our exercise framework.

There are a few reasons why this particular approach is so effective:

1. It communicates just how interested the candidate is in the job: That they’re willing to go above-and-beyond to demonstrate their interest.
2. They show that the sort of things they’re interested in working on are largely aligned with what we’re trying to do (especially if they make an exercise with our exercise framework).
3. If it’s a coding project you can get a sense for what their code quality is like and how they wish to portray themselves.

Naturally this isn’t completely ideal as it requires a candidate to put in considerably more time into their application than just generating a single-sheet resume. That being said the level of information that is conveyed is so much higher that it almost isn’t comparable.

The traditional interview process is broken. It’s high-stress and a poor representation of what you’ll actually experience working for the employer, or working with the candidate.

Over this past summer I developed a new “take-home” project-based interview for JavaScript-centric candidates to Khan Academy. I wanted a way to try to better-quantify a person’s development ability without having to necessarily suss out their best code from Github, or elsewhere.

I extracted a tangible, contained, project that I had assigned to one of my interns and turned it into a project. Thus far it’s worked extremely well as, as can be attested to by Pamela Fox (who completed the project with flying colors). I’m very pleased with this approach as it gets all the benefit of the candidate creating a custom project on their own but it has the added benefit of communicating to the candidate the exact type of projects that they’ll be working on here at Khan Academy.

I’d love to open up the project at some point to show how it’s done but it’s still an effective tool in our interviewing toolbox and I’m not sure I want to lose that just yet!

However, you can be privy to the project-based interview process if you wish to apply to Khan Academy! Specifically if you’re interested in working with Pamela Fox and me to build Khan Academy’s Computer Science platform (more details on the launch, a technical talk I gave on it after launch).

If this really excites you then you should apply for a Software Developer position at Khan Academy and specifically request to go through the JavaScript project interview with myself. I’ll give you the project to work on and we’ll discuss the results extensively.

The biggest drawback to this approach is that we’re demanding some of your free time to gather this information. We’re not sure what the best way is to offset this, as of yet. However by doing the project you’ll likely be able to skip a number of stages in our interview process and hopefully get started on changing the world with us at Khan Academy all the sooner.

One of my favorite sessions was the one run by James Halliday and Max Ogden exploring Node streams. Specifically they sat us down and had us run through the amazing Stream Adventure program. It’s an interactive series of exercises that are designed to help you understand streams better. Seeing interesting ways of using streams in practice was quite eye opening for me. (I also recommend checking out the awesome Stream Handbook if streams are totally foreign to you)

I had made limited use of streams before (namely with the request and fs modules) but I was still early on. Over the past few months I’ve spent more time researching Node modules that make good use of streams and have tried to apply them in my various side projects. In theory I absolutely love streams: they are the chainable piping amazingness that exists with UNIX and I aspired to achieve with jQuery. In practice many Node stream modules are poorly written, poorly documented, out of date, or just obscure. So while theoretically using Node streams should be really easy in practice it becomes quite challenging.

In my explorations I was starting to build up a pile of streaming Node modules that I *knew* worked correctly (or, at least, as I expected them to). Namely: They supported streams in Node 0.10+ and they supported being .pipe()‘d to and from (where applicable). I realized that in a perfect world Node streams truly are like Lego bricks – it becomes quite easy to snap a few of them together to build a chained flow for your data to pass through.

With that in mind I built an exploration that I hope will help others to understand the simplicity and power of Node streams: The Node.js Stream Playground.

More information:

• http://nodestreams.com/
• https://github.com/jeresig/node-stream-playground
• https://npmjs.org/package/stream-playground

The Node.js Stream Playground was created to help Node.js developers better understand how streams work by showing a number of use cases that are easily plug-and-play-able. Additionally detailed logging is provided at every step to help users better understand what events the streams are emitting and exactly what their contents are.

How to Use the Playground

I hope you’ll get the most out of the playground by exploring concepts for yourself – looking at what happens when you pipe different streams to each other, looking at the data being logged, and attempting to understand (for yourself) what exactly is happening and why. Note that all of the code being generated is valid Node code and can be copy-and-pasted and run on your own computer (assuming you have the appropriate NPM modules installed).

I should note that this isn’t the end-all-and-be-al of Node stream education: There is still a lot to learn with regard to error handling, back pressure, and all the intricate stream concepts that exist. I hope to write some more on these concepts some day.

With that being said here are some good examples of some of the actions that you can do:

Copying a File

1. Select Read File and input/people.json.
2. Select Write File and output/people.json.

The resulting code:

var fs = require("fs");

// Read File
fs.createReadStream("input/people.json")
    // Write File
    .pipe(fs.createWriteStream("output/people.json"));

This will copy the JSON file from one location to another. This is actually the preferred way of copying files in Node – there actually is no built-in utility method for doing this.

Downloading a File

This will download the JSON file from the specified URL and save it to the local file.

1. Select HTTP Get Request and http://nodestreams.com/input/people.json.
2. Select Write File and output/people.json.

The resulting code:

var request = require("request");
var fs = require("fs");

// HTTP GET Request
request("http://nodestreams.com/input/people.json")
    // Write File
    .pipe(fs.createWriteStream("output/people.json"));

Un-Gzipping a File

1. Select Read File and input/people.csv.gz.
2. Select Un-Gzip.
3. Select Write File and output/people.csv.

The resulting code:

var fs = require("fs");
var zlib = require("zlib");

// Read File
fs.createReadStream("input/people.csv.gz")
    // Un-Gzip
    .pipe(zlib.createGunzip())
    // Write File
    .pipe(fs.createWriteStream("output/people.csv"));

Note that the initial data that is coming out of the Read File is just an array of numbers – this is to be expected. In reality it’s a Node Buffer holding all of the raw binary data. It’s not until after we Un-Gzip the file that we start to have data that’s in a more usable form.

Converting a TSV to HTML

In this case we’re going to manually parse the TSV (without headers) just to show you some of the string operations provided by event-stream.

1. Select Read File and input/people.tsv.
2. Select Split Strings.
3. Select Split Strings into Array.
4. Select Convert Array w/ Sprintf (with the default table row HTML).
5. Select Join Strings (this will insert an endline between all the individual table rows).
6. Select Concat Strings (this will combine all the individual table rows and endlines into a single string).
7. Select Wrap Strings (with the default table HTML).
8. Select Write File and output/people.html.

The resulting code:

var fs = require("fs");
var es = require("event-stream");
var vsprintf = require("sprintf").vsprintf;

// Read File
fs.createReadStream("input/people.tsv")
    // Split Strings
    .pipe(es.split("\n"))
    // Split Strings into Array
    .pipe(es.mapSync(function(data) {
        return data.split("\t");
    }))
    // Convert Array w/ Sprintf
    .pipe(es.mapSync(function(data) {
        return vsprintf("<tr><td><a href='%2$s'>%1$s</a></td><td>%3$s</td></tr>", data);
    }))
    // Join Strings
    .pipe(es.join("\n"))
    // Concat Strings
    .pipe(es.wait())
    // Wrap Strings
    .pipe(es.mapSync(function(data) {
        return "<table><tr><th>Name</th><th>City</th></tr>\n" + data + "\n</table>";
    }));

This should give you some HTML that looks something like this:

<table><tr><th>Name</th><th>City</th></tr>
<tr><td><a href='http://vinceallen.com/'>Vince Allen</a></td><td>Brooklyn, NY</td></tr>
<tr><td><a href='http://twitter.com/jandet'>Janessa Det</a></td><td>New York, NY</td></tr>
<tr><td><a href='http://patnakajima.com/'>Pat Nakajima</a></td><td>San Francisco, CA</td></tr>
<tr><td><a href='http://sarajchipps.com/'>Sara Chipps</a></td><td>New York, NY</td></tr>
<tr><td><a href='https://johnresig.com/'>John Resig</a></td><td>Brooklyn, NY</td></tr>
</table>

Download Encoded, Gzipped, CSV and Convert to HTML

This is the big one:

1. Select HTTP Get Request and http://nodestreams.com/input/people.csv.gz.
2. Select Un-Gzip (note that some of the characters are encoded incorrectly, we need to fix this!).
3. Select Change Encoding (even though it comes out as a buffer, this is correct, as we’ll see in a moment).
4. Select Parse CSV as Object.
5. Select Convert Object w/ Handlebars (with the default table row HTML).
6. Select Join Strings (this will insert an endline between all the individual table rows).
7. Select Concat Strings (this will combine all the individual table rows and endlines into a single string).
8. Select Wrap Strings (with the default table HTML).
9. Select HTTP PUT Request and http://nodestreams.com/output/people.html.

The resulting code:

var request = require("request");
var zlib = require("zlib");
var Iconv = require("iconv").Iconv;
var csv = require("csv-streamify");
var Handlebars = require("handlebars");
var es = require("event-stream");

var tmpl = Handlebars.compile("<tr><td><a href='{{URL}}'>{{Name}}</a></td><td>{{City}}</td></tr>");

// HTTP GET Request
request("http://nodestreams.com/input/people_euc-jp.csv.gz")
    // Un-Gzip
    .pipe(zlib.createGunzip())
    // Change Encoding
    .pipe(new Iconv("EUC-JP", "UTF-8"))
    // Parse CSV as Object
    .pipe(csv({objectMode: true, columns: true}))
    // Convert Object w/ Handlebars
    .pipe(es.mapSync(tmpl))
    // Join Strings
    .pipe(es.join("\n"))
    // Concat Strings
    .pipe(es.wait())
    // Wrap Strings
    .pipe(es.mapSync(function(data) {
        return "<table><tr><th>Name</th><th>City</th></tr>\n" + data + "\n</table>";
    }))
    // HTTP PUT Request
    .pipe(request.put("http://nodestreams.com/output/people.html"));

You should have the same output as the last run. A lot is happening here but even will all of these steps Node streams still makes it relatively easy to complete it all. It’s at this point that you can truly start to see the power and expressiveness of streams.

Adding in New Streams

If you’re interested in extending the playground and adding in new pluggable stream “blocks” you can simply edit blocks.js and add in the stream functions. A common stream block would look something like this:

"Change Encoding": function(from /* EUC-JP */, to /* UTF-8 */) {
   var Iconv = require("iconv").Iconv;
   return new Iconv(from, to);
},

The property name is the full title/description of the stream. The arguments to the function are variables that you wish the user to populate. The comments immediately following the argument names are the default values (you can provide multiple values by separating them with a |).

The streams are split into 3 types: “Readable”, “Writable”, “Transform” (in that they read content, modify it, and pass it through). Typically it is assumed that “Readable” streams will be the first ones that you can choose in the playground, “Writeable” streams will end the playground, and everything else is just “.pipe()able”.

If you’ve added a new stream please send a pull request and I’ll be happy to add it!

Running Your Own Server

After downloading the code (either from Github or NPM). Be sure to run npm install to install all the dependencies. You can then use just node app.js to run a server – or if you wish to run something more robust you can install naught and then run npm start.

WARNING I have no idea how robust the application’s security is. This application is generating and executing code on the user’s behalf (although it is not allowing arbitrary code to be executed). Feel free to run it on a local server or, if you feel confident in the code, run it on your own server. At the moment I’m running it on a standalone server with nothing else on it, just in case.

Feedback Welcome!

I’d love to hear about how people are using streams and if this tool has been helpful for your understanding how streams work. Let me know if there are particular stream modules that you really like and – if possible – try and add them to the stream playground so that others can experience them as well!

One of the missing pieces, for most Open Source developers, is having consistent, reliable, income backing your development. Some developers are sponsored by their work place, others have a job but do Open Source work on the side. Even others may not have reliable incomes and still want to create software that isn’t tied to a specific revenue-generator. I personally use Gittip to fund hosting costs for some of my side projects.

Gittip is capable of freeing Open Source developers by allowing people to give them money on a week-by-week basis. This is intriguing as the money that comes in will come slowly – people will be less likely to give and they’ll likely give smaller amounts – but it’ll also be relatively stable (once someone is giving there will be not much reason to decrease that amount or stop giving).

I’ve been providing feedback to Chad Whitacre and the rest of the Gittip team wherever I can because this is something that I really want to succeed.

Taking a further step I looked to see what were the major pain points to growth. Right now they’re:

1. Getting more developers signed up and asking for donations.
2. Getting more people giving money to those developers.

I’m attempting to work with the Gittip team to help make #1 more feasible, and while I’m giving money personally, I wanted to try and get more money into the system.

I saw a great opportunity for the development team at Khan Academy to give back to the Open Source community in ways that were not just code-related.

Talking with Ben Kamens we came upon a good solution: Every developer at Khan Academy (interns included) has $5/week to give to the Open Source developers of their choice. (You’re restricted from giving that money to employees of Khan Academy though, heh.) That money would then be rolled up into a single contribution coming from the Khan Academy Gittip account.

This seemed like the ideal scenario: Developers gets full control of how their money should be allocated and Khan Academy gets the benefit of their financial contributions coming from a unified source. (Which helps to communicate to developers that we care about Open Source and that, hopefully, we’re a cool place to work, which we are.)

In the spirit of shipping beats perfection I hacked together a Google Spreadsheet which Khan Academy devs could modify to allocate their contributions.

At the moment 21 Khan Academy devs are collectively giving $57.75/week to 15 different Open Source devs. This is a great start! Unfortunately we’re trying to give $46.25 to an additional 8 devs who aren’t yet on Gittip, so our contribution is stuck in limbo. (At the moment the best we can hope for is to pester them on Twitter, etc. and get them to sign up. In some cases devs seem to be disinclined to receive donations if they already have a full-time job — in that case they should feel free to donate the money to the charity of their choice, or even just back to other devs!)

Update July 17th: We’re now up to 26 Khan Academy devs giving $89/week to 17 Open Source devs on Gittip. Another $40.25/week is being held until some additional Open Source devs join Gittip.

We’ve already started the process of contacting some of the devs that we want to support:

@jashkenas @fat @cloudhead @stuk @ariyahidayat Heya! Could you join @Gittip? We want to thank you for your OSS work: http://t.co/jr6OHRDoVm

— Khan Academy (@khanacademy) July 16, 2013

There is a script that I’ve made to automate the process of extracting the tips from the spreadsheet and uploading them to Gittip (run as a weekly cron job). You can find it on Github:

https://github.com/Khan/gittip-gdoc

I want to thank Chad for helping to add in an API that would make it easier to create this script!

I hope others are as excited about Gittip as we are at Khan Academy. I strongly think that this is the future of providing developers with a financial incentive for creating useful Open Source work. Gittip is still quite young and our understanding of how this mode of contribution should work is still open for debate so I hope that this will help inspire others to give, and strive towards a financial ideal, as well.

Getting a C++ game engine running in JavaScript, using WebGL for rendering, is a massive feat and is largely due to the toolchain that Mozilla has developed to make it all possible.

Since the release of the Unreal Engine 3 port to Asm.js I’ve been watching the response on Twitter, blogs, and elsewhere and while some developers are grasping the interesting confluence of open technologies that’ve made this advancement happen I’ve also seen a lot of confusion: Is Asm.js a plugin? Does Asm.js make my regular JavaScript fast? Does this work in all browsers? I feel that Asm.js, and related technologies, are incredibly important and I want to try and explain the technology so that developers know what’s happened and how they will benefit. In addition to my brief exploration into this subject I’ve also asked David Herman (Senior Researcher at Mozilla Research) a number of questions regarding Asm.js and how all the pieces fit together.

What is Asm.js?

In order to understand Asm.js and where it fits into the browser you need to know where it came from and why it exists.

Asm.js comes from a new category of JavaScript application: C/C++ applications that’ve been compiled into JavaScript. It’s a whole new genre of JavaScript application that’s been spawned by Mozilla’s Emscripten project.

Emscripten takes in C/C++ code, passes it through LLVM, and converts the LLVM-generated bytecode into JavaScript (specifically, Asm.js, a subset of JavaScript).

If the compiled Asm.js code is doing some rendering then it is most likely being handled by WebGL (and rendered using OpenGL). In this way the entire pipeline is technically making use of JavaScript and the browser but is almost entirely skirting the actual, normal, code execution and rendering path that JavaScript-in-a-webpage takes.

Asm.js is a subset of JavaScript that is heavily restricted in what it can do and how it can operate. This is done so that the compiled Asm.js code can run as fast as possible making as few assumptions as it can, converting the Asm.js code directly into assembly. It’s important to note that Asm.js is just JavaScript – there is no special browser plugin or feature needed in order to make it work (although a browser that is able to detect and optimize Asm.js code will certainly run faster). It’s a specialized subset of JavaScript that’s optimized for performance, especially for this use case of applications compiled to JavaScript.

The best way to understand how Asm.js works, and its limitations, is to look at some Asm.js-compiled code. Let’s look at a function extracted from a real-world Asm.js-compiled module (from the BananaBread demo). I formatted this code so that it’d be a little bit saner to digest – it’s normally just a giant blob of heavily-minimized JavaScript:

Technically this is JavaScript code but we can already see that this looks nothing like most DOM-using JavaScript that we normally see. A few things we can notice just by looking at the code:

• This particular code only deals with numbers. In fact this is the case of all Asm.js code. Asm.js is only capable of handling a selection of different number types and no other data structure (this includes strings, booleans, or objects).
• All external data is stored and referenced from a single object, called the heap. Essentially this heap is a massive array (intended to be a typed array, which is highly optimized for performance). All data is stored within this array – effectively replacing global variables, data structures, closures, and any other forms of data storage.
• When accessing and setting variables the results are consistently coerced into a specific type. For example f = e | 0; sets the variable f to equal the value of e but it also ensures that the result will be an integer (| 0 does this, converting an value into an integer). We also see this happening with floats – note the use of 0.0 and g[...] = +(...);.
• Looking at the values coming in and out of the data structures it appears as if the data structured represented by the variable c is an Int32Array (storing 32-bit integers, the values are always converted from or to an integer using | 0) and g is a Float32Array (storing 32-bit floats, the values always converted to a float by wrapping the value with +(...)).

By doing this the result is highly optimized and can be converted directly from this Asm.js syntax directly into assembly without having to interpret it, as one would normally have to do with JavaScript. It effectively shaves off a whole bunch of things that can make a dynamic language, like JavaScript, slow: Like the need for garbage collection and dynamic types.

As an example of some more-explanatory Asm.js code let’s take a look at an example from the Asm.js specification:

function DiagModule(stdlib, foreign, heap) {
    "use asm";

    // Variable Declarations
    var sqrt = stdlib.Math.sqrt;

    // Function Declarations
    function square(x) {
        x = +x;
        return +(x*x);
    }

    function diag(x, y) {
        x = +x;
        y = +y;
        return +sqrt(square(x) + square(y));
    }

    return { diag: diag };
}

Looking at this module it seems downright understandable! Looking at this code we can better understand the structure of an Asm.js module. A module is contained within a function and starts with the "use asm"; directive at the top. This gives the interpreter the hint that everything inside the function should be handled as Asm.js and be compiled to assembly directly.

Note, at the top of the function, the three arguments: stdlib, foreign, and heap. The stdlib object contains references to a number of built-in math functions. foreign provides access to custom user-defined functionality (such as drawing a shape in WebGL). And finally heap gives you an ArrayBuffer which can be viewed through a number of different lenses, such as Int32Array and Float32Array.

The rest of the module is broken up into three parts: variable declarations, function declarations, and finally an object exporting the functions to expose to the user.

The export is an especially important point to understand as it allows all of the code within the module to be handled as Asm.js but still be made usable to other, normal, JavaScript code. Thus you could, theoretically, have some code that looks like the following, using the above DiagModule code:

document.body.onclick = function() {
    function DiagModule(stdlib){"use asm"; ... return { ... };}

    var diag = DiagModule({ Math: Math }).diag;
    alert(diag(10, 100));
};

This would result in an Asm.js DiagModule that’s handled special by the JavaScript interpreter but still made available to other JavaScript code (thus we could still access it and use it within a click handler, for example).

What is the performance like?

Right now the only implementation that exists is in nightly versions of Firefox (and even then, for only a couple platforms). That being said early numbers show the performance being really, really good. For complex applications (such as the above games) performance is only around 2x slower than normally-compiled C++ (which is comparable to other languages like Java or C#). This is substantially faster than current browser runtimes, yielding performance that’s about 4-10x faster than the latest Firefox and Chrome builds.

This is a substantial improvement over the current best case. Considering how early on in the development of Asm.js is it’s very likely that there could be even greater performance improvements coming.

It is interesting to see such a large performance chasm appearing between Asm.js and the current engines in Firefox and Chrome. A 4-10x performance difference is substantial (this is in the realm of comparing these browsers to the performance of IE 6). Interestingly even with this performance difference many of these Asm.js demos are still usable on Chrome and Firefox, which is a good indicator for the current state of JavaScript engines. That being said their performance is simply not as good as the performance offered by a browser that is capable of optimizing Asm.js code.

Use Cases

It should be noted that almost all of the applications that are targeting Asm.js right now are C/C++ applications compiled to Asm.js using Emscripten. With that in mind the kind of applications that are going to target Asm.js, in the near future, are those that will benefit from the portability of running in a browser but which have a level of complexity in which a direct port to JavaScript would be infeasible.

So far most of the use cases have centered around code bases where performance is of the utmost importance: Such as in running games, graphics, programming language interpreters, and libraries. A quick look through the Emscripten project list shows many projects which will be of instant use to many developers.

• A number of game engines have already been ported. A good demo of what is possible is the BananaBread FPS Game (Source Code) which is playable directly in the browser and features multiplayer and bots.
• A port of LaTeX to JavaScript, called texlive.js, using Emscripten, allowing you to compile PDFs completely within your browser.
• A port of SQLite to JavaScript capable of running in Node.js.
• NaCL: A Networking and Cryptography Library

Asm.js Support

As mentioned before the nightly version of Firefox is currently the only browser that supports optimizing Asm.js code.

However it’s important to emphasize that Asm.js-formatted JavaScript code is still just JavaScript code, albeit with an important set of restrictions. For this reason Asm.js-compiled code can still run in other browsers as normal JavaScript code, even if that browser doesn’t support it.

The critical puzzle piece is the performance of that code: If a browser doesn’t support typed arrays or doesn’t specially-compile the Asm.js code then the performance is going to be much worse off. Of course this isn’t special to Asm.js, likely any browser that doesn’t have those features is also suffering in other ways.

Asm.js and Web Development

As you can probably see from the code above Asm.js isn’t designed to be written by hand. It’s going to require some sort of tooling to write and it’s going to require some rather drastic changes from how one would normally write JavaScript, in order to use. The most common use case for Asm.js right now is in applications complied from C/C++ to JavaScript. Almost none of these applications interact with the DOM in a meaningful way, beyond using WebGL and the like.

In order for it to be usable by regular developers there are going to have to be some intermediary languages that are more user-accessible that can compile to Asm.js. The best candidate, at the moment, is LLJS in which work is starting to get it compiling to Asm.js. It should be noted that a language like LLJS is still going to be quite different from regular JavaScript and will likely confuse many JavaScript users. Even with a nice more-user-accessible language like LLJS it’s likely that it’ll still only be used by hardcore developers who want to optimize extremely complex pieces of code.

Even with LLJS, or some other language, that could allow for more hand-written Asm.js code we still wouldn’t have an equally-optimized DOM to work with. The ideal environment would be one where we could compile LLJS code and the DOM together to create a single Asm.js blob which could be executed simultaneously. It’s not clear to me what the performance of that would look like but I would love to find out!

Q&A with David Herman

I sent some questions to David Herman (Senior Researcher at Mozilla Research) to try and get some clarification on how all the pieces of Asm.js fit together and how they expect users to benefit from it. He graciously took the time to answer the questions in-depth and provided some excellent responses. I hope you find them to be as illuminating as I did.

What is the goal of Asm.js? Who do you see as the target audience for the project?

Our goal is to make the open web a compelling virtual machine, a target for compiling other languages and platforms. In this first release, we’re focused on compiling low-level code like C and C++. In the longer run we hope to add support for higher-level constructs like structured objects and garbage collection. So eventually we’d like to support applications from platforms like the JVM and .NET.

Since asm.js is really about expanding the foundations of the web, there’s a wide range of potential audiences. One of the audiences we feel we can reach now is game programmers who want access to as much raw computational power as they can. But web developers are inventive and they always find ways to use all the tools at their disposal in ways no one predicts, so I have high hopes that asm.js will become an enabling technology for all sorts of innovative applications I can’t even imagine.

Does it make sense to create a more user-accessible version of Asm.js, like an updated version of LLJS? What about expanding the scope of the project beyond just a compiler target?

Absolutely. In fact, my colleague James Long recently announced that he’s done an initial fork of LLJS that compiles to asm.js. My team at Mozilla Research intends to incorporate James’s work and officially evolve LLJS to support asm.js.

In my opinion, you generally only want to write asm.js by hand in a very narrow set of instances, like any assembly language. More often, you want to use more expressive languages that compile efficiently to it. Of course, when languages get extremely expressive like JavaScript, you lose predictability of performance. (My friend Slava Egorov wrote a nice post describing the challenges of writing high-performance code in high-level languages.) LLJS aims for a middle ground — like a C to asm.js’s assembly — that’s easier to write than raw asm.js but has more predictable performance than regular JS. But unlike C, it still has smooth interoperability with regular JS. That way you can write most of your app in dynamic, flexible JS, and focus on only writing the hottest parts of your code in LLJS.

There is talk of a renewed performance divide between browsers that support Asm.js and browsers that don’t, similar to what happened during the last JavaScript performance race in 2008/2009. Even though technically Asm.js code can run everywhere in reality the performance difference will simply be too crippling for many cases. Given this divide, and the highly restricted subset of JavaScript, why did you choose JavaScript as a compilation target? Why JavaScript instead of a custom language or plugin?

First of all, I don’t think the divide is as stark as you’re characterizing it: we’ve built impressive demos that work well in existing browsers but will benefit from killer performance with asm.js.

It’s certainly true that you can create applications that will depend on the increased performance of asm.js to be usable. At the same time, just like any new web platform capability, applications can decide whether to degrade gracefully with some less compute-intensive fallback behavior. There’s a difference in kind between an application that works with degraded performance and an application that doesn’t work at all.

More broadly, keep in mind the browser performance race that started in the late 00’s was great for the web, and applications have evolved along with the browsers. I believe the same thing can and will happen with asm.js.

How would you compare Asm.js with Google’s Native Client? They appear to have similar goals while Asm.js has the advantage of “just working” everywhere that has JavaScript. Have there been any performance comparisons?

Well, Native Client is a bit different, since it involves shipping platform-specific assembly code; I don’t believe Google has advocated for that as a web content technology (as opposed to making it available to Chrome Web Store content or Chrome extensions), or at least not recently.

Portable Native Client (PNaCl) has a closer goal, using platform-independent LLVM bitcode instead of raw assembly. As you say, the first advantage of asm.js is compatibility with existing browsers. We also avoid having to create a system interface and repeat the full surface area of the web API’s as the Pepper API does, since asm.js gets access to the existing API’s by calling directly into JavaScript. Finally, there’s the benefit of ease of implementability: Luke Wagner got our first implementation of OdinMonkey implemented and landed in Firefox in just a few months, working primarily by himself. Because asm.js doesn’t have a big set of syscalls and API’s, and because it’s built off of the JavaScript syntax, you can reuse a whole bunch of the machinery of an existing JavaScript engine and web runtime.

We could do performance comparisons to PNaCl but it would take some work, and we’re more focused on closing the gap to raw native performance. We plan to set up some automated benchmarks so we can chart our progress compared with native C/C++ compilers.

Emscripten, another Mozilla project, appears to be the primary producer of Asm.js-compatible code. How much of Asm.js is being dictated by the needs of the Emscripten project? What benefits has Emscripten received now that improvements are being made at the engine level?

We used Emscripten as our first test case for asm.js as a way to ensure that it’s got the right facilities to accommodate the needs of real native applications. And of course benefiting Emscripten benefits everyone who has native applications they want to port — such as Epic Games, who we teamed up with to port the Unreal Engine 3 to the web in just a few days using Emscripten and asm.js.

But asm.js can benefit anyone who wants to target a low-level subset of JavaScript. For example, we’ve spoken with the folks who build the Mandreel compiler, which works similarly to Emscripten. We believe they could benefit from targeting asm.js just as Emscripten has started doing.

Alon Zakai has been compiling benchmarks that generally run around 2x slower than native, where we were previously seeing results anywhere from 5x to 10x or 20x of native. This is just in our initial release of OdinMonkey, the asm.js backend for Mozilla’s SpiderMonkey JavaScript engine. I expect to see more improvements in coming months.

How fluid is the Asm.js specification? Are you open to adding in additional features (such as more-advanced data structures) as more compiler authors being to target it?

You bet. Luke Wagner has written up an asm.js and OdinMonkey roadmap on the Mozilla wiki, which discusses some of our future plans — I should note that none of these are set in stone but they give you a sense of what we’re working on. I’m really excited about adding support for ES6 structured objects. This would provide garbage-collected but well-typed data structures, which would help compilers like JSIL that compile managed languages like C# and Java to JavaScript. We’re also hoping to use something like the proposed ES7 value types to provide support for 32-bit floats, 64-bit integers, and hopefully even fixed-length vectors for SIMD support.

Is it possible, or even practical, to have a JavaScript-to-Asm.js transpiler?

Possible, yes, but practical? Unclear. Remember in Inception how every time you nested another dream-within-a-dream, time would slow down? The same will almost certainly happen every time you try to run a JS engine within itself. As a back-of-the-envelope calculation, if asm.js runs native code at half native speed, then running a JS engine in asm.js will execute JS code at half that engine’s normal speed.

Of course, you could always try running one JS engine in a different engine, and who knows? Performance in reality is never as clear-cut as it is in theory. I welcome some enterprising hacker to try it! In fact, Stanford student Alex Tatiyants has already compiled Mozilla’s SpiderMonkey engine to JS via Emscripten — all you’d have to do is use Emscripten’s compiler flags to generate asm.js. Someone with more time on their hands than me should give it a try…

At the moment all DOM/browser-specific code is handled outside of Asm.js-land. What about creating an Emscripten-to-Asm.js-compiled version of the DOM (akin to DOM.js)?

This is a neat idea. It may be a little tricky with the preliminary version of asm.js, which doesn’t have any support for objects at all. As we grow asm.js to include support for ES6 typed objects, something like this could become feasible and quite efficient!

A cool application of this would be to see how much of the web platform could be self-hosted with good performance. One of the motivations behind DOM.js was to see if a pure JS implementation of the DOM could beat the traditional, expensive marshaling/unmarshaling and cross-heap memory management between the JS heap and the reference-counted C++ DOM objects. With asm.js support, DOM.js might get those performance wins plus the benefits of highly optimized data structures. It’s worth investigating.

Given that it’s fairly difficult to write Asm.js, compared with normal JavaScript, what sorts of tools would you like to have to help both developers and compiler authors?

First and foremost we’ll need languages like LLJS, as you mentioned, to compile to asm.js. And we’ll have some of the usual challenges of compiling to the web, such as mapping generated code back to the original source in the browser developer tools, using technologies like source maps. I’d love to see source maps developed further to be able to incorporate richer debugging information, although there’s probably a cost/benefit balance to be struck between the pretty minimal source location information of source maps and super-complex debugging metadata formats like DWARF.

For asm.js, I think we’ll focus on LLJS in the near term, but I always welcome ideas from developers about how we can improve their experience.

I assume that you are open to working with other browser vendors, what has collaboration or discussion been like thus far?

Definitely. We’ve had a few informal discussions and they’ve been encouraging so far, and I’m sure we’ll have more. I’m optimistic that we can work with multiple vendors to get asm.js somewhere that we all feel we can realistically implement without too much effort or architectural changes. As I say, the fact that Luke was able to implement OdinMonkey in a matter of just a few months is very encouraging. And I’m happy to see a bug on file for asm.js support in V8.

More importantly, I hope that developers will check out asm.js and see what they think, and provide their feedback both to us and other browser vendors.

These are a few arguments against the switch that I’ve seen so far:

A browser switching to WebKit will result in stagnation

That is demonstrably not true. KDE created KHTML, Apple created WebKit based upon that, Google created WebKit/Chromium based upon that. I don’t think anyone can successfully argue that Chome/Chromium isn’t a better browser than Safari which isn’t a better browser than Konqueror. The Chrome team has proved that stagnation when using WebKit is merely a choice, as a contributor to WebKit you have the complete ability to drive it in a direction you wish (often for the better). I see no reason why the highly-skilled development team at Opera won’t be able to do the same. They can implement a number of their Opera-specific features into WebKit and it’s likely that those features will start to trickle back into other WebKit-using browsers as well.

This is helping to making WebKit a de facto standard, bugs-and-all

I don’t see this argument as being relevant any more, WebKit is already a de facto standard. I mean, everyone remembers when the browsers decided to implement -webkit vendor prefixes? It’s obvious that the “WebKit is an de facto standard” horse has already left the gate. As to the bugs: WebKit is a common code base that a number of browsers contribute to, however every browser vendor has the ability to make changes to their own fork of the code base. I see no reason why these “now-standard” WebKit bugs wouldn’t be fixed by any single vendor. Having a bug in the engine does not mean that a single browser vendor is incapable of fixing it — they would just be willfully not fixing it (as the browser vendors currently willfully clone -webkit prefixes).

In the case of JavaScript libraries virtually everyone has standardized upon jQuery at this point. This didn’t result in stagnation, which was a major concern, instead it’s resulted in a number of interesting and hyper-popular second-tier frameworks which build upon jQuery, such as: Twitter Bootstrap, HTML5 Boilerplate, and Backbone.js.

This will affect Opera’s ability to influence standards

I don’t see the switch to WebKit causing this. I do see Anne van Kesteren‘s move to Mozilla as being a massive blow to Opera’s ability to push standards though. I don’t know anything about the situation but if his moving was caused by the switch to WebKit then yes, Opera’s move to WebKit has affected their ability to influence standards.

Opera switching to WebKit is a slippery slope and/or Opera is a small player, Firefox or IE switching to WebKit would be a bigger problem

I think one this is clear already: WebKit has completely and unequivocally won mobile at this point. They are nearly the only rendering engine used on the vast majority of mobile browsers, including the soon-to-switch Opera Mini/Mobile browsers too. There is no reason to worry about a slippery slope, the slope has already been slid down. In order for any other browser to remain relevant in the world of mobile (which, you must admit, is quickly becoming the only world we live in) they must keep feature parity with WebKit. Let’s follow this to its logical conclusion: In a world in which WebKit is now virtually the only mobile browser vendor Mozilla and Microsoft will feel increased pressure to switch their browser engines over to WebKit in order to keep pace. Google has proved that with Chrome that WebKit stagnation is simply a choice so there’s no reason why these other companies shouldn’t be able to build off of WebKit (and possibly create WebKit hybrids, such as WebKit + IonMonkey).

The big question becomes: Should they switch?

At this point it’s honestly a business/engineering decision for Mozilla and Microsoft (as it always has been). If some percentage of your developer force is spending all of their time implementing the same standards that everyone else is implementing then switching to a common code base will give you the ability to free up some of your developers to work on something else. You saw what happened in the case of Chrome: They used their extra developer time to completely crush the competition on performance. This resulted in the everyone-wins race to become the fastest browser.

Ultimately it’s important to remember that WebKit is not a monolithic entity. It’s a shared codebase that a number of corporations contribute to. (In this respect it’s different from jQuery: Almost all contributions to jQuery comes back into the main codebase, whereas with WebKit some come back to the main codebase and some stay in a fork.) There is a lot of code sharing going on but it isn’t the be-all and end-all of browser development. Innovation can clearly still occur when working on a shared codebase and performance will almost certainly continue to improve.

I’ve found that this occurs when people are doing inline event tracking and are assuming that the Google Analytics functionality will always exist. Ghostery-or-not this is generally not a good practice to have: Assuming that some external, 3rd-party, script will always correctly load will result in many a broken web page.

You see cases where people work around this issue, for example in the HTML5 Boilerplate project:

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script>window.jQuery || document.write('<script src="js/vendor/jquery-1.9.1.min.js"><\/script>')</script>

This attempts to load the jQuery library directly from Google’s CDN and if it fails (for whatever reason — Google’s down, bad Internet connection, or some other weird reason) then it falls back to attempting to load a local version of jQuery. One question that comes of this is: Why not just always load jQuery from your local source rather than rely upon a third party? In this case it makes a lot of sense to pull jQuery from a CDN as you’ll benefit from the performance and caching benefits that a CDN provides, all while reducing the total bandwidth that you’ll have to expend on your end.

In the case of Google Analytics you typically don’t have that luxury: Google recommends that you load the ga.js library directly from their site (which can make sense as then they can push out seamless API changes and bug fixes). However this still causes issues for when Google Analytics simply doesn’t load, wether due to network conditions or due to Ghostery or some other browser extension.

For this reason, in a site I recently built, I added in the following code as a light wrapper around Google’s event tracking API:

// Add in Google Analytics tracking
var track = function(category, name, value) {
	if (window._gaq) {
		window._gaq.push(["_trackEvent", category, name, value]);
	}
};

and then when I wanted to track an actual event I would do something like this:

track("Source", "Visit", this.href);

If Google Analytics has failed to load for some reason then the track() call just silently does nothing – and for users who have Google Analytics installed it tracks the event. I should note that this same technique applies for other event tracking APIs as well, such as Mixpanel or KISSmetrics (they’re blocked the same as Google Analytics).

Even libraries like Segment.io, which act as a unified wrapper around tracking, are prone to the same issue. Even though they’ve gone through all the hard work of creating a unified API for event tracking they are still blocked by Ghostery, which will likely still result in errors in your application. In that case you would just write a similar track(...) wrapper around their analytics.track('Name', properties) call.

Update: The current version of the Google Analytics script always exposes the _gaq object, even if the code isn’t loaded, so this may be a non-issue for Google Analytics (although the principles here hold true for any third-party-included libraries). I’ve gotten word from the Ghostery team that Ghostery now handles this case, automatically exposing missing global objects for cases like this (see below in the comments). I’ve also gotten word from the Segment.io team that, as of now, Ghostery does not block their analytics tracking so you may not need a wrapper there. Additionally Segment.io does check to make sure that the potentially-blocked global objects are actually there before attempting to use them. In general though, and this goes for all third-party script, you’ll probably want to have a wrapper (or an alternative script) if you’re loading code from a third-party domain name.

Happy news! My book, Secrets of the JavaScript Ninja is finally in stock on Amazon! It’s been available on Manning.com for over a month now but I think Amazon has been struggling to keep the books consistently in stock.

You can get an ePub or Kindle version of the book at Manning.com.

I’ve written about the writing process for this book previously (and why it’s taken so long to get out). I especially want to thank my co-author Bear Bibeault who worked hard to get this book out the door.

One question that I’ve been asked recently: “How relevant is this book to writing JavaScript in 2012?” I think it’s still exceedingly relevant. This book has two major focuses:

1. Teaching aspects of the JavaScript language that are under-served or poorly explained. (For example I’ve been getting a number of messages from people saying how much they appreciated the explanation of closures in the book and that it finally helped them to understand that feature of the language.)
2. Helping people understand the internal workings of cross-browser DOM-centric JavaScript libraries (such as jQuery). Considering how popular jQuery is (64% of the top 10,000 web sites use jQuery) and, relatively, how few people truly understand what the library does or is trying to do – there is still a huge market for people to learn what exactly jQuery is and how it works.

I’m really happy that this book is finally out and to celebrate I’ve been trying to blog more (as you may have noticed). Are there any particular topics that you’d like me to write about? Let me know!

It has to deal with the eternal question: How do you store sensitive configuration options (such as usernames, passwords, etc.) in source control? Typically what I’ve done is to just punt on the problem entirely. I create a dummy configuration file, such as conf/sample-settings.json which has the basic structure but none of the details filled out. For example:

conf/sample-settings.json

// Copy to conf/settings.json
// and fill these in with your login details!
{
  "db": {
    "username": "",
    "password": ""
  }
}

If someone else needed the details I would just email it to them, or some such (not ideal). Especially when it came time to add additional information to the file or make other changes.

The technique I picked up from Craig was to, instead, keep an encrypted version of the configuration file in source control and then provide a means through which the user can encrypt and decrypt that data.

In this case you can still have the a dummy config file, if you wish.

To start you’ll want to make sure you have your source control ignore the configuration file — just to make super-sure that no one ever accidentally commits it. In Git you’d add a line like this to your .gitignore file:

.gitignore

conf/settings.json

Next you’ll want to create your actual config file and populate it with the real values.

conf/settings.json (* Do not check this in to source control!!)

{
  "db": {
    "username": "cool_guy",
    "password": "A1B2C3!"
  }
}

Finally you’ll want to create a script (I’m using a Makefile) that the user can run to encrypt and decrypt the file. This script uses OpenSSL, and specifically CAST5, to encrypt/decrypt the file. OpenSSL was chosen in particular as it worked out-of-the-box on both Linux and Mac machines.

OpenSSL reads in the appropriate files (depending upon if you’re encrypting or decrypting) then will prompt you for a password to encrypt/decrypt the file. (You’re free to use any encryption scheme that OpenSSL supports, of course.)

Makefile

.PHONY: _pwd_prompt decrypt_conf encrypt_conf

CONF_FILE=conf/settings.json

# 'private' task for echoing instructions
_pwd_prompt:
    @echo "Contact jeresig@gmail.com for the password."

# to create conf/settings.json
decrypt_conf: _pwd_prompt
    openssl cast5-cbc -d -in ${CONF_FILE}.cast5 -out ${CONF_FILE}
    chmod 600 ${CONF_FILE}

# for updating conf/settings.json
encrypt_conf: _pwd_prompt
    openssl cast5-cbc -e -in ${CONF_FILE} -out ${CONF_FILE}.cast5

With all this in place the next step is simple, you’ll run:

make encrypt_conf

and you’ll enter in a password with which to encrypt the config file:

Contact jeresig@gmail.com for the password.
enter cast5-cbc decryption password:

Make sure you write this down and don’t forget it — it’ll be very hard (if not impossible) to get your config file back if you forget the password.

At this point you’ll have a conf/settings.json.cast5 file and you can commit all the changes, using something like:

git add .gitignore Makefile conf/settings.json.cast5
git commit -m "Adding in an encrypted config file."

Now whenever someone downloads the code from source control they’ll need to either fill in their own values into the config file or they’ll need to get the password from you (the one you entered when you ran make encrypt_conf — or even better, use a shared password safe to manage this). Once they have the password they just run the following and enter it:

make decrypt_conf

If you ever need to update the values in the config file, it’s really straight-forward. Just update the config file, run make encrypt_conf again, and commit the new conf/settings.json.cast5 file.

One extra bit that you can add to your application, to make this process more intuitive, is a check for a missing config file and output with instructions for using the Makefile.

For example if you were using Node.js you could do:

if (!fs.existsSync("conf/settings.json")) {
  console.error("Config file [conf/settings.json] missing!");
  console.error("Did you forget to run `make decrypt_conf`?");
  process.exit(1);
}

Also, you may want to consider having a check to see if the decrypted file is out of date (which can happen if some changes were made in the source control, then were checked out, but you didn’t also run make decrypt_conf). Perhaps something like the following:

(function() {
var conf_time = fs.statSync(“conf/settings.json”).mtime.getTime();
var cast5_time = fs.statSync(“conf/settings.json.cast5”).mtime.getTime();

if (conf_time < cast5_time) { console.error("Your config file is out of date!"); console.error("You need to run `make decrypt_conf` to update it."); process.exit(1); } })();[/js] And that's it! Simpler than passing around config files manually and you still get all the benefit of using revision control to manage the file and changes.

The module is now available on NPM and can be installed by running:

npm install i18n-2

The code and documentation for the module is available on Github here:

https://github.com/jeresig/i18n-node-2

I’ve designed the API to work as simply as possible with Express.js – while still supporting a simple Node.js usage or usage with another web framework.

The module includes a number of features built-in:

• Automatic integration with Express as middleware and integration with templating solutions.
• Support for handling the setting of locale based upon query strings and/or subdomain.
• Dynamic reading and updating of translation files during development mode.
• Automatic caching of translation files in production mode.

Usage is simple, in your Express app.js:

// load modules
var express = require('express'),
	I18n = require('i18n-2');

// Express Configuration
app.configure(function() {
	// ...

	// Attach the i18n property to the express request object
	// And attach helper methods for use in templates
	I18n.expressBind(app, {
		// setup some locales - other locales default to en silently
		locales: ['en', 'de']
	}));

	// Set up the rest of the Express middleware
	app.use(app.router);
	app.use(express.static(__dirname + '/public'));
});

Inside your Express view:

module.exports = {
	index: function(req, res) {
		req.render("index", {
			title: req.i18n.__("My Site Title"),
			desc: req.i18n.__("My Site Description")
		});
	}
};

Inside your templates (this example uses the Swig templating system).

{% extends "page.swig" %}
{% block content %}
<h1>{{ __("Welcome to: %s", title) }}</h1>
<p>{{ desc }}</p>
{% endblock %}

My module is based off of the i18n-node module by Marcus Spiegel.

I used the original i18n-node module as a basis for further work as it provided the closest solution to full Express.js integration that I needed. After some initial usage though I saw that it was very much lacking. As I described in my previous post there is potential for severe problems to occur in an asynchronous environment like Node.js.

The problem that I was having was rather severe and, unfortunately, required a complete rewrite of the module. The original i18n-node module stored the locale information in a closure. Calling setLocale changed the locale but did so for every instance that was currently active, resulting in templates being rendered in the wrong language.

Here is the issue that is especially surprising (but not so much once you know how modules work in Node.js): Since the state was stored in a closure repeated calls to require('i18n') resulted in the current locale state being held the entire time, not reset. This meant that when a new request came in, the locale was set, changing the locale for all other active requests.

The only way to fix this particular problem is to simply encapsulate all state into a single object — I did this by making an object that you instantiate: new I18n(options).

I looked at other JavaScript translation solutions, including the excellent Jed and messageformat and while their translation capabilities are top-notch what they provided was overkill for my particular application and did not provide the Express.js integration that I needed.

I hope others will find this module to be useful, especially when used in conjunction with my strategy for i18n and Node.js.