Testacular - Spectacular Test Runner for JavaScript

By Vojta Jína

[NOTE: After this post was published, Testacular was renamed Karma.]

“Testacular has changed my life. Now I test-drive everything.”

-- Matias Cudich, YouTube on TV team lead

At Google we believe in testing. On the AngularJS team, it’s even worse - we are super crazy about testing. Every feature of the framework is designed with testability in mind.

We found that we were struggling with existing tools, so we decided to write our own test runner. We wanted a test runner that would meet all of our needs for both quick development and continuous integration -- a truly spectacular test runner. We've called it Testacular.

Let's walk through some mental tests for what we believe makes for an ideal test runner...

it(‘should be fast’)
In order to be productive and creative you need instant feedback. Testacular watches the files in your application. Whenever you change any of them, it immediately executes the specified tests and reports the results. You never have to leave your text editor.

This enables a new way of developing. Instead of moving back and forth between the editor and the browser, you can simply stay in the editor and experiment. You instantly see the results at the command line whenever your changes are saved.

Besides that, our experience says that if test execution is slow, people don’t write tests. Testacular eliminates many barriers that keep folks from writing tests. When developers get instant feedback from their tests, the tests become an asset rather than annoyance.

it(‘should use real browsers’)
JavaScript itself is pretty consistent between different browsers, so one could potentially test browser code in non-browser environments like Node.js. Unfortunately, that’s not the case with the DOM APIs. AngularJS does a lot of DOM manipulation, and we need to be sure that it works across browsers. Executing tests on real browsers is a must.

And because Testacular communicates with browsers through a common protocols (eg. HTTP or WebSocket), you can test not only on desktop browsers but also on other devices such as mobile phones and tablets. For instance, the YouTube team uses Testacular to run continuous integration builds on PlayStation 3.

Another advantage of using real browsers is that you can use any of the tools that the browser provides. For example, you can jump into a debugger and step through your test.

it(‘should be reliable/stable’)
To be honest, most of these ideas were already implemented in JsTD almost three years ago. I think these are truly great ideas. Unfortunately, the implementation was flaky. It’s very easy to get JsTD into an inconsistent state, so you end up restarting it pretty much all day.

Testacular solves that. It can run for days, without restarting. That’s because every test run is executed in a fresh iframe. It reconnects browsers that have lost their connection to the server. And yep, it can gracefully recover from other issues, like syntax errors in the code under test.

We'd like to invite you to take Testacular for a spin. You can learn a bit more in this screencast. Please let us know what you think!

The project is open sourced and developed on GitHub.

ScriptCover makes Javascript coverage analysis easy

By Ekaterina Kamenskaya, Software Engineer in Test, YouTube

Today we introduce the Javascript coverage analysis tool, ScriptCover. It is a Chrome extension that provides line-by-line Javascript code coverage statistics for web pages in real time without any user modifications required. The results are collected both when the page loads and as users interact with it. The tool reports details about total web page coverage and for each external/internal script, as well as annotated code sources with individually highlighted executed lines.

Short report in Chrome extension’s popup, detailing both overall scores and per-script coverage.

Main features:

• Report current and previous total Javascript coverage percentages and total number of instrumented code instructions.
• Report Javascript coverage per individual instruction for each internal and external script.
• Display detailed reports with annotated Javascript source code.
• Recalculate coverage statistics while loading the page and on user actions.

Sample of annotated source code from detailed report. First two columns are line number and number of times each instruction has been executed.

Here are the benefits of ScriptCover over other existing tools:

• Per instructions coverage for external and internal scripts: The tool formats original external and internal Javascript code from ‘<script>’ tags to ideally place one instruction per line and then calculates and displays Javascript coverage statistics. It is useful even when the code is compressed to one line.

• Dynamic: Users can get updated Javascript coverage statistics while the web page is loading and while interacting with the page.

• Easy to use: Users with different levels of expertise can install and use the tool to analyse coverage. Additionally, there is no need to write tests, modify the web application’s code, save the inspected web page locally, manually change proxy settings, etc. When the extension is activated in a Chrome browser, users just navigate through web pages and get coverage statistics on the fly.

• It’s free and open source!

         
Want to try it out? Install ScriptCover and let us know what you think.

We envision many potential features and improvements for ScriptCover. If you are passionate about code coverage, read our documentation and participate in discussion group. Your contributions to the project’s design, code base and feature requests are welcome!

Google JS Test, now in Open Source

By Aaron Jacobs

Google JS Test is a JavaScript unit testing framework that runs on the V8 JavaScript Engine, the same open source project that is responsible for Google Chrome’s super-fast JS execution speed. Google JS Test is used internally by several Google projects, and we’re pleased to announce that it has been released as an open source project.

Features of Google JS Test include:

• Extremely fast startup and execution time, without needing to run a browser.

• Clean, readable output in the case of both passing and failing tests.

• An optional browser-based test runner that can simply be refreshed whenever JS is changed.

• Style and semantics that resemble Google Test for C++.

• A built-in mocking framework that requires minimal boilerplate code (e.g. no $tearDown or$verifyAll calls), with style and semantics based on the Google C++ Mocking Framework.

• A system of matchers allowing for expressive tests and easy to read failure output, with many built-in matchers and the ability for the user to add their own.
  
  

See the Google JS Test project home page for a quick introduction, and the getting started page for a tutorial that will teach you the basics in just a few minutes.

Super Fast JS Testing

by Shyam Seshadri

Before I jump into how exactly you can perform super fast and easy JS testing, let me give you some background on the problem.

Javascript is a finicky language (Some people even hesitate to call it a language). And it can easily grow and become a horrible and complicated beast, incapable of being tamed once let loose. And testing it is a nightmare. Once you have decided on a framework (of which there are a dime a dozen), you then have to set it up to run just right. You need to set it up to actually run your tests. Then you have to figure out how to run it in a continuous integration environment. Maybe even run it in headless mode. And everyone solves it in their own ways.

But the biggest problem I have with most of these frameworks is that executing the tests usually requires a context switch. By that, I mean to run a JSUnit test, you end up usually having to open the browser, browse to a particular url or html page which then runs the test. Then you have to look at the results there, and then come back to your editor to either proceed further or fix your tests. Works, but really slows down development.

In java, all it takes is to click the run button in your IDE to run your tests. You get instant feedback, a green / red bar and details on which tests passed and failed and at what line. No context switch, you can get it to run at every save, and proceed on your merry way. Till now, this was not possible with Javascript.

But now, we have JS Test Driver. My colleagues Jeremie and Misko ended up running into some of the issues I outlined above, and decided that going along with the flow was simply unacceptable. So they created a JS Testing framework which solves these very things. You can capture any browser on any machine, and when you tell it to run tests, it will go ahead and execute them on all these browsers and return you the results in your client. And its blazing fast. I am talking milliseconds to run 100 odd tests. And you can tell it to rerun your tests at each save. All within the comforts of your IDE. And over the last three weeks, I have been working on the eclipse plugin for JS Test Driver, and its now at the point where its in a decent working condition.

The plugin in action

The plugin allows you to, from within Eclipse, start the JS Test Driver server, capture some browsers, and then run your tests. You get pretty icons telling you what browsers were captured, the state of the server, the state of the tests. It allows you to filter and show only failures, rerun your last launch configuration, even setup the paths to your browsers so you can launch it all from the safety of eclipse. And as you can see, its super fast. Some 100 odd tests in less than 10 ms. If thats not fast, I don’t know what is.

For more details on JS Test Driver, visit its Google Code website and see how you can use it in your next project and even integrate it into a continuous integration. Misko talks a little bit more about the motivations behind writing it on his Yet Another JS Testing Framework post. To try out the plugin for yourselves, go add the following update site to eclipse:

http://js-test-driver.googlecode.com/svn/update/

For all you IntelliJ fanatics, there is something similar in the works.

Yet Another JavaScript Testing Framework

by Miško Hevery & Jeremie Lenfant-engelmann

Did you notice that there are a lot of JavaScript testing frameworks out there? Why has the JavaScript community not consolidated on a single JavaScript framework the way Java has on JUnit. My feeling is that all of these frameworks are good at something but none solve the complete package. Here is what I want out of JavaScript unit-test framework:

I want to edit my JavaScript code in my favorite IDE, and when I hit Ctrl-S, I want all of the tests to execute across all browsers and return the results immediately.

I don't know of any JavaScript framework out there which will let me do what I want. In order to achieve my goal I need a JavaScript framework with these three features:

Command Line Control

Most JavaScript test runners consist of JavaScript application which runs completely in the browser. What this means in practice is that I have to go to the browser and refresh the page to get my results. But browsers need an HTML file to display, which means that I have to write HTML file which loads all of my production code and my tests before I can run my tests. Now since browsers are sandboxes, the JavaScript tests runner can only report the result of the test run inside the browser window for human consumption only. This implies that 1) I cannot trigger running of the tests by hitting Ctrl-S in my IDE, I have to Alt-tab to Browser, hit refresh and Alt-tab back to the IDE and 2) I cannot display my test result in my IDE, the results are in the browser in human readable form only.

On my continuous build machine I need to be able to run the same tests and somehow get the failures out of the browser and on to the status page. Most JavaScript test runners have a very poor story here, which makes integrating them into a continuous build very difficult.

What we need, is the ability to control test execution from command line so that I can trigger it from my IDE, or my continuous build machine. And I need test failures to be reported on the command line (not inside the browser where they are unreachable) so that I can display them in IDE or in continuous build status page.

Parallel Execution

Since most JavaScript test runners run fully in the browser I can only run my test on one browser at a time during my development process. In practice this means that you don't find out about failures in other browser until you have checked in the code to your continuous build machine (if you were able to set it up) and your code executes on all browsers. By that point you have completely forgotten about what you have written and debugging becomes a pain. When I run my test I want to run them on all browser platforms in parallel.

Instant Feedback in IDE

After I hit Ctrl-S on my IDE, my patience for test results is about two seconds before I start to get annoyed. What this means in practice is that you can not wait until the browser launches and runs the tests. The browser needs to be already running. Hitting refresh on your browser manually is very expensive since the browser needs to reload all of the JavaScript code an re-parse it. If you have one HTML file for each TestCase and you have hundred of these TestCases, The browser may be busy for several minutes until it reloads and re-parses the same exact production code once for each TestCase. There is no way you can fit that into the patience of average developer after hitting Ctrl-S.

Introducing JsTestDriver

Jeremie Lenfant-engelmann and I have set out to build a JavaScript test runner which solves exactly these issues so that Ctrl-S causes all of my JavaScript tests to execute in under a second on all browsers. Here is how Jeremie has made this seemingly impossible dream a reality. On startup JsTestDriver captures any number of browsers from any number of platforms and turns them into slaves. As slave the browser has your production code loaded along with all of your test code. As you edit your code and hit Ctrl-S the JsTestDriver reloads only the files which you have modified into the captured browsers slaves, this greatly reduces the amount of network traffic and amount of JavaScript re-parsing which the browser has to do and therefore greatly improves the test execution time. The JsTestDriver than runs all of your test in parallel on all captured browsers. Because JavaScript APIs are non-blocking it is almost impossible for your tests to run slow, since there is nothing to block on, no network traffic and no re-parsing of the JavaScript code. As a result JsTestDriver can easily run hundreds of TestCases per second. Once the tests execute the results are sent over the network to the command which executed the tests either on the command line ready to be show in you IDE or in your continuous build.

Demo

TotT: Simulating Time in jsUnit Tests

Sometimes you need to test client-side JavaScript code that uses setTimeout() to do some work in the future. jsUnit contains the Clock.tick() method, which simulates time passing without causing the test to sleep.
For example, this function will set up some callbacks to update a status message over the course of four seconds:

function showProgress(status) {
  status.message = "Loading";
  for (var time = 1000; time <= 3000; time+= 1000) {
    // Append a '.' to the message every second for 3 secs.
    setTimeout(function() {
      status.message += ".";
    }, time);
  }
  setTimeout(function() {
    // Special case for the 4th second.
    status.message = "Done";
  }, 4000);
}

The jsUnit test for this function would look like this:

function testUpdatesStatusMessageOverFourSeconds() {
  Clock.reset(); // Clear any existing timeout functions on the event queue.
  var status = {};
  showProgress(status); // Call our function.

  assertEquals("Loading", status.message);

  Clock.tick(2000); // Call any functions on the event queue that have been
                    // scheduled for the first two seconds.
  assertEquals("Loading..", status.message);

  Clock.tick(2000); // Same thing again, for the next two seconds.
  assertEquals("Done", status.message);
}

This test will run very quickly - it does not require four seconds to run.

Clock supports the functions setTimeout(), setInterval(), clearTimeout(), and clearInterval(). The Clock object is defined in jsUnitMockTimeout.js, which is in the same directory as jsUnitCore.js.

Remember to download this episode of Testing on the Toilet and post it in your office.

TotT: JavaScript: Simulating Time in jsUnit Tests

Sometimes you need to test client-side JavaScript code that uses setTimeout() to do some work in the future. jsUnit contains the Clock.tick() method, which simulates time passing without causing the test to sleep. For example, this function will set up some callbacks to update a status message over the course of four seconds:

function showProgress(status) {
  status.message = "Loading";
  for (var time = 1000; time <= 3000; time += 1000) {
    // Append a '.' to the message every second for 3 secs.
    setTimeout(function() {
      status.message += ".";
    }, time);
  }
  setTimeout(function() {
    // Special case for the 4th second.
    status.message = "Done";
  }, 4000);
}

The jsUnit test for this function would look like this:

function testUpdatesStatusMessageOverFourSeconds() {
  Clock.reset(); // Clear any existing timeout functions on the event queue.
  var  status = {};
  showProgress(status); // Call our function.
  assertEquals("Loading", status.message);
  Clock.tick(2000); // Call any functions on the event queue that have
                    // been scheduled for the first two seconds.
  assertEquals("Loading..",  status.message);
  Clock.tick(2000); // Same thing again, for the next two seconds.
  assertEquals("Done", status.message);
}

This test will run very quickly - it does not require four seconds to run.

Clock supports the functions setTimeout(),
setInterval(), clearTimeout(), and
clearInterval(). The Clock object is defined in
jsUnitMockTimeout.js, which is in the same directory as
jsUnitCore.js.

Remember to download this episode of Testing on the Toilet and post it in your office.