We strive to practice Test-Driven Development (TDD) -- that means we try to write our tests first, and then write our code. While we don't always live up to this ideal, we do write automated tests for as much of our back-end code as possible, and in most cases, we do not accept pull requests for code changes that aren't covered by automated tests.

We use RSpec unit tests (located in the spec folder) and Cucumber integration tests (in the features folder).

In addition to running tests on our development environments, we also run them using two continuous integration services:

• Travis-CI runs our tests when a pull request is submitted or merged
• Codeship also runs tests when a pull request is merged, and in addition to that, it deploys the new code to our staging environment

Table of Contents Unit Tests Integration Tests Running Tests Prepare the database Run tests Generate coverage Known Test Failures Pending Tests

Unit Tests

Use RSpec for unit testing Ruby code. Unit tests match to code functionality and not to features, and run quickly. They are helpful to ensure that you're not changing behavior accidentally when refactoring or adding new functionality, and that your code is working the way you'd expect it to.

Writing unit tests makes it easier to catch mistakes before committing them and reduces the burden on our human testing volunteers.

Resources on writing good tests with RSpec:

• Everyday Rails Testing with RSpec
• Better Specs
• RSpec Style Guide
• Structure of RSpec Tests Slides

Integration Tests

We write specification in Cucumber. These specify behavior and are easy to understand if you're not a developer.

They run much more slowly than RSpec; don't introduce too many unnecessary steps, and be mindful of unnecessary database calls!

Resources on writing good tests with Cucumber:

• Write Great Cucumber Tests
• Writing Better Cucumber Scenarios; or, Why We're Deprecating FactoryGirl's Cucumber Steps

Running Tests

The instructions below are for a local install. If you are using Vagrant, please follow the testing instructions on our Vagrant page.

Prepare the database

Before you run the automated tests, you may need to prepare your test database. This is usually only necessary the first time you run tests or after running a migration:

  bundle exec rake db:test:prepare

Failing that, you can recreate the test database altogether:

  mysql -e "drop database vagrant_test;"
  mysql -e "create database vagrant_test DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;"
  RAILS_ENV=test bundle exec rake db:schema:load
  RAILS_ENV=test bundle exec rake db:migrate

Run tests

• Run all RSpec tests: RAILS_ENV=test bundle exec rspec spec
• Run a specific RSpec test: RAILS_ENV=test bundle exec rspec spec/my_model_spec.rb
• Run all Cucumber tests: RAILS_ENV=test bundle exec cucumber features
• Run a specific Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature
• Run a specific scenario within a Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature:10, where :10 is the line number of the scenario
• See verbose output: add --format pretty (more info)

Generate coverage

Whenever you run a test, coverage results will be at coverage/index.html, which can be viewed in a browser.

You should also check out Codecov, if you need to generate coverage for the entire test suite.

Known Test Failures

Some tests currently fail when run locally, but not in CI, due to environment differences. These include:

• LogfileReader with some logfiles reads rows correctly with various patterns (OS specific)
• WorkSearch tests (can't convert SearchResult to Array)

Other tests fail intermittently, particularly when run on Travis:

• Bookmarks
  • cucumber features/bookmarks/bookmark_create.feature:386 # Scenario: Editing a bookmark's tags should update the bookmark blurb
• Collections
  • cucumber features/collections/collection_anonymity.feature:7 # Scenario: Create a hidden collection, add new and existing works to it, reveal works
  • cucumber features/collections/collection_anonymity.feature:161 # Scenario: Create an anonymous collection, add new and existing works to it, reveal authors
  • cucumber features/collections/collection_navigation.feature:4 # Scenario: Create a collection and check the links
• Other B
  • cucumber features/other_b/series.feature:7 # Scenario: Three ways to add a work to a series
  • cucumber features/other_b/series.feature:96 # Scenario: Three ways to add a work to a series: a user with more than one pseud
  • cucumber features/other_b/subscriptions_fandoms.feature:57 # Scenario: Author of anonymous work is not shown in feed
• Users
  • cucumber features/users/user_dashboard.feature:70 # Scenario: When a user has more works, series, or bookmarks than the maximum displayed on dashboards (5), the "Recent" listbox for that type of item should contain a link to that user's page for that type of item (e.g. Works (6), Bookmarks (10)). The link should go to the user's works, series, or bookmarks page. That link should not exist for any pseuds belonging to that user until the pseud has 5 or more works/series/bookmarks, and then the pseud's link should go to the works/series/bookmarks page for that pseud.

Pending Tests

Mark failing tests pending if a test is written but isn't working for known reasons, or if you need to merge something urgently and don't want to break the build.

In RSpec, do this using xit:

xit "should only get works under that tag" do
  get :index, tag_id: @fandom.name
  expect(assigns(:works).items).to include(@work)
  expect(assigns(:works).items).not_to include(@work2)
end 

In Cucumber, use the When /^"([^\"]*)" is fixed$/ step and comment out any failing steps:

Scenario: Post Without Preview
  Given I am logged in as "whoever" with password "whatever"
    And I add the work "public" to series "be_public"
    And I follow "be_public"
    And "Issue 2169" is fixed
  # Then I should not see the "title" text "Restricted" within "h2"