This is a republished \ blog post from Manfred Stienstra. You can find it on the fingertips blog.
Why you need a fast test suite
Developers need to be able to run tests quickly or they will stop running them.
The biggest bane of test driven development, or whatever variant you practice, is long boot times. Even when you just run one test a slow boot will make it a tedious job. There are a number of ways to reduce startup times in a Ruby on Rails project.
Load less dependencies to get a faster test suite boot time
Project dependencies need to be loaded every time you start your test suite, less dependencies means faster startup. Keeping project dependencies to a minimum is always a good idea, not just because of boot time.
Let’s jump into an example. We’re assuming you’re using the following, simplified, Gemfile for your dependencies:
gem "rails" gem "bcrypt" gem "foreman" gem "thin" gem "newrelic_rpm" gem "airbrake"
First we stop loading gems which aren’t used in the code.
gem "rails" gem "bcrypt" gem "foreman", require: false gem "thin", require: false gem "newrelic_rpm" gem "airbrake"
Then we make sure dependencies are only specified for the environments where they’re used in.
gem "rails" gem "bcrypt" group :development do gem "foreman", require: false end group :production do gem "thin", require: false gem "newrelic_rpm" gem "airbrake" end
At this point you might start getting undefined constant errors about Airbrake. You can decide to either move Airbrake back into all environments or stub its implementation.
We generally do this by defining the module or class ourselves from a support file in the test directory, like so:
module Airbrake extend self def notify(exception, opts = {}) end end
Partial example of a stub implementation for Airbrake.
On the one hand you probably never want to send data from your test suite to Airbrake. On the other hand some people are more comfortable running their entire production stack in at least one or two tests.
Reduce time spent setting up tests
Basically you need to do less. There are a few strategies to accomplish this.
One strategy is moving setup code as local to your test as possible so it only runs for the tests where it’s needed. For example, don’t run setup methods specific for functional tests in unit test.
Keep in mind that your test_helper.rb or spec_helper.rb is loaded during every boot. Try to limit slow operations in this file.
Tests that require long setup tasks sometimes need to be run in a variety of contexts: library tests, model tests, and functional tests. Sometimes you can cheat a little bit and test through the entire stack in one massive integration test.
Another strategy is to perform long setup tasks only once and find a way to cache the result to speed up next runs.
In one of our applications we process YAML files with questionnaire definitions and create records for them in the database. In hindsight we should have never done this, but that’s a discussion for another day.
In unit and functional tests we use fixtures to test specific situations. For regression tests we need the production definitions. We speed up loading these by dumping the database tables with INTO OUTFILE and load them with LOAD DATA INFILE. We also hook into Rails fixture loading code so it only happens once for the entire suite.
Finally, never, ever, ever, ever, call external services from your tests. Not only is this slow, but it might accidentally erase production data. In most tests we override Net::HTTP#start to make sure it’s never called by accident.
Factories or fixtures
There is a lot to say about database setup, so it deserves its own little chapter.
Let’s get the elephant out of the room first. Factories are slow! They’re slow by design.
Factories can be used to create database records anywhere in your test suite. This makes them pretty flexible and allows you to keep your test data local to your tests. The drawback is that it makes them almost impossible to speed up in any significant way.
We can understand why factories are slow by looking at why ActiveRecord fixtures are fast.
Fixtures are bulk loaded once. A transaction is created in the database when a test starts. After the test the database is rolled back to its initial state. This means the fixtures don’t have to be reloaded for each test.
Most factory implementations start a test with an empty database and run through all the factories needed for a test each time it’s run. After the test the database is reset to its blank state by removing all records. On top of that records are created through ActiveRecord model instances, which creates a lot of objects in the Ruby VM, which need to be garbage collected.
We try to use fixtures as much as possible. We write little scenarios with them to make tests easier to understand. Paul might be a pro account holder who has paid his bills the last two years, but unfortunately missed a payment this month because his credit card expired.
Using small scenarios like this reduce the chance of someone messing up tests by using the wrong fixture or changing fixtures to not fit their intended scenario. You can even document these scenarios shortly in the fixture files or test helper.
Use a fast test framework
Your test framework needs to be loaded and initialized during every boot.
RSpec is notoriously slow. If you’re willing to part with it, you might want to consider minitest/spec. You can use minitest/spec with Rails out of the box since version 4.0. Another alternative is Peck on Rails.
In doubt, measure!
Optimizations don’t really make sense when you can’t measure the speedups. Most test frameworks print their runtimes, but this doesn’t include boot time. The time tool is your friend.
$ time ruby test/unit/post_test.rb Loaded suite test/unit/post_test Started .. Finished in 0.001313 seconds. 2 tests, 6 assertions, 0 failures, 0 errors ruby test/unit/post_test.rb 0.15s user 0.08s system 96% cpu 0.238 total
In this example test/unit reports 0.001313 seconds, but the real runtime is 0.238.
To measure particular parts of your code you can either use a profiler or the Benchmark class.
Isolate the slow code and keep the benchmark in place to measure if your fix actually helps.
We think Manfred makes some great points here. As mentioned above you should definitely check out the fingertips blog – they have some great stuff on there!
How do you keep your test suite fast? Are there any other things you found work really well for you? Let us know in the comments. We would love to get your opinion!