How to Write Smoke Tests for an Ember Rails Stack

The following story shows the importance of smoke tests when writing an app using the Ember+Rails stack. It covers the most basic example, and even then things go awry. In addition, it shows an example of an end-to-end test and some of the difficulty in setting up an environment for smoke tests. Finally, it mentions exhaust, a library that can ease this pain.

Imagine, you're ready to start writing your next great application. It's like TodoMVC meets Twitter meets eBay: Users will write down their To Do lists and auction off tasks to other users in 140 characters or less. Investors are very excited. You're very excited. High-fives all around. You're going to change the world, and you're going to do that using all the right technologies: Ember for the client side and Rails for the backend!

Here is your first user story:

cucumber
Given I am a user
When I go to the todos page
Then I should see a list of all todos

Piece of cake! You start by installing the latest Ember:

> npm install -g ember bower
> ember new todo-frontend
> cd todo-frontend

"I don't care what @dhh says," you think out loud. "I'm a rockstar developer, so I'm gonna test-drive it like I stole it!" You don't have an API to test yet, so you start by mocking it. You install pretender and generate your first acceptance test:

> ember install ember-cli-pretender
> ember generate acceptance-test todos

// tests/acceptance/todos.js
import Ember from 'ember';
import { module, test } from 'qunit';
import startApp from 'emberbook/tests/helpers/start-app';
import Pretender from 'pretender';
let TODOS = [
  {id: 1, title: "write a blog post"},
  {id: 2, title: "let people read it"},
  {id: 3, title: "... profit"}
];
module('Acceptance | todos', {
  beforeEach: function() {
    this.application = startApp();
    this.server = new Pretender(function() {
      this.get('/todos', function(){
        var json = {
          todos: TODOS
        };
        return [200, {}, JSON.stringify(json)];
      });
    });
  },
  afterEach: function() {
    Ember.run(this.application, 'destroy');
    this.server.shutdown();
  }
});
test('visiting /todos', function(assert) {
  visit('/todos');
  andThen(function() {
    var title = find('h1');
    assert.equal(title.text(), 'Todo List');
    var todos = find('.todo-item');
    assert.equal(todos.length, TODOS.length);
    assert.equal(currentURL(), '/todos');
  });
});

Time to fire up the test runner and watch it all burn:

ember test --serve

Yep, it's red like the devil. First error, no route.

// app/router.js
import Ember from 'ember';
import config from './config/environment';
var Router = Ember.Router.extend({
  location: config.locationType
});
Router.map(function() {
  this.route("todos");
});
export default Router;

Now the test can't find the text "Todo List." You fix this as well:

{{!- app/templates/todos.hbs }}
# Todo List
{{#each model as |todo|}}
  <p class='todo-item'>
    {{todo.title}}
  </p>
{{/each}}

Now the test breaks because there are no To Do items on the page. You move forward:

// app/routes/todos.js
import Ember from 'ember';
export default Ember.Route.extend({
  model() {
    return this.store.findAll("todo");
  }
});

Can't find the "todo" model. One more fix:

// app/models/todo.js
import DS from 'ember-data';
export default DS.Model.extend({
  title: DS.attr(),
  complete: DS.attr("boolean")
});

Whoa, it's green! You pat yourself on the back. But you can't actually deliver the story until there's an API to back it up. Time to grab the latest version of Rails.

This is going to be an API rather than a full-fledged application, so you only need a few gems. You spend several minutes pondering a post you just read about building Rails API apps and deciding whether to use the Rails 5 --api flag. However, you want this product to hit the market "yesterday," so you just roll with what's stable:

> gem install rails
> rails new todo_api --skip-test-unit --skip-bundle
> cd todo_api

You also skipped test-unit because you'd rather use Rspec's eloquent DSL for this project's tests.

Time to boil the Gemfile down to the few gems you really need. This API only needs to serve JSON, so out with anything related to the asset pipeline:

# Gemfile
source 'https://rubygems.org'
gem 'rails', '4.2.3'
gem 'sqlite3'
gem 'active_model_serializers'
group :development, :test do
  gem 'rspec-rails'
end

Bundle it and get ready to write some specs!

> bundle install
> rails generate rspec:install

Time to bang out the first request spec:

# spec/requests/todos_spec.rb
require "rails_helper"
RSpec.describe "Todos", :type => :request do
  # letBANG because `before {expected_todos}` looks funny
  let!(:expected_todos) do
    3.times.map do |i|
      Todo.create(title: "Todo #{i+1}")
    end
  end
  it "lists all todos" do
    get "/todos"
    todos = extract_key_from_response("todos", response)
    expect(todos.count).to eq expected_todos.count
  end
  def extract_key_from_response(key, response)
    json_response = JSON.parse(response.body)
    expect(json_response).to have_key key
    json_response[key]
  end
end

You run the test. Watch it go red! First gripe: no model. You fix that:

class Todo < ActiveRecord::Base
end

It's migration time!

<code class="language-bash rails generate migration CreateTodos title:string</code>

# db/migrate/#{timestamp}\_create\_todos.rb
class CreateTodos < ActiveRecord::Migration
  def change
    create_table :todos do |t|
      t.string :title
      t.timestamps null: false
    end
  end
end

> rake db:migrate

And of course, it needs a route.

# config/routes.rb
Rails.application.routes.draw do
  resources :todos, only: :index
end

The test is still red, because there is no controller. Next fix:

# controllers/todos_controller.rb
class TodosController < ApplicationController
  def index
    render json: todos
  end
  private
  def todos
    Todo.all
  end
end

All green! Time to boot up the app(s) and bask in your own creative genius. You run the following commands in different terminals:

> rails server
> ember server

You can't wait to see your baby running. You open "http://local:4200/todos"... and you get a blank screen. Now that was disappointing. Both test suites are green. What could have gone wrong?

After some inspection into the problem, you notice a GET http://localhost:4200/todos 404 (Not Found) in your logs. Well, crap. Ember doesn't know the API is on a different domain. Well, that's an easy fix. You just need to specify a host in the application adapter. And since you are attractive and intelligent, you know better than to hardcode the host. After all, it will change for staging and production. So you open config/environment.js in your Ember app, and you add the following:

// config/environment.js                                                                                             if (environment === 'test') {
  ENV.API_HOST = ''
} else {
  ENV.API_HOST = (process.env.API_HOST || 'http://localhost:3000')
}

You set the test environment's API_HOST to an empty string, so that the current tests keep working. All other environments use the Rails' default http://localhost:3000 unless an API_HOST environment variable is set. Now you can create an application adapter and fix the bug:

// app/adapters/application.js
import DS from 'ember-data';
import config from '../config/environment';
export default DS.RESTAdapter.extend({
  host: config.API_HOST
});

W00T! Fixed that problem. Time to run the tests. Once again, you start the Rails and Ember apps in two different terminals.

> rails server
> ember server

Dang it. The app still doesn't work. You didn't set up CORS so that modern browsers can talk to the Rails API. You add rack-cors to the Gemfile, bundle it, and add the simplest policy you can think of to get things working. Allow everything!

# Gemfile
gem 'rack-cors', :require => 'rack/cors'

bundle install

# config/application.rb
config.middleware.insert_before 0, "Rack::Cors", :debug => true, :logger => (-> { Rails.logger }) do
  allow do
    origins '*'
    resource '*',
      :headers => :any,
      :methods => [:get, :post, :delete, :put, :options, :head],
      :max_age => 0
  end
end

You cross your fingers, run the tests, and start the servers. When you visit the todos route, you see your app is working. Phew! Even though your app is working now, you aren't 100 percent happy. You spent the last hour test driving an application that didn't work, even though all tests were green. And you didn't even focus unit tests -- all of your tests were integration tests. Shouldn't integration tests prevent things like this?

At this point, you remember a post on the Hashrocket blog about test driving elixir apps with cucumber. You decide to try these techniques in practice and write some smoke tests. The type of integration tests that were just written are great for testing application(s) in isolation. These isolated integration tests run very fast. And because they run fast, the majority of your tests should be either request specs or Ember integration. There is no reason to have full coverage (testing every possibility) with smoke tests.

However, in order to ensure the app actually works, there should be at least one smoke test for every API endpoint. Knowing that cucumber-rails already loads your test environment, you think of the simplest solution you can to try writing a smoke test. Simply add cucumber-rails, manually start Ember and Rails, then overwrite how capybara works. This is the solution you come up with. You add cucumber to the Gemfile and install it.

# Gemfile
group :development, :test do
  gem 'cucumber-rails', require: false
  gem 'database_cleaner'
  gem 'capybara-webkit' # ...
end

> bundle install
> rails generate cucumber:install

Then you overwrite capybara's behavior.

# features/support/env.rb
Capybara.configure do |config|
  # Don't start rails
  config.run_server = false
  # Set capybara's driver. Use your own favorite
  config.default_driver = :webkit
  # Make all requests to the Ember app
  config.app_host = 'http://localhost:4200'
end

And you write the first feature and step definitions.

# features/todo.feature
Feature: Todos
  When a user visits "/todos", they should see all todos
  Scenario: User views todos
    Given 4 todos
    When I visit "/todos"
    Then I should see "Todo List"
    And I see 4 todos

# features/steps/todo.rb
Given(/^(\d+) todos$/) do |num|
  num = num.to_i
  num.times do |i|
    Todo.create(title: "Title #{i}")
  end
end
When(/^I visit "(.*?)"$/) do |path|
  visit path
end
Then(/^I should see "(.*?)"$/) do |text|
  expect(page).to have_text(text)
end
Then(/^I see (\d+) todos$/) do |num|
  expect(all(".todo-item").length).to eq num.to_i
end

Now in separate terminals you run both Rails and Ember. This time Rails is started in the test environment so that Ember requests hit the Rails test database.

> rails server --environment test
> ember server

You run the test suite with rake, and voilà! You see that the test suite passes. You know that both Ember and Rails are communicating end-to-end. And even though you are super proud of yourself, you're not happy. "The servers should start automatically. Why can't I just type rake?" you think to yourself. Here is the first solution you come up with.

# features/support/env.rb
ember_pid = fork do
  puts "Starting Ember App"
  Dir.chdir("/your/user/directory/todo-frontend") do
    exec({"API_HOST" => "http://localhost:3001"}, "ember server --port 4201 --live-reload false")
  end
end
rails_pid = fork do
  puts "Starting Rails App"
  Dir.chdir(Rails.root) do
    exec("rails server --port 3001 --environment test")
  end
end
sleep 2
at_exit do #kill the Ember and Rails apps
  puts("Shutting down Ember and Rails App")
  Process.kill "KILL", rails_pid
  Process.kill "KILL", ember_pid
end
Capybara.configure do |config|
  config.run_server = false
  config.default_driver = :webkit
  config.app_host = 'http://localhost:4201'
end

This solution works okay. Rails and Ember run on separate ports, so the development versions of the two servers can keep running. The only problem is the sleep 2. As the Rails and Ember apps grow, so will the time needed to wait for the servers to start. Also, this number is magic; it might take longer on another machine. How long will it take on the CI server?

What you really want to do is halt the tests until you know that Rails and Ember are running. However, after some investigation, you realize there is no way to know if Ember is running successfully. But then you notice how EmberCLI smoke tests itself.

  it('ember new foo, server, SIGINT clears tmp/', function() {
    return runCommand(path.join('.', 'node_modules', 'ember-cli', 'bin', 'ember'), 'server', '--port=54323','--live-reload=false', {
        onOutput: function(string, child) {
          if (string.match(/Build successful/)) {
            killCliProcess(child);
          }
        }
      })
      .catch(function() {
        // just eat the rejection as we are testing what happens
      });
  });

Now you know that Ember has booted when it outputs "Build successful" and have some insight as to how you might wait for Rails.

# features/support/env.rb
begin
  DatabaseCleaner.strategy = :truncation
rescue NameError
  raise "You need to add database_cleaner to your Gemfile (in the :test group) if you wish to use it."
end
ember_server = nil
rails_server = nil
Dir.chdir("/Users/mwoods/hashrocket/todo-frontend") do
  ember_server = IO.popen([{"API_HOST" => "http://localhost:3001"}, "ember", "server", "--port", "4201", "--live-reload", "false", :err => [:child, :out]])
end
Dir.chdir(Rails.root) do
  rails_server = IO.popen(['rails', 'server', '--port', '3001', '--environment', 'test', :err => [:child, :out]])
end
# Before timeout loop to prevent orphaning processes
at_exit do
  Process.kill 9, rails_server.pid, ember_server.pid
end
# if it takes longer than 30 seconds to boot throw an error
Timeout::timeout(30) do
  # wait for the magic words from ember
  while running = ember_server.gets
    if running =~ /build successful/i
      break
    end
  end
  # when rails starts logging, it's running
  while running = rails_server.gets
    if running =~ /info/i
      break
    end
  end
end
Capybara.configure do |config|
  config.run_server = false
  config.default_driver = :webkit
  config.app_host = 'http://localhost:4201'
end

This is not a perfect solution, but it's good enough for day one. On day two, you might decide to move the smoke tests to their own repository. After all, they aren't Rails-specific, so they probably should be a separate project.

Problems like these and many others have happened to me. That's the reason I'm introducing exhaust, a new gem that hopefully alleviates some of the headache associated with smoke testing an Ember+Rails stack. Enjoy!