Get Started Quickly with Docker and Sidekiq

In the past, I had tried to set up Docker during a few different programming competitions without ever having used it before. During the crunch time, I was unsuccessful in both setting it up and learning how it worked. But in those competitions I had the good fortune that my teammates were all able to configure their environment for the project. That still came at the cost of getting, installing, and configuring all relevant dependencies.

And that good fortune doesn't always occur. Some systems require you to go through exhaustive steps to make things work. With Docker, any potentially tedious work gets contained and shippable to co-developers who will be able to avoid those startup costs.

In this article, let's go a step further in how to install Docker within a Linux virtual machine that can be run on any computer operating system. So regardless of what OS you are on, you can follow along. From there, we'll get up and running with the background task manager Sidekiq as our end goal.

VirtualBox is a great virtual machine platform that makes installing additional operating systems easy-peasy. Install VirtualBox if you don't have it already; after that, grab a copy of Ubuntu Server.

Now let's install a new VM image:

1. Start up VirtualBox and click New.

2. Set the name for your server.

3. Set type to "Linux" and version to "Ubuntu (64-bit)". If you'd like, you can stick with the defaults for the rest of this VM setup and just click through.

4. When asked how much RAM you want to dedicate to it, it will have a recommendation shown. Give it more than it suggests. I have 8GB on my computer so feeling very generous I'll give this VM up to 4GB(4096MB).

5. Leave Create virtual hard disk now selected and click Create.

6. With VDI selected, click Next.

7. With Dynamically allocated selected, click Next.

8. Finally, choose how much disk space you'd like to dedicate to it and click Create.

Now you're ready to install Ubuntu Server on the VM:

1. Right click the VM you just created and open Settings.

2. Select Storage.

3. Click on the CD icon that says Empty.

4. Over on the far right, click the CD icon dropdown menu and select Choose Virtual Optical Disk File...

5. When the file dialog comes up, click the Ubuntu Server iso file you've previously downloaded and click Open.

6. Click Okay. Now when you start the VM, it will boot from the iso image file as if you had an installation CD in, and you can go through the quick install process.

7. With the VM you've created, click Start.

For the Ubuntu Server installation, you can use the defaults for everything and just breeze through the install. However, it would be a more efficient use of disk space if you manually set the VM drive partition yourself.

If you've given a decent amount of RAM to the virtual machine, you won't need swap space (which is inefficient anyway), so manually create a drive partition using the full amount of space and put its mount point as /. When you're given options as to what groups of packages to install, I'd recommend installing the SSH server so you can shell-in from the host; this will let you copy and paste. Otherwise, you'll have to manually type most everything on the command line in the VirtualBox VM window. When asked to install "grub boot loader", choose Yes.

Congrats! You now have a working Linux server virtual machine which works as an excellent sandbox and a fully capable server.

To make VirtualBox internal VMs available from your computer through a host-only network, shut down the VM with shutdown -h now and do the following in VirtualBox:

1. File -> Preferences (Alt-G)

2. Network -> Host-only

3. + (add)

This creates a vboxnet0 network that will be available across both the host system and for each configured virtual machine.

Next, open the VM's network preferences and add a second NIC (Network Interface Card).

1. Right-click on your VM and select Settings.

2. Choose Network and then Adapter 2.

3. Select Enable Network Adapter.

4. Set Attached to to Host-only Adapter and click OK. It should have the correct name for the NIC chosen by default.

On the VM, you'll need to add the network interface in the main configuration file. Run ifconfig and ifconfig -a and look for the extra one listed in the -a results. Copy that network name and place it in the /etc/network/interfaces file as follows:

# Host-only
auto enp0s8
iface enp0s8 inet dhcp

Replace enp0s8 with whatever the network card name shows up as for you. Then after a reboot, you can run ifconfig to get the host-only network IP address. This will allow you to see your Docker web apps running in the VM from the host machine at that IP address, e.g., http://192.168.56.101:3000/.

You now have a functional stand-alone machine acting as a true server should.

The simplest way to install Docker on this Ubuntu Server VM is to run:

curl -fsSL https://get.docker.com/ | sh
sudo usermod -aG docker $(whoami)

The first line gets a script for Docker to be installed on Linux and runs it. The second line gives the current user's account permission to use Docker.

If you'd like to manually install Docker and have it configured with Ubuntu's package manager, you can follow the directions here. An abbreviated way to do that is this:

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo "deb https://apt.dockerproject.org/repo ubuntu-$(. /etc/lsb-release && echo "$DISTRIB_CODENAME") main" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo apt-get install docker-engine
sudo service docker start
sudo usermod -aG docker $(whoami)

The installation for docker-compose is as easy as copying it from the GitHub repo and making it executable.

sudo curl -L "https://github.com/docker/compose/releases/download/1.9.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

And for autocompletion on the command line, run the following:

curl -L https://raw.githubusercontent.com/docker/compose/$(docker-compose version --short)/contrib/completion/bash/docker-compose > /etc/bash_completion.d/docker-compose

Besure to reboot your VM after all Docker items are installed with the command reboot.

To make the startup process with Docker much simpler, we're going to begin with a free template from Bitnami.com. Bitnami provides many different kinds of starter installers or VM images for many different kinds of technologies, both for development or production environments. They have a Ruby/Rails Docker configuration available on GitHub. If you're looking to do something with just Ruby or Sinatra, you may use https://github.com/bitnami/bitnami-docker-ruby for a container. To install this Docker image (after previous VM reboot):

mkdir ~/myapp && cd ~/myapp
curl -LO https://raw.githubusercontent.com/bitnami/bitnami-docker-rails/master/docker-compose.yml
docker-compose up

This will install a couple Docker images for a Rails application and the database. Go ahead and move this process to the background with Ctrl-Z and then running bg.

We'll now look at the example Rails site running on the host machine. Check the IP address with ifconfig and look for the NIC name you set earlier for the host-only connection. Type that IP address in with port 3000: http://192.168.56.102:3000/ and hit Enter. You will now see the success page for having Rails running.

If you have the Docker container running in the background, go ahead and shut it down with kill %1. Since the VM server is running with just one shell open, let's go over a few techniques for switching between processes.

In Linux, you can run a process in the background by putting an ampersand at the end of the command or by pressing Ctrl-Z and running bg for background. When you move an application to the background in this manner, you will see a number displayed to identify it. You can use that number to kill the process or bring it back to the foreground to continue working in it with fg %number.

When you put a process in the background, it can still write to standard output (STDOUT) and make a mess of whatever you may be doing in the terminal. For this reason, we'll want to reroute the output to a log file with &>session.log. So to start a Docker application quietly in the background, you can do:

docker-compose up &>session.log &

Now you have your containers up and running, and you may run commands in them without the output from the container's Rails server spitting out log data. To see the most recent output in the session.log file, you can run tail session.log.

If you want to scan gradually through the entire contents of the log file, you may use cat session.log | more. With this, you can use the space bar to continue by page.

Now to run anything within the Docker instance, you need to precede it with docker-compose exec myapp. Since this may become tedious to continuously type, we can shorten it with an alias: alias app='docker-compose exec myapp'.

Now instead of doing docker-compose exec myapp ruby --version, you can simply type app ruby --version.

In order to use Sidekiq, we first need to have a connection to Redis. This is super simple with Docker Compose. We just need to modify our docker-compose.yml file as follows:

version: '2'
services:
  mariadb:
    image: bitnami/mariadb:latest
  redis:
    image: redis:latest
  myapp:
    tty: true # Enables debugging capabilities when attached to this container.
    image: bitnami/rails:latest
    environment:
    depends_on:
      - mariadb
      - redis
    ports:
      - 3000:3000
    volumes:
      - .:/app
    env_file:
      - .env
  sidekiq:
    build: .
    command: bundle exec sidekiq
    depends_on:
      - mariadb
      - redis
    volumes:
      - .:/app
    env_file:
      - .env

Now create a .env file and put the following in it. We've taken the DATABASE_URL that was originally in the docker-compose.yml file and moved it into .env since this environment variable will be shared across two separate processes.

DATABASE_URL=mysql2://mariadb/my_app_development
JOB_WORKER_URL=redis://redis:6379/0

After that, uncomment the Redis line in your Gemfile: gem 'redis', '~> 3.0'. Next, edit an initializer for Sidekiq in config/initializers/sidekiq.rb:

sidekiq_config = { url: ENV['JOB_WORKER_URL'] }
Sidekiq.configure_server do |config|
  config.redis = sidekiq_config
end
Sidekiq.configure_client do |config|
  config.redis = sidekiq_config
end

Now all we need to do is create a worker, and we'll be off to the races.

#app/workers/my_worker.rb
class MyWorker
  include Sidekiq::Worker
  def perform(name, count)
    logger.info "#{name} says the count is #{count}"
  end
end
20.times do |index|
  MyWorker.perform_in(index, "Bobby", index)
end

And that's it. Just start up your docker-compose file, and you'll see 20 messages from your background worker. If you use docker-compose up &>session.log &, you'll see the messages from Bobby in there from the Sidekiq process.

There are lots of How-Tos for Docker out there, and I can see that it's clearly gotten easier to get stuff done with Docker Compose. Less configuration required for a better experience all around.

VirtualBox offers the advantage of containing an entire operating system in a single file. You can even go further and have multiple kinds of operating systems and services running via multiple VMs and experiment with different services over a virtual network. My personal favorite benefit is using it as a sandbox, which is simply a method of being more secure by keeping software separate from your primary operating system.

Now Docker comes really close to doing most of the things VirtualBox does with emulating a VM and optional network configurations. But it's a barebones imitation and won't have tools that you haven't built into your container(s). Each tool has its own specialties.

Sidekiq is a well-vetted, multi-threaded background job processor. Its performance outmatches all other Ruby background workers. And they have an enterprise system available to buy into.

There was a lot of material covered here. Be sure to experiment with things to help retain the knowledge!