Trapping Signals in Docker Containers

This article was originally published on Medium by Grigoriy Chudnov, and we are sharing it here for Codeship readers.

Have you ever stopped a Docker container?

There are essentially two commands that you can use to stop it: docker stop and docker kill. Behind the scenes, docker stop stops a running container by sending it SIGTERM signal, letting the main process process it and, after a grace period, using SIGKILL to terminate the application.

Running applications in Docker, you could use signals to communicate with an application from the host machine, reload configuration, do some cleanup on program termination or coordinate more than one executable.

Let’s see the way signals can be used in the Docker environment.

Signals represent a form of interprocess communication. A signal is a message to a process from the kernel to notify that some condition has occurred.

When a signal is issued to a process, the process is interrupted and a signal handler is executed. If there is no signal handler, the default handler is called instead.

The process tells the kernel the types of signals it's interested in handling in a nondefault way by providing handler functions (callbacks). To register a signal handler, use one of the signal API functions (reference: The Linux Programming Interface: A Linux and UNIX System Programming Handbook by Michael Kerrisk).

When you run the kill command in a terminal, you’re asking the kernel to send a signal to another process.

One common signal is SIGTERM, which tells the process to shut down and terminate. It could be used to close down sockets, database connections or remove any temporary files. Many daemons accept SIGHUP to reload configuration files. SIGUSR1 and SIGUSR2 are the user-defined signals and can be handled in an application-specific way.

For example, to set up SIGTERM signal in node.js:

process.on('SIGTERM', function() {
  console.log('shutting down...');
});

When SIGTERM is handled by a process, the normal sequence of execution is temporarily interrupted by the provided signal handler. The function is executed, and the process continues to run from the point it was interrupted.

Here's a list of common signals -- Michael Kerrisk's book, The Linux Programming Interface: A Linux and UNIX System Programming Handbook, has the full reference:

All signals except for SIGKILL and SIGSTOP can be intercepted by the process.

A docker command docker kill is used to send a signal to the main process inside a container.

Usage: docker kill [OPTIONS] CONTAINER [CONTAINER...]
Kill a running container using SIGKILL or a specified signal
  -s, --signal="KILL"    Signal to send to the container

The signal sent to the container is handled by the main process that's running (PID 1). The process could ignore the signal, let the default action occur or provide a callback function for that signal.

As an example, let’s run the following application (program.js) inside a container and examine signal handlers.

'use strict';
var http = require('http');
var server = http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(3000, '0.0.0.0');
console.log('server started');
var signals = {
  'SIGINT': 2,
  'SIGTERM': 15
};
function shutdown(signal, value) {
  server.close(function () {
    console.log('server stopped by ' + signal);
    process.exit(128 + value);
  });
}
Object.keys(signals).forEach(function (signal) {
  process.on(signal, function () {
    shutdown(signal, signals[signal]);
  });
});

Here we create an HTTP server that listens on port 3000 and set up two signal handlers for SIGINT and SIGTERM. When handled, the following message will be printed to stdout: server stopped by [SIGNAL].

When the application is the main process in a container (PID1), it could handle signals directly. Here's the Dockerfile to build an image based on io.js:

FROM iojs:onbuild
COPY ./program.js ./program.js
COPY ./package.json ./package.json
EXPOSE 3000
ENTRYPOINT ["node", "program"]

NOTE: When creating a Dockerfile, make sure you use the exec form of ENTRYPOINT or RUN commands. Otherwise the application will be started as a subcommand of /bin/sh -c, which doesn't pass signals. The container’s PID1 will be the shell, and your application will not receive any signals from the docker kill command.

Build the image:

$ docker build -t signal-fg-app .

Run the container:

$ docker run -it --rm -p 3000:3000 --name="signal-fg-app" signal-fg-app

Visit http://localhost:3000 to verify that the application is running. Then open a new terminal and issue the docker kill command:

$ docker kill --signal="SIGTERM" signal-fg-app

or

$ docker stop signal-fg-app

Both commands can be used to issue SIGTERM signal and stop the application. Application handles the signal, and you should see the following message in the terminal you run the container in:

server stopped by SIGTERM

The process to be signaled could be the one in the background, and you can't send any signals directly. One solution is to set up a shell-script as the entrypoint and orchestrate all signal processing in that script.

An updated Dockerfile:

FROM iojs:onbuild
COPY ./program.js ./program.js
COPY ./program.sh ./program.sh
COPY ./package.json ./package.json
RUN  chmod +x ./program.sh
EXPOSE 3000
ENTRYPOINT ["./program.sh"]

The new entrypoint is a bash-script program.sh that orchestrates signal processing:

 #!/usr/bin/env bash
    set -x
    pid=0
    # SIGUSR1-handler
    my_handler() {
      echo "my_handler"
    }
    # SIGTERM-handler
    term_handler() {
      if [ $pid -ne 0 ]; then
        kill -SIGTERM "$pid"
        wait "$pid"
      fi
      exit 143; # 128 + 15 -- SIGTERM
    }
    # setup handlers
    # on callback, kill the last background process, which is `tail -f /dev/null` and execute the specified handler
    trap 'kill ${!}; my_handler' SIGUSR1
    trap 'kill ${!}; term_handler' SIGTERM
    # run application
    node program &
    pid="$!"
    # wait indefinitely
    while true
    do
      tail -f /dev/null & wait ${!}
    done

Here we set up two signal handlers: one for the user-defined signal, SIGUSR1, and the second one, SIGTERM, to gracefully shut down the application.

NOTE: The Single UNIX Specification specifies the functions that are guaranteed to be safe to call from within a signal handler. These functions are reentrant and are called async-signal safe by the Single UNIX Specification.

In the script, we put our application in the background with the ampersand (&):

node program &

Finally, we use wait to suspend execution until a child process exits. wait and waitpid are the functions that are always interrupted when a signal is caught. After the signal kicks in, we process it in the specified handler and wait for a signal again.

NOTE: According to the Docker docs, SIGCHLD, SIGKILL and SIGSTOP are not proxied.

To run the code, open a new terminal window and build the image:

docker build -t signal-bg-app .

Run the container:

docker run -it --rm -p 3000:3000 --name="signal-bg-app" signal-bg-app

Open a new terminal and issue SIGUSR1:

docker kill --signal="SIGUSR1" signal-bg-app

Finally, to stop the application:

docker kill --signal="SIGTERM" signal-bg-app

The application should terminate gracefully with the corresponding message printed to stdout.

Signals provide a way of handling asynchronous events and can be used by applications running in Docker containers. You can use signals to communicate with an application from the host machine, reload configuration, do some cleanup or coordinate multiple processes.

NOTE: Source code can be found in the GitHub repo.

*We want to thank Grigory for sharing his article with us.