Docker in The New York Times Newsroom

Codeship was at DockerCon 2015! This week, we’ll be providing summaries on our blog of some of the talks we attended at this two-day conference in San Francisco. If you are interested in Docker support from Codeship, click here.

On Monday afternoon, Eric Buth spoke at DockerCon about how The New York Times’ Interactive News team leverages Docker to provide a reliable path to content production on tight deadlines.

As an interactive news developer at The New York Times, Buth explained to his audience that his team has been able to leverage Docker for standardized deployment across disparate technologies with immutable infrastructure.

[caption id="attachment_1958" align="aligncenter" width="6016"]

Eric Buth at DockerCon 2015[/caption]

The Goal

Buth pointed out that the Interactive News Department is a bit different from a traditional software engineering team. They work to create news features like election coverage, user generated content, or publishing tools to get content online.

As a result, their focus is less technical, and there’s less experience with typical engineering practices with things like continuous integration and delivery. The team needed to build a solid CI/CD solution so they could keep their focus on producing content and engaging experiences.

The Solution

To achieve a CI/CD workflow, Buth explained that the team created a deployment tool called Blade, a wrapper around Chef’s Knife. A simple command, bl server create, would run a bunch of Chef recipes.

Buth allowed that it was a good start but was rather slow, preventing the team from achieving a truly successful CI/CD workflow. The team then started running an open-source, on-premises tool for their CI, leveraging Docker containers as deployment artifacts. They now use an immutable infrastructure with Docker deployments.

The New York Times is historically a Rails shop, but Buth mentioned that as new technologies emerge, they need to support more stuff.

For example, JavaScript became important for them to support due to their work with graphics and their need to run Node.js apps. Then Python devs joined the team. And they still have giant monolith applications like The New York Times’ CMS. Add on more than 300 new microapps (news items, quizzes, etc.), and now the Interactive News Department is dealing with problems like scheduling, configuration, discovery, and routing for their Docker-deployed applications.

The Results

To manage all these moving parts, the Interactive News team combined a mixture of off-the-shelf tools (like CoreOS etcd) with their own tools, some of which they’ve open sourced:

• scheduling: they use components built into CoreOS, i.e. Fleet

• configuration: etcd + github.com/newsdev/context

• discovery: etcd + github.com/newsdev/remora (not open)

• routing: etcd + github.com/newsdev/promise

• Docker image builder (based on GitHub web hooks): github.com/newsdev/longshore

The use of Docker containers as a consistent approach to deployment across disparate tech was key to arriving at a solid CI/CD workflow -- an achievement that Buth emphasized was important to the success of his team.

Slides