Node.js Async Best Practices and Avoiding Callback Hell

This article was originally published on the RisingStack blog by Tamas Hodi. With their kind permission, we’re sharing it here for Codeship readers.

In this post, we'll cover the tools and techniques you have at your disposal when handling Node.js asynchronous operations: async.js, promises, generators, and async functions. After reading this article, you’ll know how to avoid the despised callback hell!

Previously we have gathered a strong knowledge about asynchronous programming in JavaScript and understood how the Node.js event loop works. If you did not read these articles, I highly recommend them as introductions!

Node.js itself is single threaded, but some tasks can run in parallel, thanks to its asynchronous nature. But what does running parallelly mean in practice?

Since we program a single threaded VM, it is essential that we do not block execution by waiting for I/O, but handle them concurrently with the help of Node.js's event driven APIs. Let’s take a look at some fundamental patterns and learn how we can write efficient, non-blocking code with the built-in solutions of Node.js and some third-party libraries.

Let's take a look at these simple async operations. They do nothing special, just fire a timer and call a function once the timer finished.

function fastFunction (done) {
  setTimeout(function () {
    done()
  }, 100)
}
function slowFunction (done) {
  setTimeout(function () {
    done()
  }, 300)
}

Seems easy, right?

Our higher order functions can be executed sequentially or in parallel with the basic "pattern" by nesting callbacks. But using this method can lead to an untameable callback hell.

function runSequentially (callback) {
  fastFunction((err, data) => {
    if (err) return callback(err)
    console.log(data)   // results of a
    slowFunction((err, data) => {
      if (err) return callback(err)
      console.log(data) // results of b
      // here you can continue running more tasks
    })
  })
}

To become an efficient Node.js developer, you have to avoid the constantly growing indentation level, produce clean and readable code and be able to handle complex flows.

Let me show you some of the libraries we can use to organize our code in a nice and maintainable way!

Async is a utility module that provides straightforward, powerful functions for working with asynchronous JavaScript. Async contains some common patterns for asynchronous flow control with the respect of error-first callbacks.

Let's see how our previous example would look like using async!

async.waterfall([fastFunction, slowFunction], () => {
  console.log('done')
})

What kind of witchcraft just happened?

Actually, there is no magic to reveal. You can easily implement your async job-runner, which can run tasks in parallel and wait for each to be ready.

Let's take a look at what async does under the hood!

// taken from https://github.com/caolan/async/blob/master/lib/waterfall.js
function(tasks, callback) {
    callback = once(callback || noop);
    if (!isArray(tasks)) return callback(new Error('First argument to waterfall must be an array of functions'));
    if (!tasks.length) return callback();
    var taskIndex = 0;
    function nextTask(args) {
        if (taskIndex === tasks.length) {
            return callback.apply(null, [null].concat(args));
        }
        var taskCallback = onlyOnce(rest(function(err, args) {
            if (err) {
                return callback.apply(null, [err].concat(args));
            }
            nextTask(args);
        }));
        args.push(taskCallback);
        var task = tasks[taskIndex++];
        task.apply(null, args);
    }
    nextTask([]);
}

Essentially, a new callback is injected into the functions, and this is how async knows when a function is finished.

In case you wouldn't like to stick to the solid callback protocol, then co can be a good choice for you. co is a generator-based flow-control tool for Node.js and the browser, using promises and letting you write non-blocking code in a nice-ish way. It's a powerful alternative that takes advantage of generator functions tied with promises without the overhead of implementing custom iterators.

const fastPromise = new Promise((resolve, reject) => {
  fastFunction(resolve)
})
const slowPromise = new Promise((resolve, reject) => {
  slowFunction(resolve)
})
co(function * () {
  yield fastPromise
  yield slowPromise
}).then(() => {
  console.log('done')
})

As for now, I suggest going with co, since one of the most waited Node.js async/await functionality is only available in the nightly, unstable v7.x builds. But if you are already using Promises, switching from co to async function will be easy.

This syntactic sugar on top of Promises and Generators will eliminate the problem of callbacks and even help you to build nice flow-control structures. Almost like writing synchronous code, right?

Stable Node.js branches will receive this update in the near future, so you will be able to remove co and just do the same.

As we have just learned several tools and tricks to handle async, it is time to do some practice with fundamental control flows to make our code more efficient and clean.

Let’s take an example and write a route handler for our web app, where the request can be resolved after three steps:

• validateParams

• dbQuery

• serviceCall

If you'd like to write them without any helper, you'd most probably end up with something like this. Not so nice, right?

// validateParams, dbQuery, serviceCall are higher-order functions
// DONT
function handler (done) {
  validateParams((err) => {
    if (err) return done(err)
    dbQuery((err, dbResults) => {
      if (err) return done(err)
      serviceCall((err, serviceResults) => {
        done(err, { dbResults, serviceResults })
      })
    })
  })
}

Instead of callback hell, we can use the async library to refactor our code, as we have already learned:

// validateParams, dbQuery, serviceCall are higher-order functions
function handler (done) {
  async.waterfall([validateParams, dbQuery, serviceCall], done)
}

Let's take it a step further! Rewrite it to use Promises:

// validateParams, dbQuery, serviceCall are thunks
function handler () {
  return validateParams()
    .then(dbQuery)
    .then(serviceCall)
    .then((result) => {
      console.log(result)
      return result
    })
}

Also, you can use co powered generators with Promises:

// validateParams, dbQuery, serviceCall are thunks
const handler = co.wrap(function * () {
  yield validateParams()
  const dbResults = yield dbQuery()
  const serviceResults = yield serviceCall()
  return { dbResults, serviceResults }
})

It feels like a "synchronous" code but still doing async jobs one after each other. Let's see how this snippet should work with async / await.

// validateParams, dbQuery, serviceCall are thunks
async function handler () {
  await validateParams()
  const dbResults = await dbQuery()
  const serviceResults = await serviceCall()
  return { dbResults, serviceResults }
})

Fortunately, Node.js eliminates the complexities of writing thread-safe code. You just have to stick to these rules to keep things smooth:

• Prefer async over sync API. Using a non-blocking approach gives superior performance over the synchronous scenario.

• Always use the best fitting flow control or a mix of them. This will reduce the time spent waiting for I/O to complete.

You can find all of the code from this article in this repository.