Async patterns in Node.js: only 5+ different ways to do it!#

NOTE: The original version of this was written back in 2017(!), but in 2022 it's all still valid (despite all newer books, courses and articles mentioning async/await, understanding promises is still a requirement and there still is and will continue to be a lot of callbacks code around that you'll have to integrate with).

TL;DR: callbacks, async module, promises, async/await, coroutines/generators, which to use and when.

So you’re a Node.js developer, right? Then you know Tim Toady! Nice fella, right? …NO?! Jokes aside, we know that TIMTOWTDI or Tim Toady stands for 2 things nowadays: (1) “There is more than one way to do it” and (2) the Twitter handle of Larry Wall (inventor of Perl, the first programming language to truly embrace (1)). The next popular language/platform to embrace TIMTOWTDI was obviously Node.js.

And it embraced it to extreme extents: the core aspect of Node.js, asynchronous programming, has TIMTOWTDI all over it! In this article I’ll explain the 4+ways of writing async code in Node.js (most work in the browser too btw).

So let’s start our journey through the jungle of Node.js async patterns with:

1. Callbacks: simple, obvious, …hellish#

const fs = require('fs');

fs.readFile('./hello.txt', 'utf-8', (err, data) => {
  if (err) return console.error('Failed reading file:', err);

  console.log('File contents:', data);
});

Not much to be said about this. To the method doing something async, you pass a callback (taking the customary err, dataWhateverEtc arguments) which is to be called after the thing is done (or failed)… The first argument is null except when an error happens (generally you process the error path first in you code and return or throw if you have an error). Simple. Well fit for Javascript since functions are first class objects. Obvious. Why the hell would anyone want more?!

Well, ‘cause regular programs tend to need to do stuff in sequence, and when you have callbacks-async actions in sequence you end up with the “beloved” “pyramid”:

doSuff1(arg1, (err, ret1) => {
  if (err) {
    ifErrorDoStufE1(argE1, (errE1, retE1) => {
      // ...
    });
    // ...
    return stuffToReturnWhenErr1;
  }
  // ...
  thenAfterDoStuff2(arg2, (err, ret2) => {
    // ...
    thenAfterDoStuff3(arg3, (err, ret3) => {
      // ...
      thenAfterDoStuff4(arg4, (err, ret4) => {
        // ...
        thenAfterDoStuff5(arg5, (err, ret5) => {
          // ...
          thenAfterDoStuff6(arg6, (err, ret6) => {
            // ...
            thenAfterDoStuff7(arg7, (err, ret7) => {
              // ...
            });
          });
        });
      });
    });
  });
  // ...
});

You can imagine that the real-life production version of this is way more complicated. And you can imagine debugging this and how the stack traces look like. This is why this resulting pattern is less affectionately called “callback hell”.

But wait… this is still the simplest case you can think of. Real-life code would need to do things like “when both async actions A and B are done (hopefully in parallel if they do IO) do C” or “when the first of async actions A, B and C finishes, do D”. Let’s check out how the first case looks with callbacks-async code:

let leftToDo = 2; // or use booleans doneA and doneB instead...

doA(argA, (errA, resA) => {
  // ...
  leftToDo--;
  // ...to do after A
  if (leftToDo === 0) {
    todoAfterAB();
  }
});

doB(argB, (errB, resB) => {
  // ...
  leftToDo--;
  // ...to do after B
  if (leftToDo === 0) {
    todoAfterAB();
  }
});

function todoAfterAB() {
  // ...
}

This seems almost OK… until you realize that this is the simplest imaginable case and that those // ...s can be tens/hundreds of LOC. Enter:

2. Async module: the callbacks wrangling cowboy to the rescue!#

The well known and loved 3rd party module async comes to the rescue with a well thought of set of async helpers that allows one to organize async callbacks in sane ways even in complex scenarios. It’s basically what you yourself would’ve invented after working with code like the one above for long enough… but the problem is that each developer would come up with his/her own slightly different way of handling these patterns! Standardization in handling callbacks-async code is the true feature of the async module. If you’re not familiar with it, you should, because you’ll definitely come across code using it, and you’ll have to be able to understand it, even if imho you should jump over it in your own code: if callbacks get too messy, just jump to Promises! …that’s my 2 cents of advice ;)

Here’s how a slightly more complicated variant of the code above looks like using async:

const async = require('async');

async.parallel(
  [
    doA, // if A needs no args and no special logic afterwards
    doB.bind(null, argB), // if B needs argB
    (callback) => { // if C needs both arg and special logic
      // ...stuff to do immediately before C
      doC(argC, (errC, resC) => {
        // ...stuff to do immediately after C
        callback(resC);
      });
    }
  ],
  (err, results) => { // callback
    // ...stuff to do after A, B and C are all done
  }
);

This uses 3 async callbacks, it also takes care of collecting the results of all of them (in the results array), and it also shows how to handle async functions what need params and special logic happening immediately before and after an async callback.

The name, “parallel” is a bit confusing, because as you know Node.js is single threaded, so the only thing happening in parallel is the IO, done outside Javascript code (so don’t ever think of using async.parallel for parallelizing CPU intensive code in Node.js, despite its name).

The other nice consequence is that code doesn’t “float to the right” that much, even though you still need some nesting.

So, uhm… that’s it, right? “Callback hell” / “pyramid code” was the problem, and the async module is the solution, no?

…not really: those two were just the most obvious of problems. Thing is, real-life async code doesn’t sit nicely isolated, in one module, where you know what every line of code does and you can refactor to your heart’s desire. Your code will look more like this:

In my-kitchen-module.js:

// ...
exports.init = function (arg1, arg2, cb) {
  // ...
  cookMeat(cmarg1, (err, r) => { // 🐌 async, this will take time
    if (err) {
      // ...
      return;
    }
    // ...
    exports.cookedMeat = r; // 🚩
    // ...
  });
  // ...
  if (nothingExplodedYet) {
    // nothing exploded, the meat is cooking,
    // and in the meantime most of the kitchen is usable
    cb();
  }
}
// ...

…then in your program.js:

const myKitchenModule = require('./my-kitchen-module');
// ...
myKitchenModule.init(err => {
  // ...
  /* 🌮 My awesome taco receipe: */
  let tortilla = initTortilla();
  let habaneros = chopHabaneros();
  let avocado = chopAvocado();
  // ...
  let cookedMeat = myKitchenModule.cookedMeat;
  let ingredients = [tortilla, habaneros, avocado, cookedMeat];
  assembleTaco(ingredients, (err, taco) => {
    if (err) {
      console.error("Can't assemble taco because:", err);
      return;
    }
    // ...
    console.log("Your taco is served:", taco);
  });
});

And now you’d be hungrily waiting for your taco, but instead you’d likely get a Can't assemble taco because: meat is not cooked yet (cookedMeat is undefined)! Ouch…

The obvious solution to this problem that doesn’t involve changing when myKitchenModule.init happens etc., is to add another callback parameter to myKitchenModule.init, turning it to .init(kitchenReadyCb, meatCookedCb). Nice, but… in an actual project, you’ll have dozens of such callbacks to added to dozens of different modules. Do you really want to do that, go on adding modifications upon modification to external module code that you don’t completely understand, until something inevitable breaks? And this is just one example of a situation where writing async code forces you to refactor code external to your app. Yuck!

(Note: Yeah, you could say that “other people should write better and more async friendly code” so you don’t have to patch shit up all day long! But… that’s not a very constructive attitude: imho one should thank other people for writing the code they’ve written so far and making it freely available and saving you hundreds of hours of drudge work! Be honest, if the code would really be that bad, you wouldn’t even bother patching it up to work for your app. This ain’t Haskell my friend, and you should adapt your expectations to the community and ecosystem you are working in.)

Now, let’s solve the problem realistically. But what wast the problem? …well, let me rephrase it a bit:

**Code needs to be able to refer to async created/acquired resources, without having to care/know whether:

the resource has already been create/acquired
the resource is in the process of being created/acquired
the resource creation/acquisition process has not even been initiated yet, without requiring patching refactoring other code that sits outside our app or module.** (We should be more general than this actually… but this is good enough to make the point, and we’re not writing a Computer Science textbook here.)

More explicitly, I should be able to (1) write code referring to cookedMeat without having to care whether (a) cookedMeat exists, (b) is in the process of being prepared, or maybe (c) we haven’t even started the process of preparing it (maybe we’re waiting for it to be delivered) and it should (2) be the same code path, not an ugly multi-if/switch and (3) we shouldn’t have to put this code in a special place or to modify an external module for this either!

Being slightly more theoretical we can say that: in most practical situations, complex callbacks-async code breaks incapsulation. But let’s stop rephrasing the problem(s) and move on to see the solution(s):

3. Promises: the promised solution …delivered as a DIY recipe#

It turns out, that we can refer to values that may not exist yet, using only what the Javascript language offers (and without ugly hacks like a setTimeut and pooling every 50ms to see if “it’s done yet”). We just need to wrap our values that don’t exist yet into special container objects.

Let’s simplify the code above a bit to see how things would look like this way:

// magically wrap cookedMeat in a "promise"
let cookedMeatPromise = ...✨ magic here...;

/* 🌮 My awesome taco receipe: */
let tortilla = initTortilla();
let habaneros = chopHabaneros();
let avocado = chopAvocado();
// ...
// 🚩 this must still work in the case where by the time we reach
// 🚩 this line, meat has already been cooked
myKitchenModule.cookedMeatPromise.whenMeatCooked((err, cookedMeat) => {
  // ...
  assembleTaco([tortilla, habaneros, avocado], (err, taco) => {
    // ...
    console.log("Your taco is served:", taco);
  });
});

Only problem is that ...magic here... is not valid Javascript (at least until ES-yearWhenAIReachesSuperHumanLevel). And with the code arranged this way, you can’t really see any obvious way to make replace this with ES-7 code while still satisfying all requirements. Hmm… But maybe we can move things around a bit:

let cookedMeatPromise = new CookedMeatPromise((res, rej) => {
  cookMeat((err, r) => {
    if (err) {
      rej(err);
      return;
    }
    // ...
    res(r);
  };
});

/* 🌮 My awesome taco receipe: */
let tortilla = initTortilla();
let habaneros = chopHabaneros();
let avocado = chopAvocado();
// ...
// 🚩 this must still work in case by the time we reach
// 🚩 this line, meat has already been cooked
myKitchenModule.cookedMeatPromise.whenMeatCooked(
  cookedMeat => {
    // ...
    assembleTaco([tortilla, habaneros, avocado], taco => {
      // ...
      console.log("Your taco is served:", taco);
    });
  },
  err => console.error("Problem cooking meat:", err)
);

and now we can even figure out a tentative of a working implementation for the CookedMeatPromise class:

class CookedMeatPromise {
  constructor(cb) {
    this._done = false;
    cb(this._resolve, this._reject);
  }
  whenMeatCooked(resolve, reject) {
    if (this._done) {
      if (this._err) reject(this._err);
      else resolve(this._cookedMeat);
      return;
    }
    setTimeout(self.whenMeatCooked.bind(this, resolve, reject), 100);
  }
  _resolve(cookedMeat) {
    this._cookedMeat = cookedMeat;
    tis._done = true;
  }
  _reject(err) {
    this._err = err;
    this._done = true;
  }
}

Now, don’t even imagine that the implementation above will work correctly in all possible cases… But it’s good enough to prove a point: that this can be done with nothing but the basic building blocks your language provides. An that it’s useful.

Of course, there are problems, the most superficially obvious of which being that the specific names like whenMeatCooked will make it necessary to re-document this pattern for every case you use it slightly differently. We can standardize a bit, starting with renaming whenMeatCooked to then, and making it return this to make stuff chainable (fancy folks would say we’re building a “monad” here…), then adding another similar method called catch that only takes the error callback, and you can easily end up writing slightly nicer and more general code like this:

let cookedMeatPromise = new Promise((resolve, reject) => {
  cookMeat((err, r) => {
    if (err) {
      reject(err);
      return;
    }
    // ...
    resolve(r);
  };
});

/* 🌮 My awesome taco receipe: */
let tortilla = initTortilla();
let habaneros = chopHabaneros();
let avocado = chopAvocado();
// ...
// 🚩 this must still work in case by the time we reach
// 🚩 this line, meat has already been cooked
myKitchenModule.cookedMeatPromise.then(cookedMeat => {
  // ...
  assembleTaco([tortilla, habaneros, avocado], taco => {
    // ...
    console.log("Your taco is served:", taco);
  });
}).catch(err => console.error("Problem cooking meat:", err));

It doesn’t seem much, but nicer matters when line after line of code piles up. And, it turns out you get more: with just a few tweaks you can add methods like Promise.race and Promise.all and just like that get like ~90% of the practically useful functionality provided by the async module. Add a couple more basic helpers like Promise.resolve (to write something like answer = Promise.resolve(42) and have a one liner promise that instantly resolves to the answer to the ultimate question of life, the universe and everything) and Promise.reject and you (almost) get the Promise/A+ specifications that Node.js and modern browsers implement (and can be polyfilled down to IE6 if you need).

For comparison, this is the async.parallel example translated into promises-async code:

let prA = doA(); // do A and return a promise for A
let prB = doB(argB);
let prC = doC(argC);

prC.then(resC => { /* ...stuff to do immediately after C */ });

Promise.all([prA, prB, prC]).then(results => {
  // ...stuff to do after A, B and C are all successfully done
}).catch(err => {
  // ...handle first error that happened
});

Nice, but… what about all the older APIs that support callbacks instead of Promises? Well, it turns up you can easily convert/mix any callbacks-async API to/with a Promise-based API. Actually, if you ponder a bit at the Promise constructor code, you’ll figure out for yourself the “manual” way to do it (scroll down to the start of the section on async/await for an actual example). But you don’t even have to do this: popular Promises libraries like Bluebird.js (that provide a bunch of extra functionality, basically empowering the Promise class with the extra missing 10% of async code control provided by modules like async that you’d otherwise have to code yourself when using Promises) provide a Promise.promisifyAll method that can be called on an entire module, like fs = Promise.promisifyAll(require("fs")) and Boom!, you can use any Node.js module, standard lib or otherwise, with promises (it basically creates methods like readFileAsync which returns promises and leaves the default ones unmodified, without the Async suffix).

And btw, mixing callbacks-async and promise-async code, even in the same file is A-OK imho. Consistency rulez, bien sûr, but mixed callbacks-promises code is easy enough to read, reason about, and debug.

I’m going to stop here, even though promises are a topic you absolutely must understand in modern Javascript. No, you can’t just learn callbacks and then async/away. A deep understanding of promises is crucial for all the other things, like corutines/generators/async/await/whatever.

So go and learn JS Promises from some of these excellent resources:

JavaScript Promises: an Introduction @ Google Web Fundamentals: includes nice visual flow explanation, if unsure start with this one
Using Promises @ MDN: concise/condensed
Master the JavaScript Interview: What is a Promise? @ Eric Elliott’s Medium blog: lots of nice diagrams
You’re Missing the Point of Promises: confusing for newbies, necessary reading for professional developers
JavaScript Promise API @ David Walsh: if you like DW’s long but thorough and insightful explanations you might want to start with this
promisejs.org: good intro and standard

Now, moving on, it turns out this is still not the best one could do. And I’m gonna pull a “back from the future” trick on you to explain two more async patterns: (4) first the future (actually present and fully stable and ready to use since Node.js 8.0) and then (3) the coroutine/generators-based backported-future that allows one to write async/await-like code for browsers and older Node.js versions that don’t support the async and await keywords. And no, one can’t just skip learning the co/generators-async pattern in JS, as it’s already featured in prominent places like the official MongoDB-Node.js-driver docs, and there are tones of modules using it, some of which will inevitably remain unupdated/unmaintained while you’ll still need them. So, moving on…

5. Async/Await: the right way …waaay too late!#

Let’s start explaining this by looking at an actual example:

const fs = require('fs');

function getFileContents(fileName) {
  return new Promise((resolve, reject) => {
    fs.readFile('hello.txt', 'utf-8', (err, data) => {
      if (err) reject(err);
      else resolve(data);
    });
  });
}

getFileContents('hello.txt').then(data => {
  console.log('File contents:', data)
}).catch(err => {
  console.error("Error reading file:", err);
});

I hope this makes sense. Though it seems slightly over-engineered, I know. But let’s imagine that getFileContents is not something we wrote ourselves, but part of some module’s API that is standardized to always return promises.

Now… leaving aside the promise producing code, which will often be inside a library somewhere, even the promise consuming code looks ugly af. Lots of extra parens and boilerplate even for this extremely simplified example. Imagine production code using this pattern.

What if the language itself could give you a helping hand and make it more natural to say things like "data is the value you get after waiting for getFileContents" or “the awaited result of getFileContents”. What if you could say something like this:

console.log('File contents:',
            await getFileContents('hello.txt'));

See the callback… that’s not there? You could even put it in a temporary variable since putting funny expressions as method parameters is annoying to read, and have a let data = await getFileContents(...).

But it turns out that having this syntactic sugar can actually make code harder to read and also make it harder for the language to optimize your code… so we need to be a tad bit more explicit: any await expression must be inside a function defined with the async keyword. So we can easily spot functions code that may look sync but actually does async stuff. The actual correct async/await version of the code above is more like this:

// ...
async function showFileContents() {
  let data = await getFileContents('hello.txt');
  console.log('File contents:', data);
}
showFileContents();

But wait… there’s something missing here: error handling! Well, it turns out you get one more nice thing with async/await: you can ditch the ugly .catch(err => ...) callback and handle errors in your async code with the same ol’ try/catch you know and love:

// ...
async function showFileContents() {
  try {
    let data = await getFileContents('hello.txt');
    console.log('File contents:', data);
  } catch (err) {
    console.error("Error reading file:", err);
  }
}
showFileContents();

Here you go! Life is sweet when the language you code in actually helps you. And, if you still can’t feel how much it helps, see how this more realistic example (still simplified because you know production-code error handling flow will never be that simple):

getFileContents('hello.txt').then(data => {
  // ...
  return validator.validateFormat(data).then(isValid => {
    // ...
    return auth.getCurrentUser().then(user => {
      // ...
      return validator.validateSignature(data, user.sig).then(isSigValid => {
        // ...
        return dataParser.parse(data).then(parsedData => {
          console.log('Action to do:', parsedData.action);
          // ...
        });
      });
    });
  });
}).catch(err => {
  console.error("Error getting data from file:", err);
});

Turns nicely into this:

try {
  let data = await getFileContents('hello.txt');
  // ...
  await validator.validateFormat(data); // throws if invalid
  // ...
  let user = await auth.getCurrentUser();
  // ...
  await validator.validateSignature(data, user.sig); // throws if invalid
  // ...
  let parsedData = await dataParser.parse(data);
  console.log('Action to do:', parsedData.action);
  // ...
} catch (err) {
  console.error("Error getting data from file:", err);
}

See how linear and sequential the code reads now? And, while debugging code with promises is not such a pita, debugging async/await code is actually nice: you can actually use the debugger to step through code (with the thens of promises you’ll have to place breakpoints at the start of each, manually, because the debugger only steps through sync code). One can say that async/await finally brought sanity to Node.js!

Oh, and one more thing: to use async/await, you actually need code that returns promises. Ain’t no other way to do it. I didn’t lie when I said before that you absolutely need to understand promises in JS! And I’m not the only one to say this.

The important links between promises and async/await are:

everything you await for is (usually) a Promise
the returned value from an async function is a Promise, so you can gloriously end the previous example with:
showFileContents().then(() => console.log('We did it!'))

Btw, if you haven’t arrived here with some prior understanding of async/await, pause and read up on it:

6 Reasons Why JavaScript’s Async/Await Blows Promises Away (Tutorial) @ hackernoon.com
Async/Await: The Hero JavaScript Deserved @ twillio.com/blog

Now… if you’ve gone so far, there’s one more step to do: a step back from this wonderful future!

4. co/generators for async: the future, in past present#

What if you wanted all the goodness of async/await, but in a JS version without support for these keywords? Let’s see… could we engineer it without any help from the language? Where would we even start from?

To answer this question it’s worth asking: what other slightly older JS feature allows pausing and resuming execution like async/await does? Uhm… generators? And, if you have an advanced enough understanding of JS generators (you can get it from here and here or here among other places), you could end up with a rewrite of the previous getFileContents example looking like this:

const fs = require('fs');

function getFileContents(gen) {
  let g = gen();
  let fileName = g.next().value;
  console.log('File name:', fileName);
  fs.readFile(fileName, 'utf-8', (err, data) => {
    if (err) return g.throw(err);
    g.next(data);
  });
}

getFileContents(function* () {
  try {
    let data = yield 'hello.txt';
    console.log('File contents:', data);
  } catch (err) {
    console.error("Error reading file:", err);
  }
});

That’s cool, but… hardly any good: you’ve traded simple callbacks for generator callbacks, just to get to use the yield keyword as some context-dependent synonym of await. Now it’s more complicated and it would need more boilerplate and same nesting callbacks if you want a succession of async actions: “callback hell” + “generators madness” …yuck!

But, if you’re a smart cookie, you’d have notice that this could be rewritten in a more general way and be made to work with promises… somehow. We’re not going to explain here how. By piling trick upon trick, you would reinvent what the famous co module does, and be able to write code like this:

const co = require('co');

co.wrap(function*() {
  try {
    let data = yield getFileContents('hello.txt');
    // ...
    yield validator.validateFormat(data); // throws if invalid
    // ...
    let user = yield auth.getCurrentUser();
    // ...
    yield validator.validateSignature(data, user.sig); // throws if invalid
    // ...
    let parsedData = yield dataParser.parse(data);
    console.log('Action to do:', parsedData.action);
    // ...
  } catch (err) {
    console.error("Error reading file:", err);
  }
}).then(() => console.log('We did it!'));

So one can basically use generators to implement async/await in JS. I’ll let you decide whether this is awesome or scary.

Things like co are the work of very smart people who can’t stand living in the present, once they’ve seen the future! For regular folks like me, this may sometimes seem like some grotesque “intellectual masturbation”. But, it’s due to the work of people like them that we can have the sweet future of async/await: once they’ve got to play with co, people realized that async/await is the future, and they’ve worked hard to make it part of the language! So, despite your first instinct, thank them, don’t curse them!

In the end, this is one more JS async pattern that you have to understand whether you like it or not: you will come across code using it!

But… your dizzy mind wonders: which of these should I actually use?

There’s no perfect answer, but here goes my opinionated rant:

Which async patterns to use and when?#

My guide is to walk through these questions in order:

Is your code very simple (less than 200 LOC, not much successive async operations leading to nested callbacks) OR extremely performance sensitive (like the inner loop of a game engine, where even creating promise objects would eat too much memory or be too slow)?
THEN: use callbacks-async! Keep it simple, stupid!
Do you target pre-node-8.0 or browsers (without transpiling) OR you add stuff to a heavily-mixed-style codebase (which already uses callback, async, promises etc.)?
THEN: use promises! This will keep things obvious and mix well with anything. And if standard promises are not full-featured enough for your use cases, just pull in something like Bluebird.
Are you starting a new project targeting node ≥8.0 (or using a transpiler)?
THEN: use async/awayt ffs! It’s 2017, let’s start acting like it…

You’ll probably notice that I skipped two approaches:

async module: I don’t really see a room for this, unless you’re working in an old-school callbacks-only project with a strict guideline against promises or anything newer… then by all means use it, it’s 10x better than reinventing a buggy version of it yourself.
co/generators: again, this is a middle ground solution, for which I don’t see any room nowadays. If you can’t use async/await, then stick with promises! The code is readable enough, waaay more newbie friendly (especially when it comes to debugging) and can grow well into async/await later on. Really, forget about this, it was a useful experiment but its time has passed!

One more question lingers: why “5+” ways, and not just “5”? Cause I’m modest enough to assume there are other ways I’m not yet aware yet… or ’cause Tim Toady says so…