Everything You Need To Know About Async & Meteor
Async & Meteor
Meteor runs on top of Node.js. This means I/O activities such as reading a file or sending a request over the network won’t block the whole program. Instead, we provide callbacks that will be executed in the Event Loop later when those activities finish. Ok, I agree, that may not make a lot of sense. How about some cartoons!
Let’s say our task is to read an encrypted file, then decrypt it and get the secret content:
var aes = Meteor.require('aes-helper')
, fs = Meteor.require('fs');
var getSecretData = function(key) {
fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
if (err) console.log(err);
else console.log( 'Secret: ' + aes.decrypt(res, key) );
}
};
getSecretData('my-secret-key');Here is what a generic, garden-variety Event Loop looks like:
The Event Loop is just a queue of functions waiting to be executed. Every time we call a function, it is put onto the Event Loop
When we execute getSecretData to decrypt and print out the secret, the function readFile get called and appear on the Event Loop:
That readFile guy doesn’t care about what happens later at all, he just tells the OS to send the file and then go away!
Some moment later, the readFile operation is completed. A guy with the name ‘callback’ will jump into the Event Loop:
Later, when the file is received, our hero appears and finishes off the job
That’s quite nice and get the job done. But what if our task is more sophisticated and require many level of async operations? We might end up with something like this:
picture from Sagi Isha‘s Medium
The problem of async control flow is it makes the code more difficult to read and maintain. It would be nicer if we can have getSecretData return the secret content and print it out synchronously, like this:
/* This code looks nicer, but sadly it doesn't work */
getSecretData = function(key) {
var decryptedSecret;
fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
if (err) console.log(err);
else decryptedSecret = aes.decrypt(res, key);
}
return decryptedSecret; // undefined This code does not work, because getSecretData will not wait for the readFile operation to finish. It just goes on and return decryptedSecret as an undefined value. To solve this problem, we need a new hero. Here comes Fiber!
Meet Fiber, the big guy who can carry many functions inside him
A Fiber is a special container function. He can be put into the Event Loop like other normal functions. But Fiber has a special power: He can halt at any point in the middle of his execution, get out of the Event Loop to take a rest, then come back at any time, all at his will (or in fact, the developer’s will). When a Fiber halts, control will be passed to the next function in the Event Loop (which may be a normal function or yet another Fiber).
You properly already see the advantage here: If our Fiber contains a function that performs a time-consuming I/O activity, he can just get out of the Event Loop and wait for the result. In the mean time, we can go on and run the next functions waiting in the queue. Life is short and time is precious! When the I/O activity finishes, our Fiber can jump back in again and resume what he was doing the last time.
Here is our code, Fiber-powered:
var Fiber = Npm.require('fibers');
// Our Fiber-powered getSecretData function
getSecretData = function(key) {
var fiber = Fiber.current; // get the currently-running Fiber
fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
if (err) console.log(err);
else fiber.run( aes.decrypt(res, key) ); // resume execution of this fiber.
// The argument passed to fiber.run
// (i.e. the secret data) will become
// the value returned by Fiber.yield
// below
}
// halt this Fiber for now. When the execution is resumed later, return whatever
// passed to fiber.run
var result = Fiber.yield();
return result;
};
// We wrap our code in a Fiber, then run it
Fiber(function() {
var result = getSecretData('my-secret-key');
console.log(result); // the decrypted secret
}).run();Alright, that may not make a lot of sense. Here are your cartoons:
When Fiber encounters a yield, he knows it’s time to take a rest!
Calling run() will signal the resuming of execution for this Fiber. Whatever passed to run() will become the value returned by yield()
I hear you saying: ‘Ok, that’s looks good. But the yield and run stuffs still sound weird to me’.
I got you. We’ll see something even nicer than a Fiber. It’s a Future!
You can see Future as an abstraction layer on top of Fiber. This gives us the power of Fiber with a nicer API. Like a well-groomed Fiber.
var Future = Npm.require('fibers/future');
// Our Future-powered getSecretData function
getSecretData = function(key) {
var future = new Future; // create a new, bright future
fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
if (err) console.log(err);
else future.return( aes.decrypt(res, key) ); // signal that the future has
// finished (resolved)
// the passed argument (the
// decrypted secret) will become
// the value returned by wait()
// below
}
return future; // we return the future instance so other code can wait()
// for this future
};
// The future method is added to the prototype object of every function
// Calling future() on a function will return a Fiber-wrapped version of it
(function() {
// we wait for the future to finish. While we're waiting, control will be yielded
// when this future finishes, wait() will return the value passed to
// future.return()
var result = getSecretData('my-secret-key').wait();
console.log(result);
}.future()) ();Wait! In the examples above, we have freely modified our getSecretData function. What if you come across an async function that you can’t modify (like functions from external APIs)? No worry, instead of modifying it, we can wrap it up!
// A native, garden-variety async function
getSecretData = function(key, callback) {
fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
if (err) throw new Error(err.message);
else callback && callback( null, aes.decrypt(res, key) );
}
};
// Let's wrap it up
// What returned here is actually a future instance. When the async operation
// completes, this future instance will be marked as finished and the result
// of the async operation will become the value returned by wait()
var getSecretDataSynchronously = Future.wrap(getSecretData);
(function() {
// we call wait() on the returned future instance to, well, wait for this future
// to finish and get the result later when it does
var result = getSecretDataSynchronously ('my-secret-key').wait();
console.log(result);
}.future()) ();Hmm, looks like we’ll need to remember to call wait() every time. What a hassle! Fortunately, it’s even simpler if we use Meteor.wrapAsync:
getSecretData = function(key, callback) { fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) { if (err) throw new Error(err.message); else callback && callback( null, aes.decrypt(res, key) ); } }; var getSecretDataSynchronously = Meteor.wrapAsync(getSecretData); var result = getSecretDataSynchronously(key); // <-- no wait() here! return result;
Actually there’s more to async than meets the eye. Other useful things that are worth mentioning:
Future.wrap and Meteor.wrapAsync are very selective
They only do business with native, pure async functions. That is, functions that expect a callback with error and result as arguments. Also, they only work on the server-side (since yielding is not possible on the client – there are no Fibers living there).
Meteor.wrapAsync will turn your innocent function into Two-Face !!!
Fortunately, two-faced functions are not as destructive as Harvey Dent. In fact, they’re pretty useful: They can be called synchronously (like what we were doing above) or asynchronously (with a callback passed to them).
On server-side, methods such as HTTP.call and collection.insert/update/remove are all already pre-wrapped this way. Take HTTP.call for example: If you call it without a callback, the method will block until the response is received. If called with a callback, HTTP.call returns immediately, and will later execute the provided callback when network response has arrived.
On client-side, since blocking/yielding is not possible, we always have to provide a callback to these methods.
Fiber hiccups
By default, method calls from a client are run inside a single Fiber – they’re run one at a time. This Fiber gets access to a set of environment variables that are specific to the currently connected client (e.g. Meteor.userId()). This may result in two common problems:
1) On server-side, calling methods like HTTP.call synchronously will block other subsequent method calls from the current client. This may not be a good thing. If subsequent methods are independent from the current running method, we can save time by using this.unblock(), which will allow other method calls to be run in a new Fiber:
JavaScript
Meteor.methods({
requestSecret: function() {
this.unblock();
return HTTP.call('GET', 'http://www.nsa.gov/top-secrets');
}
});2) “Meteor code must always run within a Fiber”
Looks familiar? This error often occurs when you try to call a third-party API with async callback. You’re not allowed to do this, since the callback function would be executed outside Fiber, without access to necessary environment variables. One way to solve this is wrapping the callback function with Meteor.bindEnvironment, which will return a Fiber-wrapped and environment-packed version of the function. The second way is using Meteor.wrapAsync like what we were doing above (actually wrapAsync already called bindEnvironment internally for us!).
I hope you’ve learned something useful about async and Meteor in this article. Happy coding!
