• Node.js
  • Tutorial

Pattern: Asynchronous Iteration

If you've been using Node.js and Connect, you're familiar with this idea of calling the next function to move on to the next piece of middleware. And even if you've been using Meteor and Iron Router, this is a concept that applies as well-- the idea of calling a function to move to the next item in a list.

I've given this pattern a name called asynchronous iteration. I don't know if that's an official name, but that's what I'm going to call it. And in the rest of this video, I'm going to show you a little bit more in detail what that means.

All right. I've pasted in some new code here because I want to focus specifically on this concept and get rid of all the other abstractions. If you're following along and you want to run this program, type npm start into the cloned repository, and that'll run the program.

So let's walk through what this program is actually doing. So I've created a variable called middleware. And I've called it that because I want you to think about Connect middleware when we're talking about this concept. It's just an array of functions named 1 through 3.

And then what I want to do is to iterate or to loop over all of these functions and for each one to call that function. So another word for this is we're calling these functions in sequence or in order. So I want to call the first function, and then the second one, and then the third one.

Another concept to point out here is that each function is executing synchronously. So each function is synchronous. And what this means is that the second function doesn't get called until the first one returns. And then the third function doesn't get called until the second one returns. So they execute sequentially and also synchronously. Another word for this is that each function is actually blocking until it returns.

So to make this a little bit more clear, what we could do is when this is all done, print out to the console that we're done. And then I'll hop over to the right and rerun this program. And notice that the numbers 1 through 3 are printed in sequence. And then once those are done, we print done out to the console. So all of this runs synchronously.

Now, let's throw a little hiccup in here and make one of these functions asynchronous. If you recall, asynchronous means to run something at a later time. And in JavaScript, we can do that by using setTimeout or using one of the Node.js input/output, or I/O functions, to execute some input/output and then call the function at a later time when it's done.

But for now, just to simulate the concept we'll use the setTimeout function. We'll pass it a callback that we want to run at a later time, let's just say 3,000 milliseconds or 3 seconds later. And I'll pop this console.log line right into the setTimeout. So this is asynchronous because this function inside of here is going to run at a later time. But as soon as this gets called, it'll return and then our return statement will execute next.

This time if I run the program, notice the result is a bit strange. It's out of order. So we start off with 1. But then it prints out 3 and then done, and then finally at the very end 2. And the reason for that is because we've introduced this asynchronous function, setTimeout.

And so function 2 returns, and then function 3 runs, and then we print out done. And then two seconds or three seconds later is when this function actually gets run. So because it's asynchronous and we're looping synchronously, we've lost our order.

So this describes the problem that we're trying to solve. What I'd like to do is to be able to keep the order of these functions. I want them to run sequentially. But I also want to be able to use asynchronous functions like setTimeout, or more likely an I/O function like readfile or writefile.

The way we can achieve this is by passing in an iterator function as a parameter and then allowing these functions themselves to control when we move on to the next item. So let's code out the example together. I can change this so that each one of these middleware functions takes a parameter. And I'm going to call it next just like Connect calls it.

Then what I want to be able to do is to call the next function when this function is done and ready to pass on control to the next item. In fact, I'll just get rid of the return statement, and we'll just call next right away here. And we'll do the same thing in function 3. Instead of returning, I'll just call next. But in function 2, instead of calling next at the ends like this, we'll call it inside of the setTimeout. And then we'll get rid of the return statement.

Now I need a function that's actually going to start this iteration and control the iteration. So I'm going to create a function called run, and then call it so that I don't forget. And what this function is going to do is to control the iteration through our list.

So to start off, I'm going to create a variable for i that will be the index of the item that we're currently in. And then I'm going to create our next function. And what this will do is to say, if the index is still less than the size of the list-- in this case, we could say middleware.length, or we could just hard code it to 3-- then we'll find the right middleware.

So we'll go to that index. And we may as well increment the index pointer while we're at it and then call that function. And we're going to pass in this function as a parameter. So we'll pass in the iterator function as a parameter. So this will start off at middleware 0, call that function passing in next as a value, and then it'll move on to 1 and 2.

And then when it gets to 3, it's no longer less than the middleware length. And so this will just bail out and return. So we could put a return statement here to be a little bit more clear, but it's not required. So I'll just say else, we'll just return.

And now what I need to do is to kick this process off. And so I'll call the next function. That'll start our iteration. And then down at the bottom, I'm calling run. And this time when I run the program, notice that it almost seems like we're waiting here for the next value. And once we have it, it prints out 2 and then 3.

But it's not actually blocking. What's happening is the next function has just not been called yet. And it gets called three seconds later, and then we continue iterating through the function list.

So using this technique, we're able to maintain our sequence and also use asynchronous functions like setTimeout. Now, one thing this is lacking is we don't know when this entire sequence has completed. And what I'd really like to be able to do is to bring back our done string here when the entire middleware stack is completely run.

So what we could do to achieve that is just pass in a done function to run. And let's just make it a callback that I'll call done. And when we're completely done, I'll print out to the console that we're done like this. And what we can do inside of the run function is when we're completely done iterating through this list, I'll call the done method that's passed in as a parameter.

When I run this over in the right, now we'll print out 1, wait for the second function to complete, and then it goes through the rest of the list and prints out done like what we had to begin with. So now we've kept our sequence, and we're able to use asynchronous functions. And this is the pattern that's used in Connect middleware.