• Meteor
  • Tutorial

Subscriptions and DDP

In this screencast, we'll get some intuition about how Meteor's DDP protocol works with subscriptions and collections. The DDP protocol is changing. But our goal for now is just to see how the parts fit together. So even when the protocol changes, these concepts should still be helpful.

Let's visualize the parts of the Meteor system we'll be looking at. First, I have the client, or the browser. In the browser, we can create collections, which are driven by Mini Mongo. We can also subscribe to messages broadcast from the server.

Then I have the Meteor server. In the server, we can create collections, which use MongoDB, and we can publish messages to connected clients. The client and the server pass messages back and forth using the DDP protocol, and the messages are simple JSON objects. We'll start off with a simple subscription to a Mongo DB collection called Posts with a user interface that just shows a message for each post.

Let's take a quick tour through the code, and then we'll take a look at the DDP messages that are being sent by the server. On the left is the app.html file. In the body, I'm rendering the post template. Inside the template, I iterate over all the posts, creating a list item and displaying the message for each one.

On the right is the app.s file. First, I create a new media collection called Posts. This code will run on both the client and the server. But on the client, the collection will be driven by Mini Mongo, which is a light, in memory version of MongoDB that runs completely in the browser.

Inside the client, I've defined the post helper on the posts template, which reactively returns the results of the Mini Mongo post collection. Then I create a subscription by calling meteor.subscribe and passing the name of the subscription as the first parameter, an array of parameters as the second, and finally, a callback that will get called when the subscription is complete. This happens when all the initial documents have been sent.

On the server when it starts, I'll load up my Mongo post collection with a few dummy documents. Then I'll create a publication by calling meteor.publish with the name of the publication as the first parameter and a callback function as the second. The callback function will get called any time a new client subscribes. In this example, the callback returns post.find. This will automatically publish all existing post documents as well send messages for added, changed, or removed documents.

Let's try to get a better sense of what's going on by inspecting the DDP messages being sent when we subscribe to posts. To do that, we'll create a simple DDP client that prints out the messages to the console. I've created a fork of Tom Coleman's node DDP client and added the ability to inspect DDP messages as they come off the wire. Since I haven't published my fork to NPM, I'm going to first add it as a sub module to my Get project.

I'll type Get Sub Module Add followed by the location of the repository and then the destination-- in this case, inside my DDP client folder. I get a message saying it already exists, because I've already added the sub module. But if you're doing it for the first time, you won't get this error.

Next, I'm going to grab the code for the sub module by typing Get Sub Module Update with the init switch. Once again, I get an error saying I've already done this. But if you're initializing the sub module for the first time, you won't get this error. Finally, I'll change into my DDP Client folder and type NPM link followed by the path to the DDP client module. That will install the module and all of its dependencies in a central folder and then sim link to it from the Node Modules directory.

Great. Now I can require the DDP module in my code. Now let's create our client.js file and pop into Vim. First, I"ll require the DDP module. Next, I'll create a new DDP client, passing an Options object with a host of local host and a port of 3000. Then I'll add a callback using the On Message method I created in my fork to print DDP messages to the console as they come off the wire.

Finally, I'll call the Connect method. And once we're connected, I'll call the Subscribe method with the name of the subscription as the first parameter, an array of parameters as the second, and a callback function that'll get called when the subscription is marked as complete. Let's try it out.

In the upper right, I'm running my Meteor server. In the upper left, we see things working as expected in the browser. In the bottom terminal, let's fire up our DDP client. A key thing to notice is that for each post in the post collection, we get a DDP message telling us to set certain fields in a document with a given ID and for a collection named Posts. Then we get a final message telling us we've received all the initial posts. In other words, our subscription is complete and our subscription callback is called.

OK. Let's try creating our own publisher that doesn't use a server side Mongo collection. Over in the app.js file, the first thing I'll do is move the post collection into the client. This means it will only exist locally on the browser, not in our server side Mongo store. We, won't need the server startup code anymore since we've removed the server side collection.

Now let's send some custom messages in our Publisher function. First, by convention, I'll store this in a variable called Self. Then I'll create a new unique identifier that we'll use a little later. Next, I'll call this set method of Self to send a self message. The first parameter will be the name of the collection we wish to perform a set operation on. The next parameter is the ID of the document. The last parameter is an object containing the fields we want to set.

So far, we've only been queueing up the messages. We tell Meteor to send them down to the client by calling the Flush method. Let's take a look.

We can see our messages appear in the browser as expected. If we start our DDP client, we can see the two messages we set in the publisher function. But wait. We're missing something.

Notice in the browser our Subscription Complete callback didn't print anything to the console. And in our DDP client, we don't see the callback message printed either. Although it's not preventing us from receiving the messages, let's update our code to send the Subscription Complete message to the client.

In our Publish callback, all we need to do is call the Complete method on self. This will queue up a complete message that will be sent to the client along with the rest of our messages. Over in the browser, we can see the callback prints the message to the console. And we can see in our DDP client that the last message is the Subscription Complete message.

Next, let's look at how to send an unset message. Inside my publish callback after the initial flush, I'll use the meteor.settimeout method to wait two seconds and then send a message that will remove a document. To do that, I'll call the unset method of the subscription, passing the name of the collection as the first parameter, the ID of the document as the second, and an array of fields to unset as the third.

If all fields are unset, the document will be removed on the client. I'll call the Flush method again to tell the subscription I'm done creating messages and to flush them all to the client. In the browser, I see the initial subscription is complete, and then two seconds later, the messages removed. In my DDP client, I can see the subscription complete callback is called. And then I see what the unset message looks like.

To conclude, let's look at the difference between subscriptions and collections. So far, I have a subscription called Posts, and I've been calling the set method with posts as the first parameter. This tells the client we want to modify the post collection. Instead, let's tell the client to modify the Items collection, but keep the subscription name the same.

In the browser, you can see that the subscription complete callback is called, but no messages are shown. That's because we don't have a collection named Items defined on the client, so Meteor doesn't know what it's supposed to update. In the DDP client, I can verify that the messages are still being sent, but the collection name is now Items.

If we change our client collection name to Items, we should see the messages again. I'll go ahead and change all the client references from posts to items, but I'll continue to subscribe to posts. In the browser, now that I have a collection called items, we see the messages again. So this was a bit theoretical, but I'm hoping it illustrates the difference between the name of a subscription and the name of a collection used in a set or unset message.

I'd like to give credit to two members of the media community. My node DDP client project was forked from Tom Coleman's project. You might also know of Tom's work on the Telescope project. And Tom's DDP client work was based on Alan Sikora's original DDP client project. Both of these guys are awesome contributors, and you should check out their work on GitHub.

Stay tuned for more videos next week. I've got material lined up for a tour of the Media Router project as well as hooking Meteor up to the queue job framework. If you have a preference, leave a comment in the Feedback box on the right of the Evented Mind page. Have a great weekend.