• Meteor

Merging the Client View of Data

From the class:  Publish Subscribe in Meteor

We've talked about how a client gets a view into server side data. And the way that it gets the view is by subscribing to a subscription or a publish function. I've created two example subscriptions here, one called best item, and the other called the most recent item.

Let's pretend that both of these subscription functions use the same collection. So we're going to publish data from something called the items collection. And then let's pretend that both of these publish functions publish out the same exact document. And that could happen, for example, if the best item and most recent item are the same.

To simulate this, I've created a document up here with an ID and a title. And inside of both publish functions, I'll call the added function of the subscription. And we'll see that we're adding a document to the items collection with the document ID. And then I'll just pass the document.

And there will only be one document in this best item subscription, so I'll send the ready message right after that. I'm going to do the same thing down here for the most recent items subscription. We'll send the same exact document, and then we'll send a ready message.

Now, one of the jobs of live data and of the pub subsystem of Meteor is to make sure that the client only gets the minimum set of messages required to update its view of the data. And so in this case-- let me clear out the drawings a little bit. In this case, what it needs to do is to take this added message, and it's going to send that to the client. And then by the time we get here and we try and send it again, Meteor needs to realize that that's already been sent. And so what the client will get is just one added message. Because that's the minimum set of messages required to get the client up to date with having this document locally.

So sometimes this process of taking these two identical messages and merging them is called-- or the mechanism that does it is called the merge box. And you might see that mentioned on Stack Overflow or in other places. But we'll look at the actual object in source code that does this, and it's called a session collection view.

And it's called a session collection view because it's a view that a client has into a particular collection for that session. And in this case, the collection would be the items collection. So let's take a look at this in action by using the DDP tools to subscribe to both of these publish functions at once.

To make it more clear, I'm going to use the DDP command line tools instead of a browser. And I can subscribe to multiple publish functions at once by using the subscribe-multi command, and then passing the names. So we'll subscribe to the best item subscription and then also to the most recent item subscription.

All right. Once we connect, you can see the client subscribes first to best item and then to most recent item. And when it subscribes to best item, the server runs the handler and sends the added message for the item document. And we can see that message here with the collection items and the ID of one, and then the actual fields that have been added. And then we send the ready message from the server, and you can see that reflected over here on the client.

Next step, it subscribes to the most recent item. And in that case, the added function gets called again, but we don't see another added message down here. And that's because the server is smart enough to know that the client has already seen this message, and so we don't need to send it again. So the ready message gets called to indicate that the initial document set has been sent already, and the client should be up to date.

Let's try another example. This time around in the first published function, instead of publishing the entire doc, I'm going to publish just the title. But we'll use the same document ID. And in the second publish function, I'm going to publish the title and also a description. And so now the first publish function is publishing just a part of the fields, and we're adding another field down in the second subscription called description or desc for short.

Over in the client, console I'll subscribe again to both of these publish functions. This time something interesting happens. This time around we subscribe to best item, and the added message gets sent for the document, and then the ready message for the first subscription get sent as well.

And then the second publish function is run. But this time around, the client gets a changed message, even though on the server we're calling the added function. And that's because the pub subsystem of Meteor is smart enough to know that the client has already received this document. But in order to get the client view up to date, it needs to change the document by adding an additional field, this description field. And so the message that the client gets is that we need to change a document in the items collection with an idea of one, and the fields that need to get added are this description field with a value of hello world. And once that gets sent, we send the ready message indicating that the initial document set is complete and has been sent.

I've opened up a browser client on the left so that we can play in the JavaScript console. I want to show you what happens when we unsubscribe from a publish function, especially when there's overlap like we have here. To start, let's go into the browser. And I'm going to subscribe to the first publish function. So we'll call this handle best item sub.

And let me update the WebSocket so we can see what messages were sent. OK, great. It looks like we got the initial added message, and then the ready message from the first subscription. Now let's create the most recent item sub by subscribing to most recent item.

And I'll refresh the WebSocket tab. And just like we saw in the console, this time around we're going to get a changed message because Meteor knows that we've already got the document from this added message. And so all we need to do is add this description field. Next up, what I want to do is to unsubscribe from the second publish function.

And if we unsubscribe, what do you think will happen? Well, we already have a document, the same document published from a different publish function. And so we don't want to get rid of the document altogether. But notice that this document is different from this one because this one just has a title, and this one just has a description.

So the minimum set of messages required to get the client up to date is to just remove this field, and then we'll be good to go with just this view of the data. Let's see if that's actually what happens. So I'll type most recent item sub, and we'll call the stop function.

When that happens, you can see the client sends up the unsubscribe message for this subscription. And the server sends back a changed message. And this time, what we're changing is the items collection, and we're changing the document with an ID of one. But Meteor is telling the client that it should clear the field called description. So that's how it's going to tell the client to get rid of this field, and that will get us up to date with the document that was published from the first subscription.

Now let's stop the first subscription as well. So I'll say best item sub, and we'll call the stop function on it. Now we have no more subscriptions that are using this document. And so if I were to refresh the WebSocket tab and click on Frames, this time around the server tells the client to remove the document completely. Here's the removed message for the collection items with the ID of 1, and then finally our no sub method to indicate that the server has unsubscribed us.

So to recap, what we do on the client when we want a view of data is we subscribe. And so we'll subscribe, in this case, to this subscription. And then also we'll subscribe to this subscription. And then when these subscription send their added messages, Meteor will merge the result into a single stream of added, changed, and removed messages, the minimum set required to get the client's view of the data up to date.

Now, this system has some memory and CPU characteristics. So I'd like to jump to the source code of Meteor to give you a better sense of both of those. To demonstrate this, I've put the code over on the left, and I've created a console down on the right that we can use for using the DDP tools. And on the right, I have the node inspector Chrome Debugger that we'll be using to debug the server.

And I dropped a breakpoint in the second publish function so that we can see what the state of our server looks like at this point. So down in the bottom, let's subscribe to both of these publications, or both of these subscriptions. And that'll drop us right into the second subscription. And the first one has already run. And so this function has run. It sent the added message for the document, and then its ready message. And now here we are in the second publish function.

In my watch expression, I've expanded the server. So this is our DDP server that's running. And if I go into Sessions, I can see my one connected client here. And if I expand that, we're going to take a look at the collection views property.

This is where Meteor stores the view that the client has into a particular collection. And in this case, that collection is items. If we expand that, you can see it has a couple of properties. But the one that we'll be looking at is the documents property.

These are all the documents that the client has access to based on the subscription functions that it's subscribed to. In this case, we're looking at one document. And if we expand that, you can see each field in the document that the client can see, and for each field, which subscriptions use that field.

There's also another property for the document that shows which subscriptions the field exists in, or the document exists in. I set another breakpoint right below. So now we've used this document from two publish functions or two subscriptions. And if we come over into the collection views now, we'll have the same document in there. But if we look at the data by key, you can see the title is used in two subscriptions, and the description is only used in one. And I can expand that so you can get a look at it.

So these are the two subscriptions that use this title attribute. And for each one, Meteor stores the value that was present at the time that it was used in the subscription. So the important memory characteristic is that for each connected client, the server needs to store a copy of all of the documents and the documents' fields that that client can see. And so you might actually have quite a bit of duplication because each connected client has their own view of the data.

And the CPU characteristics are that every time we call the added, changed, or removed message or methods on a publish function, Meteor needs to go ahead and calculate the minimum number of messages required to reflect the ultimate change to the client. I hope this video gave you a better sense of how the so-called merge box works and the server side memory and CPU requirements to publish out and keep consistent the client's view of data from the server.