Transcript
  • Meteor
  • Tutorial

File Uploader Part 7: Streaming Writes

In the last episode, we finished up the streaming upload method of the Meteor file. And in this episode, we're going to work on streaming writes. So we're going to update the Save method of Meteor file on the server.

So to recall, on the server, we added a method called Save to the Meteor file prototype. And the Save method gets a file path, creates a new buffer with the file's data, and then calls the write file sync method of FS, which is just a node module, and passes in the full file path, the buffer, and then whatever options are passed into our method.

So one thing I should point out that's different from this method from the last time I did an episode on it is this Sanitize method. So I'd like to thank Wojas, who was the first commenter in the last File Save episode. And he pointed out that because this dot name is coming from the client, it can't be trusted. And so we do need to sanitize the file name, and that's the function that I created up here, which just gets rid of any slashes or repeating periods in the file name. So if you noticed anything else here that needs to be updated, just let me know in the comments.

The key changes that we need to make are to support either creating a new file if we're starting at the beginning of an upload or just appending to an existing file if we're, say, halfway through an upload stream. So right now, we're assuming that we're getting the entire file into the buffer and then just writing that to disk all at once. So what we're going to do is we're going to call a method called write sync on FS. And you can read more about these methods in the Node documentation.

But the first parameter of write sync is going to be a file descriptor, which I'll explain in a second. And the second is going to be the same buffer that we have above. The third is going to be the offset of the buffer that we want to write, and then the length that we want to write, and then the offset in the file where we want to start writing. And so that's going to be this.start. And that's going to come from the file which is passed up to the server from the wire. And then when we're done, we need to close this file descriptor.

So what is a file descriptor? A file descriptor, if you're not coming from Unix, is just a lightweight handle to the file. So it doesn't read the file into memory, but it allows you to do something and have a reference to the file in a small amount of memory. So I'm going to create a variable called file descriptor, FD for short, and set that equal to the filesystem.opensync.

So we're going to open up a file descriptor synchronously, and we'll pass the file path. And then it's expecting a mode. And the mode can be a write or append or a number of different modes. So I'm just going to pass mode in here.

And then I'm going to create another variable called Mode. And what I'm going to do is say, if this.start is equal to 0-- so we're starting a new file-- I'm going to just open the file descriptor in write mode. Otherwise, I'm going to open it in Append mode.

OK. So we can get rid of our old write file sync method, and our Save method on the server should be complete. So let's just recap it. So the first parameter of Save on the server is Directory Path. And that's because remember that the Save method is going to be on an instance. And so we're going to do something like file.save. And we'll pass the directory path as the first parameter and then some optional options as a second.

And then what we'll do is we'll create a full file path by joining the directory path with the file name. And we need to sanitize that, because the file name is coming from the client. And so we need to make sure it doesn't contain any weird characters. Then just like last time, we'll create a new node buffer with the data for the file. And for a File Descriptor mode, if we're just starting a new file-- in other words, Start is at 0, so we're just starting at the zeroeth byte-- then we'll create a new file or overwrite an existing one by using the W flag for mode. Otherwise, we'll open a file descriptor in Append mode which is what A stands for.

And the next thing we do is we create a file descriptor by calling the open sync method on the FS node module, passing the file path and then the mode as the second parameter. And then instead of calling write file sync, we just call write sync. But we pass a file descriptor as the first parameter, the buffer that we want to write, where we want to start in the buffer, how much of the buffer we want to write, and then on the actual file, where we want to start writing.

And so this is where we're going to start appending to the end of the file piece by piece as we upload each chunk of the file from the client. And when we're done, we'll just close the file descriptor and return. And so we have all of the parts that we need at this point to read a stream from the client, upload that stream, and then write that stream piece by piece to disk on the server.