Reading Large Files In Node.js From Another Server

I have two servers that communicate with each other. Server1 requests for parts of the file from Server2 and store the data received into one file. Server2 is supposed to receive e

Solution 1:

Your main problem here is that you treating your data as string by appending chunks to a string.

By rewriting this should be

var req = http.request(options, function(res) {
  var data = [];
  res.on('data', function(chunk) {
    data.push(chunk);
  });

  res.on('end', function() {
    fs.writeFile(fileName, Buffer.concat(data), function() {
      console.log("write end")
    });
  });
});

That way we are creating a big array of binary chunks, and when the download is complete we write the concatenation of all the chunks to a file.

But notice the word big

If you stick with this implementation you are risking to get out of memory, especially if you are dealing with large (>500mb) files.

Streams to the rescue

var req = https.request(options, function(res) {
  res.pipe(fs.createWriteStream(fileName)).on("close", function() {
    console.log("write end");
  });
});

Using the above implementation memory footprint should stay low. Because the moment you get a specific amount of data from your download, you write them to the file. That way you never keep the whole file into the program's memory.