"error: Write After End" With Csv-write-stream

I am sure this is a fundamentally misunderstanding of how streams work, but I am banging my head on the wall. I have some sensor data in json format that I want to append to a csv

Solution 1:

Without seeing the full code, I can imagine that you're calling write_csv multiple times, since you're trying to write multiple objects to that file.

The issue is that the first time you call write_csv you're ending the writer, that's why the second time you call it you get:

Error [ERR_STREAM_WRITE_AFTER_END]: write after end

---

functionwrite_csv(obj) {
    writer.pipe(fs.createWriteStream('out.csv', { flags: 'a' }))
    writer.write()
    writer.end();
}
write_csv({ hello: 'world', foo: 'bar', baz: 'taco'});
// When you call it again, writer.end(); is already closed// The following line will trigger the errorwrite_csv({ hello: 'world', foo: 'bar', baz: 'taco'});

What you should do instead, is close the writer only when you're done writing to it.

const writer = csvWriter(); // Create new writer// open file
writer.pipe(fs.createWriteStream('out.csv', { flags: 'a' }));

for(const obj of objects) // Write as many times as you wish
   writer.write(obj);

writer.end(); // I'm done writing.

---

Now the problem you have, is that if you try to perform multiple .writes you will reach memory limit, because you're not dealing with backpressure correctly.

I recommend reading the following question:

why does attempting to write large a large file cause js heap to run out of memory

To deal with that, you will need to wait for the drain event to be emitted.

Here is a wrapper around csvWriter that will handle backpressure.

const fs = require('fs');
const csvWriter = require('csv-write-stream');

classWriter {

    constructor(file) {
        this.writer = csvWriter();
        this.writer.pipe(fs.createWriteStream(file, { flags: 'a' }));
    }

    write(obj) {
        // if .write returns false we have to wait until `drain` is emittedif(!this.writer.write(obj))
            returnnewPromise(resolve =>this.writer.once('drain', resolve))

        returntrue;
    }

    end() {
        // Wrap it in a promise if you wish to wait for the callback.this.writer.end(); 
    }

}

(async() => {
    const writer = newWriter('out.csv');

    for(let i = 0; i < 1e8; i++) {
        const res = writer.write({ hello: 'world', foo: 'bar', baz: 'taco' });
        if(res instanceofPromise) {
            // You can remove this if, and leave just: await writer.write...// but the code will be slowerawait res; // This will wait for the stream to emit the drain event
        }
    }

    writer.end();

})();

---

Update: Now with the actual code, the above answer still stands, but since you're writing to the file when receiving a request. You can choose whether to open the file once, and write on every request, close it when the server shuts down (or when you choose). or just open the file, write to it, and close it on every request,

For the former, you should use the answer above, for the latter, all you need to do, is create a new writer every time you call write_csv instead of having one global writer.

functionwrite_csv(obj) {
    // Create a new writer every timeconst writer = csvWriter({ sendHeaders: false }); 
    writer.pipe(fs.createWriteStream('out.csv', { flags: 'a' }));
    writer.write(obj);
    writer.end();
};

Solution 2:

You can try this code instead, I thinks it solves your problem. Use this function in your route and pass the json data.

First install this package: npm i --save json2csv

constJson2csvParser = require("json2csv").Parser;
const fs = require("fs");

functioncsvConverter(jsonData, cb) {

  const json2csvParser = newJson2csvParser();
  const csv_data = json2csvParser.parse(jsonData);
  const file_name = "report";

  // TODO: Change file path accordinglyconst file_path = `/Users/public/csv_files/${file_name}.csv`;

  fs.appendFile(file_path, csv_data, err => {
    if (err) {
      console.log(err);
      cb(err, null);
      return;
    }
    const response = {
    msg: "successful",
    file_address: file_path
  };
    cb(null, response);
    return;
  });
}