Writing to Files in Node.Js

Writing to files in Node.js

There are a lot of details in the File System API. The most common way is:

const fs = require('fs');

fs.writeFile("/tmp/test", "Hey there!", function(err) {
    if(err) {
        return console.log(err);
    }
    console.log("The file was saved!");
}); 

// Or
fs.writeFileSync('/tmp/test-sync', 'Hey there!');

Get data from fs.readFile

To elaborate on what @Raynos said, the function you have defined is an asynchronous callback. It doesn't execute right away, rather it executes when the file loading has completed. When you call readFile, control is returned immediately and the next line of code is executed. So when you call console.log, your callback has not yet been invoked, and this content has not yet been set. Welcome to asynchronous programming.

Example approaches

const fs = require('fs');
// First I want to read the file
fs.readFile('./Index.html', function read(err, data) {
    if (err) {
        throw err;
    }
    const content = data;

    // Invoke the next step here however you like
    console.log(content);   // Put all of the code here (not the best solution)
    processFile(content);   // Or put the next step in a function and invoke it
});

function processFile(content) {
    console.log(content);
}

Or better yet, as Raynos example shows, wrap your call in a function and pass in your own callbacks. (Apparently this is better practice) I think getting into the habit of wrapping your async calls in function that takes a callback will save you a lot of trouble and messy code.

function doSomething (callback) {
    // any async callback invokes callback with response
}

doSomething (function doSomethingAfter(err, result) {
    // process the async result
});

Write a line into a .txt file with Node.js

Inserting data into the middle of a text file is not a simple task. If possible, you should append it to the end of your file.

The easiest way to append data some text file is to use build-in fs.appendFile(filename, data[, options], callback) function from fs module:

var fs = require('fs')
fs.appendFile('log.txt', 'new data', function (err) {
  if (err) {
    // append failed
  } else {
    // done
  }
})

But if you want to write data to log file several times, then it'll be best to use fs.createWriteStream(path[, options]) function instead:

var fs = require('fs')
var logger = fs.createWriteStream('log.txt', {
  flags: 'a' // 'a' means appending (old data will be preserved)
})

logger.write('some data') // append string to your file
logger.write('more data') // again
logger.write('and more') // again

Node will keep appending new data to your file every time you'll call .write, until your application will be closed, or until you'll manually close the stream calling .end:

logger.end() // close string

Note that logger.write in the above example does not write to a new line. To write data to a new line:

var writeLine = (line) => logger.write(`\n${line}`);
writeLine('Data written to a new line');

nodejs: write multiple files in for loop

You can save promises in an array and use Promise.all to wait for them to all finish:

const fs = require('fs');
const path = require('path');

const files = [f1, f2, ...];

function copyFile(source, destination) {
    const input = fs.createReadStream(source);
    const output = fs.createWriteStream(destination);
    return new Promise((resolve, reject) => {

        output.on('error', reject);
        input.on('error', reject);
        input.on('end', resolve);
        input.pipe(output);
    });
}

const promises = files.map(file => {
    const source = file.path;
    const destination = path.join('/tmp', file.path);
    // Use these instead of line above if you have files in different
    // directories and want them all at the same level:
    // const filename = path.parse(file.path).base;
    // const destination = path.join('/tmp', filename);
    return copyFile(source, destination);
});

Promise.all(promises).then(_ => {
    // do what you want
    console.log('done');
}).catch(err => {
    // handle I/O error
    console.error(err);
});

Nodejs: How can I optimize writing many files?

First off, you never want to use fs.writefileSync() in handling real-time requests because that blocks the entire node.js event loop until the file write is done.

OK, based on writing each block of data to a different file, then you want to allow multiple disk writes to be in process at the same time, but not unlimited disk writes. So, it's still appropriate to use a queue, but this time the queue doesn't just have one write in process at a time, it has some number of writes in process at the same time:

const EventEmitter = require('events');

class Queue extends EventEmitter {
    constructor(basePath, baseIndex, concurrent = 5) {
        this.q = [];
        this.paused = false;
        this.inFlightCntr = 0;
        this.fileCntr = baseIndex;
        this.maxConcurrent = concurrent;
    }

    // add item to the queue and write (if not already writing)
    add(data) {
        this.q.push(data);
        write();
    }

    // write next block from the queue (if not already writing)
    write() {
        while (!paused && this.q.length && this.inFlightCntr < this.maxConcurrent) {
            this.inFlightCntr++;
            let buf = this.q.shift();
            try {
                fs.writeFile(basePath + this.fileCntr++, buf, err => {
                    this.inFlightCntr--;
                    if (err) {
                        this.err(err);
                    } else {
                        // write more data
                        this.write();
                    }
                });
            } catch(e) {
                this.err(e);
            }
        }
    }

    err(e) {
        this.pause();
        this.emit('error', e)
    }

    pause() {
        this.paused = true;
    }

    resume() {
        this.paused = false;
        this.write();
    }
}

let q = new Queue("file-", 0, 5);

// This fires 30 times/sec and runs for 30-45 min
dataSender.on('gotData', function(data){
    q.add(data);
}

q.on('error', function(e) {
    // go some sort of write error here
    console.log(e);
});

Things to consider:

1. Experiment with the concurrent value you pass to the Queue constructor. Start with a value of 5. Then see if raising that value any higher gives you better or worse performance. The node.js file I/O subsystem uses a thread pool to implement asynchronous disk writes so there is a max number of concurrent writes that will allow so cranking the concurrent number up really high probably does not make things go faster.

2. You can experiement with increasing the size of the disk I/O thread pool by setting the UV_THREADPOOL_SIZE environment variable before you start your node.js app.

3. Your biggest friend here is disk write speed. So, make sure you have a fast disk with a good disk controller. A fast SSD on a fast bus would be best.

4. If you can spread the writes out across multiple actual physical disks, that will likely also increase write throughput (more disk heads at work).

---

This is a prior answer based on the initial interpretation of the question (before editing that changed it).

Since it appears you need to do your disk writes in order (all to the same file), then I'd suggest that you either use a write stream and let the stream object serialize and cache the data for you or you can create a queue yourself like this:

const EventEmitter = require('events');

class Queue extends EventEmitter {
    // takes an already opened file handle
    constructor(fileHandle) {
        this.f = fileHandle;
        this.q = [];
        this.nowWriting = false;
        this.paused = false;
    }

    // add item to the queue and write (if not already writing)
    add(data) {
        this.q.push(data);
        write();
    }

    // write next block from the queue (if not already writing)
    write() {
        if (!nowWriting && !paused && this.q.length) {
            this.nowWriting = true;
            let buf = this.q.shift();
            fs.write(this.f, buf, (err, bytesWritten) => {
                this.nowWriting = false;
                if (err) {
                    this.pause();
                    this.emit('error', err);
                } else {
                    // write next block
                    this.write();
                }
            });
        }
    }

    pause() {
        this.paused = true;
    }

    resume() {
        this.paused = false;
        this.write();
    }
}

// pass an already opened file handle
let q = new Queue(fileHandle);

// This fires 30 times/sec and runs for 30-45 min
dataSender.on('gotData', function(data){
    q.add(data);
}

q.on('error', function(err) {
    // got disk write error here
});

---

You could use a writeStream instead of this custom Queue class, but the problem with that is that the writeStream may fill up and then you'd have to have a separate buffer as a place to put the data anyway. Using your own custom queue like above takes care of both issues at once.

Other Scalability/Performance Comments

1. Because you appear to be writing the data serially to the same file, your disk writing won't benefit from clustering or running multiple operations in parallel because they basically have to be serialized.

2. If your node.js server has other things to do besides just doing these writes, there might be a slight advantage (would have to be verified with testing) to creating a second node.js process and doing all the disk writing in that other process. Your main node.js process would receive the data and then pass it to the child process that would maintain the queue and do the writing.

3. Another thing you could experiment with is coalescing writes. When you have more than one item in the queue, you could combine them together into a single write. If the writes are already sizable, this probably doesn't make much difference, but if the writes were small this could make a big difference (combining lots of small disk writes into one larger write is usually more efficient).

4. Your biggest friend here is disk write speed. So, make sure you have a fast disk with a good disk controller. A fast SSD would be best.

Write a file into specific folder in node js?

Ensure that the directory is available & accessible in the working directory.
In this case, a function like below needs to be called at the start of the application.

function initialize() {
  const exists = fs.existsSync('./niktoResults');
  if(exists === true) {
    return;
  }
  fs.mkdirSync('./niktoResults')
}

---

Related Topics