How Do I Iterate Through A Large Dataset With Firebase / Node.js?

Currently I'm using Fireproof to make it promise-based and here's what I have: fbase.child('questions').once('value').then(function(questionsSnapshot) { return questionsSnapshot.

Solution 1:

This question is really about iteration performance in Node rather than anything to do with Firebase, if I understand correctly. The question is, once you have this 20000+ array of objects, what is the best way to iterate through them and perform an operation on each one?

There are a few possible performance optimizations you could make. First, you could use a for loop rather than Array.prototype.forEach(), which tends to be faster for very large arrays.

...
varlen = questionsSnapshot.length;
for (var i = 0; i < len; i++) {
  console.log(questionsSnapshot[i].val());
}

Or, using ES6 for...of syntax:

for (let question of questionsSnapshot) {
  console.log(question.val());
}

However, in a quick test using console.time() in Node on an array of 20000 objects, I noticed no consistent performance gain. In fact Array.prototype.forEach() was often faster than using a for loop.

The other optimization you might make is to move the nested callback passed to forEach() into the module scope. That way, it will only be created once, rather than being recreated each time its parent function is called.

fbase.child("questions").once("value").then(function(questionsSnapshot) {
  return questionsSnapshot.forEach(logValue);
});

functionlogValue(questionSnapshot) {
  returnconsole.log(questionSnapshot.val());
}

However, this will only result in a significant performance gain if you make many calls to the parent function, i.e., if perform your Firebase query very often. It sounds to me like the question is more about iterating over a large array once you already have it, not about performing the same query many times over.

So, if you want to perform an operation for each item in your array, I think what you are doing is fine. 20000 objects is not all that huge; it takes about 5ms to iterate through such an array on my unimpressive computer. In fact, whatever it is you are doing to each item is likely to take longer than iterating through the array.

If you notice a particular lag in performance, you might want to elaborate by editing your post. Otherwise, I would suggest not worrying about it.