Execute Batch Of Promises In Series. Once Promise.all Is Done Go To The Next Batch

April 21, 2024 Post a Comment

I have an array that contains an array of promises, and each inner array could have either 4k, 2k or 500 promises. In total there are around 60k promises and I may test it with oth

Solution 1:

Your question is a bit misnamed which may have confused some folks in this question and in the previous version of this question. You are trying to execute a batch of async operations in series, one batch of operations, then when that is done execute another batch of operations. The results of those async operations are tracked with promises. Promises themselves represent async operations that have already been started. "Promises" aren't executed themselves. So technically, you don't "execute a batch of promises in series". You execute a set of operations, track their results with promises, then execute the next batch when the first batch is all done.

Anyway, here's a solution to serializing each batch of operations.

You can create an inner function which I usually call next() that lets you process each iteration. When the promise resolves from processing one innerArray, you call next() again:

functionmainFunction() {
    returnnewPromise(function(resolve, reject) {
        var bigArray = [[argument1, argument2, argument3, argument4], [argument5, argument6, argument7, argument8], ....];
        //the summ of all arguments is over 60k...var results = [];

        var index = 0;
        functionnext() {
            if (index < bigArray.length) {
                getInfoForEveryInnerArgument(bigArray[index++]).then(function(data) {
                    results.push(data);
                    next();
                }, reject);
            } else {
                resolve(results);
            }
        }
        // start first iterationnext();
    });
}

This also collects all the sub-results into a results array and returns a master promise who's resolved value is this results array. So, you could use this like:

mainFunction().then(function(results) {
    // final results array here and everything done
}, function(err) {
    // some error here
});

You could also use the .reduce() design pattern for iterating an array serially:

functionmainFunction() {
    var bigArray = [[argument1, argument2, argument3, argument4], [argument5, argument6, argument7, argument8], ....];
    return bigArray.reduce(function(p, item) {
        return p.then(function(results) {
            returngetInfoForEveryInnerArgument(item).then(function(data) {
                results.push(data);
                return results;
            })
        });
    }, Promise.resolve([]));
}

This creates more simultaneous promises than the first option and I don't know if that is an issue for such a large set of promises (which is why I offered the original option), but this code is cleaner and the concept is convenient to use for other situations too.

FYI, there are some promise add-on features built for doing this for you. In the Bluebird promise library (which is a great library for development using promises), they have Promise.map() which is made for this:

functionmainFunction() {
    var bigArray = [[argument1, argument2, argument3, argument4], [argument5, argument6, argument7, argument8], ....];
    returnPromise.map(bigArray, getInfoForEveryInnerArgument);

}

Solution 2:

An answer from October 2020. Async/await makes it short: only 10 code lines+JSDoc.

Baca Juga

/**
 * Same as Promise.all(items.map(item => task(item))), but it waits for
 * the first {batchSize} promises to finish before starting the next batch.
 *
 * @templateA
 * @templateB
 * @param {function(A): B} task The task to run for each item.
 * @param {A[]} items Arguments to pass to the task for each call.
 * @param {int} batchSize
 * @returns {B[]}
 */asyncpromiseAllInBatches(task, items, batchSize) {
    let position = 0;
    let results = [];
    while (position < items.length) {
        const itemsForBatch = items.slice(position, position + batchSize);
        results = [...results, ...awaitPromise.all(itemsForBatch.map(item =>task(item)))];
        position += batchSize;
    }
    return results;
}

Solution 3:

@jfriend00 Just adding to your answer using async/await with reduce:

functionrunPromisesInSeries(bigArray, getInfoForEveryInnerArgument) {
  try {
    return bigArray.reduce(async (acc, cItem) => {
      const results = await acc
      const data = awaitgetInfoForEveryInnerArgument(cItem)
      results.push(data)
      return results
    }, Promise.resolve([]))
  } catch (err) {
    throw err
  }
}

Solution 4:

In addition, if original array is not of promises but of objects that should be processed, batch processing can be done without an external dependency using combination of Array.prototype.map(), Array.prototype.slice() and Promise.all():

// Main batch parallelization function.functionbatch(tasks, pstart, atonce, runner, pos) {
  if (!pos) pos = 0;
  if (pos >= tasks.length) return pstart;
  var p = pstart.then(function() {
    output('Batch:', pos / atonce + 1);
    returnPromise.all(tasks.slice(pos, pos + atonce).map(function(task) {
      returnrunner(task);
    }));
  });
  returnbatch(tasks, p, atonce, runner, pos + atonce);
}

// Output function for the examplefunctionoutput() {
  document.getElementById("result").innerHTML += Array.prototype.slice.call(arguments).join(' ') + "<br />";
  window.scrollTo(0, document.body.scrollHeight);
}

/*
 * Example code.
 * Note: Task runner should return Promise.
 */functiontaskrunner(task) {
  returnnewPromise(function(resolve, reject) {
    setTimeout(function() {
      output('Processed:', task.text, 'Delay:', task.delay);
      resolve();
    }, task.delay);
  });
}

var taskarray = [];
functionpopulatetasks(size) {
  taskarray = [];
  for (var i = 0; i < size; i++) {
    taskarray.push({
      delay: 500 + Math.ceil(Math.random() * 50) * 10,
      text: 'Item ' + (i + 1)
    });
  }
}

functionclean() {
  document.getElementById("result").innerHTML = '';
}

var init = Promise.resolve();
functionstart() {
  var bsize = parseInt(document.getElementById("batchsize").value, 10),
    tsize = parseInt(document.getElementById("taskssize").value, 10);
  populatetasks(tsize);
  init = batch(taskarray.slice() /*tasks array*/ , init /*starting promise*/ , bsize /*batch size*/ , taskrunner /*task runner*/ );
}

<inputtype="button"onclick="start()"value="Start" /><inputtype="button"onclick="clean()"value="Clear" />&nbsp;Batch size:&nbsp;<inputid="batchsize"value="4"size="2"/>&nbsp;Tasks:&nbsp;<inputid="taskssize"value="10"size="2"/><preid="result" />

Solution 5:

You can do it recursively, for example here I needed to put about 60k documents in mongo, but it was too big, to do it in one step, therefore I take 1k documents, send them to the mongo, after it is finished I take another 1k documents etc.

exports.rawRecursive = (arr, start) => {
        //ending conditionif (start > arr.length) {
            return;
        }

        Rawmedicament.insertManyAsync(_.slice(arr, start, start + 1000)).then(() => {
            //recursiveexports.rawRecursive(arr, start + 1000);
        });
};

If you want to notice, when everything is done, you can in ending condition put callback or if you like Promises you can call resolve() there.

JavaScript Free