beerman•3w ago

batching tasks

I'm calling an api and get an array of 2000+ items. I want to chunk these into batches of 100 and then process them all concurrently. I've tried BatchQueue but it looks like it stops after processing only the first batch.

const batchQueue = new BatchQueue(100)
const data = await response.json() as ApiResponse[];

let processedCount = 0;

this.batchQueue.queue(
  ...data.map((spin) => async () => {
    try {
      await this.processSpin(spin);
      processedCount++;
    } catch (error) {
        console.error(error);
    }
  }),
);

await this.batchQueue.run();

const batchQueue = new BatchQueue(100)
const data = await response.json() as ApiResponse[];

let processedCount = 0;

this.batchQueue.queue(
  ...data.map((spin) => async () => {
    try {
      await this.processSpin(spin);
      processedCount++;
    } catch (error) {
        console.error(error);
    }
  }),
);

await this.batchQueue.run();

What's the best way to do this preferably without third party modules?

10 Replies

Leokuma•2w ago

Something like this (untested):

import { chunk } from "jsr:@std/collections/chunk";

for (const spinGroup of chunk(...data, 100)) {
  const spins = spinGroup.map((spin) => this.processSpin(spin));
  await Promise.all(spins);
}

import { chunk } from "jsr:@std/collections/chunk";

for (const spinGroup of chunk(...data, 100)) {
  const spins = spinGroup.map((spin) => this.processSpin(spin));
  await Promise.all(spins);
}

CodyC•2w ago

I wrote a library to do this kind of thing because I wasn't a fan of the ones I found.
https://jsr.io/@nfnitloop/better-iterators

JSR

@nfnitloop/better-iterators - JSR

@nfnitloop/better-iterators on JSR: Chainable iterators (sync and async) for TypeScript, with support for opt-in, bounded parallelism

CodyC•2w ago

So that would be something like:

const results = await 
lazy(yourItems).chunked(100).map({
    parallel: 5,
    async mapper(yourItemChunk) { ... }
})
.toArray()

const results = await 
lazy(yourItems).chunked(100).map({
    parallel: 5,
    async mapper(yourItemChunk) { ... }
})
.toArray()

beermanOP•2w ago

this is awesome

CodyC•2w ago

Thanks! If you end up using it, let me know! Feel free to add issues to request new features you mind find helpful.

beermanOP•2w ago

I migrated the project to bun due to Deno’s comparibility issues with Convex. I will probably end up using it sometime down the road though

CodyC•2w ago

Oh. Did you open an issue about convex compatibility? The Deno team seems keen to make sure node compatibility works so it’s not a blocker to adopting Deno.

beermanOP•2w ago

Many things don't work, including SvelteKit. Which is why almost every project I tried to build with Deno was inevitably migrated to Bun at one point or another. Community and adoption are also severely lagging. Deno is great but it's still a litte early. Give it another year or two.

CodyC•2w ago

Ah. Yeah I wasn’t happy with sveltekit support either. But I decided to abandon it rather than deno. 😆

beermanOP•2w ago

I struggle to see how sticking with Deno over Bun or even Node would bring you any tangible benefits—at least not on the same level as choosing Svelte over React, for example. I can’t quite get behind that decision, but to each their own.

batching tasks

Did you find this page helpful?