LFCavalcanti Posts

LFCavalcanti

•Created by LFCavalcanti on 3/19/2024 in #help

Is there a way to read big files using something like .seek but with ending position as well?

So, I'm doing a version of the billion line challenge in JS. I've already implemented a version with Node basically calculating offsets to "slice" the file in a amount og segments matching the available parallelism in the system. To achieve that I'm calling Node's "fs" library like so:

const fileReadStream = fs.createReadStream(workerData.filePath, {
  start: workerData.start,
  end: workerData.end,
});

const fileReadStream = fs.createReadStream(workerData.filePath, {
  start: workerData.start,
  end: workerData.end,
});

I'm now trying to do the same in Deno, both for the challenge and learning Deno. In the docs there's a way to use the file.seek(offset, whence) that allows for a offset that can be either from the end, start or from current. I'm not sure how to use the same logic having different threads/workers reading from the same files in different byte ranges.

36 replies