nef•4mo ago

Possible to use CsvParseStream in Node.js

Hi, is it possible to use nodejs file streams with deno's stdlib?

import fs from "node:fs";
import { CsvParseStream } from "@std/csv";

fs.createReadStream("./data.csv")
  .pipe(new CsvParseStream()); // incompatible types

import fs from "node:fs";
import { CsvParseStream } from "@std/csv";

fs.createReadStream("./data.csv")
  .pipe(new CsvParseStream()); // incompatible types

13 Replies

nefOP•4mo ago

as far as I can tell there's nothing in @std/path or @std/fs that opens a file I'm sure loading to a string is possible, but the file is fairly big

marvinh.•4mo ago

There is no function to open a file in any of the @std packages, because that's already included in Deno itself, see https://docs.deno.com/examples/streaming_files/

nefOP•4mo ago

yeah sorry I'm trying to use the libraries from node.js with jsr's node compatibility I'm still interested if it's possible but for now I'm using npm's csv-parse

marvinh.•4mo ago

Right, in that case it's likely that you're running into the issue that Node by default uses NodeStreams (older API) and @std is written for WebStreams. In Node you can convert between the two, see https://nodejs.org/api/webstreams.html#nodejs-streams-interoperability

nefOP•4mo ago

oooh that looks like it, thank you! I'm looking into this again and got it working, but typescript is complaining about the types:

import fs from 'node:fs';
import { Readable } from 'node:stream';
import { CsvParseStream } from '@std/csv';

const stream = Readable
    .toWeb(fs.createReadStream('./data.csv', 'utf8'))
    .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

import fs from 'node:fs';
import { Readable } from 'node:stream';
import { CsvParseStream } from '@std/csv';

const stream = Readable
    .toWeb(fs.createReadStream('./data.csv', 'utf8'))
    .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

ts: Argument of type 'CsvParseStream<{ readonly skipFirstRow: true; }>' is not assignable to parameter of type 'ReadableWritablePair<Record<string, string> | undefined, any>'.
  Types of property 'readable' are incompatible.
    Type 'ReadableStream<Record<string, string>>' is missing the following properties from type 'ReadableStream<Record<string, string> | undefined>': values, [Symbol.asyncIterator] [2345]

ts: Argument of type 'CsvParseStream<{ readonly skipFirstRow: true; }>' is not assignable to parameter of type 'ReadableWritablePair<Record<string, string> | undefined, any>'.
  Types of property 'readable' are incompatible.
    Type 'ReadableStream<Record<string, string>>' is missing the following properties from type 'ReadableStream<Record<string, string> | undefined>': values, [Symbol.asyncIterator] [2345]

marvinh.•4mo ago

I'm not sure at the top off my head how to resolve that error, but you could also go with the Deno-native version when you're using Deno in the first place:

import { CsvParseStream } from "@std/csv";

const source = await Deno.open("./data.csv");

const stream = source.readable
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

import { CsvParseStream } from "@std/csv";

const source = await Deno.open("./data.csv");

const stream = source.readable
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

nefOP•4mo ago

I am tempted to switch from node to deno, but last time (a month ago) I had a bunch of issues with the lsp in neovim...

marvinh.•4mo ago

Oh so the issue is that you're getting the type error with plain ts and not with Deno's LSP?

nefOP•4mo ago

is there some tsconfig setting that would help?

marvinh.•4mo ago

need to check. Up until now I thought you were using Deno's LSP Looks like the types in Node are a bit wonky. It infers the stream as any with .toWeb(), so you need to manually override the type:

import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
  .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
  .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

nefOP•4mo ago

that still works, but this time the error has moved to the iteration

import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
  .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

// Type 'ReadableStream<Record<string, string>>' must have a '[Symbol.asyncIterator]()' method that returns an async iterator.
for await (const entry of stream) {
  if (!entry) continue;
  console.log(entry);
}

import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
  .pipeThrough(new CsvParseStream({ skipFirstRow: true }));

// Type 'ReadableStream<Record<string, string>>' must have a '[Symbol.asyncIterator]()' method that returns an async iterator.
for await (const entry of stream) {
  if (!entry) continue;
  console.log(entry);
}

the type inference also vanished, entry used to be Record<string, string> and now it's any

marvinh.•4mo ago

I'm afraid I can't help further. Sounds like Node's web stream types are pretty alpha

nefOP•4mo ago

they're classified as "stable" haha thank you for the help though, I'll look into moving my project to deno

Possible to use CsvParseStream in Node.js

Did you find this page helpful?