nef
nef4w ago

Possible to use CsvParseStream in Node.js

Hi, is it possible to use nodejs file streams with deno's stdlib?
import fs from "node:fs";
import { CsvParseStream } from "@std/csv";

fs.createReadStream("./data.csv")
.pipe(new CsvParseStream()); // incompatible types
import fs from "node:fs";
import { CsvParseStream } from "@std/csv";

fs.createReadStream("./data.csv")
.pipe(new CsvParseStream()); // incompatible types
13 Replies
nef
nefOP4w ago
as far as I can tell there's nothing in @std/path or @std/fs that opens a file I'm sure loading to a string is possible, but the file is fairly big
marvinh.
marvinh.4w ago
There is no function to open a file in any of the @std packages, because that's already included in Deno itself, see https://docs.deno.com/examples/streaming_files/
nef
nefOP4w ago
yeah sorry I'm trying to use the libraries from node.js with jsr's node compatibility I'm still interested if it's possible but for now I'm using npm's csv-parse
marvinh.
marvinh.4w ago
Right, in that case it's likely that you're running into the issue that Node by default uses NodeStreams (older API) and @std is written for WebStreams. In Node you can convert between the two, see https://nodejs.org/api/webstreams.html#nodejs-streams-interoperability
nef
nefOP2w ago
oooh that looks like it, thank you! I'm looking into this again and got it working, but typescript is complaining about the types:
import fs from 'node:fs';
import { Readable } from 'node:stream';
import { CsvParseStream } from '@std/csv';

const stream = Readable
.toWeb(fs.createReadStream('./data.csv', 'utf8'))
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));
import fs from 'node:fs';
import { Readable } from 'node:stream';
import { CsvParseStream } from '@std/csv';

const stream = Readable
.toWeb(fs.createReadStream('./data.csv', 'utf8'))
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));
ts: Argument of type 'CsvParseStream<{ readonly skipFirstRow: true; }>' is not assignable to parameter of type 'ReadableWritablePair<Record<string, string> | undefined, any>'.
Types of property 'readable' are incompatible.
Type 'ReadableStream<Record<string, string>>' is missing the following properties from type 'ReadableStream<Record<string, string> | undefined>': values, [Symbol.asyncIterator] [2345]
ts: Argument of type 'CsvParseStream<{ readonly skipFirstRow: true; }>' is not assignable to parameter of type 'ReadableWritablePair<Record<string, string> | undefined, any>'.
Types of property 'readable' are incompatible.
Type 'ReadableStream<Record<string, string>>' is missing the following properties from type 'ReadableStream<Record<string, string> | undefined>': values, [Symbol.asyncIterator] [2345]
marvinh.
marvinh.2w ago
I'm not sure at the top off my head how to resolve that error, but you could also go with the Deno-native version when you're using Deno in the first place:
import { CsvParseStream } from "@std/csv";

const source = await Deno.open("./data.csv");

const stream = source.readable
.pipeThrough(new TextDecoderStream())
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));
import { CsvParseStream } from "@std/csv";

const source = await Deno.open("./data.csv");

const stream = source.readable
.pipeThrough(new TextDecoderStream())
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));
nef
nefOP2w ago
I am tempted to switch from node to deno, but last time (a month ago) I had a bunch of issues with the lsp in neovim...
marvinh.
marvinh.2w ago
Oh so the issue is that you're getting the type error with plain ts and not with Deno's LSP?
nef
nefOP2w ago
is there some tsconfig setting that would help?
marvinh.
marvinh.2w ago
need to check. Up until now I thought you were using Deno's LSP Looks like the types in Node are a bit wonky. It infers the stream as any with .toWeb(), so you need to manually override the type:
import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));
import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));
nef
nefOP2w ago
that still works, but this time the error has moved to the iteration
import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));

// Type 'ReadableStream<Record<string, string>>' must have a '[Symbol.asyncIterator]()' method that returns an async iterator.
for await (const entry of stream) {
if (!entry) continue;
console.log(entry);
}
import fs from "node:fs";
import { Readable } from "node:stream";
import { CsvParseStream } from "@std/csv";

const source = Readable.toWeb(fs.createReadStream("./data.csv", "utf8"));
const stream = (source as ReadableStream<string>)
.pipeThrough(new CsvParseStream({ skipFirstRow: true }));

// Type 'ReadableStream<Record<string, string>>' must have a '[Symbol.asyncIterator]()' method that returns an async iterator.
for await (const entry of stream) {
if (!entry) continue;
console.log(entry);
}
the type inference also vanished, entry used to be Record<string, string> and now it's any
marvinh.
marvinh.2w ago
I'm afraid I can't help further. Sounds like Node's web stream types are pretty alpha
nef
nefOP2w ago
they're classified as "stable" haha thank you for the help though, I'll look into moving my project to deno

Did you find this page helpful?