Robbie•4w ago

Replacing `fs.createWriteStream()` with Deno equivalent

Hi, I am trying to download files from S3 to disk and it works if I use fs.createWriteStream() as follows:

import * as fs from "node:fs";
import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const result = await response.Body.pipe(fs.createWriteStream("out.dat"));

import * as fs from "node:fs";
import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const result = await response.Body.pipe(fs.createWriteStream("out.dat"));

How would I achieve the same thing with Deno's standard library instead of node:fs? I have tried the following:

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const outputWriter = output.writable.getWriter();
await outputWriter.ready;
const result = await response.Body.pipe(outputWriter);

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const outputWriter = output.writable.getWriter();
await outputWriter.ready;
const result = await response.Body.pipe(outputWriter);

But I get an error:

Stack trace:
TypeError: dest.on is not a function
    at IncomingMessageForClient.Readable.pipe (ext:deno_node/_stream.mjs:2688:12)
    at <anonymous>:11:36
    at eventLoopTick (ext:core/01_core.js:175:7)

Stack trace:
TypeError: dest.on is not a function
    at IncomingMessageForClient.Readable.pipe (ext:deno_node/_stream.mjs:2688:12)
    at <anonymous>:11:36
    at eventLoopTick (ext:core/01_core.js:175:7)

Any ideas?

6 Replies

crowlKats•4w ago

deno (and the web) and node's streaming APIs are different; so first you need to convert it using one of the node functions to get a web one: https://nodejs.org/api/stream.html#streamreadabletowebstreamreadable-options then from there you can use .pipeTo on the readable with the value of the writable

RobbieOP•4w ago

Thanks. When I try:

const result = await Readable.toWeb(response.Body).pipeTo(outputWriter);

const result = await Readable.toWeb(response.Body).pipeTo(outputWriter);

I get an new error:

Stack trace:
TypeError: Failed to execute 'pipeTo' on 'ReadableStream': Argument 1 is not of type WritableStream
    at makeException (ext:deno_webidl/00_webidl.js:93:10)
    at Array.WritableStream (ext:deno_webidl/00_webidl.js:1120:13)
    at ReadableStream.pipeTo (ext:deno_web/06_streams.js:5303:39)
    at <anonymous>:13:52
    at eventLoopTick (ext:core/01_core.js:175:7)

Stack trace:
TypeError: Failed to execute 'pipeTo' on 'ReadableStream': Argument 1 is not of type WritableStream
    at makeException (ext:deno_webidl/00_webidl.js:93:10)
    at Array.WritableStream (ext:deno_webidl/00_webidl.js:1120:13)
    at ReadableStream.pipeTo (ext:deno_web/06_streams.js:5303:39)
    at <anonymous>:13:52
    at eventLoopTick (ext:core/01_core.js:175:7)

I get the same error with:

const result = await response.Body.transformToWebStream().pipeTo(outputWriter);

const result = await response.Body.transformToWebStream().pipeTo(outputWriter);

crowlKats•4w ago

you want to pass the writable, not the writer (yes i know, very confusing nomenclature), so in your case output.writable

RobbieOP•4w ago

That works! Thanks 🙏

Fifth-Normal-Form•4w ago

Naming is hard 😣

RobbieOP•4w ago

If anyone is interested, this is the final working code:

import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const result = await response.Body.transformToWebStream().pipeTo(output.writable);

import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const result = await response.Body.transformToWebStream().pipeTo(output.writable);