Robbie
Robbie3w ago

Replacing `fs.createWriteStream()` with Deno equivalent

Hi, I am trying to download files from S3 to disk and it works if I use fs.createWriteStream() as follows:
import * as fs from "node:fs";
import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const result = await response.Body.pipe(fs.createWriteStream("out.dat"));
import * as fs from "node:fs";
import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const result = await response.Body.pipe(fs.createWriteStream("out.dat"));
How would I achieve the same thing with Deno's standard library instead of node:fs? I have tried the following:
const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const outputWriter = output.writable.getWriter();
await outputWriter.ready;
const result = await response.Body.pipe(outputWriter);
const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const outputWriter = output.writable.getWriter();
await outputWriter.ready;
const result = await response.Body.pipe(outputWriter);
But I get an error:
Stack trace:
TypeError: dest.on is not a function
at IncomingMessageForClient.Readable.pipe (ext:deno_node/_stream.mjs:2688:12)
at <anonymous>:11:36
at eventLoopTick (ext:core/01_core.js:175:7)
Stack trace:
TypeError: dest.on is not a function
at IncomingMessageForClient.Readable.pipe (ext:deno_node/_stream.mjs:2688:12)
at <anonymous>:11:36
at eventLoopTick (ext:core/01_core.js:175:7)
Any ideas?
6 Replies
crowlKats
crowlKats3w ago
deno (and the web) and node's streaming APIs are different; so first you need to convert it using one of the node functions to get a web one: https://nodejs.org/api/stream.html#streamreadabletowebstreamreadable-options then from there you can use .pipeTo on the readable with the value of the writable
Robbie
RobbieOP3w ago
Thanks. When I try:
const result = await Readable.toWeb(response.Body).pipeTo(outputWriter);
const result = await Readable.toWeb(response.Body).pipeTo(outputWriter);
I get an new error:
Stack trace:
TypeError: Failed to execute 'pipeTo' on 'ReadableStream': Argument 1 is not of type WritableStream
at makeException (ext:deno_webidl/00_webidl.js:93:10)
at Array.WritableStream (ext:deno_webidl/00_webidl.js:1120:13)
at ReadableStream.pipeTo (ext:deno_web/06_streams.js:5303:39)
at <anonymous>:13:52
at eventLoopTick (ext:core/01_core.js:175:7)
Stack trace:
TypeError: Failed to execute 'pipeTo' on 'ReadableStream': Argument 1 is not of type WritableStream
at makeException (ext:deno_webidl/00_webidl.js:93:10)
at Array.WritableStream (ext:deno_webidl/00_webidl.js:1120:13)
at ReadableStream.pipeTo (ext:deno_web/06_streams.js:5303:39)
at <anonymous>:13:52
at eventLoopTick (ext:core/01_core.js:175:7)
I get the same error with:
const result = await response.Body.transformToWebStream().pipeTo(outputWriter);
const result = await response.Body.transformToWebStream().pipeTo(outputWriter);
crowlKats
crowlKats3w ago
you want to pass the writable, not the writer (yes i know, very confusing nomenclature), so in your case output.writable
Robbie
RobbieOP3w ago
That works! Thanks 🙏
Fifth-Normal-Form always
Naming is hard 😣
Robbie
RobbieOP3w ago
If anyone is interested, this is the final working code:
import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const result = await response.Body.transformToWebStream().pipeTo(output.writable);
import { S3 } from "npm:@aws-sdk/client-s3@3.701.0";

const s3 = new S3();
const response = await s3.getObject({ Bucket: "mybucket", Key: "mykey" });
const output = await Deno.open("out.dat", { write: true, create: true });
const result = await response.Body.transformToWebStream().pipeTo(output.writable);