D
Deno

help

Streaming large files with fetch and FormData

Ddisintegrator12/13/2023
Hi, I'm trying to find a way to build multiplart/form-data requests in Deno that contain large files. Ideally, I'll like these requests and the underlying files to be streamed out instead of needing to read the whole file into memory and adding them as parts to the request. I can't seem to find a built-in API for acquiring a Blob-like file handle. I also tried using npm:fetch-blob/from.js but that isn't compatible with Deno. Compared to other platforms: Node.js v20+: the fs.openAsBlob function in node:fs can return a web File backed by the file system Bun: The Bun.file function also returns a web File backed by the file system
import { fileFrom } from "npm:fetch-blob/from.js";

async function run() {
const file = await fileFrom("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
import { fileFrom } from "npm:fetch-blob/from.js";

async function run() {
const file = await fileFrom("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
Any suggestions on how I can make this work in Deno?
ASAntonio Sampaio12/13/2023
wait, you want to stream files via formdata?
Ddisintegrator12/13/2023
Yep, it's meant to be doable
ASAntonio Sampaio12/13/2023
have you tried to send a readablestream? i dont know about stream and formdata compatibility, but readablestream is the way to stream file via fetch
Ddisintegrator12/13/2023
That would work if the request body is entirely the file's contents Here's a complete example that works in Node.js v20+:
import fs from "node:fs";

async function run() {
const file = await fs.openAsBlob("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
import fs from "node:fs";

async function run() {
const file = await fs.openAsBlob("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
and in Bun:
async function run() {
const file = Bun.file("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
async function run() {
const file = Bun.file("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
ASAntonio Sampaio12/13/2023
it sends the file as stream?
Ddisintegrator12/13/2023
yep, not buffered into memory The Bun docs even call this out explicitly: https://bun.sh/docs/api/file-io#reading-files-bun-file
A BunFile represents a lazily-loaded file; initializing it does not actually read the file from disk. ... The reference conforms to the Blob interface, so the contents can be read in various formats.
ASAntonio Sampaio12/13/2023
thats what im searching... blob is like an abstract representation of data?
Ddisintegrator12/13/2023
and note that FormData.append(k, v) / FormData.set(k, v) accept string | Blob https://developer.mozilla.org/en-US/docs/Web/API/Blob
ASAntonio Sampaio12/13/2023
im reading this i thought that blob was the full file loaded into memory
Ddisintegrator12/13/2023
blob is like an abstract representation of data
That's exactly right. It's the most abstract sense of data that can be moved around.
i thought that blob was the full file loaded into memory
It doesn't have to be. In fact, in the browser when you select a file to upload with <input type="file">, that is not instantly read into memory. If it's used in a FormData as part of fetch on the client-side, the browser will stream it out. The requirements are that you know the size of the stream ahead of time. So if you dig into some of the other implementations, they compute the file size and get a streaming handle to it and that all gets represented as a File (which inherits Blob).
ASAntonio Sampaio12/13/2023
but that isn't compatible with Deno
can you tell us what goes wrong when you try to use it? im looking the code of fetch-blob
Ddisintegrator12/13/2023
For sure, one sec and I'll get a look to the problematic code... and thank you so much for your patience ❤️
Ddisintegrator12/13/2023
GitHub
deno/ext/fetch/21_formdata.js at 76a6ea57753be420398d3eba8f313a6c98...
A modern runtime for JavaScript and TypeScript. Contribute to denoland/deno development by creating an account on GitHub.
Ddisintegrator12/13/2023
When I pass a fetch-blob Blob it ends up entering the else branch of that code and gets serialized as a string [object Blob] because it doesn't inherit from Deno's BlobPrototype What would be ideal is if Deno did some duck typing where it checks that the value has the shape:
type BlobLike = {
[Symbol.toStringTag]: 'File' | 'Blob',
name: 'music.mp3',
stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}
type BlobLike = {
[Symbol.toStringTag]: 'File' | 'Blob',
name: 'music.mp3',
stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}
Note 1: I believe this can be an async iterable. In Node.js stream can be () => fs.createReadStream('./sample.txt') (see: https://github.com/nodejs/undici/issues/2202#issuecomment-1664134203).
ASAntonio Sampaio12/13/2023
so i guess you can use blobFrom func can u try?
ASAntonio Sampaio12/13/2023
No description
ASAntonio Sampaio12/13/2023
No description
Ddisintegrator12/13/2023
I did try that out but the Blob it's creating is not a Deno Blob
ASAntonio Sampaio12/13/2023
maybe something related to node compatibility things?
ASAntonio Sampaio12/13/2023
aaaaa it creates its own blob, sowry
Ddisintegrator12/13/2023
Ye and there's a typescript error as a result too One solution to this is that I can see is if Deno updated its FormData API to be more liberal in what it accepts. Similar to undici (node.js) and Bun. Or possibly introduce Deno.openAsBlob similar to node.js
ASAntonio Sampaio12/13/2023
or maybe we could replicate fetch-blob for deno haha i have to work now hope you find the solution soon glad for the knowledge about blobs ❤️

Looking for more? Join the community!

Recommended Posts
If I want to integrate a fully featured Deno runtime into my application, where do I start?I want to create a scripting environment for my application that will enable people to create customTrying to deploy a project on Deno Deploy with a private Github repository dependencyBy reading the docs at https://docs.deno.com/runtime/manual/basics/modules/private I found that I caHow can I make sure the code gets run on the client-side??!```tsx import { useEffect, useState } from "preact/hooks"; export default function Callback() { VSCode + Deno + Vite + TypescriptJust setup a project with `create-vite-extra` but all the starter files are full of typescript errorMemory LimitHello guys, in some situations i'am receiving one of the following error message `dd-hard-memory-limGoogle CalendarHi! I am looking for a minimal example how to access google calendar API. I already obtained credenI can't access my deno cloud github accountPlease help, by mistake I converted my personal github account to an organization account, and now IUploaded a module, need help!I uploaded my module: https://deno.land/x/vipps_mobilepay_sdk@0.0.1 But then I realised that I shoulBetter way of finding element from mapIm making an api and all routes are stored in a map. is there a better and more optimised way to loodeno info depthhow do i limit maximum output depth for `deno info` command? for example, it's easier to analyze verDate libraryCan anyone recommend a decent working date library for deno? All the ones I'm trying (datefns, dayjsawait writer.write(buffer) never resolves on Deno.CommandHiho. I'm creating a `Deno.Command` that writes a buffer to `vipsthumbnail` via stdin (and reads theNot sure if I understand kv watch right?Not sure if I understand kv watch right, is like named queues? o more like a reliable Broadcast chanDisplaying a ModalI am trying to get a minimal example of using Bootstrap Modals, and managing that modal using state.How do I run an npm script with multiple dependencies, e.g. `drizzle-kit generate:pg`?I thought about wrapping it into a task `"generate": "deno run -A npm:drizzle-kit generate:pg"` in tHow to use a class name as a qualifier for an enum ?Hi there, it's more of a generic typescript question but I'm using it in the Deno context and the lihow can I make him see the process?```ts error: Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'on') proImporting NPM Packages in Deno Throws 'Uncaught SyntaxError' for Specific ModulesHi there, I'm relatively new to Deno. I'm running into an error with an NPM package import. ```JS imHow to override type definitions provided by a third party module@Deno AI Helper How do I override type definitions provided by a third party moduleSync Child Stdin ReadI'd like to make a JS wrapper around the repl of another language. For example: ```js let a = other