disintegrator
disintegrator7mo ago

Streaming large files with fetch and FormData

Hi, I'm trying to find a way to build multiplart/form-data requests in Deno that contain large files. Ideally, I'll like these requests and the underlying files to be streamed out instead of needing to read the whole file into memory and adding them as parts to the request. I can't seem to find a built-in API for acquiring a Blob-like file handle. I also tried using npm:fetch-blob/from.js but that isn't compatible with Deno. Compared to other platforms: Node.js v20+: the fs.openAsBlob function in node:fs can return a web File backed by the file system Bun: The Bun.file function also returns a web File backed by the file system
import { fileFrom } from "npm:fetch-blob/from.js";

async function run() {
const file = await fileFrom("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
import { fileFrom } from "npm:fetch-blob/from.js";

async function run() {
const file = await fileFrom("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
Any suggestions on how I can make this work in Deno?
23 Replies
Antonio Sampaio
Antonio Sampaio7mo ago
wait, you want to stream files via formdata?
disintegrator
disintegrator7mo ago
Yep, it's meant to be doable
Antonio Sampaio
Antonio Sampaio7mo ago
have you tried to send a readablestream? i dont know about stream and formdata compatibility, but readablestream is the way to stream file via fetch
disintegrator
disintegrator7mo ago
That would work if the request body is entirely the file's contents Here's a complete example that works in Node.js v20+:
import fs from "node:fs";

async function run() {
const file = await fs.openAsBlob("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
import fs from "node:fs";

async function run() {
const file = await fs.openAsBlob("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
and in Bun:
async function run() {
const file = Bun.file("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
async function run() {
const file = Bun.file("./sample.txt");
const fd = new FormData();
fd.append("file", file);

const res = await fetch("https://httpbin.org/anything/upload", {
method: "POST",
body: fd,
});

console.log(await res.json());
}

run();
Antonio Sampaio
Antonio Sampaio7mo ago
it sends the file as stream?
disintegrator
disintegrator7mo ago
yep, not buffered into memory The Bun docs even call this out explicitly: https://bun.sh/docs/api/file-io#reading-files-bun-file
A BunFile represents a lazily-loaded file; initializing it does not actually read the file from disk. ... The reference conforms to the Blob interface, so the contents can be read in various formats.
Antonio Sampaio
Antonio Sampaio7mo ago
thats what im searching... blob is like an abstract representation of data?
disintegrator
disintegrator7mo ago
and note that FormData.append(k, v) / FormData.set(k, v) accept string | Blob https://developer.mozilla.org/en-US/docs/Web/API/Blob
Antonio Sampaio
Antonio Sampaio7mo ago
im reading this i thought that blob was the full file loaded into memory
disintegrator
disintegrator7mo ago
blob is like an abstract representation of data
That's exactly right. It's the most abstract sense of data that can be moved around.
i thought that blob was the full file loaded into memory
It doesn't have to be. In fact, in the browser when you select a file to upload with <input type="file">, that is not instantly read into memory. If it's used in a FormData as part of fetch on the client-side, the browser will stream it out. The requirements are that you know the size of the stream ahead of time. So if you dig into some of the other implementations, they compute the file size and get a streaming handle to it and that all gets represented as a File (which inherits Blob).
Antonio Sampaio
Antonio Sampaio7mo ago
but that isn't compatible with Deno
can you tell us what goes wrong when you try to use it? im looking the code of fetch-blob
disintegrator
disintegrator7mo ago
For sure, one sec and I'll get a look to the problematic code... and thank you so much for your patience ❤️
disintegrator
disintegrator7mo ago
GitHub
deno/ext/fetch/21_formdata.js at 76a6ea57753be420398d3eba8f313a6c98...
A modern runtime for JavaScript and TypeScript. Contribute to denoland/deno development by creating an account on GitHub.
disintegrator
disintegrator7mo ago
When I pass a fetch-blob Blob it ends up entering the else branch of that code and gets serialized as a string [object Blob] because it doesn't inherit from Deno's BlobPrototype What would be ideal is if Deno did some duck typing where it checks that the value has the shape:
type BlobLike = {
[Symbol.toStringTag]: 'File' | 'Blob',
name: 'music.mp3',
stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}
type BlobLike = {
[Symbol.toStringTag]: 'File' | 'Blob',
name: 'music.mp3',
stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}
Note 1: I believe this can be an async iterable. In Node.js stream can be () => fs.createReadStream('./sample.txt') (see: https://github.com/nodejs/undici/issues/2202#issuecomment-1664134203).
Antonio Sampaio
Antonio Sampaio7mo ago
so i guess you can use blobFrom func can u try?
Antonio Sampaio
Antonio Sampaio7mo ago
No description
Antonio Sampaio
Antonio Sampaio7mo ago
No description
disintegrator
disintegrator7mo ago
I did try that out but the Blob it's creating is not a Deno Blob
Antonio Sampaio
Antonio Sampaio7mo ago
maybe something related to node compatibility things?
Antonio Sampaio
Antonio Sampaio7mo ago
aaaaa it creates its own blob, sowry
disintegrator
disintegrator7mo ago
Ye and there's a typescript error as a result too One solution to this is that I can see is if Deno updated its FormData API to be more liberal in what it accepts. Similar to undici (node.js) and Bun. Or possibly introduce Deno.openAsBlob similar to node.js
Antonio Sampaio
Antonio Sampaio7mo ago
or maybe we could replicate fetch-blob for deno haha i have to work now hope you find the solution soon glad for the knowledge about blobs ❤️