disintegrator•13mo ago

Streaming large files with fetch and FormData

Hi, I'm trying to find a way to build multiplart/form-data requests in Deno that contain large files. Ideally, I'll like these requests and the underlying files to be streamed out instead of needing to read the whole file into memory and adding them as parts to the request. I can't seem to find a built-in API for acquiring a Blob-like file handle. I also tried using npm:fetch-blob/from.js but that isn't compatible with Deno. Compared to other platforms: Node.js v20+: the fs.openAsBlob function in node:fs can return a web File backed by the file system Bun: The Bun.file function also returns a web File backed by the file system

import { fileFrom } from "npm:fetch-blob/from.js";

async function run() {
  const file = await fileFrom("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

import { fileFrom } from "npm:fetch-blob/from.js";

async function run() {
  const file = await fileFrom("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

Any suggestions on how I can make this work in Deno?

23 Replies

Antonio Sampaio•13mo ago

wait, you want to stream files via formdata?

disintegratorOP•13mo ago

Yep, it's meant to be doable

Antonio Sampaio•13mo ago

have you tried to send a readablestream? i dont know about stream and formdata compatibility, but readablestream is the way to stream file via fetch

disintegratorOP•13mo ago

That would work if the request body is entirely the file's contents Here's a complete example that works in Node.js v20+:

import fs from "node:fs";

async function run() {
  const file = await fs.openAsBlob("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

import fs from "node:fs";

async function run() {
  const file = await fs.openAsBlob("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

and in Bun:

async function run() {
  const file = Bun.file("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

async function run() {
  const file = Bun.file("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

Antonio Sampaio•13mo ago

it sends the file as stream?

disintegratorOP•13mo ago

yep, not buffered into memory The Bun docs even call this out explicitly: https://bun.sh/docs/api/file-io#reading-files-bun-file

A BunFile represents a lazily-loaded file; initializing it does not actually read the file from disk. ... The reference conforms to the Blob interface, so the contents can be read in various formats.

Antonio Sampaio•13mo ago

thats what im searching... blob is like an abstract representation of data?

disintegratorOP•13mo ago

and note that FormData.append(k, v) / FormData.set(k, v) accept string | Blob https://developer.mozilla.org/en-US/docs/Web/API/Blob

Antonio Sampaio•13mo ago

im reading this i thought that blob was the full file loaded into memory

disintegratorOP•13mo ago

blob is like an abstract representation of data

That's exactly right. It's the most abstract sense of data that can be moved around.

i thought that blob was the full file loaded into memory

It doesn't have to be. In fact, in the browser when you select a file to upload with <input type="file">, that is not instantly read into memory. If it's used in a FormData as part of fetch on the client-side, the browser will stream it out. The requirements are that you know the size of the stream ahead of time. So if you dig into some of the other implementations, they compute the file size and get a streaming handle to it and that all gets represented as a File (which inherits Blob).

Antonio Sampaio•13mo ago

but that isn't compatible with Deno

can you tell us what goes wrong when you try to use it? im looking the code of fetch-blob

disintegratorOP•13mo ago

For sure, one sec and I'll get a look to the problematic code... and thank you so much for your patience ❤️

disintegratorOP•13mo ago

This is the exact line: https://github.com/denoland/deno/blob/76a6ea57753be420398d3eba8f313a6c98eab8c3/ext/fetch/21_formdata.js#L110

GitHub

deno/ext/fetch/21_formdata.js at 76a6ea57753be420398d3eba8f313a6c98...

A modern runtime for JavaScript and TypeScript. Contribute to denoland/deno development by creating an account on GitHub.

disintegratorOP•13mo ago

When I pass a fetch-blob Blob it ends up entering the else branch of that code and gets serialized as a string [object Blob] because it doesn't inherit from Deno's BlobPrototype What would be ideal is if Deno did some duck typing where it checks that the value has the shape:

type BlobLike = {
    [Symbol.toStringTag]: 'File' | 'Blob',
    name: 'music.mp3',
    stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}

type BlobLike = {
    [Symbol.toStringTag]: 'File' | 'Blob',
    name: 'music.mp3',
    stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}

Note 1: I believe this can be an async iterable. In Node.js stream can be () => fs.createReadStream('./sample.txt') (see: https://github.com/nodejs/undici/issues/2202#issuecomment-1664134203).

Antonio Sampaio•13mo ago

so i guess you can use blobFrom func can u try?

Antonio Sampaio•13mo ago

Antonio Sampaio•13mo ago

disintegratorOP•13mo ago

I did try that out but the Blob it's creating is not a Deno Blob

Antonio Sampaio•13mo ago

maybe something related to node compatibility things?

disintegratorOP•13mo ago

https://github.com/node-fetch/fetch-blob/blob/main/index.js#L49 It's this Blob

Antonio Sampaio•13mo ago

aaaaa it creates its own blob, sowry

disintegratorOP•13mo ago

Ye and there's a typescript error as a result too One solution to this is that I can see is if Deno updated its FormData API to be more liberal in what it accepts. Similar to undici (node.js) and Bun. Or possibly introduce Deno.openAsBlob similar to node.js

Antonio Sampaio•13mo ago

or maybe we could replicate fetch-blob for deno haha i have to work now hope you find the solution soon glad for the knowledge about blobs ❤️