disintegrator Comments

disintegrator

Posts Comments

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

Or possibly introduce Deno.openAsBlob similar to node.js

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

One solution to this is that I can see is if Deno updated its FormData API to be more liberal in what it accepts. Similar to undici (node.js) and Bun.

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

Ye and there's a typescript error as a result too

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

https://github.com/node-fetch/fetch-blob/blob/main/index.js#L49 It's this Blob

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

I did try that out but the Blob it's creating is not a Deno Blob

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

What would be ideal is if Deno did some duck typing where it checks that the value has the shape:

type BlobLike = {
    [Symbol.toStringTag]: 'File' | 'Blob',
    name: 'music.mp3',
    stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}

type BlobLike = {
    [Symbol.toStringTag]: 'File' | 'Blob',
    name: 'music.mp3',
    stream: () => ReadableStream<Uint8Array>, // [NOTE 1]
}

Note 1: I believe this can be an async iterable. In Node.js stream can be () => fs.createReadStream('./sample.txt') (see: https://github.com/nodejs/undici/issues/2202#issuecomment-1664134203).

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

When I pass a fetch-blob Blob it ends up entering the else branch of that code and gets serialized as a string [object Blob] because it doesn't inherit from Deno's BlobPrototype

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

This is the exact line: https://github.com/denoland/deno/blob/76a6ea57753be420398d3eba8f313a6c98eab8c3/ext/fetch/21_formdata.js#L110

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

For sure, one sec and I'll get a look to the problematic code... and thank you so much for your patience ❤️

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

The requirements are that you know the size of the stream ahead of time. So if you dig into some of the other implementations, they compute the file size and get a streaming handle to it and that all gets represented as a File (which inherits Blob).

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

i thought that blob was the full file loaded into memory

It doesn't have to be. In fact, in the browser when you select a file to upload with <input type="file">, that is not instantly read into memory. If it's used in a FormData as part of fetch on the client-side, the browser will stream it out.

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

blob is like an abstract representation of data

That's exactly right. It's the most abstract sense of data that can be moved around.

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

https://developer.mozilla.org/en-US/docs/Web/API/Blob

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

and note that FormData.append(k, v) / FormData.set(k, v) accept string | Blob

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

The Bun docs even call this out explicitly: https://bun.sh/docs/api/file-io#reading-files-bun-file

A BunFile represents a lazily-loaded file; initializing it does not actually read the file from disk. ... The reference conforms to the Blob interface, so the contents can be read in various formats.

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

yep, not buffered into memory

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

Here's a complete example that works in Node.js v20+:

import fs from "node:fs";

async function run() {
  const file = await fs.openAsBlob("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

import fs from "node:fs";

async function run() {
  const file = await fs.openAsBlob("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

and in Bun:

async function run() {
  const file = Bun.file("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

async function run() {
  const file = Bun.file("./sample.txt");
  const fd = new FormData();
  fd.append("file", file);

  const res = await fetch("https://httpbin.org/anything/upload", {
    method: "POST",
    body: fd,
  });

  console.log(await res.json());
}

run();

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

That would work if the request body is entirely the file's contents

37 replies

DDeno

•Created by disintegrator on 12/13/2023 in #help

Streaming large files with fetch and FormData

Yep, it's meant to be doable

37 replies