Zidan•11mo ago

Serving multiple static files to an HTTP request

If a client sends a request for an array of static files names like so.

["component01.js", "component01.css", "component02.js", "component02.css"]

["component01.js", "component01.css", "component02.js", "component02.css"]

Using a Deno.serve server, is it possible to serve all files in response to that one request, instead of having the client request them one by one? Would I need something like HTTP/2 or HTTP/3? And if so, are any of those supported in Deno?

28 Replies

pyrote•11mo ago

What is your use case? Browser to server or API client to server? Is returning a ZIP file with all the requested files an option?

guest271314•11mo ago

One way to do that is serve the files as a single JSON response.

Zidan•11mo ago

I don't know how to do that. Can you illustrate or link an example? My use case: I'm building an SPA. and have JS files for UI components and their corresponding CSS files. A function that renders a form needs 4 UI components for different field, buttons and so on. The form rendering function checks if those 4 UI components are available before attempting to use them. If not found, it requests them with an import statement one by one, then requests the corresponding CSS files one by one too. A ZIP file with all the requested files sounds good. But I don't know how to do that. Can you illustrate or link an example?

cyracrimes•11mo ago

It sounds like you're already doing the right thing. Putting the files into some kind of zip/json package wouldn't work well here because you'd need to pile on a couple of additional hacks to then still be able to actually import the scripts/stylesheets. Importing the resources individually isn't necessarily harmful at all - is your question motivated by a measured performance issue? Have you maybe looked into bundling your application in a build step using something like webpack?

Zidan•11mo ago

Not a performance issue, no. It's about efficiency really. A request and a response per file sounds very wasteful to me. So I wondered if I can send multiple responses per request if the request already has an array of the needed files. @pyrote mentioned a zip file with all the requested files. So that would be 1 request and 1 response per several files, which sounded pretty good to me 😄. @guest271314 mentioned a single JSON response with the files. No idea how that works 😕.

cyracrimes•11mo ago

Both ideas won't work easily because you want the browser to load the files as scripts and stylesheets. They would be potential solutions if the files were just used programmatically by your app, but you're importing them, which is different. You also can't send multiple "responses" per request as far as HTTP is concerned. But: Browsers are good at doing requests. Importing multiple things at the same time isn't generally less efficient than importing only one thing, unless your logic blocks the loading of some of the resources until others are loaded

Zidan•11mo ago

I see.. Well, I'm using Promise.all() at the moment. That shouldn't be blocking so I'll just keep using that. Thanks for the clarification.

cyracrimes•11mo ago

That sounds fine to me :) If you run into something like this again where you really do want to avoid many individual imports, especially if they're nested, I'd encourage you to look into module bundling with tools like webpack

guest271314•11mo ago

Simple {"0":[0, 255...], "1":[0, 255...]}. See https://plnkr.co/plunk/gCjYSt. Since Deno supports full duplex streaming you can stream all the files or whatever else you want using a single fetch() request, e.g., https://github.com/guest271314/native-messaging-deno/tree/fetch-duplex.

Josh•11mo ago

You could try opening a WebSocket, sending each file at a time and have the user and server notify each other when a file is received / sent, may be too much overhead though

guest271314•11mo ago

Technically you can just serve the entire combination of N files as a single Uint8Array as long as you send the offsets, too. Here I combine a Uint32Array that includes the length of the following JSON configuration, which can optionally include images, album, artist, track information to display with Media Session API in global media controls, and offsets of the following discrete ArrayBuffer s which are Opus encoded audio output by WebCodecs AudioEncoder into a single file, where I extract the images and other media information and offsets of the discrete media chunks, and play back the file in the browser https://github.com/guest271314/WebCodecsOpusRecorder, largely inspired by the Native Messaging protocol which encodes the length of a message then the message. Bonus: The resulting file is less size than Opus audio encoded in WebM container. Alternatively, include multiple files in a .tar.gz or .zip file then extract the files from archive using Compression Streams, e.g., here I extract the node executable from the Node.js nightly release and get rid of everything else - in the browser https://github.com/guest271314/download-node-nightly-executable. This is not a particularly difficult case to solve. Just choose your approach and stick to that on the fron-end and back-end.

guest271314•11mo ago

That will work. Create the script elements and set textContent. With CSSOM we can set CSS Rules directly in a stylesheet, see Modify element :before CSS rules programmatically in React and Empty style element with working CSS rules?.

Stack Overflow

Modify element :before CSS rules programmatically in React

I have an element whose :before style has to be modified based on calculations. export class Content extends React.Component { render() { return ( <div className="ring-b...

cyracrimes•11mo ago

I fully agree that you can do this, I just think it's an unnecessary hack in this case. In fairness of course, much of the technology that makes the modern web work could be described this way :P Well, I guess it's often necessary hacks. I don't think this particular thing is worth doing in this particular case

guest271314•11mo ago

Somebody said in some article Emscripten started off as a hack. Now there's WebAssembly and Bytecode Alliance. It's not really a hack. No more than YouTube using Media Source Extensions to serve multiple files for media playback. Again, at the extreme side there are Web Bundles https://github.com/GoogleChromeLabs/webbundle-plugins/blob/main/packages/webbundle-webpack-plugin/README.md, though Crromium/Chrome did remove support for navigation to Web Bundles. Choose your approach; JSON, Streams, whatever. They each work, as I demostrate in the linked working examples. Nothing is stopping you from encoding you data as JSON then using a single import assertion to import all of the files at once. Here's another example of piping multiple files to the same stream https://github.com/guest271314/AudioWorkletStream, something like

let port;
onmessage = async e => {
  'use strict';
  if (!port) {
    [port] = e.ports;
    port.onmessage = event => postMessage(event.data);
  }
  const { urls } = e.data;
  // https://github.com/whatwg/streams/blob/master/transferable-streams-explainer.md
  const { readable, writable } = new TransformStream();
  (async _ => {
    for await (const _ of (async function* stream() {
      while (urls.length) {
        yield (await fetch(urls.shift(), {cache: 'no-store'})).body.pipeTo(writable, {
          preventClose: !!urls.length,
        });
      }
    })());
  })();
  port.postMessage(
    {
      readable,
    },
    [readable]
  );
};

let port;
onmessage = async e => {
  'use strict';
  if (!port) {
    [port] = e.ports;
    port.onmessage = event => postMessage(event.data);
  }
  const { urls } = e.data;
  // https://github.com/whatwg/streams/blob/master/transferable-streams-explainer.md
  const { readable, writable } = new TransformStream();
  (async _ => {
    for await (const _ of (async function* stream() {
      while (urls.length) {
        yield (await fetch(urls.shift(), {cache: 'no-store'})).body.pipeTo(writable, {
          preventClose: !!urls.length,
        });
      }
    })());
  })();
  port.postMessage(
    {
      readable,
    },
    [readable]
  );
};

guest271314•11mo ago

Just for completeness here is an example of using signed Web Bundles in an Isolated Web App - with a Deno and txiki.js TCP server https://github.com/GoogleChromeLabs/telnet-client/pull/18. For some reason Direct Sockets have been gated behind Progressive Web Apps and now Isolated Web Apps. Somebody thinks that leads to greater "security", however I have already incorporated browser extension code into the signed Web Bundle which provides a means to communicate between an arbitrary Web page and the IWA for the purpose of controlling a TCPSocket() connection from any arbitrary Web site I choose.

GitHub

Build software better, together

GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 330 million projects.

guest271314•11mo ago

Note, for standardized approach you can serve a FormData object from the server and just read the data as File objects in the browser/client, see https://stackoverflow.com/a/47067787.

guest271314•11mo ago

Here you go https://plnkr.co/edit/jluHMgfqdPD1z6Y1?open=lib%2Fscript.js Server

onfetch = (e) => {
  if (e.request.url.includes('files')) {
    const fd = new FormData();
    fd.append(
      '0',
      new Blob([`console.log('script 0')`], { type: 'text/javascript' }),
      '0.js'
    );

    fd.append(
      '1',
      new Blob([`console.log('script 1')`], { type: 'text/javascript' }),
      '1.js'
    );

    fd.append(
      '2',
      new Blob([`body {color:blue}`], { type: 'text/css' }),
      '2.css'
    );
    let client = clients.get(e.clientId);
    // console.log(e.request.headers.get('content-type'));
    e.respondWith(
      new Response(fd, {
        headers: {},
      })
    );
  }
};

onfetch = (e) => {
  if (e.request.url.includes('files')) {
    const fd = new FormData();
    fd.append(
      '0',
      new Blob([`console.log('script 0')`], { type: 'text/javascript' }),
      '0.js'
    );

    fd.append(
      '1',
      new Blob([`console.log('script 1')`], { type: 'text/javascript' }),
      '1.js'
    );

    fd.append(
      '2',
      new Blob([`body {color:blue}`], { type: 'text/css' }),
      '2.css'
    );
    let client = clients.get(e.clientId);
    // console.log(e.request.headers.get('content-type'));
    e.respondWith(
      new Response(fd, {
        headers: {},
      })
    );
  }
};

Browser

      async function processFormData() {
        fetch('./?files')
          .then(async (r) => {
            console.log(r.headers.get('content-type'));
            /*
            return new Response(await r.text(), {
              headers: { 'content-type': r.headers.get('content-type') },
            })
            */
            return r.formData();
          })
          .then(async (fd) => {
            console.log(fd);
            for (const [, file] of fd) {
              switch (file.type) {
                case 'text/javascript':
                  const script = document.createElement('script');
                  script.textContent = await file.text();
                  document.body.appendChild(script);
                  break;
                case 'text/css':
                  const style = document.createElement('style');
                  style.textContent = await file.text();
                  document.head.appendChild(style);
                  break;
                default:
                  console.log(file.type);
              }
            }
          })
          .catch(console.warn);
      }

      async function processFormData() {
        fetch('./?files')
          .then(async (r) => {
            console.log(r.headers.get('content-type'));
            /*
            return new Response(await r.text(), {
              headers: { 'content-type': r.headers.get('content-type') },
            })
            */
            return r.formData();
          })
          .then(async (fd) => {
            console.log(fd);
            for (const [, file] of fd) {
              switch (file.type) {
                case 'text/javascript':
                  const script = document.createElement('script');
                  script.textContent = await file.text();
                  document.body.appendChild(script);
                  break;
                case 'text/css':
                  const style = document.createElement('style');
                  style.textContent = await file.text();
                  document.head.appendChild(style);
                  break;
                default:
                  console.log(file.type);
              }
            }
          })
          .catch(console.warn);
      }

guest271314

Plunker

Serve and process multiple files in FormData

Created on https://plnkr.co: Helping you build the web.

Leokuma•11mo ago

The browser will download the files only once in a lifetime and then cache them for future access. Some of the solutions proposed here will prevent the browser from caching them, forcing a redownload of all the files everytime your SPA is opened. Also I believe the cost of downloading and processing a Zip file is higher than downloading the files separately compressed as gzip or Brotli which are web standards

Zidan•11mo ago

Wow, hack/worth or not, I did not know about any of the approachs you mentioned. Thanks a lot. Now, I did some reading on FormData and Blobs. And wondering if it's possible to use the JS/CSS files from a FormData Blob without creating/appending script tags like when using import? Or is this the only way for that approach? Like auto caching? Man, do I have a lot of reading to do xD Thanks for the heads up

Leokuma•11mo ago

Yes browsers cache files automatically, so only the first time the user accesses your SPA the files will be actually downloaded. On the following times when you do import() the browser will retrieve from cache

Zidan•11mo ago

In my novice mind. A user sends a sign in request with their credentials, if successful, I would respond with body like this:

{ user: { id: '', name: '', phone: '' }, files: [ file/blob, file/blob ] }

{ user: { id: '', name: '', phone: '' }, files: [ file/blob, file/blob ] }

and the files would work without the need to create and append script elements like when done using import. Instead of responding with just user data + files names so that a pre existing function can import them one by one.

guest271314•11mo ago

Now, I did some reading on FormData and Blobs. And wondering if it's possible to use the JS/CSS files from a FormData Blob without creating/appending script tags like when using import? Or is this the only way for that approach?

Technically you can create a Blob URL and use import(). What is the use case?

Zidan•11mo ago

I have divided my SPA's UI into what I named components, modules and events. A component is function that returns an HTML element. A module is function that uses components to build pages, forms, popups Etc. An event is a function that is called by a component. All of the above is loaded on demand only, so each module function has an array for component dependencies that are checked for availability (are X, Y, Z already loaded?) before the module attempts to use them. And if they are available, the availability function (which returns a promise) just fulfills. And if one or more of the component dependencies are missing, it runs a Promise.all() on an array of import() statements to request the component files and their corresponding CSS files one by one, then fulfills or rejects accordingly. And this is working just fine for now, but I wanted to see if I can request all files at once instead of one by one. So I wanted to try HTTP/2 server push in Deno but then found that its no longer supported. Then you suggested using the FormData object which would allow me to send a list of files like I wanted to achieve, expect that it involves creating and appending <script> and <link> HTML tags to the DOM for each file, which I would like to avoid.

AapoAlas•11mo ago

Web bundles is also a thing that was promising this exact feature but it was apparently sort of abandoned this year.

AapoAlas•11mo ago

https://developer.chrome.com/docs/web-platform/web-bundles/

Chrome Developers

Get started with Web Bundles - Chrome Developers

Web Bundles enable you to share websites as a single file over Bluetooth and run them offline in your origin's context.

guest271314•11mo ago

Web Bundles was not abandoned.

AapoAlas•11mo ago

I was going to link the Chromium bug where they say this:

The code for navigation to Web Bundles has been behind a flag for a long time, but we have no plans to ship it.

but indeed I'd misunderstood, it's not the full Web Bundles but just "navigation to Web Bundles", using something like file:///home/user/web_bundle to load a page.

Our focus on WebBundle has shifted to subresource loading and Isolated Web Apps use cases.

So yeah, still going but with a somewhat limited scope. My bad.

guest271314•11mo ago

Chromium did remove the ability to navigate directly to a (Signed) Web Bundle. It is still possible to create a (Signed) Web Bundle including whatever files you want https://github.com/GoogleChromeLabs/telnet-client/issues/25 (although I don't think it is possible to register a ServiceWorker in isolated-app: protocol; at least I have not been able to do so using the Webpack and Web Bundle plugins here https://github.com/GoogleChromeLabs/telnet-client/).

GitHub

FYI: How to connect to a TCP server from an arbitrary Web page in t...

https://gist.github.com/guest271314/ac9e26f71f5b299baa1b4663002ba9e6 Using only Web API's https://github.com/guest271314/telnet-client/tree/user-defined-tcpsocket-controller-web-api Using brows...