Zidan
Zidan•16mo ago

Serving multiple static files to an HTTP request

If a client sends a request for an array of static files names like so.
["component01.js", "component01.css", "component02.js", "component02.css"]
["component01.js", "component01.css", "component02.js", "component02.css"]
Using a Deno.serve server, is it possible to serve all files in response to that one request, instead of having the client request them one by one? Would I need something like HTTP/2 or HTTP/3? And if so, are any of those supported in Deno?
28 Replies
pyrote
pyrote•16mo ago
What is your use case? Browser to server or API client to server? Is returning a ZIP file with all the requested files an option?
guest271314
guest271314•16mo ago
One way to do that is serve the files as a single JSON response.
Zidan
ZidanOP•16mo ago
I don't know how to do that. Can you illustrate or link an example? My use case: I'm building an SPA. and have JS files for UI components and their corresponding CSS files. A function that renders a form needs 4 UI components for different field, buttons and so on. The form rendering function checks if those 4 UI components are available before attempting to use them. If not found, it requests them with an import statement one by one, then requests the corresponding CSS files one by one too. A ZIP file with all the requested files sounds good. But I don't know how to do that. Can you illustrate or link an example?
cyracrimes
cyracrimes•16mo ago
It sounds like you're already doing the right thing. Putting the files into some kind of zip/json package wouldn't work well here because you'd need to pile on a couple of additional hacks to then still be able to actually import the scripts/stylesheets. Importing the resources individually isn't necessarily harmful at all - is your question motivated by a measured performance issue? Have you maybe looked into bundling your application in a build step using something like webpack?
Zidan
ZidanOP•16mo ago
Not a performance issue, no. It's about efficiency really. A request and a response per file sounds very wasteful to me. So I wondered if I can send multiple responses per request if the request already has an array of the needed files. @pyrote mentioned a zip file with all the requested files. So that would be 1 request and 1 response per several files, which sounded pretty good to me 😄. @guest271314 mentioned a single JSON response with the files. No idea how that works 😕.
cyracrimes
cyracrimes•16mo ago
Both ideas won't work easily because you want the browser to load the files as scripts and stylesheets. They would be potential solutions if the files were just used programmatically by your app, but you're importing them, which is different. You also can't send multiple "responses" per request as far as HTTP is concerned. But: Browsers are good at doing requests. Importing multiple things at the same time isn't generally less efficient than importing only one thing, unless your logic blocks the loading of some of the resources until others are loaded
Zidan
ZidanOP•16mo ago
I see.. Well, I'm using Promise.all() at the moment. That shouldn't be blocking so I'll just keep using that. Thanks for the clarification.
cyracrimes
cyracrimes•16mo ago
That sounds fine to me :) If you run into something like this again where you really do want to avoid many individual imports, especially if they're nested, I'd encourage you to look into module bundling with tools like webpack
guest271314
guest271314•16mo ago
Simple {"0":[0, 255...], "1":[0, 255...]}. See https://plnkr.co/plunk/gCjYSt. Since Deno supports full duplex streaming you can stream all the files or whatever else you want using a single fetch() request, e.g., https://github.com/guest271314/native-messaging-deno/tree/fetch-duplex.
Josh
Josh•16mo ago
You could try opening a WebSocket, sending each file at a time and have the user and server notify each other when a file is received / sent, may be too much overhead though
guest271314
guest271314•16mo ago
Technically you can just serve the entire combination of N files as a single Uint8Array as long as you send the offsets, too. Here I combine a Uint32Array that includes the length of the following JSON configuration, which can optionally include images, album, artist, track information to display with Media Session API in global media controls, and offsets of the following discrete ArrayBuffer s which are Opus encoded audio output by WebCodecs AudioEncoder into a single file, where I extract the images and other media information and offsets of the discrete media chunks, and play back the file in the browser https://github.com/guest271314/WebCodecsOpusRecorder, largely inspired by the Native Messaging protocol which encodes the length of a message then the message. Bonus: The resulting file is less size than Opus audio encoded in WebM container. Alternatively, include multiple files in a .tar.gz or .zip file then extract the files from archive using Compression Streams, e.g., here I extract the node executable from the Node.js nightly release and get rid of everything else - in the browser https://github.com/guest271314/download-node-nightly-executable. This is not a particularly difficult case to solve. Just choose your approach and stick to that on the fron-end and back-end.
guest271314
guest271314•16mo ago
That will work. Create the script elements and set textContent. With CSSOM we can set CSS Rules directly in a stylesheet, see Modify element :before CSS rules programmatically in React and Empty style element with working CSS rules?.
Stack Overflow
Modify element :before CSS rules programmatically in React
I have an element whose :before style has to be modified based on calculations. export class Content extends React.Component { render() { return ( <div className="ring-b...
cyracrimes
cyracrimes•16mo ago
I fully agree that you can do this, I just think it's an unnecessary hack in this case. In fairness of course, much of the technology that makes the modern web work could be described this way :P Well, I guess it's often necessary hacks. I don't think this particular thing is worth doing in this particular case
guest271314
guest271314•16mo ago
Somebody said in some article Emscripten started off as a hack. Now there's WebAssembly and Bytecode Alliance. It's not really a hack. No more than YouTube using Media Source Extensions to serve multiple files for media playback. Again, at the extreme side there are Web Bundles https://github.com/GoogleChromeLabs/webbundle-plugins/blob/main/packages/webbundle-webpack-plugin/README.md, though Crromium/Chrome did remove support for navigation to Web Bundles. Choose your approach; JSON, Streams, whatever. They each work, as I demostrate in the linked working examples. Nothing is stopping you from encoding you data as JSON then using a single import assertion to import all of the files at once. Here's another example of piping multiple files to the same stream https://github.com/guest271314/AudioWorkletStream, something like
let port;
onmessage = async e => {
'use strict';
if (!port) {
[port] = e.ports;
port.onmessage = event => postMessage(event.data);
}
const { urls } = e.data;
// https://github.com/whatwg/streams/blob/master/transferable-streams-explainer.md
const { readable, writable } = new TransformStream();
(async _ => {
for await (const _ of (async function* stream() {
while (urls.length) {
yield (await fetch(urls.shift(), {cache: 'no-store'})).body.pipeTo(writable, {
preventClose: !!urls.length,
});
}
})());
})();
port.postMessage(
{
readable,
},
[readable]
);
};
let port;
onmessage = async e => {
'use strict';
if (!port) {
[port] = e.ports;
port.onmessage = event => postMessage(event.data);
}
const { urls } = e.data;
// https://github.com/whatwg/streams/blob/master/transferable-streams-explainer.md
const { readable, writable } = new TransformStream();
(async _ => {
for await (const _ of (async function* stream() {
while (urls.length) {
yield (await fetch(urls.shift(), {cache: 'no-store'})).body.pipeTo(writable, {
preventClose: !!urls.length,
});
}
})());
})();
port.postMessage(
{
readable,
},
[readable]
);
};
guest271314
guest271314•16mo ago
Just for completeness here is an example of using signed Web Bundles in an Isolated Web App - with a Deno and txiki.js TCP server https://github.com/GoogleChromeLabs/telnet-client/pull/18. For some reason Direct Sockets have been gated behind Progressive Web Apps and now Isolated Web Apps. Somebody thinks that leads to greater "security", however I have already incorporated browser extension code into the signed Web Bundle which provides a means to communicate between an arbitrary Web page and the IWA for the purpose of controlling a TCPSocket() connection from any arbitrary Web site I choose.
GitHub
Build software better, together
GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 330 million projects.
guest271314
guest271314•16mo ago
Note, for standardized approach you can serve a FormData object from the server and just read the data as File objects in the browser/client, see https://stackoverflow.com/a/47067787.
guest271314
guest271314•16mo ago
Here you go https://plnkr.co/edit/jluHMgfqdPD1z6Y1?open=lib%2Fscript.js Server
onfetch = (e) => {
if (e.request.url.includes('files')) {
const fd = new FormData();
fd.append(
'0',
new Blob([`console.log('script 0')`], { type: 'text/javascript' }),
'0.js'
);

fd.append(
'1',
new Blob([`console.log('script 1')`], { type: 'text/javascript' }),
'1.js'
);

fd.append(
'2',
new Blob([`body {color:blue}`], { type: 'text/css' }),
'2.css'
);
let client = clients.get(e.clientId);
// console.log(e.request.headers.get('content-type'));
e.respondWith(
new Response(fd, {
headers: {},
})
);
}
};
onfetch = (e) => {
if (e.request.url.includes('files')) {
const fd = new FormData();
fd.append(
'0',
new Blob([`console.log('script 0')`], { type: 'text/javascript' }),
'0.js'
);

fd.append(
'1',
new Blob([`console.log('script 1')`], { type: 'text/javascript' }),
'1.js'
);

fd.append(
'2',
new Blob([`body {color:blue}`], { type: 'text/css' }),
'2.css'
);
let client = clients.get(e.clientId);
// console.log(e.request.headers.get('content-type'));
e.respondWith(
new Response(fd, {
headers: {},
})
);
}
};
Browser
async function processFormData() {
fetch('./?files')
.then(async (r) => {
console.log(r.headers.get('content-type'));
/*
return new Response(await r.text(), {
headers: { 'content-type': r.headers.get('content-type') },
})
*/
return r.formData();
})
.then(async (fd) => {
console.log(fd);
for (const [, file] of fd) {
switch (file.type) {
case 'text/javascript':
const script = document.createElement('script');
script.textContent = await file.text();
document.body.appendChild(script);
break;
case 'text/css':
const style = document.createElement('style');
style.textContent = await file.text();
document.head.appendChild(style);
break;
default:
console.log(file.type);
}
}
})
.catch(console.warn);
}
async function processFormData() {
fetch('./?files')
.then(async (r) => {
console.log(r.headers.get('content-type'));
/*
return new Response(await r.text(), {
headers: { 'content-type': r.headers.get('content-type') },
})
*/
return r.formData();
})
.then(async (fd) => {
console.log(fd);
for (const [, file] of fd) {
switch (file.type) {
case 'text/javascript':
const script = document.createElement('script');
script.textContent = await file.text();
document.body.appendChild(script);
break;
case 'text/css':
const style = document.createElement('style');
style.textContent = await file.text();
document.head.appendChild(style);
break;
default:
console.log(file.type);
}
}
})
.catch(console.warn);
}
Leokuma
Leokuma•16mo ago
The browser will download the files only once in a lifetime and then cache them for future access. Some of the solutions proposed here will prevent the browser from caching them, forcing a redownload of all the files everytime your SPA is opened. Also I believe the cost of downloading and processing a Zip file is higher than downloading the files separately compressed as gzip or Brotli which are web standards
Zidan
ZidanOP•16mo ago
Wow, hack/worth or not, I did not know about any of the approachs you mentioned. Thanks a lot. Now, I did some reading on FormData and Blobs. And wondering if it's possible to use the JS/CSS files from a FormData Blob without creating/appending script tags like when using import? Or is this the only way for that approach? Like auto caching? Man, do I have a lot of reading to do xD Thanks for the heads up
Leokuma
Leokuma•16mo ago
Yes browsers cache files automatically, so only the first time the user accesses your SPA the files will be actually downloaded. On the following times when you do import() the browser will retrieve from cache
Zidan
ZidanOP•16mo ago
In my novice mind. A user sends a sign in request with their credentials, if successful, I would respond with body like this:
{ user: { id: '', name: '', phone: '' }, files: [ file/blob, file/blob ] }
{ user: { id: '', name: '', phone: '' }, files: [ file/blob, file/blob ] }
and the files would work without the need to create and append script elements like when done using import. Instead of responding with just user data + files names so that a pre existing function can import them one by one.
guest271314
guest271314•16mo ago
Now, I did some reading on FormData and Blobs. And wondering if it's possible to use the JS/CSS files from a FormData Blob without creating/appending script tags like when using import? Or is this the only way for that approach?
Technically you can create a Blob URL and use import(). What is the use case?
Zidan
ZidanOP•16mo ago
I have divided my SPA's UI into what I named components, modules and events. A component is function that returns an HTML element. A module is function that uses components to build pages, forms, popups Etc. An event is a function that is called by a component. All of the above is loaded on demand only, so each module function has an array for component dependencies that are checked for availability (are X, Y, Z already loaded?) before the module attempts to use them. And if they are available, the availability function (which returns a promise) just fulfills. And if one or more of the component dependencies are missing, it runs a Promise.all() on an array of import() statements to request the component files and their corresponding CSS files one by one, then fulfills or rejects accordingly. And this is working just fine for now, but I wanted to see if I can request all files at once instead of one by one. So I wanted to try HTTP/2 server push in Deno but then found that its no longer supported. Then you suggested using the FormData object which would allow me to send a list of files like I wanted to achieve, expect that it involves creating and appending <script> and <link> HTML tags to the DOM for each file, which I would like to avoid.
AapoAlas
AapoAlas•16mo ago
Web bundles is also a thing that was promising this exact feature but it was apparently sort of abandoned this year.
AapoAlas
AapoAlas•16mo ago
Chrome Developers
Get started with Web Bundles - Chrome Developers
Web Bundles enable you to share websites as a single file over Bluetooth and run them offline in your origin's context.
guest271314
guest271314•16mo ago
Web Bundles was not abandoned.
AapoAlas
AapoAlas•16mo ago
I was going to link the Chromium bug where they say this:
The code for navigation to Web Bundles has been behind a flag for a long time, but we have no plans to ship it.
but indeed I'd misunderstood, it's not the full Web Bundles but just "navigation to Web Bundles", using something like file:///home/user/web_bundle to load a page.
Our focus on WebBundle has shifted to subresource loading and Isolated Web Apps use cases.
So yeah, still going but with a somewhat limited scope. My bad.
guest271314
guest271314•16mo ago
Chromium did remove the ability to navigate directly to a (Signed) Web Bundle. It is still possible to create a (Signed) Web Bundle including whatever files you want https://github.com/GoogleChromeLabs/telnet-client/issues/25 (although I don't think it is possible to register a ServiceWorker in isolated-app: protocol; at least I have not been able to do so using the Webpack and Web Bundle plugins here https://github.com/GoogleChromeLabs/telnet-client/).