Josh•2y ago

Better way to allow downloading of files besides serving entire file

I'm trying to allow users to download files that are in a private folder which cannot be accessed publicly. A special key is provided which allows them to download the file. Would I have to put the file into a public folder that can be accessed directly through a browser, or is there a way to use a router to serve files? I'm currently using a router to serve entire files to the client, but this creates a delay since some files can be gigabytes large and the site hangs while the file is being read + sent. I've attached a pic of how I serve files, any help would be appreciated! If I need to make a public folder called "download" instead, I can.

9 Replies

JoshOP•2y ago

ctx.send doesn't seem to be working as intended either

ioB•2y ago

I think the issue is that you're buffering the whole file into memory before sending it ideally it would be streamed Have you looked into https://deno.land/std@0.179.0/http/file_server.ts? I've never used oak before but I think this would solve a lot of your problems

JoshOP•2y ago

Yeah this is the issue, I'm not sure how I could stream a file through Oak. I'll keep looking and I'll check out that link to see if I can figure it out, thank you! Oak's send method supposedly streams files, but whenever I try to stream it, it throws saying the response isn't writeable. But if I await the send, it goes back to the same behaviour I'm trying to avoid. send is obviously streaming but since the middleware returns, the response is closed. I'm still looking around Bumping this Bumping again, I've resorted to using an XMLHttpRequest to show the progress of the file transfer which helps with the frozen site appearance, but this isn't the behaviour I'm aiming for still

ioB•2y ago

How big is this file?

JoshOP•2y ago

It can be any size, from a couple mb to a couple gb The issue is when it's a couple gb, the entire file is sent to the client in a blob and that makes the site look like it froze when in reality it's waiting for the file And the file isn't in a public folder so I can't have the client redirect to it and download it normally

ioB•2y ago

You definitely still could do a redirect just verify a cookie or something

JoshOP•2y ago

The file is in a folder called "downloading", and it's not in a public folder so I can't do "https://<domain>/downloading/file.txt", that would return a 404. What I do instead is have a route that loads the file into a stream

.get("/download/:token", async (ctx) => {
    const download = pendingDownloads.get(ctx.params.token);
    if (!download) {
        ctx.response.status = Status.NotFound;
        return;
    }

    pendingDownloads.delete(ctx.params.token);

    try {
        const file = await Deno.open(`./downloading/${download}`);
        const fileInfo = await file.stat();
        ctx.response.headers.set('X-Decompressed-Content-Length', fileInfo.size.toString());
        ctx.response.headers.set('Content-Disposition', `attachment; filename=${download}`);
        ctx.response.type = "stream";
        ctx.response.status = Status.OK;
        ctx.response.body = streams.readableStreamFromReader(file);
    } catch {
        ctx.throw(Status.InternalServerError);
    }
});

.get("/download/:token", async (ctx) => {
    const download = pendingDownloads.get(ctx.params.token);
    if (!download) {
        ctx.response.status = Status.NotFound;
        return;
    }

    pendingDownloads.delete(ctx.params.token);

    try {
        const file = await Deno.open(`./downloading/${download}`);
        const fileInfo = await file.stat();
        ctx.response.headers.set('X-Decompressed-Content-Length', fileInfo.size.toString());
        ctx.response.headers.set('Content-Disposition', `attachment; filename=${download}`);
        ctx.response.type = "stream";
        ctx.response.status = Status.OK;
        ctx.response.body = streams.readableStreamFromReader(file);
    } catch {
        ctx.throw(Status.InternalServerError);
    }
});

Doctor 🤖•2y ago

If the site is giving a freezing effect then that’s a different issue with the JavaScript code downloading in sync instead of async. I think something like this should work.

const file = await Deno.open("some file")
ctx.response.body = file.readable

const file = await Deno.open("some file")
ctx.response.body = file.readable

JoshOP•2y ago

The file doesn't seem to download at all with this approach. For reference I'm using the anchor element download method, unless there's a way to begin a streamed download on the client with another method I'm possibly just going to stick to showing the stream sending progress and then downloading the file all at once instead of it being a normal file download, it's not the behavior I want but I can't figure out how to get it to work regularly Only other thing I can think to do is stream the file's contents to the client during the file creation, but that seems like a hassle I found a way to pipe directly to the file system, only issue is that download progress isn't shown

const res = await fetch(`/download/${token}`);

const ws_dest = window.showSaveFilePicker().then(handle => handle.createWritable());

const ts_dec = new TransformStream({
  transform(chunk, controller) {
    controller.enqueue(chunk);
  }
});

return res.body.pipeThrough(ts_dec).pipeTo(await ws_dest);

const res = await fetch(`/download/${token}`);

const ws_dest = window.showSaveFilePicker().then(handle => handle.createWritable());

const ts_dec = new TransformStream({
  transform(chunk, controller) {
    controller.enqueue(chunk);
  }
});

return res.body.pipeThrough(ts_dec).pipeTo(await ws_dest);

Ended up going the route of public folders