Deno•3y ago

await writer.write(buffer) never resolves on Deno.Command

Hiho. I'm creating a

Deno.Command

Deno.Command

that writes a buffer to

vipsthumbnail

vipsthumbnail

via stdin (and reads the result via stdout).

It works most of the time, but sometimes gets stuck on

await writer.write(source);

await writer.write(source);

. Source is the result of

await Deno.readFile(path)

await Deno.readFile(path)

.

The big problem is that there is no error, the writer.write() promise is just never resolved. So I have no idea how to debug, catch errors or even cancel this after timeout or something.

I can process a couple of files (all with a new

Deno.Command

Deno.Command

) but then after a few files it gets stuck and the promise never resolves.

Any ideas how to begin solving what the problem is.

    const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
      args: args,
      stdin: "piped",
      stderr: "piped",
      stdout: "piped",
    }).spawn();

    const writer = cmd.stdin.getWriter();
    console.log("START WRITING", typeof source, source.byteLength);
    await writer.write(source); // <--- STUCK HERE MAY NEVER RESOLVE
    console.log("YAY! WE GOT HERE");
    await writer.ready;
    await writer.close();

    const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
      args: args,
      stdin: "piped",
      stderr: "piped",
      stdout: "piped",
    }).spawn();

    const writer = cmd.stdin.getWriter();
    console.log("START WRITING", typeof source, source.byteLength);
    await writer.write(source); // <--- STUCK HERE MAY NEVER RESOLVE
    console.log("YAY! WE GOT HERE");
    await writer.ready;
    await writer.close();

Joe Hildebrand•12/11/23, 7:18 PM

Are you sure that vipsthumbnail is actually reading stdin if stdin hasn't closed?

Joe Hildebrand•12/11/23, 7:19 PM

If the output buffer fills up,

writer.write()

writer.write()

will block.

Joe Hildebrand•12/11/23, 7:21 PM

you could try

await Promise.all([writer.write(), writer.close()])

await Promise.all([writer.write(), writer.close()])

to find out

rauniorooOP•12/11/23, 7:22 PM

hm. no I'm not sure :) it appears to read, I can process a couple of files just fine. is it possible vipsthumbnail does read stdin, but then stops reading it when it encounters some kind of error parsing a certain file?

Joe Hildebrand•12/11/23, 7:23 PM

anything's possible, of course. All programs have bugs.

JJoe Hildebrand you could try `await Promise.all([writer.write(), writer.close()])` to find out

rauniorooOP•12/11/23, 7:24 PM

this doesnt seem to help with getting it unstuck

Joe Hildebrand•12/11/23, 7:24 PM

however, when vipsthumbnail exits without reading pending stdin, I would have expected

await writer.write

await writer.write

to throw.

rauniorooOP•12/11/23, 7:27 PM

the weird thing is that vipsthumbnail prints stuff to stderr, like _file "" does not exist, but then it still gives a proper result. maybe it first tries to open a file named "stdin", prints some errors, and only then reverts to the code path that actually reads stdin. (you are supposed to call it with "stdin" in place of the input file name argument)

jeff.hykin•12/11/23, 7:29 PM

might be unrelated but this sounds similar to this old problem

Deno.run Strange Issue

rauniorooOP•12/11/23, 7:29 PM

how would I go about writing a timeout to abort the

writer.write()

writer.write()

if it takes x amount of time

Joe Hildebrand•12/11/23, 7:29 PM

What if you throw

cmd.status()

cmd.status()

into the Promise.all? Do you get a zero exit code?

rauniorooOP•12/11/23, 7:32 PM

    await Promise.all([writer.write(source), writer.close(), cmd.status]);

    await Promise.all([writer.write(source), writer.close(), cmd.status]);

just makes it fail on the first file (same result, never resolves)

Rraunioroo how would I go about writing a timeout to abort the `writer.write()` if it take...

jeff.hykin•12/11/23, 7:33 PM

This uses Deno.run but it should be the same general logic

https://github.com/jeff-hykin/quickr/blob/139ebd639bb474be1b0b77a3c3f273ca36c93d6d/main/run.js#L592C23-L592C23

GitHub

quickr/main/run.js at 139ebd639bb474be1b0b77a3c3f273ca36c93d6d · je...

Tools for Deno. Contribute to jeff-hykin/quickr development by creating an account on GitHub.

Rraunioroo how would I go about writing a timeout to abort the `writer.write()` if it take...

rauniorooOP•12/11/23, 7:41 PM

Copilot to the rescue, it wrote this nicely working timeout function, so at least I can move on when it fails.

    const timeout = new Promise((_, reject) =>
      setTimeout(() => reject(new Error("Operation timed out")), 1000)
    );
    try {
      await Promise.race([writer.write(source), timeout]);
    } catch (error) {
      console.log("writing to vipsthumbnail timed out");
      return null;
    }

    const timeout = new Promise((_, reject) =>
      setTimeout(() => reject(new Error("Operation timed out")), 1000)
    );
    try {
      await Promise.race([writer.write(source), timeout]);
    } catch (error) {
      console.log("writing to vipsthumbnail timed out");
      return null;
    }

Rraunioroo Copilot to the rescue, it wrote this nicely working timeout function, so at leas...

jeff.hykin•12/11/23, 7:43 PM

did you not actually want to cancel the process on timeout?

rauniorooOP•12/11/23, 7:44 PM

oh yeah I should do that, close everything before returning

jeff.hykin•12/11/23, 7:45 PM

okay. if you try to kill a process that already stopped it'll get an error. The code I linked handles that case, although you'll have to integrate it with the write-timeout code

rauniorooOP•12/11/23, 7:46 PM

When it works, it prints stuff like this to stderr, but does still give a beautiful image back :)

/usr/bin/vipsthumbnail: unable to thumbnail 
VipsForeignLoad: file "" does not exist
/usr/bin/vipsthumbnail: unable to thumbnail 

VipsForeignLoad: file "" does not exist

/usr/bin/vipsthumbnail: unable to thumbnail 
VipsForeignLoad: file "" does not exist
/usr/bin/vipsthumbnail: unable to thumbnail 

VipsForeignLoad: file "" does not exist

Jjeff.hykin okay. if you try to kill a process that already stopped it'll get an error. The ...

rauniorooOP•12/11/23, 7:46 PM

would like to avoid using the deprecated Deno.run if at all possible

Rraunioroo would like to avoid using the deprecated Deno.run if at all possible

jeff.hykin•12/11/23, 7:47 PM

the code will be basically the the same for Deno.Commad. There's a .kill() method and an .status() promise

rauniorooOP•12/11/23, 8:03 PM

I think this is kinda solved (I can now process images fine most of the time and "cleanly" abort when it fails for whatever reason). I learned a lot more than expected:) Thank you both!

JJoe Hildebrand If the output buffer fills up, `writer.write()` will block.

rauniorooOP•12/11/23, 8:04 PM

this was probably the most key piece of info. not 100% sure but I'm assuming vips stops reading stdin when it encounters.... something, and then this happens

rauniorooOP•12/11/23, 9:07 PM

wait a sec. actually all my problems went away with this much shorter and leaner solution:

instead of

buffer

buffer

source is now

ReadableStream

ReadableStream

(which is nicer already)

Then I just use

source.pipeTo(cmd.stdin)

source.pipeTo(cmd.stdin)

. Apparently

pipeTo

pipeTo

handles things like backpressure in a way that makes

vipsthumbnailer

vipsthumbnailer

stay happy on the receiving end. And this further confirms that vips wasn't failing on some files, just my way of feeding it (I guess the earlier

writer.write

writer.write

method only worked for very small files or something).

    const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
      args: args,
      stdin: "piped",
      stderr: "piped",
      stdout: "piped",
    }).spawn();

    source.pipeTo(cmd.stdin);
    const out = await cmd.output();

    const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
      args: args,
      stdin: "piped",
      stderr: "piped",
      stdout: "piped",
    }).spawn();

    source.pipeTo(cmd.stdin);
    const out = await cmd.output();

rauniorooOP•12/12/23, 11:22 AM

For full disclosure if anyone attempts similar stuff with vips. Spent like 6 full days to integrate vips, trying so many different approaches. It was a proper adventure. Firstly: wasm. So, wasm-vips sounds nice but seems not ready yet. Main reasons: leaks memory like crazy, is fast but slower than native cli, except for AVIF which is just suuuper-slow on wasm. Also it seems deno version is not quite at the level of node version (importing node version to deno through npm compat does not work, tried that too). Also I somehow thought using wasm would be the "pure/clean" approach, but did a 180 on that ideologically, and now think that one should use native apps whenever possible. Trouble with native vips is that it is very painful to install on Amazon's own AWS-optimized linux distro (no prebuilt binaries and even getting compiling to work is a frustrating experience). Solution here is to just switch to Ubuntu, which has vips ready among many other niceties.

rauniorooOP•12/12/23, 11:29 AM

Using native cli it is then. Passing data through STDIN as described above kinda works, but is unreliable for some reason and somewhat rarely but randomly fails in mysterious ways (even the pipeTo approach). Using just plain local file paths and/or temporary files is 100% reliable. Back to basics I guess :) I was dead set on trying non-file-backed piped approach to passing stuff to, and back from vips, as my files come from various non-file sources like S3. Luckily I realized I run every request to external services through a local file-based caching setup, so I do in fact have a local file path for everything after all, so no need to use temp files. Just need to ask the caching layer for the secret path to read from :) STDOUT seems to work reliably for getting the converted files back to deno, no problem there