D
Deno

help

await writer.write(buffer) never resolves on Deno.Command

Rraunioroo12/11/2023
Hiho. I'm creating a Deno.Command that writes a buffer to vipsthumbnail via stdin (and reads the result via stdout). It works most of the time, but sometimes gets stuck on await writer.write(source);. Source is the result of await Deno.readFile(path). The big problem is that there is no error, the writer.write() promise is just never resolved. So I have no idea how to debug, catch errors or even cancel this after timeout or something. I can process a couple of files (all with a new Deno.Command) but then after a few files it gets stuck and the promise never resolves. Any ideas how to begin solving what the problem is.
const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
args: args,
stdin: "piped",
stderr: "piped",
stdout: "piped",
}).spawn();

const writer = cmd.stdin.getWriter();
console.log("START WRITING", typeof source, source.byteLength);
await writer.write(source); // <--- STUCK HERE MAY NEVER RESOLVE
console.log("YAY! WE GOT HERE");
await writer.ready;
await writer.close();
const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
args: args,
stdin: "piped",
stderr: "piped",
stdout: "piped",
}).spawn();

const writer = cmd.stdin.getWriter();
console.log("START WRITING", typeof source, source.byteLength);
await writer.write(source); // <--- STUCK HERE MAY NEVER RESOLVE
console.log("YAY! WE GOT HERE");
await writer.ready;
await writer.close();
JHJoe Hildebrand12/11/2023
Are you sure that vipsthumbnail is actually reading stdin if stdin hasn't closed? If the output buffer fills up, writer.write() will block. you could try await Promise.all([writer.write(), writer.close()]) to find out
Rraunioroo12/11/2023
hm. no I'm not sure :) it appears to read, I can process a couple of files just fine. is it possible vipsthumbnail does read stdin, but then stops reading it when it encounters some kind of error parsing a certain file?
JHJoe Hildebrand12/11/2023
anything's possible, of course. All programs have bugs.
Rraunioroo12/11/2023
this doesnt seem to help with getting it unstuck
JHJoe Hildebrand12/11/2023
however, when vipsthumbnail exits without reading pending stdin, I would have expected await writer.write to throw.
Rraunioroo12/11/2023
the weird thing is that vipsthumbnail prints stuff to stderr, like _file "" does not exist, but then it still gives a proper result. maybe it first tries to open a file named "stdin", prints some errors, and only then reverts to the code path that actually reads stdin. (you are supposed to call it with "stdin" in place of the input file name argument)
Jjeff.hykin12/11/2023
might be unrelated but this sounds similar to this old problem https://discord.com/channels/684898665143206084/1074484890562736188/1074682730073313340
Rraunioroo12/11/2023
how would I go about writing a timeout to abort the writer.write() if it takes x amount of time
JHJoe Hildebrand12/11/2023
What if you throw cmd.status() into the Promise.all? Do you get a zero exit code?
Rraunioroo12/11/2023
await Promise.all([writer.write(source), writer.close(), cmd.status]); just makes it fail on the first file (same result, never resolves)
Jjeff.hykin12/11/2023
GitHub
quickr/main/run.js at 139ebd639bb474be1b0b77a3c3f273ca36c93d6d · je...
💾 📦 Tools for Deno. Contribute to jeff-hykin/quickr development by creating an account on GitHub.
Rraunioroo12/11/2023
Copilot to the rescue, it wrote this nicely working timeout function, so at least I can move on when it fails.
const timeout = new Promise((_, reject) =>
setTimeout(() => reject(new Error("Operation timed out")), 1000)
);
try {
await Promise.race([writer.write(source), timeout]);
} catch (error) {
console.log("writing to vipsthumbnail timed out");
return null;
}
const timeout = new Promise((_, reject) =>
setTimeout(() => reject(new Error("Operation timed out")), 1000)
);
try {
await Promise.race([writer.write(source), timeout]);
} catch (error) {
console.log("writing to vipsthumbnail timed out");
return null;
}
Jjeff.hykin12/11/2023
did you not actually want to cancel the process on timeout?
Rraunioroo12/11/2023
oh yeah I should do that, close everything before returning
Jjeff.hykin12/11/2023
okay. if you try to kill a process that already stopped it'll get an error. The code I linked handles that case, although you'll have to integrate it with the write-timeout code
Rraunioroo12/11/2023
When it works, it prints stuff like this to stderr, but does still give a beautiful image back :)
/usr/bin/vipsthumbnail: unable to thumbnail
VipsForeignLoad: file "" does not exist
/usr/bin/vipsthumbnail: unable to thumbnail

VipsForeignLoad: file "" does not exist
/usr/bin/vipsthumbnail: unable to thumbnail
VipsForeignLoad: file "" does not exist
/usr/bin/vipsthumbnail: unable to thumbnail

VipsForeignLoad: file "" does not exist
would like to avoid using the deprecated Deno.run if at all possible
Jjeff.hykin12/11/2023
the code will be basically the the same for Deno.Commad. There's a .kill() method and an .status() promise
Rraunioroo12/12/2023
I think this is kinda solved (I can now process images fine most of the time and "cleanly" abort when it fails for whatever reason). I learned a lot more than expected:) Thank you both! this was probably the most key piece of info. not 100% sure but I'm assuming vips stops reading stdin when it encounters.... something, and then this happens wait a sec. actually all my problems went away with this much shorter and leaner solution: instead of buffer source is now ReadableStream (which is nicer already) Then I just use source.pipeTo(cmd.stdin). Apparently pipeTohandles things like backpressure in a way that makes vipsthumbnailer stay happy on the receiving end. And this further confirms that vips wasn't failing on some files, just my way of feeding it (I guess the earlier writer.write method only worked for very small files or something).
const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
args: args,
stdin: "piped",
stderr: "piped",
stdout: "piped",
}).spawn();

source.pipeTo(cmd.stdin);
const out = await cmd.output();
const cmd = new Deno.Command(this.vipspath + this.vipsthumbnail, {
args: args,
stdin: "piped",
stderr: "piped",
stdout: "piped",
}).spawn();

source.pipeTo(cmd.stdin);
const out = await cmd.output();
For full disclosure if anyone attempts similar stuff with vips. Spent like 6 full days to integrate vips, trying so many different approaches. It was a proper adventure. Firstly: wasm. So, wasm-vips sounds nice but seems not ready yet. Main reasons: leaks memory like crazy, is fast but slower than native cli, except for AVIF which is just suuuper-slow on wasm. Also it seems deno version is not quite at the level of node version (importing node version to deno through npm compat does not work, tried that too). Also I somehow thought using wasm would be the "pure/clean" approach, but did a 180 on that ideologically, and now think that one should use native apps whenever possible. Trouble with native vips is that it is very painful to install on Amazon's own AWS-optimized linux distro (no prebuilt binaries and even getting compiling to work is a frustrating experience). Solution here is to just switch to Ubuntu, which has vips ready among many other niceties. Using native cli it is then. Passing data through STDIN as described above kinda works, but is unreliable for some reason and somewhat rarely but randomly fails in mysterious ways (even the pipeTo approach). Using just plain local file paths and/or temporary files is 100% reliable. Back to basics I guess :) I was dead set on trying non-file-backed piped approach to passing stuff to, and back from vips, as my files come from various non-file sources like S3. Luckily I realized I run every request to external services through a local file-based caching setup, so I do in fact have a local file path for everything after all, so no need to use temp files. Just need to ask the caching layer for the secret path to read from :) STDOUT seems to work reliably for getting the converted files back to deno, no problem there

Looking for more? Join the community!

Recommended Posts
Not sure if I understand kv watch right?Not sure if I understand kv watch right, is like named queues? o more like a reliable Broadcast chanDisplaying a ModalI am trying to get a minimal example of using Bootstrap Modals, and managing that modal using state.How do I run an npm script with multiple dependencies, e.g. `drizzle-kit generate:pg`?I thought about wrapping it into a task `"generate": "deno run -A npm:drizzle-kit generate:pg"` in tHow to use a class name as a qualifier for an enum ?Hi there, it's more of a generic typescript question but I'm using it in the Deno context and the lihow can I make him see the process?```ts error: Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'on') proImporting NPM Packages in Deno Throws 'Uncaught SyntaxError' for Specific ModulesHi there, I'm relatively new to Deno. I'm running into an error with an NPM package import. ```JS imHow to override type definitions provided by a third party module@Deno AI Helper How do I override type definitions provided by a third party moduleSync Child Stdin ReadI'd like to make a JS wrapper around the repl of another language. For example: ```js let a = otherUsing custom font using fresh and tailwindcssHello ! I recently followed [this guide](https://fresh.deno.dev/docs/examples/migrating-to-tailwindwasm-vips file does not existOk this is not directly a deno thing, but anyway. Has anyone successfully used wasm-vips with `vips.update imports to latest version in `deno.json`i defined imports as URLs with the version of the packages in my `deno.json` file. i would like to kAttempt on adding ref/unref on UDPHello, issue [#20138](https://github.com/denoland/deno/issues/20138) is about `unref` not being implFresh migrating from serveTlsIn my `main.ts` file for Fresh I have some code I borrowed from source a few minor versions ago to sServing static websiteI have a static website with an index.html that imports a main.js and css file. When serving index.hpositional arguments deno taskCan I use positional arguments in deno task? I have a number of scripts that I frequently want to rLinter does not warn about lack of constructor arguments.The vscode shows that there are no arguments, but linter does not show it as an error: ``` root@instCan you make a fresh route detect, if it is being requested by `deno run`Can you make a fresh route detect, if it is being requested by `deno run` like `deno run https://locDeno dependency top level await work aroundI have the following error while building with dnt ``` [dnt] Top level await cannot be used when disBundling a CLI written in JS with DenoHi team, Thanks for writing this [excellent blog-post (and video)](https://deno.com/blog/roll-your-Leaking operation op_http_close in testsI tried to update my repositories from Deno 1.38.1 to 1.38.4. After the update my test keeps failing