Sync Child Stdin Read

I'd like to make a JS wrapper around the repl of another language. For example:

let a = otherLang`10`
let b = otherLang`{ c = 11; d = 12; }`;
console.log(b.c) // 11

let a = otherLang`10`
let b = otherLang`{ c = 11; d = 12; }`;
console.log(b.c) // 11

I can spawn a repl in a subprocess, keep it open and hook into the stdin/out/err of it. Then the idea would be to do something like the following:

otherLang`{ c = 11; d = 12; }`

// roughly becomes
child.stdin.write(`
  print(json(listAttrNames(
    { c = 11; d = 12; }
  )))
  print(${endUuid})
`)
while (1) {
   child.stdout.read() // until endUuid
}

otherLang`{ c = 11; d = 12; }`

// roughly becomes
child.stdin.write(`
  print(json(listAttrNames(
    { c = 11; d = 12; }
  )))
  print(${endUuid})
`)
while (1) {
   child.stdout.read() // until endUuid
}

The problem is something like that^ needs to run inside of a getter for console.log(b.a) to work. E.g. no async. I can do some incredibly hacky work, like funneling stdout to a file and then sync-reading (.seek()) the file over and over. But surely there's a better way. Relevant documentation on streams: https://deno.land/api@v1.38.5?s=ReadableStreamBYOBReader

Deno

ReadableStreamBYOBReader | Runtime APIs | Deno

4 Replies

Deno AI Helper•13mo ago

I wasn't able to find the answer to your question in the official Deno documentation. However, based on the code snippet you've provided, I can offer some general advice, but please note that this advice is not based on the Deno documentation, and you should double-check the accuracy of the advice given. In your code, you are attempting to read from child.stdout in a synchronous loop. In Deno, operations on streams are inherently asynchronous, and there is no synchronous API for reading from a stream. The child.stdout.read() method returns a Promise, which you would typically handle with await or .then(). However, since you need this to work in a synchronous context (e.g., inside a getter), this poses a challenge because you cannot use await inside a synchronous function. One potential issue with your approach is that you are using an infinite loop (while (1)) to read from child.stdout. This will block the event loop and prevent other asynchronous operations from running, which is generally not recommended in JavaScript environments. Since you are looking for a synchronous solution, and Deno does not provide synchronous file or stream reading capabilities, you might indeed be forced to use a workaround like the one you mentioned (funneling stdout to a file and then sync-reading the file). However, this is not an ideal solution and could lead to performance issues or race conditions. As a reminder, please verify this information independently, as it is not sourced from the official Deno documentation, and the Deno runtime may have specific behaviors or APIs that differ from what is described here.

Helpful Links: Deno 1.21 Release Notes An Implementation of the Unix "cat" Program | Deno Docs Deno 1.24 Release Notes Creating a Subprocess | Deno Docs

raunioroo•13mo ago

Why is needs to run inside of a getter for console.log(b.a) to work.a requirement, if I may ask? Is that actually part of the problem, or can you consider other (async) approaches? I can not help with turning async stuff to sync, but may be able to help with talking and writing to a repl-style child process in async manner since I'm writing a thing that does a lot of that and solved some struggles.

jeff.hykinOP•13mo ago

Because it becomes ergonomically impractical, and has negative performance implications (I know it's weird to talk about perf on something so inherently slow but it's true). The point of the JS wrapper in this project is to be a convenient and easy interface. However, that can't really be done when basic stuff like this is impossible:

class DataFrame {
  constructor(obj) {
    this.inner = obj
    if (obj instanceof otherLangExpr) {
      this.c = obj.melted.names.byAxis.column
    } else {
       // stuff
    }
  }
  get length() {
    return otherLang`builtins.getLength ${this.inner}`
  }
}

class DataFrame {
  constructor(obj) {
    this.inner = obj
    if (obj instanceof otherLangExpr) {
      this.c = obj.melted.names.byAxis.column
    } else {
       // stuff
    }
  }
  get length() {
    return otherLang`builtins.getLength ${this.inner}`
  }
}

Even after taking a huge hit of throwing away all class wrappers, all sync getters, all sync static methods, its extremely awkward to do:

let builtins = await otherLang`builtins`
let c1 = await otherLang`i`
let c2 = await otherLang`3i`

let real = await (await builtins.isReal)(await (await builtins.sqrt)([ await (await builtins.pow)([ c1, c2 ]) ])

// vs
let real = builtins.isReal(
  builtins.sqrt([
     builtins.pow([ c1, c2 ])
  ])
)

let zip = (await (await (await (await builtins.methods).iter).withLength).zip)

let builtins = await otherLang`builtins`
let c1 = await otherLang`i`
let c2 = await otherLang`3i`

let real = await (await builtins.isReal)(await (await builtins.sqrt)([ await (await builtins.pow)([ c1, c2 ]) ])

// vs
let real = builtins.isReal(
  builtins.sqrt([
     builtins.pow([ c1, c2 ])
  ])
)

let zip = (await (await (await (await builtins.methods).iter).withLength).zip)

And even taking both of those major hits, I also cant allow something like:

await Promise.all([
  otherLang`doThing1`,
  otherLang`doThing2`,
])

await Promise.all([
  otherLang`doThing1`,
  otherLang`doThing2`,
])

Because it would create garbage input/output in the repl. So I would need to add my own locking system to ensure all language requests were handled sequentially. And even then, the performance is actually worse. Performing multiple awaits on every line, and then doing extra work to ensure they all execute sequentially creates so much overhead, believe it's notably slower than the hacky sync-read-from-file solution. Other than the repl I'm not using any async deno api's. So it's not like I can read a file or fetch a URL while I'm waiting on the repl. I'm only waiting on the repl.

raunioroo•13mo ago

Yeah, ok. I was thinking about queue type of thing, which can help with getting responses or requests to another process properly ordered so that calls can be ergonomically handled. But it doesn't help with await at all. I have read somewhere that perhaps a thing called Atomics can help turn async to sync, but I have absolutely zero idea how that works, whether it works, or is it supported in Deno. Here's a random article I googled, that talks about it maybe https://samthor.au/2021/block-nodejs-main-thread/