dan.the.discloser•3y ago

piping for async sub process

In Deno.process, is there a way to set its STDOUT to something I can read line by line while that primary process is still running?

12 Replies

AapoAlas•3y ago

Set stdout to piped and use the readable stream, I think? https://doc.deno.land/deno/stable/~/Deno.run https://doc.deno.land/deno/stable/~/Deno.Process

Deno CLI APIs – Deno.run | Deno Doc

Spawns new subprocess. RunOptions must contain at a minimum the opt.cmd, an array of program arguments, the first of which is the binary. const p = Deno.run({ cmd: ["curl", "https://example.com"], }); const status = await p.status(); Subprocess uses same working directory as parent process unless opt.cwd is specified. Environmental variable...

Deno CLI APIs – Deno.Process | Deno Doc

Represents an instance of a sub process that is returned from {@linkcode Deno.run} which can be used to manage the sub-process.

AapoAlas•3y ago

Or stdin if you meant reading line by line inside the spawned process.

dan.the.discloserOP•3y ago

Thank you! This works, but is VERY ugly * Is there a better way to terminate the reading loop? * Is there a better way to deal with the reads not filling the buffer?

const cmd = ["sh", "long-running-script"]

const proc = Deno.run({stdout: "piped" , cmd: cmd});

async function handleAll(){
    const decoder = new TextDecoder()
    const buf =new Uint8Array(10000);
    while (true){
    const n = await proc.stdout.read(buf)
    if (n === null){ break;}
    const outout = decoder.decode(buf)
    console.log(outout.slice(0,n));
    }
}

const promises = [handleAll() , proc.status()]

await Promise.allSettled(promises)

const cmd = ["sh", "long-running-script"]

const proc = Deno.run({stdout: "piped" , cmd: cmd});

async function handleAll(){
    const decoder = new TextDecoder()
    const buf =new Uint8Array(10000);
    while (true){
    const n = await proc.stdout.read(buf)
    if (n === null){ break;}
    const outout = decoder.decode(buf)
    console.log(outout.slice(0,n));
    }
}

const promises = [handleAll() , proc.status()]

await Promise.allSettled(promises)

AapoAlas•3y ago

https://deno.land/std@0.161.0/streams/mod.ts will probably have the helpers you want.

AapoAlas•3y ago

Also, TextDecoderStream exists: https://deno.land/api@v1.26.2?s=TextDecoderStream

Deno

TextDecoderStream | Runtime APIs | Deno

dan.the.discloserOP•3y ago

I read through the doc's and the sources (to the extent I can) and I just don't get it. Clearly what I am doig is not right. error: TS2339 [ERROR]: Property 'pipeThrough' does not exist on type 'Reader & Closer & { readable: ReadableStream<Uint8Array>; }'.

async function handleAll(){
    const output = await proc.stdout.pipeThrough((new TextDecoderStream()))
    console.dir(output)
}

async function handleAll(){
    const output = await proc.stdout.pipeThrough((new TextDecoderStream()))
    console.dir(output)
}

Can you point me at a simple example?

AapoAlas•3y ago

stdout.readable.pipeThrough

dan.the.discloserOP•3y ago

Thank you -- i will try it Thank you! This is much less ugly! Is there also a way to make the control of the while loop less ugly?

async function handleAll() {
  const outputStream = await proc.stdout.readable.pipeThrough(
    new TextDecoderStream(),
  );
  const r = outputStream.getReader();
  let firstOutput = { done: false };
  while (!firstOutput.done) {
    firstOutput = await r.read();
    console.log(firstOutput);
  }
  return r;
}

async function handleAll() {
  const outputStream = await proc.stdout.readable.pipeThrough(
    new TextDecoderStream(),
  );
  const r = outputStream.getReader();
  let firstOutput = { done: false };
  while (!firstOutput.done) {
    firstOutput = await r.read();
    console.log(firstOutput);
  }
  return r;
}

AapoAlas•3y ago

I presume your end goal is not to just log the output? You might consider implementing your eventual use case as a writable stream. But if you just need to get the output then yeah, a while loop doing awaits is pretty much the way. You can improve it a bit by doing:

let firstOut;
do {
  firstOut = await r.read();
} while (!firstOut.done);

let firstOut;
do {
  firstOut = await r.read();
} while (!firstOut.done);

dan.the.discloserOP•3y ago

thank you a, a "do while" was exactly what I was hoping for indeed there is more work to do with the results, but I am trying to understands. the basics. when I tried

async function handleAll() {
  const outputStream = await proc.stdout.readable.pipeThrough(
    new TextDecoderStream(),
  );
  for await (const firstOutput of outputStream) {
    console.log(firstOutput);
  }
}

async function handleAll() {
  const outputStream = await proc.stdout.readable.pipeThrough(
    new TextDecoderStream(),
  );
  for await (const firstOutput of outputStream) {
    console.log(firstOutput);
  }
}

it waited for the process to finish and then gives me all the reads rapidly rather tthan giving to me as they become available

AapoAlas•3y ago

Humm...

dan.the.discloserOP•3y ago

The problem was with my environment, the iterator works fine. Sorry to be stupid.