DNA•10mo ago

Puppeteer: "BadResource: Bad resource ID" on Ubuntu

Hello, i just switched my server os over to ubuntu server 23.10. When executing my script, it immediately throws an error an exits. Before, on my windows machine, the script worked completely fine. Error:

error: Uncaught (in promise) BadResource: Bad resource ID
    const result = await reader.read(inspectArr);
                                ^
    at read (ext:deno_io/12_io.js:116:28)
    at FsFile.read (ext:deno_fs/30_fs.js:706:12)
    at readDelim (https://deno.land/std@0.93.0/io/bufio.ts:652:33)
    at readDelim.next (<anonymous>)
    at readStringDelim (https://deno.land/std@0.93.0/io/bufio.ts:702:20)
    at readStringDelim.next (<anonymous>)
    at readLines (https://deno.land/std@0.93.0/io/bufio.ts:711:18)
    at readLines.next (<anonymous>)
    at waitForWSEndpoint (https://deno.land/x/puppeteer@16.2.0/src/deno/BrowserRunner.ts:168:20)
    at eventLoopTick (ext:core/01_core.js:166:7)

error: Uncaught (in promise) BadResource: Bad resource ID
    const result = await reader.read(inspectArr);
                                ^
    at read (ext:deno_io/12_io.js:116:28)
    at FsFile.read (ext:deno_fs/30_fs.js:706:12)
    at readDelim (https://deno.land/std@0.93.0/io/bufio.ts:652:33)
    at readDelim.next (<anonymous>)
    at readStringDelim (https://deno.land/std@0.93.0/io/bufio.ts:702:20)
    at readStringDelim.next (<anonymous>)
    at readLines (https://deno.land/std@0.93.0/io/bufio.ts:711:18)
    at readLines.next (<anonymous>)
    at waitForWSEndpoint (https://deno.land/x/puppeteer@16.2.0/src/deno/BrowserRunner.ts:168:20)
    at eventLoopTick (ext:core/01_core.js:166:7)

Relevant code:

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";
import * as path from "https://deno.land/std@0.197.0/path/mod.ts";

const browser = await puppeteer.launch({
    headless: true,
    product: "chrome",
    defaultViewport: {width: 1280, height: 720},
    args: [
        //
        `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        "--mute-audio",
    ],
});

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";
import * as path from "https://deno.land/std@0.197.0/path/mod.ts";

const browser = await puppeteer.launch({
    headless: true,
    product: "chrome",
    defaultViewport: {width: 1280, height: 720},
    args: [
        //
        `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        "--mute-audio",
    ],
});

Troubleshooting steps taken: - Re-ran installation script - Cleared cache - Rebooted machine uname -a:

Linux dnarpi 6.5.0-1011-raspi #14-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 9 14:06:28 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

deno --version:

deno 1.41.1 (release, aarch64-unknown-linux-gnu) v8 12.1.285.27 typescript 5.3.3

36 Replies

cknight•10mo ago

This doesn't answer your question I'm afraid, but if you're not deeply invested in Puppeteer, you could consider using the excellent and Deno native Astral https://astral.deno.dev/

Introduction | Astral

A simple introduction to Astral

DNAOP•10mo ago

I'll take a look at that real quick, give me a few minutes

cknight•10mo ago

But my best guess as to your problem is the really old version of std being imported (like 120 versions or so old) which I'm guessing is not compatible with the latest Deno.

DNAOP•10mo ago

The problem is just, that puppeteer for deno hasnt been updated in 2 years Maybe i'd try the node version?

cknight•10mo ago

That was going to be my next suggestion. @lino-levan any thoughts?

ioB•10mo ago

Yeah, I mean, Astral will work for arm64, but you'd need to supply a browser binary yourself Everything after that will work just fine I have no idea why chrome for testing doesn't ship linux arm64 binaries In the medium term I'm adding support for Firefox but FF is even more undocumented than Chromium is when it comes to browser automation.

DNAOP•10mo ago

I just installed/downloaded chrome using npx @puppeteer/browsers install chrome@stable, but that returned me the same error

DNAOP•10mo ago

That was pretty much the only source of the chrome binary i found online

ioB•10mo ago

chrome doesn't ship an arm linux binary chromium does iirc I believe you should be able to sudo apt-get install chromium-browser or something

DNAOP•10mo ago

That worked! Thank you so much

DNAOP•10mo ago

Using Astral, how can i use request interception?

ioB•10mo ago

what are you trying to do? I'd recommend looking at the celestial bindings you can get them on a page using https://deno.land/x/astral@0.3.5/mod.ts?s=Page&p=prototype.unsafelyGetCelestialBindings.

const celestial = page.unsafelyGetCelestialBindings();

const celestial = page.unsafelyGetCelestialBindings();

that opens up access to the raw chrome devtools protocol which gives you a ton of control it's fully typed too which is quite nice

ioB•10mo ago

the docs for the chrome devtools protocol can be found https://chromedevtools.github.io/devtools-protocol/

Chrome DevTools Protocol

Chrome DevTools Protocol - version tot

ioB•10mo ago

depending on what you're trying to do, you should look at https://chromedevtools.github.io/devtools-protocol/tot/Fetch/

Chrome DevTools Protocol

Chrome DevTools Protocol - version tot - Fetch domain

ioB•10mo ago

something like

const celestial = page.unsafelyGetCelestialBindings();
await celestial.Fetch.enable();
celestial.addEventListener("Fetch.requestPaused", (event)=>{
  console.log("got event", event);
  // run await celestial.Fetch.continueRequest({ requestId }); to continue request
});

const celestial = page.unsafelyGetCelestialBindings();
await celestial.Fetch.enable();
celestial.addEventListener("Fetch.requestPaused", (event)=>{
  console.log("got event", event);
  // run await celestial.Fetch.continueRequest({ requestId }); to continue request
});

might be a good start

DNAOP•10mo ago

Code:

const videoPage = await browser.newPage();

const endTimeMin = Date.now() + 5000;
// await videoPage.setRequestInterception(true);
const celestial = page.unsafelyGetCelestialBindings();

await celestial.Fetch.enable({});

const videoPage = await browser.newPage();

const endTimeMin = Date.now() + 5000;
// await videoPage.setRequestInterception(true);
const celestial = page.unsafelyGetCelestialBindings();

await celestial.Fetch.enable({});

Error:

error: Uncaught (in promise) InvalidStateError: readyState not OPEN
    this.ws.send(JSON.stringify({
            ^
    at WebSocket.send (ext:deno_websocket/01_websocket.js:326:13)
    at Celestial.#sendReq (https://deno.land/x/astral@0.3.5/bindings/celestial.ts:13261:13)
    at Object.enable (https://deno.land/x/astral@0.3.5/bindings/celestial.ts:21634:33)
    at file:///E:/asdfasdfasdfasdf/download-account.ts:141:26
    at eventLoopTick (ext:core/01_core.js:166:7)

error: Uncaught (in promise) InvalidStateError: readyState not OPEN
    this.ws.send(JSON.stringify({
            ^
    at WebSocket.send (ext:deno_websocket/01_websocket.js:326:13)
    at Celestial.#sendReq (https://deno.land/x/astral@0.3.5/bindings/celestial.ts:13261:13)
    at Object.enable (https://deno.land/x/astral@0.3.5/bindings/celestial.ts:21634:33)
    at file:///E:/asdfasdfasdfasdf/download-account.ts:141:26
    at eventLoopTick (ext:core/01_core.js:166:7)

ioB•10mo ago

perhaps unsafelyGetCelestialBindings should wait for the websocket to be open hm

DNAOP•10mo ago

I cant use await on page.unsafelyGetCelestialBindings() however

ioB•10mo ago

okay, try adding

export async function websocketReady(ws: WebSocket) {
  await new Promise<void>((res) => {
    ws.onopen = () => {
      res();
    };
  });
}

export async function websocketReady(ws: WebSocket) {
  await new Promise<void>((res) => {
    ws.onopen = () => {
      res();
    };
  });
}

and then doing

const videoPage = await browser.newPage();

const endTimeMin = Date.now() + 5000;
// await videoPage.setRequestInterception(true);
const celestial = page.unsafelyGetCelestialBindings();
await websocketReady(celestial.ws);
await celestial.Fetch.enable({});

const videoPage = await browser.newPage();

const endTimeMin = Date.now() + 5000;
// await videoPage.setRequestInterception(true);
const celestial = page.unsafelyGetCelestialBindings();
await websocketReady(celestial.ws);
await celestial.Fetch.enable({});

this will be fixed in the next release of astral

DNAOP•10mo ago

I did implement this a bit differently than you did, but the result should be the same anyway. However, for some reason, its stuck after "done 0", basically at the Promise Code:

const celestial = videoPage.unsafelyGetCelestialBindings();
console.log("done 0");
await new Promise<void>((resolve) => (celestial.ws.onopen = () => resolve()));
console.log("done 1");
await celestial.Fetch.enable({});
console.log("done 2");

celestial.addEventListener("Fetch.requestPaused", async (event) => {
    console.log(event);
});

console.log("done 3");
await videoPage.goto(video);
console.log("done 4");

const celestial = videoPage.unsafelyGetCelestialBindings();
console.log("done 0");
await new Promise<void>((resolve) => (celestial.ws.onopen = () => resolve()));
console.log("done 1");
await celestial.Fetch.enable({});
console.log("done 2");

celestial.addEventListener("Fetch.requestPaused", async (event) => {
    console.log(event);
});

console.log("done 3");
await videoPage.goto(video);
console.log("done 4");

ioB•10mo ago

you have to do somethign with the request basically you "caught" the request, and you have to decide what to do with it

DNAOP•10mo ago

But wouldnt the promise get resolved anyway, even if the request isnt handled yet? Im not getting "deno 1", which is right after the promise

ioB•10mo ago

well by default goto waits for network activity to stop you can change that behavior by using the waitUntil option

await videoPage.goto(video, { waitUntil: "load" });

await videoPage.goto(video, { waitUntil: "load" });

or something oh wait hold on I misread your message okay well the issue is the socket opened by the time you set up the promise so it's hanging

DNAOP•10mo ago

ws.onopen is never called

ioB•10mo ago

something like

export async function websocketReady(ws: WebSocket) {
  await new Promise<void>((res) => {
    if(ws.readyState === 1) {
      res();
    } else {
      ws.onopen = () => {
        res();
      };
    }
  });
}

export async function websocketReady(ws: WebSocket) {
  await new Promise<void>((res) => {
    if(ws.readyState === 1) {
      res();
    } else {
      ws.onopen = () => {
        res();
      };
    }
  });
}

should fix it? if not I have another guess

DNAOP•10mo ago

Yeah that worked Thank you so much for helping me with my problems

ioB•10mo ago

No worries, that's what I'm here for.

DNAOP•10mo ago

I still got a small question? How do i pass the request?

celestial.addEventListener("Fetch.requestPaused", async (event) => {
    console.log(event);
});

celestial.addEventListener("Fetch.requestPaused", async (event) => {
    console.log(event);
});

ioB•10mo ago

If you always want to pass it, you can simply call

await celestial.Fetch.continueRequest({ requestId });

await celestial.Fetch.continueRequest({ requestId });

where requestId is somewhere in the event, I'd look at the console.log to see where maybe event.data.requestId? not sure

DNAOP•10mo ago

Thanks once again, this worked

celestial.addEventListener("Fetch.requestPaused", async (event) => {
    console.log(event.detail.request.url);
    celestial.Fetch.continueRequest({requestId: event.detail.requestId});
});

celestial.addEventListener("Fetch.requestPaused", async (event) => {
    console.log(event.detail.request.url);
    celestial.Fetch.continueRequest({requestId: event.detail.requestId});
});

ioB•10mo ago

👍

DNAOP•10mo ago

So i just ran into the next problem. When intercepting the requests, on windows, the request for master.m3u8 comes in and is also console.logged, but on my linux machine, the request never shows up. Does anyone have an idea, why it is like that? Also, the screenshot taken right before the while loop looks just the same on both machines Code:

const videoPage = await browser.newPage();
await videoPage.setViewportSize({width: 1920, height: 1080});
console.log("Video page prepared");

const videoPageCelestial = videoPage.unsafelyGetCelestialBindings();
await new Promise<void>((resolve) => (videoPageCelestial.ws.readyState === 1 ? resolve() : (videoPageCelestial.ws.onopen = () => resolve())));
await videoPageCelestial.Fetch.enable({});

console.log("Enabled request interception");

videoPageCelestial.addEventListener("Fetch.requestPaused", (event) => {
    console.log("\n" + event.detail.request.url);
    if (event.detail.request.url.includes("master.m3u8")) data.master = event.detail.request.url;

    if (event.detail.request.url.includes(".png") || event.detail.request.url.includes(".jpeg") || event.detail.request.url.includes(".jpg") || event.detail.request.url.includes(".css") || event.detail.request.url.includes(".ico") || event.detail.request.url.includes(".gif") || event.detail.request.url.includes("data:image") || event.detail.request.url.includes("data:font")) return videoPageCelestial.Fetch.failRequest({requestId: event.detail.requestId, errorReason: "Aborted"});
    else videoPageCelestial.Fetch.continueRequest({requestId: event.detail.requestId});
});

console.log("Going to video page: " + video);
await videoPage.goto(video);
console.log("Video page loaded, waiting for master.m3u8");

Deno.writeFileSync("screenshot.png", await videoPage.screenshot({format: "png"}));

while (!data.master) null; // <-- Stuck here on ubuntu server 23.10, proceeds after a few seconds on my main machine (windows 11) though

const videoPage = await browser.newPage();
await videoPage.setViewportSize({width: 1920, height: 1080});
console.log("Video page prepared");

const videoPageCelestial = videoPage.unsafelyGetCelestialBindings();
await new Promise<void>((resolve) => (videoPageCelestial.ws.readyState === 1 ? resolve() : (videoPageCelestial.ws.onopen = () => resolve())));
await videoPageCelestial.Fetch.enable({});

console.log("Enabled request interception");

videoPageCelestial.addEventListener("Fetch.requestPaused", (event) => {
    console.log("\n" + event.detail.request.url);
    if (event.detail.request.url.includes("master.m3u8")) data.master = event.detail.request.url;

    if (event.detail.request.url.includes(".png") || event.detail.request.url.includes(".jpeg") || event.detail.request.url.includes(".jpg") || event.detail.request.url.includes(".css") || event.detail.request.url.includes(".ico") || event.detail.request.url.includes(".gif") || event.detail.request.url.includes("data:image") || event.detail.request.url.includes("data:font")) return videoPageCelestial.Fetch.failRequest({requestId: event.detail.requestId, errorReason: "Aborted"});
    else videoPageCelestial.Fetch.continueRequest({requestId: event.detail.requestId});
});

console.log("Going to video page: " + video);
await videoPage.goto(video);
console.log("Video page loaded, waiting for master.m3u8");

Deno.writeFileSync("screenshot.png", await videoPage.screenshot({format: "png"}));

while (!data.master) null; // <-- Stuck here on ubuntu server 23.10, proceeds after a few seconds on my main machine (windows 11) though

ioB•10mo ago

Hey, sorry for getting back to you so late. I suspect this is site-to-site variance. This site might avoid loading videos for someone on linux.

DNAOP•10mo ago

Is it possible to manually set the useragent?

ioB•10mo ago

Yes, it is it's just Celestial.Emulation.setUserAgentOverride