rnbguy
rnbguyβ€’11mo ago

puppeteer makes script to hang for a while at the end

I am using npm:puppeteer in my script. Everything is working fine. But often the script hangs at the end of the exit. Is this normal with puppeteer? I think that some async op is leaking. But I am not sure if I made any mistake or it is coming from puppeteer. I do call browser.close() in a finally block.
19 Replies
ioB
ioBβ€’11mo ago
Could you share some code?
jeff.hykin
jeff.hykinβ€’11mo ago
There's also a deno puppeteer which might fix your problems
ioB
ioBβ€’11mo ago
there's also https://deno.land/x/astral if you just need a browser automation library
rnbguy
rnbguyβ€’11mo ago
@lino-levan here is a code reproducing the behavior on my machine.
import puppeteer from "npm:puppeteer";

const browser = await puppeteer.launch({
executablePath: "google-chrome-stable",
headless: "new",
});

const page = await browser.newPage();

try {
await page.goto("https://www.google.com");
} finally {
page.close();
browser.close();
}

console.log("bye")
import puppeteer from "npm:puppeteer";

const browser = await puppeteer.launch({
executablePath: "google-chrome-stable",
headless: "new",
});

const page = await browser.newPage();

try {
await page.goto("https://www.google.com");
} finally {
page.close();
browser.close();
}

console.log("bye")
I am actually using puppeteer-extra for fortified browser. the same behavior persists.
import "npm:puppeteer";
import { default as puppeteer } from "npm:puppeteer-extra";
import StealthPlugin from "npm:puppeteer-extra-plugin-stealth";
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({
executablePath: "google-chrome-stable",
headless: "new",
});

const page = await browser.newPage();

try {
await page.goto("https://www.google.com");
} finally {
page.close();
browser.close();
}

console.log("bye")
import "npm:puppeteer";
import { default as puppeteer } from "npm:puppeteer-extra";
import StealthPlugin from "npm:puppeteer-extra-plugin-stealth";
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({
executablePath: "google-chrome-stable",
headless: "new",
});

const page = await browser.newPage();

try {
await page.goto("https://www.google.com");
} finally {
page.close();
browser.close();
}

console.log("bye")
@jeff.hykin I can confirm import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts"; solves the issue. πŸ™‚ but I need to use puppeteer-extra for fortification. 😦 hey @lino-levan thanks for suggesting astral. looks like it is fortified by default. πŸ™‚ https://bot.sannysoft.com tests are passed but astral is not customizable enough like puppeteer 😦 I need to start the browser with proxy and send page.evaluate() with function arguments. but nonetheless, it looks solid. maybe I will come back to it later πŸ™‚
ioB
ioBβ€’11mo ago
probably worth opening an issue! Have you tried setting the HTTP_PROXY env var? It works for me in Astral. Released a new update with function arguments to page.evaluate(). Let me know if you run into any other blockers.
rnbguy
rnbguyβ€’11mo ago
ooh awesome! @lino-levan if you're the maintainer, let me look into it more and open issues πŸ™‚
ioB
ioBβ€’11mo ago
Would love feedback on the project. I think browser automation libraries should be hardened by default, no configuration neccessary.
rnbguy
rnbguyβ€’11mo ago
btw, I will try with HTTP_PROXY, but can you please add support for argument passing for browser executable? like for chrome, I can just pass --proxy-server=${proxyUrl}. no pressure πŸ™‚
ioB
ioBβ€’11mo ago
This isn't portable unfortunately. I'm hoping to get FF/Safari support going pretty soon and they don't have equivalents. HTTP_PROXY seems to be portable accross both chrome and FF at least
rnbguy
rnbguyβ€’11mo ago
I meant, passing argument to browser executable in general. I am fine with passing proxy-server argument by myself. like, this can have another field called browserArgs?: string[].
rnbguy
rnbguyβ€’11mo ago
GitHub
astral/src/browser.ts at main Β· lino-levan/astral
A high-level puppeteer/playwright-like library for Deno - lino-levan/astral
ioB
ioBβ€’11mo ago
Would probably be in LaunchOptions, but that sounds more than reasonable. I'll open a PR.
ioB
ioBβ€’11mo ago
Any other immediate feedback? Would love to roll this out into a new release.
Γ“scar
Γ“scarβ€’11mo ago
I have the same issue with puppeteer. In fact I filed an issue here: https://github.com/denoland/deno/issues/20179
rnbguy
rnbguyβ€’11mo ago
@lino-levan awesome! with your latest commit, I managed to completely move from puppeteer to astral 🍻 only thing I missed is waitUntil in Page.goto() ref. https://github.com/puppeteer/puppeteer/blob/2fee2b0adc7fd5f26c7b1d75a3efa29c2fe12dab/packages/puppeteer-core/src/common/Frame.ts#L109
GitHub
puppeteer/packages/puppeteer-core/src/common/Frame.ts at 2fee2b0adc...
Node.js API for Chrome . Contribute to puppeteer/puppeteer development by creating an account on GitHub.
rnbguy
rnbguyβ€’11mo ago
also, I was using .$$eval before - which is not available in astral. but I worked it around by just using .evaluate. so it's not a biggie πŸ™‚
ioB
ioBβ€’11mo ago
I believe page.goto takes in a GoToOptions which includes a waitUntil (of type "load" | "networkidle0" | "networkidle2").
rnbguy
rnbguyβ€’11mo ago
ah. I was getting errors because I was using "domcontentloaded". all good in that case πŸ™‚