rnbguy•16mo ago

puppeteer makes script to hang for a while at the end

I am using npm:puppeteer in my script. Everything is working fine. But often the script hangs at the end of the exit. Is this normal with puppeteer? I think that some async op is leaking. But I am not sure if I made any mistake or it is coming from puppeteer. I do call browser.close() in a finally block.

19 Replies

ioB•16mo ago

Could you share some code?

jeff.hykin•16mo ago

There's also a deno puppeteer which might fix your problems

ioB•16mo ago

there's also https://deno.land/x/astral if you just need a browser automation library

rnbguyOP•16mo ago

@lino-levan here is a code reproducing the behavior on my machine.

import puppeteer from "npm:puppeteer";

const browser = await puppeteer.launch({
  executablePath: "google-chrome-stable",
  headless: "new",
});

const page = await browser.newPage();

try {
  await page.goto("https://www.google.com");
} finally {
  page.close();
  browser.close();
}

console.log("bye")

import puppeteer from "npm:puppeteer";

const browser = await puppeteer.launch({
  executablePath: "google-chrome-stable",
  headless: "new",
});

const page = await browser.newPage();

try {
  await page.goto("https://www.google.com");
} finally {
  page.close();
  browser.close();
}

console.log("bye")

I am actually using puppeteer-extra for fortified browser. the same behavior persists.

import "npm:puppeteer";
import { default as puppeteer } from "npm:puppeteer-extra";
import StealthPlugin from "npm:puppeteer-extra-plugin-stealth";
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({
  executablePath: "google-chrome-stable",
  headless: "new",
});

const page = await browser.newPage();

try {
  await page.goto("https://www.google.com");
} finally {
  page.close();
  browser.close();
}

console.log("bye")

import "npm:puppeteer";
import { default as puppeteer } from "npm:puppeteer-extra";
import StealthPlugin from "npm:puppeteer-extra-plugin-stealth";
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({
  executablePath: "google-chrome-stable",
  headless: "new",
});

const page = await browser.newPage();

try {
  await page.goto("https://www.google.com");
} finally {
  page.close();
  browser.close();
}

console.log("bye")

@jeff.hykin I can confirm import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts"; solves the issue. 🙂 but I need to use puppeteer-extra for fortification. 😦 hey @lino-levan thanks for suggesting astral. looks like it is fortified by default. 🙂 https://bot.sannysoft.com tests are passed but astral is not customizable enough like puppeteer 😦 I need to start the browser with proxy and send page.evaluate() with function arguments. but nonetheless, it looks solid. maybe I will come back to it later 🙂

ioB•16mo ago

probably worth opening an issue! Have you tried setting the HTTP_PROXY env var? It works for me in Astral. Released a new update with function arguments to page.evaluate(). Let me know if you run into any other blockers.

rnbguyOP•16mo ago

ooh awesome! @lino-levan if you're the maintainer, let me look into it more and open issues 🙂

ioB•16mo ago

Would love feedback on the project. I think browser automation libraries should be hardened by default, no configuration neccessary.

rnbguyOP•16mo ago

btw, I will try with HTTP_PROXY, but can you please add support for argument passing for browser executable? like for chrome, I can just pass --proxy-server=${proxyUrl}. no pressure 🙂

ioB•16mo ago

This isn't portable unfortunately. I'm hoping to get FF/Safari support going pretty soon and they don't have equivalents. HTTP_PROXY seems to be portable accross both chrome and FF at least

rnbguyOP•16mo ago

I meant, passing argument to browser executable in general. I am fine with passing proxy-server argument by myself. like, this can have another field called browserArgs?: string[].

rnbguyOP•16mo ago

https://github.com/lino-levan/astral/blob/main/src/browser.ts#L45

GitHub

astral/src/browser.ts at main · lino-levan/astral

A high-level puppeteer/playwright-like library for Deno - lino-levan/astral

ioB•16mo ago

Would probably be in LaunchOptions, but that sounds more than reasonable. I'll open a PR.

ioB•16mo ago

https://github.com/lino-levan/astral/pull/7

GitHub

feat: launch args by lino-levan · Pull Request #7 · lino-levan/astr...

ioB•16mo ago

Any other immediate feedback? Would love to roll this out into a new release.

Óscar•16mo ago

I have the same issue with puppeteer. In fact I filed an issue here: https://github.com/denoland/deno/issues/20179

rnbguyOP•16mo ago

@lino-levan awesome! with your latest commit, I managed to completely move from puppeteer to astral 🍻 only thing I missed is waitUntil in Page.goto() ref. https://github.com/puppeteer/puppeteer/blob/2fee2b0adc7fd5f26c7b1d75a3efa29c2fe12dab/packages/puppeteer-core/src/common/Frame.ts#L109

GitHub

puppeteer/packages/puppeteer-core/src/common/Frame.ts at 2fee2b0adc...

Node.js API for Chrome . Contribute to puppeteer/puppeteer development by creating an account on GitHub.

rnbguyOP•16mo ago

also, I was using .$$eval before - which is not available in astral. but I worked it around by just using .evaluate. so it's not a biggie 🙂

ioB•16mo ago

I believe page.goto takes in a GoToOptions which includes a waitUntil (of type "load" | "networkidle0" | "networkidle2").

rnbguyOP•16mo ago

ah. I was getting errors because I was using "domcontentloaded". all good in that case 🙂