Aaditya Mathur
Aaditya Mathur5mo ago

Getting errors while using puppeteer in deno environment

Here is my code(supabase edge function index.ts):
import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

Deno.serve(async (req) => {
try {
const { url } = await req.json();

const scrape = async (url: string) => {
const browser = await puppeteer.launch();
const page = await browser.newPage();

await page.goto(url);

const title = await page.title();

const emails = await page.evaluate(() => {
const emailRegex =
/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/g;
const bodyText = document.body.innerText;
return bodyText.match(emailRegex);
});

const divData = await page.evaluate(() => {
const divs = Array.from(document.querySelectorAll("div"));

const extractDataFromDiv = (div: HTMLDivElement) => {
const divContent = div.innerText;

const splitContent = divContent
.split("\n")
.map((content) => content.trim())
.filter((content) => content !== "" && content.trim() !== "");

return splitContent;
};

return divs.map(extractDataFromDiv);
});

await browser.close();

return {
title,
emails,
content: divData,
};
};

const data = await scrape(url);

return new Response(
JSON.stringify(data),
{ headers: { "Content-Type": "application/json" } },
);
} catch (error) {
return new Response(`Error: ${error}`, { status: 500 });
}
});
import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

Deno.serve(async (req) => {
try {
const { url } = await req.json();

const scrape = async (url: string) => {
const browser = await puppeteer.launch();
const page = await browser.newPage();

await page.goto(url);

const title = await page.title();

const emails = await page.evaluate(() => {
const emailRegex =
/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/g;
const bodyText = document.body.innerText;
return bodyText.match(emailRegex);
});

const divData = await page.evaluate(() => {
const divs = Array.from(document.querySelectorAll("div"));

const extractDataFromDiv = (div: HTMLDivElement) => {
const divContent = div.innerText;

const splitContent = divContent
.split("\n")
.map((content) => content.trim())
.filter((content) => content !== "" && content.trim() !== "");

return splitContent;
};

return divs.map(extractDataFromDiv);
});

await browser.close();

return {
title,
emails,
content: divData,
};
};

const data = await scrape(url);

return new Response(
JSON.stringify(data),
{ headers: { "Content-Type": "application/json" } },
);
} catch (error) {
return new Response(`Error: ${error}`, { status: 500 });
}
});
@everyone please help me with this issue
No description
3 Replies
Deno AI Helper
Deno AI Helper5mo ago
It looks like you're trying to use Puppeteer within a Deno environment. According to the documentation, Puppeteer is one of the npm packages that now work with Deno thanks to recent improvements to compatibility with Node.js modules. 🦕 However, if you're encountering errors, there are a few steps you can take to troubleshoot the issue: 1. Check the Error Message: The error message you receive can often give you a clue about what's going wrong. Make sure to read it carefully. 2. Verify Puppeteer Compatibility: Ensure that the version of Puppeteer you are using is compatible with Deno. As per the documentation, improvements have been made, but it's always good to check the latest compatibility status. 3. Permissions: Puppeteer requires certain permissions to run, such as access to the network and the ability to spawn subprocesses. Make sure you're granting the necessary permissions when you start your Deno script. For example:
deno run --allow-net --allow-read --allow-write --allow-run index.ts

deno run --allow-net --allow-read --allow-write --allow-run index.ts

Adjust the permissions according to your needs and security considerations. 4. Resource Limits: If you're running this in a serverless environment like Supabase Edge Functions, be aware of the resource limits (memory, CPU, execution time) that might be in place. Puppeteer can be resource-intensive, and if the environment doesn't provide enough resources, it might lead to errors. 5. Debugging: Add more console.log statements to your code to get a better understanding of where it fails. This can help you pinpoint the exact line or operation that's causing the issue. 6. Deno Version: Ensure you are using a recent version of Deno that includes the latest fixes and improvements for npm compatibility. 7. Puppeteer Version:
Deno AI Helper
Deno AI Helper5mo ago
Check if the version of Puppeteer you are using is the one that's known to work with Deno. Sometimes, using a different version can resolve compatibility issues. 8. Check for Open Issues: Look at the Deno repository on GitHub to see if there are any open issues related to Puppeteer usage that might be relevant to your problem. If you've gone through these steps and are still facing issues, you might want to provide the specific error message you're encountering so that the community can offer more targeted assistance. Remember to keep any sensitive information private when sharing error logs or code snippets.
Leokuma
Leokuma5mo ago
I guess Deno Puppeteer is not generally compatible with serverless environments, meaning it's a stateful module Also make sure that it is legal to run Pupetteer in Supabase or Deploy for what you're doing. Some providers prohibit certain types of bots There's already a similar issue: https://github.com/lucacasonato/deno-puppeteer/issues/91
GitHub
Example to deploy in Deno · Issue #91 · lucacasonato/deno-puppeteer
Can you append a example to deploy en Deno, try deploy but, i recive this message on console