DNA•4mo ago

Capturing network requests using Puppeteer?

I want to capture any requests, which include .m3u8 in their URL, but its neither captured by the normal requestInterception nor using the createCDPSession method. In the normal chrome devtools, the master.m3u8 shows up, but my program cant catch it for some reason. I also tried using Astral, but the it also didn't capture the request.

Puppeteer version: 16.2.0 (latest)
Deno version:
> deno 1.42.1 (release, x86_64-pc-windows-msvc)
> v8 12.3.219.9
> typescript 5.4.3

Puppeteer version: 16.2.0 (latest)
Deno version:
> deno 1.42.1 (release, x86_64-pc-windows-msvc)
> v8 12.3.219.9
> typescript 5.4.3

Using createDCPSession:

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
    executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
    headless: false,
    args: [
        //
        `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        "--mute-audio",
    ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

// Log all network requests
const interception = await page.target().createCDPSession();
await interception.send("Page.enable");
await interception.send("Network.enable");

for (const event of [
    //
    "Page.domContentEventFired",
    "Page.fileChooserOpened",
    "Page.frameAttached",
    "Page.frameDetached",
    "Page.frameNavigated",
    "Page.interstitialHidden",
    "Page.interstitialShown",
    "Page.javascriptDialogClosed",
    "Page.javascriptDialogOpening",
    "Page.lifecycleEvent",
    "Page.loadEventFired",
    "Page.windowOpen",
    "Page.frameClearedScheduledNavigation",
    "Page.frameScheduledNavigation",
    "Page.compilationCacheProduced",
    "Page.downloadProgress",
    "Page.downloadWillBegin",
    "Page.frameRequestedNavigation",
    "Page.frameResized",
    "Page.frameStartedLoading",
    "Page.frameStoppedLoading",
    "Page.navigatedWithinDocument",
    "Page.screencastFrame",
    "Page.screencastVisibilityChanged",
    "Network.dataReceived",
    "Network.eventSourceMessageReceived",
    "Network.loadingFailed",
    "Network.loadingFinished",
    "Network.requestServedFromCache",
    "Network.requestWillBeSent",
    "Network.responseReceived",
    "Network.webSocketClosed",
    "Network.webSocketCreated",
    "Network.webSocketFrameError",
    "Network.webSocketFrameReceived",
    "Network.webSocketFrameSent",
    "Network.webSocketHandshakeResponseReceived",
    "Network.webSocketWillSendHandshakeRequest",
    "Network.requestWillBeSentExtraInfo",
    "Network.resourceChangedPriority",
    "Network.responseReceivedExtraInfo",
    "Network.signedExchangeReceived",
    "Network.requestIntercepted",
])
    interception.on(event, (event) => console.log(event));

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
    executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
    headless: false,
    args: [
        //
        `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        "--mute-audio",
    ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

// Log all network requests
const interception = await page.target().createCDPSession();
await interception.send("Page.enable");
await interception.send("Network.enable");

for (const event of [
    //
    "Page.domContentEventFired",
    "Page.fileChooserOpened",
    "Page.frameAttached",
    "Page.frameDetached",
    "Page.frameNavigated",
    "Page.interstitialHidden",
    "Page.interstitialShown",
    "Page.javascriptDialogClosed",
    "Page.javascriptDialogOpening",
    "Page.lifecycleEvent",
    "Page.loadEventFired",
    "Page.windowOpen",
    "Page.frameClearedScheduledNavigation",
    "Page.frameScheduledNavigation",
    "Page.compilationCacheProduced",
    "Page.downloadProgress",
    "Page.downloadWillBegin",
    "Page.frameRequestedNavigation",
    "Page.frameResized",
    "Page.frameStartedLoading",
    "Page.frameStoppedLoading",
    "Page.navigatedWithinDocument",
    "Page.screencastFrame",
    "Page.screencastVisibilityChanged",
    "Network.dataReceived",
    "Network.eventSourceMessageReceived",
    "Network.loadingFailed",
    "Network.loadingFinished",
    "Network.requestServedFromCache",
    "Network.requestWillBeSent",
    "Network.responseReceived",
    "Network.webSocketClosed",
    "Network.webSocketCreated",
    "Network.webSocketFrameError",
    "Network.webSocketFrameReceived",
    "Network.webSocketFrameSent",
    "Network.webSocketHandshakeResponseReceived",
    "Network.webSocketWillSendHandshakeRequest",
    "Network.requestWillBeSentExtraInfo",
    "Network.resourceChangedPriority",
    "Network.responseReceivedExtraInfo",
    "Network.signedExchangeReceived",
    "Network.requestIntercepted",
])
    interception.on(event, (event) => console.log(event));

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

Using normal requestInterception:

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
      executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
      headless: false,
      args: [
            //
            `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            "--mute-audio",
      ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

await page.setRequestInterception(true);
page.on("request", (request) => {
      console.log(request.url());
      request.continue();
});

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
      executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
      headless: false,
      args: [
            //
            `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            "--mute-audio",
      ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

await page.setRequestInterception(true);
page.on("request", (request) => {
      console.log(request.url());
      request.continue();
});

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

3 Replies

Deno AI Helper•4mo ago

I wasn't able to find the answer to your question in the official Deno documentation. However, based on the code snippets you've provided, I can offer some general advice, but please note that this advice is not based on the Deno documentation, and you should double-check the accuracy of the advice given. In your createCDPSession code, you are using a lot of events, but not all of them are related to network requests. If you are specifically looking to capture network requests that include .m3u8 in their URL, you should focus on the Network.requestWillBeSent event, which is triggered when a network request is about to be sent. You can then inspect the request object to check if the URL contains .m3u8. Here's a simplified version of the event listener that you might use:

interception.on("Network.requestWillBeSent", (event) => {
    if (event.request.url.includes('.m3u8')) {
        console.log(event.request.url);
    }
});

interception.on("Network.requestWillBeSent", (event) => {
    if (event.request.url.includes('.m3u8')) {
        console.log(event.request.url);
    }
});

For the normal request interception code, your approach seems correct. You are setting up request interception and logging the URL of each request. If .m3u8 requests are not being captured, it could be due to several reasons, such as the requests being made by a service worker, or the requests not being initiated by the main page. Please ensure that the page.setRequestInterception(true); is set before any network activity occurs, as this can sometimes cause issues if set too late. Remember to verify the paths you are using for the --disable-extensions-except and --load-extension arguments, as incorrect paths could cause unexpected behavior. Since this is not an exhaustive solution and is not based on the Deno documentation, I recommend further investigation and testing to pinpoint the issue. You may also want to

Deno AI Helper•4mo ago

consult the Puppeteer documentation or community for more specific guidance on capturing network requests with .m3u8 URLs.

Helpful Links: Node API Compatibility List | Deno Docs HTTP Server APIs | Deno Docs A list of every web API in Deno Deno 1.7 Release Notes

DNA•4mo ago

Also, if that matters, the master.m3u8 is requested from inside an iframe