Deno•2y ago

Capturing network requests using Puppeteer?

I want to capture any requests, which include

.m3u8

.m3u8

in their URL, but its neither captured by the normal requestInterception nor using the

createCDPSession

createCDPSession

method. In the normal chrome devtools, the

master.m3u8

master.m3u8

shows up, but my program cant catch it for some reason. I also tried using Astral, but the it also didn't capture the request.

Puppeteer version: 16.2.0 (latest)
Deno version:
> deno 1.42.1 (release, x86_64-pc-windows-msvc)
> v8 12.3.219.9
> typescript 5.4.3

Puppeteer version: 16.2.0 (latest)
Deno version:
> deno 1.42.1 (release, x86_64-pc-windows-msvc)
> v8 12.3.219.9
> typescript 5.4.3

Using

createDCPSession

createDCPSession

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
    executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
    headless: false,
    args: [
        //
        `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        "--mute-audio",
    ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

// Log all network requests
const interception = await page.target().createCDPSession();
await interception.send("Page.enable");
await interception.send("Network.enable");

for (const event of [
    //
    "Page.domContentEventFired",
    "Page.fileChooserOpened",
    "Page.frameAttached",
    "Page.frameDetached",
    "Page.frameNavigated",
    "Page.interstitialHidden",
    "Page.interstitialShown",
    "Page.javascriptDialogClosed",
    "Page.javascriptDialogOpening",
    "Page.lifecycleEvent",
    "Page.loadEventFired",
    "Page.windowOpen",
    "Page.frameClearedScheduledNavigation",
    "Page.frameScheduledNavigation",
    "Page.compilationCacheProduced",
    "Page.downloadProgress",
    "Page.downloadWillBegin",
    "Page.frameRequestedNavigation",
    "Page.frameResized",
    "Page.frameStartedLoading",
    "Page.frameStoppedLoading",
    "Page.navigatedWithinDocument",
    "Page.screencastFrame",
    "Page.screencastVisibilityChanged",
    "Network.dataReceived",
    "Network.eventSourceMessageReceived",
    "Network.loadingFailed",
    "Network.loadingFinished",
    "Network.requestServedFromCache",
    "Network.requestWillBeSent",
    "Network.responseReceived",
    "Network.webSocketClosed",
    "Network.webSocketCreated",
    "Network.webSocketFrameError",
    "Network.webSocketFrameReceived",
    "Network.webSocketFrameSent",
    "Network.webSocketHandshakeResponseReceived",
    "Network.webSocketWillSendHandshakeRequest",
    "Network.requestWillBeSentExtraInfo",
    "Network.resourceChangedPriority",
    "Network.responseReceivedExtraInfo",
    "Network.signedExchangeReceived",
    "Network.requestIntercepted",
])
    interception.on(event, (event) => console.log(event));

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
    executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
    headless: false,
    args: [
        //
        `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        "--mute-audio",
    ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

// Log all network requests
const interception = await page.target().createCDPSession();
await interception.send("Page.enable");
await interception.send("Network.enable");

for (const event of [
    //
    "Page.domContentEventFired",
    "Page.fileChooserOpened",
    "Page.frameAttached",
    "Page.frameDetached",
    "Page.frameNavigated",
    "Page.interstitialHidden",
    "Page.interstitialShown",
    "Page.javascriptDialogClosed",
    "Page.javascriptDialogOpening",
    "Page.lifecycleEvent",
    "Page.loadEventFired",
    "Page.windowOpen",
    "Page.frameClearedScheduledNavigation",
    "Page.frameScheduledNavigation",
    "Page.compilationCacheProduced",
    "Page.downloadProgress",
    "Page.downloadWillBegin",
    "Page.frameRequestedNavigation",
    "Page.frameResized",
    "Page.frameStartedLoading",
    "Page.frameStoppedLoading",
    "Page.navigatedWithinDocument",
    "Page.screencastFrame",
    "Page.screencastVisibilityChanged",
    "Network.dataReceived",
    "Network.eventSourceMessageReceived",
    "Network.loadingFailed",
    "Network.loadingFinished",
    "Network.requestServedFromCache",
    "Network.requestWillBeSent",
    "Network.responseReceived",
    "Network.webSocketClosed",
    "Network.webSocketCreated",
    "Network.webSocketFrameError",
    "Network.webSocketFrameReceived",
    "Network.webSocketFrameSent",
    "Network.webSocketHandshakeResponseReceived",
    "Network.webSocketWillSendHandshakeRequest",
    "Network.requestWillBeSentExtraInfo",
    "Network.resourceChangedPriority",
    "Network.responseReceivedExtraInfo",
    "Network.signedExchangeReceived",
    "Network.requestIntercepted",
])
    interception.on(event, (event) => console.log(event));

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

Using normal requestInterception:

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
      executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
      headless: false,
      args: [
            //
            `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            "--mute-audio",
      ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

await page.setRequestInterception(true);
page.on("request", (request) => {
      console.log(request.url());
      request.continue();
});

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
      executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
      headless: false,
      args: [
            //
            `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            "--mute-audio",
      ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

await page.setRequestInterception(true);
page.on("request", (request) => {
      console.log(request.url());
      request.continue();
});

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);

Deno AI Helper•4/8/24, 4:16 PM

I wasn't able to find the answer to your question in the official Deno documentation. However, based on the code snippets you've provided, I can offer some general advice, but please note that this advice is not based on the Deno documentation, and you should double-check the accuracy of the advice given.

In your

createCDPSession

createCDPSession

code, you are using a lot of events, but not all of them are related to network requests. If you are specifically looking to capture network requests that include

.m3u8

.m3u8

in their URL, you should focus on the event, which is triggered when a network request is about to be sent. You can then inspect the

request

request

object to check if the URL contains

.m3u8

.m3u8

.

Here's a simplified version of the event listener that you might use:

For the normal request interception code, your approach seems correct. You are setting up request interception and logging the URL of each request. If

.m3u8

.m3u8

requests are not being captured, it could be due to several reasons, such as the requests being made by a service worker, or the requests not being initiated by the main page.

Please ensure that the is set before any network activity occurs, as this can sometimes cause issues if set too late.

Remember to verify the paths you are using for the and arguments, as incorrect paths could cause unexpected behavior.

Since this is not an exhaustive solution and is not based on the Deno documentation, I recommend further investigation and testing to pinpoint the issue. You may also want to

Deno AI Helper•4/8/24, 4:16 PM

consult the Puppeteer documentation or community for more specific guidance on capturing network requests with

.m3u8

.m3u8

URLs.

Helpful Links:
Node API Compatibility List | Deno Docs
HTTP Server APIs | Deno Docs
A list of every web API in Deno
Deno 1.7 Release Notes

DNAOP•4/9/24, 5:37 PM

Also, if that matters, the master.m3u8 is requested from inside an iframe