DNAD
Deno•2y ago•
3 replies
DNA

Capturing network requests using Puppeteer?

I want to capture any requests, which include .m3u8 in their URL, but its neither captured by the normal requestInterception nor using the createCDPSession method. In the normal chrome devtools, the master.m3u8 shows up, but my program cant catch it for some reason. I also tried using Astral, but the it also didn't capture the request.

Puppeteer version: 16.2.0 (latest)
Deno version:
> deno 1.42.1 (release, x86_64-pc-windows-msvc)
> v8 12.3.219.9
> typescript 5.4.3


Using createDCPSession:
import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
    executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
    headless: false,
    args: [
        //
        `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
        "--mute-audio",
    ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

// Log all network requests
const interception = await page.target().createCDPSession();
await interception.send("Page.enable");
await interception.send("Network.enable");

for (const event of [
    //
    "Page.domContentEventFired",
    "Page.fileChooserOpened",
    "Page.frameAttached",
    "Page.frameDetached",
    "Page.frameNavigated",
    "Page.interstitialHidden",
    "Page.interstitialShown",
    "Page.javascriptDialogClosed",
    "Page.javascriptDialogOpening",
    "Page.lifecycleEvent",
    "Page.loadEventFired",
    "Page.windowOpen",
    "Page.frameClearedScheduledNavigation",
    "Page.frameScheduledNavigation",
    "Page.compilationCacheProduced",
    "Page.downloadProgress",
    "Page.downloadWillBegin",
    "Page.frameRequestedNavigation",
    "Page.frameResized",
    "Page.frameStartedLoading",
    "Page.frameStoppedLoading",
    "Page.navigatedWithinDocument",
    "Page.screencastFrame",
    "Page.screencastVisibilityChanged",
    "Network.dataReceived",
    "Network.eventSourceMessageReceived",
    "Network.loadingFailed",
    "Network.loadingFinished",
    "Network.requestServedFromCache",
    "Network.requestWillBeSent",
    "Network.responseReceived",
    "Network.webSocketClosed",
    "Network.webSocketCreated",
    "Network.webSocketFrameError",
    "Network.webSocketFrameReceived",
    "Network.webSocketFrameSent",
    "Network.webSocketHandshakeResponseReceived",
    "Network.webSocketWillSendHandshakeRequest",
    "Network.requestWillBeSentExtraInfo",
    "Network.resourceChangedPriority",
    "Network.responseReceivedExtraInfo",
    "Network.signedExchangeReceived",
    "Network.requestIntercepted",
])
    interception.on(event, (event) => console.log(event));

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);


Using normal requestInterception:
import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts";

const browser = await puppeteer.launch({
      executablePath: "C:/Program Files/Google/Chrome/Application/chrome.exe",
      headless: false,
      args: [
            //
            `--disable-extensions-except=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            `--load-extension=${path.join(Deno.cwd(), "extensions", "cjpalhdlnbpafiamejdnhcphjbkeiagm", "1.50.0_0")}`,
            "--mute-audio",
      ],
});

const browserExtensionWaitPromise = new Promise((resolve) => setTimeout(resolve, 5000));

await browserExtensionWaitPromise;
const page = await browser.newPage();

await page.setRequestInterception(true);
page.on("request", (request) => {
      console.log(request.url());
      request.continue();
});

await page.goto(totalEpisodeUrls[data.season - 1][data.episode - 1].split("--")[0]);
image.png
image.png
Was this page helpful?