How to capture network response body in Playwright without blocking navigation

by

Playwright blocks navigation when a page.on("response") handler or route.fetch() call awaits response.body() inline with the navigation promise, because body() resolves only after the full payload is received and page.goto() can wait on the same response lifecycle.

Root Cause Diagnosis

response.body() returns a promise that settles only after the response stream finishes. If you await it inside a route handler, the request is held open (the browser never receives the fulfilled response), so page.goto() stalls until load timeout. Inside a response event handler, an await does not block navigation by itself, but errors thrown there (for example, Protocol error (Network.getResponseBody): No resource with given identifier found after a redirect or navigation) are unhandled and often surface as hangs or flaky failures. Capturing the body must be decoupled from the navigation path.

Quick Fix

Register a response listener, store the promise without awaiting it in the handler, and resolve it after goto() completes:

const bodies: Promise<Buffer | null>[] = [];

page.on("response", (response) => {
  if (response.url().includes("/api/data")) {
    bodies.push(response.body().catch(() => null));
  }
});

await page.goto("https://example.com", { waitUntil: "domcontentloaded" });
const results = await Promise.all(bodies);

Step-by-Step Resolution

1. Prefer waitForResponse for a single known request. Start the waiter before triggering navigation, but do not await it until after goto():

const respPromise = page.waitForResponse(
  (r) => r.url().includes("/api/data") && r.status() === 200
);

await page.goto("https://example.com", { waitUntil: "domcontentloaded" });

const response = await respPromise;
const json = await response.json();

2. Use Promise.all when the response is triggered by the navigation itself.

const [response] = await Promise.all([
  page.waitForResponse((r) => r.url().includes("/api/data")),
  page.goto("https://example.com", { waitUntil: "commit" }),
]);
const body = await response.text();

3. If you must intercept via route, use route.fetch() and fulfill immediately. Read the body from the fetched response, then fulfill with the same object so the browser is not left waiting:

await page.route("**/api/data", async (route) => {
  const response = await route.fetch();
  const body = await response.body();
  captured.push(body);
  await route.fulfill({ response, body });
});

Keep any heavy processing (parsing, disk writes) out of this handler; push to a queue and process after fulfillment.

See also  Python Requests SSLError: UNEXPECTED_EOF_WHILE_READING on specific HTTPS hosts

4. Choose a lighter waitUntil value. The default "load" waits for all subresources. Use "domcontentloaded" or "commit" when you only need the document and are collecting API responses separately.

await page.goto(url, { waitUntil: "commit", timeout: 30000 });

5. Filter listeners narrowly. Match on URL, method, and resource type to avoid calling body() on every response:

page.on("response", (r) => {
  const req = r.request();
  if (req.resourceType() === "fetch" || req.resourceType() === "xhr") {
    if (r.url().includes("/api/")) {
      bodies.push(r.body().catch(() => null));
    }
  }
});

6. Remove listeners when finished to prevent accumulation across navigations:

const handler = (r) => { /* ... */ };
page.on("response", handler);
// ...
page.off("response", handler);

Why This Fails for Some Users

  • Redirects (301/302/307/308): Redirect responses have no body. Calling body() throws Response body is unavailable for redirect responses. Guard with response.status() >= 300 && response.status() < 400.
  • Navigation discarding the body: If the page navigates away before body() is called, Chromium may evict the resource buffer and throw No resource with given identifier found. Call body() immediately in the listener (without awaiting) rather than after a delay.
  • Streaming / SSE / chunked responses: body() never resolves for responses that stay open (Server-Sent Events, long-polling, WebSocket upgrade). Exclude them by content-type (text/event-stream) or use page.on("websocket") instead.
  • Service workers: Responses served by a service worker are not visible to page.on("response") by default. Set serviceWorkers: "block" in the context options:const context = await browser.newContext({ serviceWorkers: "block" });
  • Cached responses: Responses served from memory or disk cache may lack a retrievable body in some Chromium versions. Disable cache for debugging via context.route with route.fetch(), or launch with --disable-cache.
  • Browser differences: Firefox and WebKit handle body() timing differently from Chromium; large bodies in WebKit can raise Request content was evicted from inspector cache. Capture early and keep buffers small.
  • Large payloads and memory limits: Capturing multi-MB bodies for every response in a long session can exhaust Node heap (--max-old-space-size). Stream to disk or filter by content-length.
  • HAR as an alternative: If you need all bodies without custom handlers, use recordHar and read the HAR afterward:const context = await browser.newContext({ recordHar: { path: "network.har", content: "embed", urlFilter: "**/api/**" }, }); // ... await context.close(); // HAR is flushed on close

Leave a Comment