Cut proxy bandwidth in a headless browser

Short answers

What a page pulls in, and what you actually need

Extraction almost always needs the document and the XHR or fetch responses behind it. Everything else exists for a human to look at. Blocking is not an optimisation trick: it is declining to download things you were never going to read.

A second filter earns as much as the first: allow the site’s own hosts and drop the rest. Advertising, consent platforms, session replay and tag managers are frequently more traffic than the article they sit around.

Playwright: one route handler

First establish a passing journey with the Playwright proxy setup guide. Verify the route using a page in the configured browser context, then add filtering and rerun the same assertions so the saving does not hide a broken test.

import { chromium } from 'playwright';

const BLOCK_TYPES = new Set(['image', 'media', 'font', 'stylesheet']);
const ALLOW_HOSTS = [/(^|\.)example\.com$/];

const browser = await chromium.launch();
const context = await browser.newContext({
  proxy: {
    server: 'http://gw.portproof.org:7000',
    username: 'USERNAME-peer-us-rot-ondemand',
    password: process.env.PROXY_PASSWORD,
  },
});

await context.route('**/*', (route) => {
  const request = route.request();
  const host = new URL(request.url()).hostname;
  const thirdParty = !ALLOW_HOSTS.some((re) => re.test(host));
  if (BLOCK_TYPES.has(request.resourceType()) || thirdParty) return route.abort();
  return route.continue();
});

const page = await context.newPage();
await page.goto('https://example.com/', { waitUntil: 'domcontentloaded' });
console.log((await page.content()).length, 'characters of HTML');
await browser.close();

Two details matter more than the filter itself. Wait for domcontentloaded rather than networkidle, because idle never arrives on a page whose analytics you just aborted. And reuse one context per session instead of launching a browser per page, so TLS handshakes are not paid twice.

Puppeteer: interception with the same rules

Start with the Puppeteer proxy setup guide to check the connection and authentication scope before adding resource filtering.

import puppeteer from 'puppeteer';

const BLOCK_TYPES = new Set(['image', 'media', 'font', 'stylesheet']);

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://gw.portproof.org:7000', '--blink-settings=imagesEnabled=false'],
});
const page = await browser.newPage();
await page.authenticate({
  username: 'USERNAME-peer-us-rot-ondemand',
  password: process.env.PROXY_PASSWORD,
});

await page.setRequestInterception(true);
page.on('request', (request) => {
  if (BLOCK_TYPES.has(request.resourceType())) return request.abort();
  return request.continue();
});

await page.goto('https://example.com/', { waitUntil: 'domcontentloaded' });
await browser.close();

What breaks, and how to notice

• Client-rendered pages go blank when scripts are blocked. Check for the text you expect before trusting an empty result.
• Consent walls sometimes live in a third-party script: blocking it can leave a page that never reveals its content, or can skip the wall entirely. Both happen; test the site rather than assuming.
• Screenshots stop being representative once images, fonts and stylesheets are gone. Keep a small unfiltered run for anything visual, including ad verification work.
• Infinite scroll and lazy loading usually survive image blocking, because the trigger is the placeholder, not the picture.

Measure the saving, do not estimate it

const cdp = await context.newCDPSession(page);
await cdp.send('Network.enable');

let bytes = 0;
cdp.on('Network.loadingFinished', (event) => {
  bytes += event.encodedDataLength;
});

await page.goto('https://example.com/', { waitUntil: 'domcontentloaded' });
console.log(Math.round(bytes / 1024), 'kB on the wire for this page');

Then confirm against the bill. Read gb_used from GET /v1/traffic before and after a longer run and take the difference; usage refreshes every five minutes, so leave a window either side. What the meter includes is explained in what counts as proxy traffic. Once the changed workload still returns the required data, recalculate your bandwidth estimate using the measured sample.

From kilobytes to GB and euros

The arithmetic is worth doing once. A hundred thousand pages at 2.2 MB each is about 220 GB. The same run at 260 kB is about 26 GB. That is the same job, the same pages and the same data, at a different order of magnitude on the invoice.

Sizing an order from page counts, page weight and retries is worked through in how many GB do you need.

When the right answer is no browser

Open the site once with the network panel on and look at what it calls. If the data arrives as JSON, call that endpoint directly with an HTTP client for the rest of the run: no rendering, no assets, a fraction of the bytes, and far less to go wrong.

Keep the browser for what needs a browser: rendered advertising, geo-dependent layouts, QA journeys. Which pool to point it at is in mobile 4G/5G against residential, and the per-GB ladder is on pricing, with the connection format on the setup docs.

What is not allowed

What is not allowed: blocking assets to save your own traffic is fine. Using a browser farm to hammer a site, ignore its stated limits, evade a block or create accounts is not, and nothing on the declined list becomes acceptable because it is cheaper. See the acceptable-use policy.

Read next