Choosing proxies for web scraping

Short answers

Write the record you need before choosing a proxy

Start with a permitted source and a concrete output: a catalogue entry with its market and capture time, a comparable product offer, or a QA result from your own page. Prefer an authorised feed or API when it already supplies that output. Define missing fields and duplicate handling before running the collector.

An HTTP 200 can contain an empty template or the wrong product variant. Validate content before counting a record. Preserve missing values rather than turning them into zero, and keep the collection timestamp separate from any date printed by the source.

Choose ordinary HTTP or a browser from the source

Inspect the permitted response first. If it contains the fields you need, an HTTP client can fetch them without loading a whole browser page. Scrapy’s dynamic-content guidance starts with locating the data source; a browser becomes useful when the job needs rendered content, interaction or a screenshot.

Use the Scrapy setup guide for a crawler or the HTTPX guide for an HTTP application. For rendered checks, use the Playwright guide. Test in the client that will run the job. An echo check confirms that request’s connection, not the collector’s parsing or every later browser request.

A browser may load scripts, images and background requests. Remove a resource only after checking that the accepted output stays correct. Keep certificate verification enabled, protect proxy credentials, and close responses and browser contexts when finished.

Match the proxy session to the observation

Independent observations can use per-connection rotation. Connection reuse means a new application request need not open a new proxy connection. A multi-step observation may need a sticky session alongside its own cookies and application state. Sticky keeps the selected device while available; its IP can still change or the device can disconnect.

Country routing, storefront settings and browser locale are separate inputs. Keep them consistent when comparing samples. Record requested settings without treating them as proof of the page’s actual market. The session guide explains the connection behaviour; price monitoring covers offer comparability.

Set limits that stop an unproductive run

Begin with low concurrency within the source’s permitted limits. Set a whole-task deadline, response-size ceiling and finite attempt count. For an initial diagnostic, disable retries so one failure remains one observation. Resolve proxy authentication, certificate or parser errors before repeating the sample.

A 429 response means too many requests and may include Retry-After. Honour the delay, reduce load, and stop repeated failures. Retry-After can specify seconds or an HTTP date. If the delay exceeds the remaining job deadline, defer the task instead of waiting indefinitely.

Retry only operations you know are safe to repeat; HTTP semantics distinguish idempotent requests from operations that may create another effect. Link retries to the original task and deduplicate saved output. A target refusal calls for review, not more addresses or more workers.

Measure traffic per task and per usable record

The following numbers are hypothetical arithmetic, not a Portproof benchmark. Suppose a sample plans one record per task. Its traffic total includes initial attempts, retries and failures, measured consistently across the whole sample.

At the same mix, 10,000 initial tasks would project 6 GB and about 9,000 usable records. That projection does not promise either outcome. If the requirement is 10,000 usable records, this schedule falls short: investigate missing results and resample before buying more traffic. If usable records are zero, the per-record figure is undefined.

Retries are already included here, so another retry multiplier would count them twice. Use the bandwidth calculator with consistent units, and choose headroom from observed variation. Compare client measurements with dashboard usage after an isolated run; the two counters need not measure exactly the same bytes.

Choose an order after the sample passes

Check current locations, then evaluate a small order using your acceptance conditions. The paid trial guide covers that decision. Both pools draw from one GB balance; the offer below shows current pricing and validity. Recheck the sample when the source, parser, browser resources or schedule changes.

What is not allowed

Collect only where you have permission. Follow the source’s terms, robots.txt and request limits. Stop when access is refused, and keep credentials and unnecessary personal data out of logs and exported records.

Read next