Most scrapers don't get blocked for one big reason. They get blocked for several small ones at once: a datacenter IP, too many requests too fast, headers that don't match a real browser, and a fingerprint that doesn't match the IP's location. This guide covers each cause, what to change, and a practical setup with one mobile proxy per scraper worker.
Sites protect themselves from overload, abuse and data resale, and their defenses score every request on a handful of signals. One weak signal is usually tolerated. Several together trigger a block, a CAPTCHA or a quietly empty page.
User-Agent, a library default like python-requests/2.32, no Accept-Language, or a User-Agent that changes on every request within one session.Treat blocks as feedback. A 429, a 403 or a CAPTCHA tells you the site thinks you are asking for too much, too fast or from the wrong place. The fix is to slow down and look more like a normal visitor, not to force your way through.
Your IP address is the first thing a site sees, before a single header is read. The three common sources behave very differently:
| IP type | How sites see it | Typical block rate |
|---|---|---|
| Datacenter | Cloud or hosting range, easy to identify, rarely a real person | High |
| Residential (proxy pools) | Home broadband IPs, but pools are shared and some IPs carry a history of abuse | Medium, varies by pool |
| Mobile carrier | Verizon, AT&T or T-Mobile IPs shared by many real phone users | Low |
The reason mobile IPs hold up so well is CGNAT (carrier-grade NAT). Mobile carriers don't have enough IPv4 addresses for every phone, so many subscribers share each public IP at the same time. To a website, one mobile IP is legitimately hundreds of real people scrolling, shopping and logging in. Blocking it would lock out real customers, so sites are far more careful about it than about a datacenter IP that no human uses.
That doesn't make a mobile IP immune. Hammer a site from one and it will still be rate limited. But it starts with a clean reputation instead of a suspicious one. The full comparison, including VPNs, is in Mobile vs Residential vs Datacenter Proxy vs VPN.
"Always rotate" is common advice, and often wrong. The right choice depends on whether the site expects continuity:
Static vs. Rotating Proxies goes deeper on the trade-offs.
Most blocks can be avoided with habits that cost nothing:
Retry-After header if there is one; otherwise wait and double the delay each time (exponential backoff). Repeated 403s on an IP mean pause that worker or rotate its IP, not retry instantly.User-Agent plus Accept and Accept-Language headers, and keep them the same for the whole session. Match the language to the proxy's country, for example en-US for a US proxy.robots.txt before crawling, skip disallowed paths, and honor any crawl delay. Read the site's terms, prefer an official API when one exists, and avoid collecting personal data you don't need.Plain HTTP (Python requests or httpx, Node fetch) is fast and light. Use it whenever the data is in the page HTML or in a JSON endpoint the page calls. Switch to a real browser when:
Playwright is the best default for new projects (Python and Node, reliable waits, per-context proxies). Puppeteer suits Node teams already on Chrome. Selenium fits existing test suites and other languages. For AI pipelines that turn pages into clean text, Crawl4AI wraps a browser for you. Setup guides with mobile proxies: Playwright, Puppeteer, Selenium (Python) and Crawl4AI.
A browser that says its time zone is Europe/Berlin while its IP is in Chicago is an easy mismatch to spot. Set the browser context's time zone and language to match the proxy's location, and keep them fixed for the session:
If you switch a Multi Location proxy to another city, update the time zone to match before the next run.
The simplest setup that holds up at scale is one dedicated mobile proxy per scraper worker. Each worker gets its own IP, its own cookie session and its own rate limit, so one worker getting rate limited never affects the others. To scale, add workers and proxies instead of speeding up existing workers.
8000 and SOCKS5 is port 9000 on Multi Location and Single Location proxies.Running several workers? Give each its own proxy and reset link, and never share a proxy between two workers hitting the same site at the same time. For agents and LLM-driven browsers, the same principles apply; see Mobile Proxies for AI Agents.
Be a good guest. Scrape at the pace a site can comfortably serve, cache what you already fetched, avoid peak hours for small sites, and stop when a site clearly asks you to. Well-behaved scrapers get blocked far less, and they keep working for years.
It depends on what you collect, how you collect it and where you are. Reading publicly available pages is generally lower risk than scraping behind a login, collecting personal data, or copying copyrighted content. A site's terms of service can still bind you even when the data is public, and privacy laws such as GDPR and CCPA apply to personal data however you obtained it. This is not legal advice: for a commercial project, check with a lawyer.
There is no universal number. A safe starting point is one request every 2 to 5 seconds per site from each IP, with random jitter, then adjust based on what the site returns. If you see 429 or 403 responses, CAPTCHAs, or empty pages, slow down. Large sites with heavy traffic tolerate more than small ones.
Use a sticky (static) IP for anything with a session: logins, carts, multi-step forms and pagination that depends on cookies. Rotate between independent jobs, for example after each batch of product pages or when an IP starts getting 429s. Rotating on every single request is rarely necessary and breaks anything that relies on cookies.
Only when the data is rendered by JavaScript, the site runs a browser check before showing content, or you need to click and scroll. If the data is in the page HTML or a JSON endpoint, plain HTTP requests are faster, cheaper and easier to keep consistent.
Your home connection has a residential IP with a clean history. Cloud servers use datacenter IP ranges that many sites treat as automated by default. Routing the scraper through a mobile proxy gives it a carrier IP instead of a datacenter one.
Dedicated US 4G/5G proxies on Verizon, AT&T and T-Mobile, with one-click and by-link IP rotation. Pay as you go, no monthly commitment.