Proxy Guide · Web Scraping

How to Scrape Websites Without Getting Blocked

November 2026·11 min read·web scraping, mobile proxies, rate limits

Most scrapers don't get blocked for one big reason. They get blocked for several small ones at once: a datacenter IP, too many requests too fast, headers that don't match a real browser, and a fingerprint that doesn't match the IP's location. This guide covers each cause, what to change, and a practical setup with one mobile proxy per scraper worker.

In this guide

  1. Why scrapers get blocked
  2. IPs: mobile vs datacenter vs residential
  3. Rotate per request or keep a sticky session?
  4. Request hygiene
  5. Headless browsers vs plain HTTP
  6. Practical setup: one mobile proxy per worker
  7. FAQ

1. Why scrapers get blocked

Sites protect themselves from overload, abuse and data resale, and their defenses score every request on a handful of signals. One weak signal is usually tolerated. Several together trigger a block, a CAPTCHA or a quietly empty page.

Treat blocks as feedback. A 429, a 403 or a CAPTCHA tells you the site thinks you are asking for too much, too fast or from the wrong place. The fix is to slow down and look more like a normal visitor, not to force your way through.

2. IPs: mobile vs datacenter vs residential

Your IP address is the first thing a site sees, before a single header is read. The three common sources behave very differently:

IP typeHow sites see itTypical block rate
DatacenterCloud or hosting range, easy to identify, rarely a real personHigh
Residential (proxy pools)Home broadband IPs, but pools are shared and some IPs carry a history of abuseMedium, varies by pool
Mobile carrierVerizon, AT&T or T-Mobile IPs shared by many real phone usersLow

The reason mobile IPs hold up so well is CGNAT (carrier-grade NAT). Mobile carriers don't have enough IPv4 addresses for every phone, so many subscribers share each public IP at the same time. To a website, one mobile IP is legitimately hundreds of real people scrolling, shopping and logging in. Blocking it would lock out real customers, so sites are far more careful about it than about a datacenter IP that no human uses.

That doesn't make a mobile IP immune. Hammer a site from one and it will still be rate limited. But it starts with a clean reputation instead of a suspicious one. The full comparison, including VPNs, is in Mobile vs Residential vs Datacenter Proxy vs VPN.

3. Rotate per request or keep a sticky session?

"Always rotate" is common advice, and often wrong. The right choice depends on whether the site expects continuity:

Static vs. Rotating Proxies goes deeper on the trade-offs.

4. Request hygiene

Most blocks can be avoided with habits that cost nothing:

Python: check robots.txt before crawling
from urllib import robotparser rp = robotparser.RobotFileParser("https://example.com/robots.txt") rp.read() UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0 Safari/537.36" if not rp.can_fetch(UA, "https://example.com/products/123"): print("Disallowed by robots.txt, skipping") print("Crawl delay:", rp.crawl_delay(UA)) # None if the site doesn't set one

5. Headless browsers vs plain HTTP

Plain HTTP (Python requests or httpx, Node fetch) is fast and light. Use it whenever the data is in the page HTML or in a JSON endpoint the page calls. Switch to a real browser when:

Playwright is the best default for new projects (Python and Node, reliable waits, per-context proxies). Puppeteer suits Node teams already on Chrome. Selenium fits existing test suites and other languages. For AI pipelines that turn pages into clean text, Crawl4AI wraps a browser for you. Setup guides with mobile proxies: Playwright, Puppeteer, Selenium (Python) and Crawl4AI.

Keep the fingerprint consistent with the proxy

A browser that says its time zone is Europe/Berlin while its IP is in Chicago is an easy mismatch to spot. Set the browser context's time zone and language to match the proxy's location, and keep them fixed for the session:

Playwright (Python): context matched to a Chicago proxy
from playwright.sync_api import sync_playwright PROXY = {"server": "http://HOST:8000", "username": "USER", "password": "PASS"} with sync_playwright() as p: browser = p.chromium.launch(proxy=PROXY) context = browser.new_context( timezone_id="America/Chicago", # match the proxy's city locale="en-US", # sets navigator.language and Accept-Language viewport={"width": 1366, "height": 768}, ) page = context.new_page() page.goto("https://api.ipify.org?format=json") print(page.text_content("body")) # should show the proxy's IP browser.close()

If you switch a Multi Location proxy to another city, update the time zone to match before the next run.

6. Practical setup: one mobile proxy per worker

The simplest setup that holds up at scale is one dedicated mobile proxy per scraper worker. Each worker gets its own IP, its own cookie session and its own rate limit, so one worker getting rate limited never affects the others. To scale, add workers and proxies instead of speeding up existing workers.

  1. Buy a RelayKit mobile proxy for each worker. Multi Location and Single Location proxies are Verizon with unlimited data; Multi-Carrier offers AT&T, T-Mobile and Verizon with a 15 GB/day allowance, so pick Multi or Single Location for high-volume scraping.
  2. Copy the host, username and password from the proxy card. HTTP is port 8000 and SOCKS5 is port 9000 on Multi Location and Single Location proxies.
  3. Choose how to rotate. The card has a Reset IP button and an automatic rotation timer (every N minutes). Multi Location and Single Location proxies also have a private Reset IP by Link URL: a GET request to it rotates the IP and returns the new one as JSON, at most once every 8 seconds. Keep that link private; anyone with it can rotate your proxy.
  4. Rotate between batches, never in the middle of a session, and wait a few seconds for the new IP before the next request.
Python: one worker, sticky session per batch, polite rate, backoff, rotate between batches
import random, time, requests PROXY = "http://USER:PASS@HOST:8000" # this worker's own proxy RESET_LINK = "https://relaykit.net/api/proxies/reset?token=YOUR_TOKEN" HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/131.0 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.9", } def new_session(): s = requests.Session() # one cookie jar per session s.proxies = {"http": PROXY, "https": PROXY} s.headers.update(HEADERS) return s def fetch(s, url, max_tries=5): delay = 5 for _ in range(max_tries): r = s.get(url, timeout=30) if r.status_code in (429, 403): ra = r.headers.get("Retry-After", "") wait = int(ra) if ra.isdigit() else delay # it can also be a date time.sleep(wait + random.uniform(0, 2)) delay = min(delay * 2, 300) # exponential backoff continue r.raise_for_status() return r.text raise RuntimeError(f"Gave up on {url}") def rotate_ip(): r = requests.get(RESET_LINK, timeout=30) # new IP is returned as JSON print("New IP:", r.json().get("ip")) time.sleep(10) # let the new IP settle urls = [f"https://example.com/products?page={n}" for n in range(1, 101)] for i in range(0, len(urls), 25): # batches of 25 pages s = new_session() for url in urls[i:i + 25]: html = fetch(s, url) # ... parse html ... time.sleep(random.uniform(2, 5)) # jittered delay rotate_ip() # fresh IP + fresh session per batch

Running several workers? Give each its own proxy and reset link, and never share a proxy between two workers hitting the same site at the same time. For agents and LLM-driven browsers, the same principles apply; see Mobile Proxies for AI Agents.

Be a good guest. Scrape at the pace a site can comfortably serve, cache what you already fetched, avoid peak hours for small sites, and stop when a site clearly asks you to. Well-behaved scrapers get blocked far less, and they keep working for years.

7. FAQ

Is web scraping legal?

It depends on what you collect, how you collect it and where you are. Reading publicly available pages is generally lower risk than scraping behind a login, collecting personal data, or copying copyrighted content. A site's terms of service can still bind you even when the data is public, and privacy laws such as GDPR and CCPA apply to personal data however you obtained it. This is not legal advice: for a commercial project, check with a lawyer.

How many requests can I send per IP?

There is no universal number. A safe starting point is one request every 2 to 5 seconds per site from each IP, with random jitter, then adjust based on what the site returns. If you see 429 or 403 responses, CAPTCHAs, or empty pages, slow down. Large sites with heavy traffic tolerate more than small ones.

Should I use a rotating or a static proxy for scraping?

Use a sticky (static) IP for anything with a session: logins, carts, multi-step forms and pagination that depends on cookies. Rotate between independent jobs, for example after each batch of product pages or when an IP starts getting 429s. Rotating on every single request is rarely necessary and breaks anything that relies on cookies.

Do I need a headless browser?

Only when the data is rendered by JavaScript, the site runs a browser check before showing content, or you need to click and scroll. If the data is in the page HTML or a JSON endpoint, plain HTTP requests are faster, cheaper and easier to keep consistent.

Why does my scraper work locally but get blocked on a server?

Your home connection has a residential IP with a clean history. Cloud servers use datacenter IP ranges that many sites treat as automated by default. Routing the scraper through a mobile proxy gives it a carrier IP instead of a datacenter one.

Give each scraper its own mobile IP

Dedicated US 4G/5G proxies on Verizon, AT&T and T-Mobile, with one-click and by-link IP rotation. Pay as you go, no monthly commitment.

Related guides