Proxies + AI

Use a Mobile Proxy with Crawl4AI

November 2026·7 min read·Crawl4AI, Python

Crawl4AI turns web pages into clean Markdown or structured JSON for LLM pipelines. It drives a real Chromium browser, so a dedicated 4G/5G mobile proxy plugs straight in with its ProxyConfig. This guide covers where to put the proxy (whole crawler or one run), the SOCKS5 limit, a check that the crawl really went through the proxy, and a small multi-city example.

In this guide

  1. Install and get your proxy details
  2. Proxy for one run (CrawlerRunConfig)
  3. Proxy for the whole crawler (BrowserConfig)
  4. HTTP vs SOCKS5
  5. Verify the exit IP and city
  6. Structured extraction through the proxy
  7. Multi-city crawl with one proxy
  8. Timeouts and politeness

1. Install and get your proxy details

Shell
pip install -U crawl4ai crawl4ai-setup # installs the Chromium build Crawl4AI uses

In the RelayKit dashboard, open a proxy under My Proxies and click 🤖 Use with AI tools → Crawl4AI. The snippet appears with your IP, port, username and password filled in. Below they're placeholders: YOUR_PROXY_IP, HTTP port 8000, SOCKS5 port 9000, USER, PASS. The examples read them from environment variables so you can update them in one place:

Shell
export PROXY_HOST=YOUR_PROXY_IP PROXY_PORT=8000 PROXY_USER=USER PROXY_PASS=PASS

2. Proxy for one run (CrawlerRunConfig)

This is the shape the dashboard snippet uses: the proxy is attached to the run config, so you can use different proxies for different runs.

crawl_once.py
import asyncio, os from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig, CacheMode PROXY = ProxyConfig( server=f"http://{os.environ['PROXY_HOST']}:{os.environ.get('PROXY_PORT', '8000')}", username=os.environ["PROXY_USER"], password=os.environ["PROXY_PASS"], ) async def main(): run = CrawlerRunConfig(proxy_config=PROXY, cache_mode=CacheMode.BYPASS) async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://api.ipify.org?format=json", config=run) print(result.markdown) # should show the proxy's IP asyncio.run(main())

CacheMode.BYPASS matters here: without it, Crawl4AI can return a cached copy of a page fetched earlier from another IP or city.

3. Proxy for the whole crawler (BrowserConfig)

If every page in the session should use the same proxy, set it once on the browser instead:

Browser-level proxy
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode browser = BrowserConfig(headless=True, proxy_config=PROXY) # PROXY from the previous example async with AsyncWebCrawler(config=browser) as crawler: result = await crawler.arun( url="https://example.com", config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS), )

4. HTTP vs SOCKS5

RelayKit proxies answer on HTTP (8000) and SOCKS5 (9000). Crawl4AI uses Chromium, and Chromium can't send a username and password to a SOCKS5 proxy. So:

SOCKS5, whitelist mode
run = CrawlerRunConfig(proxy_config=ProxyConfig(server="socks5://YOUR_PROXY_IP:9000"))

The dashboard's Crawl4AI snippet offers the SOCKS5 version only when whitelisting is on, for this reason.

5. Verify the exit IP and city

Check where the crawler is before the real work, and stop if it isn't on the proxy.

verify.py
import json from crawl4ai import CrawlerRunConfig, CacheMode async def where_am_i(crawler, proxy): run = CrawlerRunConfig(proxy_config=proxy, cache_mode=CacheMode.BYPASS) r = await crawler.arun(url="https://ipapi.co/json/", config=run) start, end = r.html.find("{"), r.html.rfind("}") info = json.loads(r.html[start:end + 1]) return {"ip": info.get("ip"), "city": info.get("city"), "region": info.get("region"), "org": info.get("org")}

Mobile IPs aren't static: Reset IP, an auto-rotation timer or a city switch changes them. Compare the city (and that org is the mobile carrier) rather than one exact IP.

6. Structured extraction through the proxy

For price or listing research you usually want fields, not Markdown. JsonCssExtractionStrategy pulls them with CSS selectors, no LLM call needed:

extract.py
from crawl4ai import CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy schema = { "name": "products", "baseSelector": ".product", # one element per item; adjust to the site "fields": [ {"name": "title", "selector": ".title", "type": "text"}, {"name": "price", "selector": ".price", "type": "text"}, ], } run = CrawlerRunConfig( proxy_config=PROXY, cache_mode=CacheMode.BYPASS, extraction_strategy=JsonCssExtractionStrategy(schema), wait_for="css:.product", # wait until the items are on the page page_timeout=60_000, # ms; mobile links have more latency ) # result = await crawler.arun(url=TARGET, config=run) # items = json.loads(result.extracted_content)

7. Multi-city crawl with one proxy

With a RelayKit Multi-Location proxy you don't need a proxy per city. Run the crawl, move the proxy to the next city on its card in the dashboard, copy the details shown on the card into your environment (a move connects you to a modem in the new city, so they change), and run again. Each run records the city it really came from:

city_run.py
import asyncio, json, os, time from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig, CacheMode, JsonCssExtractionStrategy PROXY = ProxyConfig( server=f"http://{os.environ['PROXY_HOST']}:{os.environ.get('PROXY_PORT', '8000')}", username=os.environ["PROXY_USER"], password=os.environ["PROXY_PASS"], ) URLS = ["https://example.com/pricing", "https://example.com/plans"] # pages you're researching SCHEMA = {"name": "prices", "baseSelector": ".plan", "fields": [{"name": "plan", "selector": "h3", "type": "text"}, {"name": "price", "selector": ".price", "type": "text"}]} async def main(): base = dict(proxy_config=PROXY, cache_mode=CacheMode.BYPASS, page_timeout=60_000) async with AsyncWebCrawler() as crawler: geo = await crawler.arun(url="https://ipapi.co/json/", config=CrawlerRunConfig(**base)) info = json.loads(geo.html[geo.html.find("{"): geo.html.rfind("}") + 1]) run = CrawlerRunConfig(**base, extraction_strategy=JsonCssExtractionStrategy(SCHEMA)) results = await crawler.arun_many(URLS, config=run) with open("city_prices.jsonl", "a") as f: for r in results: f.write(json.dumps({"city": info.get("city"), "region": info.get("region"), "url": r.url, "ok": r.success, "data": json.loads(r.extracted_content or "[]"), "at": time.time()}) + "\n") print("logged", len(results), "pages from", info.get("city")) asyncio.run(main())

Want an agent to drive the browser instead of fixed selectors? The same proxy works with browser-use; see the multi-city research agent guide.

8. Timeouts and politeness

Crawl4AI on a dedicated US mobile IP

Real Verizon 4G/5G carrier IPs with unlimited data, HTTP and SOCKS5, and a Multi-Location plan that switches US cities from the dashboard. Pay with crypto; sign up with an email.

See the Multi-Location proxy →

Related guides