Proxies + AI
Use a Mobile Proxy with Crawl4AI
November 2026·7 min read·Crawl4AI, Python
Crawl4AI turns web pages into clean Markdown or structured JSON for LLM pipelines. It drives a real Chromium browser, so a dedicated 4G/5G mobile proxy plugs straight in with its ProxyConfig. This guide covers where to put the proxy (whole crawler or one run), the SOCKS5 limit, a check that the crawl really went through the proxy, and a small multi-city example.
1. Install and get your proxy details
Shell
pip install -U crawl4ai
crawl4ai-setup # installs the Chromium build Crawl4AI uses
In the RelayKit dashboard, open a proxy under My Proxies and click 🤖 Use with AI tools → Crawl4AI. The snippet appears with your IP, port, username and password filled in. Below they're placeholders: YOUR_PROXY_IP, HTTP port 8000, SOCKS5 port 9000, USER, PASS. The examples read them from environment variables so you can update them in one place:
Shell
export PROXY_HOST=YOUR_PROXY_IP PROXY_PORT=8000 PROXY_USER=USER PROXY_PASS=PASS
2. Proxy for one run (CrawlerRunConfig)
This is the shape the dashboard snippet uses: the proxy is attached to the run config, so you can use different proxies for different runs.
crawl_once.py
import asyncio, os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig, CacheMode
PROXY = ProxyConfig(
server=f"http://{os.environ['PROXY_HOST']}:{os.environ.get('PROXY_PORT', '8000')}",
username=os.environ["PROXY_USER"],
password=os.environ["PROXY_PASS"],
)
async def main():
run = CrawlerRunConfig(proxy_config=PROXY, cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://api.ipify.org?format=json", config=run)
print(result.markdown) # should show the proxy's IP
asyncio.run(main())
CacheMode.BYPASS matters here: without it, Crawl4AI can return a cached copy of a page fetched earlier from another IP or city.
3. Proxy for the whole crawler (BrowserConfig)
If every page in the session should use the same proxy, set it once on the browser instead:
Browser-level proxy
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
browser = BrowserConfig(headless=True, proxy_config=PROXY) # PROXY from the previous example
async with AsyncWebCrawler(config=browser) as crawler:
result = await crawler.arun(
url="https://example.com",
config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS),
)
4. HTTP vs SOCKS5
RelayKit proxies answer on HTTP (8000) and SOCKS5 (9000). Crawl4AI uses Chromium, and Chromium can't send a username and password to a SOCKS5 proxy. So:
- With a username and password, use
http://YOUR_PROXY_IP:8000. It carries HTTPS sites without any problem. - For SOCKS5, turn on IP Whitelisting on the proxy card for the machine running the crawler. With no login needed, this works:
SOCKS5, whitelist mode
run = CrawlerRunConfig(proxy_config=ProxyConfig(server="socks5://YOUR_PROXY_IP:9000"))
The dashboard's Crawl4AI snippet offers the SOCKS5 version only when whitelisting is on, for this reason.
5. Verify the exit IP and city
Check where the crawler is before the real work, and stop if it isn't on the proxy.
verify.py
import json
from crawl4ai import CrawlerRunConfig, CacheMode
async def where_am_i(crawler, proxy):
run = CrawlerRunConfig(proxy_config=proxy, cache_mode=CacheMode.BYPASS)
r = await crawler.arun(url="https://ipapi.co/json/", config=run)
start, end = r.html.find("{"), r.html.rfind("}")
info = json.loads(r.html[start:end + 1])
return {"ip": info.get("ip"), "city": info.get("city"), "region": info.get("region"), "org": info.get("org")}
Mobile IPs aren't static: Reset IP, an auto-rotation timer or a city switch changes them. Compare the city (and that org is the mobile carrier) rather than one exact IP.
For price or listing research you usually want fields, not Markdown. JsonCssExtractionStrategy pulls them with CSS selectors, no LLM call needed:
extract.py
from crawl4ai import CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy
schema = {
"name": "products",
"baseSelector": ".product", # one element per item; adjust to the site
"fields": [
{"name": "title", "selector": ".title", "type": "text"},
{"name": "price", "selector": ".price", "type": "text"},
],
}
run = CrawlerRunConfig(
proxy_config=PROXY,
cache_mode=CacheMode.BYPASS,
extraction_strategy=JsonCssExtractionStrategy(schema),
wait_for="css:.product", # wait until the items are on the page
page_timeout=60_000, # ms; mobile links have more latency
)
# result = await crawler.arun(url=TARGET, config=run)
# items = json.loads(result.extracted_content)
7. Multi-city crawl with one proxy
With a RelayKit Multi-Location proxy you don't need a proxy per city. Run the crawl, move the proxy to the next city on its card in the dashboard, copy the details shown on the card into your environment (a move connects you to a modem in the new city, so they change), and run again. Each run records the city it really came from:
city_run.py
import asyncio, json, os, time
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig, CacheMode, JsonCssExtractionStrategy
PROXY = ProxyConfig(
server=f"http://{os.environ['PROXY_HOST']}:{os.environ.get('PROXY_PORT', '8000')}",
username=os.environ["PROXY_USER"], password=os.environ["PROXY_PASS"],
)
URLS = ["https://example.com/pricing", "https://example.com/plans"] # pages you're researching
SCHEMA = {"name": "prices", "baseSelector": ".plan",
"fields": [{"name": "plan", "selector": "h3", "type": "text"},
{"name": "price", "selector": ".price", "type": "text"}]}
async def main():
base = dict(proxy_config=PROXY, cache_mode=CacheMode.BYPASS, page_timeout=60_000)
async with AsyncWebCrawler() as crawler:
geo = await crawler.arun(url="https://ipapi.co/json/", config=CrawlerRunConfig(**base))
info = json.loads(geo.html[geo.html.find("{"): geo.html.rfind("}") + 1])
run = CrawlerRunConfig(**base, extraction_strategy=JsonCssExtractionStrategy(SCHEMA))
results = await crawler.arun_many(URLS, config=run)
with open("city_prices.jsonl", "a") as f:
for r in results:
f.write(json.dumps({"city": info.get("city"), "region": info.get("region"), "url": r.url,
"ok": r.success, "data": json.loads(r.extracted_content or "[]"),
"at": time.time()}) + "\n")
print("logged", len(results), "pages from", info.get("city"))
asyncio.run(main())
Want an agent to drive the browser instead of fixed selectors? The same proxy works with browser-use; see the multi-city research agent guide.
8. Timeouts and politeness
- Set
page_timeout to 45–60 seconds. Carrier connections have more latency than a datacenter. - Keep
arun_many batches small for one proxy: a dedicated mobile link is one modem, not a pool of thousands of IPs. - Retry a failed URL once before giving up; mobile networks have occasional slow seconds.
- Respect each site's terms and robots rules, keep request rates modest, and collect data for your own research, monitoring and QA.
Crawl4AI on a dedicated US mobile IP
Real Verizon 4G/5G carrier IPs with unlimited data, HTTP and SOCKS5, and a Multi-Location plan that switches US cities from the dashboard. Pay with crypto; sign up with an email.
See the Multi-Location proxy →Related guides