Python httpx and asyncio Proxies for Production Jobs

This article is educational. Buy Mobile, Residential, or Corporate on the product catalog and compare rates on Pricing.

The existing Python HTTP/HTTPS proxy guide covers requests. This article is the next layer: httpx (sync and async), asyncio, bounded concurrency, and retries. Generate the HTTP or SOCKS5 endpoint in the dashboard, then point httpx at it. Use this pattern for legitimate work: regional SEO checks, localization QA, public catalog research, and scheduled monitors. Follow applicable law and each site’s terms.

When httpx instead of requests

requests is still the right choice for a short script, a notebook, or a one-off SERP check. Reach for httpx when you need HTTP/1.1 and HTTP/2 from one client, a single API for sync Client and async AsyncClient, connection pooling you can tune, first-class timeouts (connect, read, write, pool), and optional SOCKS via httpx[socks].

If your codebase is already requests plus Session, keep it. Do not rewrite working jobs only to “use async.” Async helps when you have many independent HTTP calls and a proxy pool that can absorb parallel sessions. A single sticky checkout flow is often simpler as sync httpx or requests.

Install and credentials

pip install "httpx[http2,socks]"

Dashboard credentials go in the environment, not in git:

PG_PROXY=http://USER:PASS@host:port

HTTP and HTTPS targets both use an HTTP proxy URL on Proxy Grove unless you explicitly choose SOCKS5. Username/password auth is the usual path; IP allowlisting is for servers with a stable egress. See credentials vs allowlisting.

Sync httpx Client

Reuse one Client for a sticky window. Creating a new client per URL throws away the pool and makes session behavior harder to reason about.

import os
import httpx

PROXY = os.environ["PG_PROXY"]

timeout = httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=10.0)
limits = httpx.Limits(max_connections=20, max_keepalive_connections=10)

with httpx.Client(
    proxy=PROXY,
    timeout=timeout,
    limits=limits,
    http2=True,
    follow_redirects=True,
) as client:
    r = client.get("https://example.com", headers={"User-Agent": "ProxyGroveResearch/1.0"})
    r.raise_for_status()
    print(r.status_code, r.http_version)

http2=True is a negotiation. Many sites still speak HTTP/1.1; httpx will fall back. That is expected. Do not treat HTTP/2 as a ranking trick. For a small batch of URLs on a sticky residential or corporate IP, a sync client is enough. You get one identity, explicit timeouts, and fewer moving parts than asyncio.

AsyncClient and asyncio

Use AsyncClient when the job is “fetch N independent public pages” and you want overlap. Cap concurrency. An unbounded gather will open more connections than your plan or the target will tolerate.

import asyncio
import os
import httpx

PROXY = os.environ["PG_PROXY"]
URLS = [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/c",
]

async def fetch(client, sem, url):
    async with sem:
        r = await client.get(url)
        return url, r.status_code

async def main():
    timeout = httpx.Timeout(10.0, read=30.0)
    limits = httpx.Limits(max_connections=12, max_keepalive_connections=6)
    sem = asyncio.Semaphore(6)
    async with httpx.AsyncClient(
        proxy=PROXY,
        timeout=timeout,
        limits=limits,
        follow_redirects=True,
    ) as client:
        rows = await asyncio.gather(*(fetch(client, sem, u) for u in URLS))
    for url, code in rows:
        print(code, url)

asyncio.run(main())

Semaphore and Limits both matter. The semaphore is your job policy (“six in flight”). Limits is the HTTP pool. If you only set one, the other can still spike. For rotating sessions, parallel tasks are the usual fit: each request can leave from a different IP. For sticky sessions, keep a low semaphore (1–3) so a multi-step flow does not look like a burst from one identity.

Timeouts, status handling, retries

Production jobs fail on connect timeouts, read timeouts, 407 (bad auth), 429, and 5xx. Retry idempotent GETs. Do not blindly retry POST.

import random
import time
import httpx

RETRY_ON = {408, 425, 429, 500, 502, 503, 504}

def get_with_retry(client, url, attempts=4):
    delay = 0.5
    last_exc = None
    for i in range(attempts):
        try:
            r = client.get(url)
            if r.status_code in RETRY_ON and i < attempts - 1:
                time.sleep(delay + random.random() * 0.2)
                delay *= 2
                continue
            return r
        except (httpx.ConnectError, httpx.ReadTimeout, httpx.PoolTimeout) as exc:
            last_exc = exc
            if i == attempts - 1:
                raise
            time.sleep(delay + random.random() * 0.2)
            delay *= 2
    raise last_exc or RuntimeError("retry loop exited")

Backoff with jitter avoids a retry stampede when many workers share the same proxy. Cap attempts at 3–5. If auth is wrong, retries will not help: fix the dashboard user and password first. Log status code, elapsed time, and whether the attempt used a sticky or rotating endpoint. That is how you tell “the target is slow” from “the pool is exhausted.”

Sticky vs rotating (not an httpx flag)

httpx does not switch Proxy Grove session mode. Sticky or rotating is configured on the plan, then you copy the endpoint. In code you only decide how many concurrent clients to run.

Sticky: one Client or AsyncClient, low concurrency, reuse for a checkout, login, or rank-tracker session that must keep the same IP. Rotating: many short requests, higher concurrency, good for catalog sweeps and large public URL lists. If a site localizes by IP, sticky is how you compare “the same visitor” across three pages. Rotating is how you sample many IPs in one country.

Pick the product to match the identity you need: residential for consumer SERP and storefronts, mobile when carrier reputation matters, corporate for business-grade space. Protocol (HTTP vs SOCKS5) is independent of that SKU.

SOCKS5 with httpx

Install httpx[socks]. Point proxy= at the SOCKS5 URL from the dashboard (socks5://USER:PASS@host:port). Use SOCKS5 when the client is not HTTP-only, or when an existing toolchain already expects SOCKS. For browsers, rank trackers, and most Python HTTP jobs, the HTTP endpoint is simpler. Do not mix HTTP and SOCKS5 URLs on the same client. One client, one proxy URL. For protocol trade-offs see SOCKS5 vs HTTP proxies rather than duplicating that comparison here.

Concurrency without a stampede

A common failure mode is asyncio.gather over thousands of URLs with a huge semaphore. That is not faster. It is a burst against one country pool and against the target. Start with 4–8 in-flight requests per worker, max_connections a little above that, timeouts you would accept in QA (10s connect, 30s read), and one country at a time when you care about local SERPs. Scale up only after error rates are boring. Pair this with the rotating proxies article when the job is high-volume public collection.

Mapping to Proxy Grove products

httpx does not care whether the IP is mobile, residential, or corporate. You do. Residential: Google and storefront checks that should look like a home ISP. Mobile: ad preview and app QA where carrier IPs matter. Corporate: business-grade space and predictable routing for professional automation. Generate the endpoint in the dashboard, put it in PG_PROXY, and keep the same httpx code. Country targeting is selected with the plan, not in Python. If you are new to product choice, start on residential proxies for most SERP and storefront jobs, then add mobile only when the workflow is carrier-specific.

Authentication mistakes show up as 407 before any target site error. Confirm the username, password, host, and port in the dashboard, then retry a single sync GET. Do not debug 407 by raising asyncio concurrency. Allowlisting is a better fit for a server with a stable public egress; credential auth is a better fit for laptops and CI. Either way, keep secrets out of the repository and rotate them if a log file ever captured a proxy URL.

Timeouts should match the job. A rank-tracker batch can wait 30 seconds to read. A health check should fail faster so the worker moves on. Separate connect timeout from read timeout so a dead proxy does not look like a slow HTML page. When you log failures, include the exception class, the URL, and the attempt number. That log is how you distinguish pool exhaustion from a blocked target from a bad credential.

If you already ship a requests client, do not rewrite it in one night. Keep that process for interactive scripts and one-off checks. Add an httpx worker beside it when a job must overlap dozens of I/O waits. Share the same PG_PROXY secret so both stacks point at one pool. Measure wall time and 407 rates before you raise concurrency. That measurement is the difference between a production job and a burst that looks like abuse to the target.

What to do next

Put dashboard credentials in the environment. Copy the sync Client snippet and confirm a 200 on a public page you are allowed to fetch. Only then enable AsyncClient with a semaphore of 4–6. For Scrapy, wait for a dedicated Scrapy article: do not bolt asyncio onto a Twisted spider. That is the production shape: one client, explicit timeouts, bounded concurrency, retries on idempotent GETs, and the SKU chosen for identity rather than for the Python library.