Scrapy Proxy Middleware for Production Crawls

This article is educational. Buy Mobile, Residential, or Corporate on the product catalog and compare rates on Pricing. Use this pattern for legitimate work: public catalog research, regional SEO sampling, localization QA, and scheduled monitors. Follow applicable law, robots.txt, and each site’s terms.

The requests guide and the httpx and asyncio guide are still the right posts for those clients. This page is Scrapy proxy setup only: Twisted, HttpProxyMiddleware, settings.py, and crawl concurrency. Do not bolt asyncio onto a Scrapy spider. Do not copy a requests Session into a pipeline and call it Scrapy.

When Scrapy instead of requests or httpx

Stay on requests or httpx for a short script, a notebook, or a handful of URLs. Reach for Scrapy when you have a crawl graph (start URLs, rules, item pipelines), you want built-in robots handling, throttling, and retries, and the job is “walk a public site you are allowed to fetch,” not “fire 50 async GETs.” Scrapy is a crawler. httpx is an HTTP client. Mixing them usually means two failure modes in one process.

If the spider is already shipping, do not rewrite it in httpx to “use async.” If the job is not a crawl, do not adopt Scrapy to “use a proxy.” Generate the HTTP or SOCKS5 endpoint in the dashboard, then point the crawler at it.

Official middleware, not a custom exploit stack

Scrapy’s own docs describe HttpProxyMiddleware: it sets the per-request proxy meta from http_proxy / https_proxy / no_proxy, or from request.meta["proxy"] (that value wins). The usual proxy URL scheme is http://. SOCKS is not the default download handler. Scrapy documents that HttpxDownloadHandler supports SOCKS proxy URLs while other built-in handlers do not. For most Proxy Grove crawls, use the HTTP endpoint. For protocol trade-offs see SOCKS5 vs HTTP rather than inventing a second protocol article.

Install and credentials

pip install scrapy

Dashboard credentials go in the environment, not in git:

export http_proxy="http://USER:PASS@host:port"
export https_proxy="http://USER:PASS@host:port"

HTTP and HTTPS targets both use an HTTP proxy URL on Proxy Grove unless you explicitly choose SOCKS5 and a handler that supports it. Username/password auth is the usual path; IP allowlisting is for servers with a stable egress. See credentials vs allowlisting.

settings.py that will not stampede

# settings.py
import os

BOT_NAME = "public_research"
ROBOTSTXT_OBEY = True
COOKIES_ENABLED = False

CONCURRENT_REQUESTS = 8
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 1.0
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0
RETRY_TIMES = 2

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

# Optional: explicit proxy if you do not want process-wide env vars
PG_PROXY = os.environ.get("PG_PROXY", "")

ROBOTSTXT_OBEY = True is the default you should keep for public research. If a site disallows the path, do not “fix” it by turning robots off. That is not a proxy problem. CONCURRENT_REQUESTS and DOWNLOAD_DELAY are how you avoid looking like a burst. An unbounded crawl through a rotating pool is still a burst against the target. Pair this with the rotating proxies article when the job is high-volume public collection. Do not create a second generic rotating post.

Per-request proxy meta

When you need one spider to use the dashboard URL without relying on shell env in every worker:

import os
import scrapy

PROXY = os.environ["PG_PROXY"]

class PublicCatalogSpider(scrapy.Spider):
    name = "public_catalog"
    start_urls = ["https://example.com/"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"proxy": PROXY},
                callback=self.parse,
            )

    def parse(self, response):
        title = response.css("title::text").get()
        yield {"url": response.url, "title": title}

Keep example.com as a stand-in. Point start URLs only at sites you are allowed to fetch. Log status codes and proxy 407s separately from target 429s. A 407 is credentials or port. Raising CONCURRENT_REQUESTS will not fix 407.

Sticky vs rotating is not a Scrapy setting

Scrapy does not switch Proxy Grove session mode. Sticky or rotating is configured on the plan, then you copy the endpoint. In the spider you only decide concurrency and delay.

Sticky: low CONCURRENT_REQUESTS_PER_DOMAIN (1-2), reuse for a multi-page flow that must keep the same IP. Rotating: more start URLs, still capped concurrency, good for sampling many public pages. If a site localizes by IP, sticky is how you compare “the same visitor” across three paths. Rotating is how you sample many IPs in one country. Country targeting is selected with the plan, not in Python.

SOCKS5 in Scrapy

Default Scrapy HTTP11 handlers are HTTP-proxy-centric. Official docs: you can use a socks5:// URL when the download handler supports it; HttpxDownloadHandler does, other built-in handlers do not. For most crawls, the HTTP endpoint is simpler. Do not mix HTTP and SOCKS5 on the same spider without a clear handler choice. One spider, one proxy URL.

Retries, delays, and what not to do

Retry idempotent GETs. Do not blindly retry POST. Autothrottle exists so you do not invent a custom “stealth” delay that still hammers one path. Do not disable robots.txt to “make the proxy work.” Do not write a downloader middleware whose only job is to bypass blocks or CAPTCHAs. If the target requires a managed unlocker, that is a different product (see the Bright Data / Oxylabs / IPRoyal comparisons). Proxy Grove sells IPs, not an unblocker.

Timeouts live in Scrapy’s download timeout settings. A rank-tracker style fetch can wait longer than a health check. Log the request URL, status, and whether the endpoint was sticky or rotating. That log is how you tell pool exhaustion from a blocked target from a bad credential.

Mapping to Proxy Grove products

Scrapy does not care whether the IP is mobile, residential, or corporate. You do. Residential: Google and storefront checks that should look like a home ISP. Mobile: ad preview and app QA where carrier IPs matter. Corporate: business-grade space. Generate the endpoint, put it in PG_PROXY or http_proxy, keep the same spider. If you are new to product choice, start on residential proxies for most SERP and storefront jobs.

If you already ship a requests or httpx client, keep it for one-off checks. Add a Scrapy project beside it when the job is a crawl graph. Share the same secret. Measure 407 rates and robots denials before you raise concurrency. That measurement is the difference between a production crawl and a burst.

Logging: 407, 429, robots, and timeouts

Treat the Scrapy log as four buckets, not one “it failed” counter. A 407 is the proxy rejecting credentials or the protocol. A 429 or 503 is the target. A robots denial is ROBOTSTXT_OBEY doing its job. A timeout is DOWNLOAD_TIMEOUT or a hung tunnel. Raising CONCURRENT_REQUESTS only helps if the failure is idle capacity. It makes 407 and 429 worse.

# settings.py extras
DOWNLOAD_TIMEOUT = 30
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]
LOG_LEVEL = "INFO"

After a crawl, read Scrapy stats: downloader/response_status_count/407, downloader/exception_type_count, robotstxt/forbidden, and httperror/response_count. Log the proxy host (not the password) and whether the plan was sticky or rotating. That line is how a teammate debugs a failed job without guessing. Do not scrape those stats into a second article; keep them on this page.

custom_settings, not one global concurrency

A rank-sample spider and a catalog spider should not share CONCURRENT_REQUESTS = 32. Put conservative defaults in settings.py. Override per spider with custom_settings. Sticky SERP checks stay at 1-2 concurrent requests per domain. Broader public catalog jobs can go higher, still capped, still delayed.

class SerpSampleSpider(scrapy.Spider):
    name = "serp_sample"
    custom_settings = {
        "CONCURRENT_REQUESTS": 2,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 1,
        "DOWNLOAD_DELAY": 2.0,
        "COOKIES_ENABLED": True,
    }

Cookies belong on sticky multi-step flows, not on every rotating catalog request. If you enable cookies on a rotating plan, you mix identities. Pick one: sticky plus cookies, or rotating plus cookies off.

Pipelines never set the proxy

Item pipelines transform items after the response is already downloaded. Copying a requests proxies= dict into a pipeline does nothing for the next request and confuses the next engineer. Keep the proxy on HttpProxyMiddleware, process-wide http_proxy, or request.meta["proxy"]. A pipeline is the right place to drop empty titles, write JSON lines, or push to a queue. It is the wrong place to open sockets.

The same split applies to middlewares you write yourself. Downloader middleware can adjust headers or drop a request. Do not add a middleware whose only purpose is to defeat blocks or CAPTCHAs. If the target needs a managed unlocker, that is a different vendor product (see the Bright Data, Oxylabs, Decodo, and IPRoyal comparisons). Proxy Grove sells IPs.

HTTPS targets still use an HTTP proxy URL

For HTTPS pages, Scrapy’s default handler opens a CONNECT tunnel through an http:// proxy URL. That is expected. You do not switch the proxy scheme to https:// just because the target is HTTPS. You switch to socks5:// only when you also switch to a download handler that supports SOCKS, such as the documented HttpxDownloadHandler. One spider, one proxy URL, one handler.

Set a descriptive USER_AGENT for the job (a research bot name and contact), not a random Chrome string. Impersonating a browser to “look real” is not a proxy feature and is not what this guide covers. Keep TLS verification on. A proxy does not mean you skip certificate checks.

CrawlSpider rules still obey the same proxy

CrawlSpider plus LinkExtractor does not rotate IPs and does not bypass robots. Rules only decide which links become requests. Those requests still go through HttpProxyMiddleware. If you leave allow too broad, you will fetch paths you did not mean to fetch, faster, through more IPs. Restrict rules to the public paths you are allowed to collect. Then keep DOWNLOAD_DELAY and autothrottle on. A wide follow rule is not an excuse to drop robots.txt.

What to do next

Put dashboard credentials in the environment. Copy the spider snippet, keep ROBOTSTXT_OBEY = True, and confirm a 200 on a public page you are allowed to fetch. Only then raise concurrency from 2 toward 8. For Playwright or Puppeteer, wait for that dedicated article. For AdsPower, wait for the profile-tool article. This page stays Scrapy: one middleware, explicit delays, bounded concurrency, retries on idempotent GETs, and the SKU chosen for identity rather than for the Python library.