Handling sticky dynamic session IDs in Python Scrapy without losing referer headers

Sticky session IDs break Referer propagation in Scrapy because the session token typically lives in the URL (e.g., ;jsessionid=... or ?sid=...) or in a cookie that the server rotates mid-crawl, while RefererMiddleware sets Referer from response.url of the parent. When the server rewrites the URL with a new session ID, the dupefilter treats it as a new request and the Referer points to a stale or tokenized URL the server no longer accepts.

Root Cause Diagnosis

Three mechanisms combine to cause this:

1. RefererMiddleware copies response.url verbatim. If the response URL contains a session token, the next request’s Referer carries that token. When the server invalidates the token, the Referer check fails and the server returns a redirect to a fresh session URL, a 403, or a login page.
2. Redirects discard the original Referer. RedirectMiddleware creates a new request from the redirect target. On a session-reset redirect, the Referer becomes the pre-redirect URL, which contains the dead token.
3. The dupefilter fingerprints the full URL. The same page with a different session ID is treated as a new request, causing duplicate crawling or, when dont_filter=False, silently dropped requests that were meant to be retries.

Quick Fix

Strip the session token from URLs before fingerprinting and before Referer assignment, and disable the default referer policy that leaks tokens.

settings.py

# Never send the full URL as Referer; origin only avoids leaking stale tokens
REFERRER_POLICY = "scrapy.spiderextensions.referer.OriginPolicy"

# Keep cookies enabled so the server-side session stays bound to the cookie jar
COOKIES_ENABLED = True

Then override Referer explicitly per request (see Step 3) when the target site requires the full path.

REFERRER_POLICY accepts the policy class path or the W3C policy names (e.g., "origin", "same-origin", "strict-origin-when-cross-origin"). Verify the accepted values for your installed Scrapy version with scrapy version -v and the docs for that release.

Step-by-Step Resolution

Step 1: Identify where the session ID lives

Run a single request through the shell and compare the URL, cookies, and redirect chain.

scrapy shell "https://example.com/start" --nolog -c \
  "print(response.url); print(response.headers.getlist('Set-Cookie')); print(request.meta.get('redirect_urls'))"

Classify the result:

Location	Example	Fix path
URL path parameter	/page;jsessionid=ABC123	Step 2 (URL canonicalization)
Query string	/page?sid=ABC123	Step 2
Cookie only	Set-Cookie: JSESSIONID=ABC123	Step 4 (cookie jar control)

Step 2: Canonicalize URLs by stripping the token

Create a custom dupefilter so the same logical page is not re-requested under a different session ID.

myproject/dupefilter.py

import re
from w3lib.url import url_query_cleaner
from scrapy.dupefilters import RFPDupeFilter
from scrapy.utils.request import fingerprint

PATH_SID = re.compile(r";jsessionid=[^?#]*", re.IGNORECASE)
QUERY_SID_PARAMS = ("sid", "jsessionid", "phpsessid")


def canonical_url(url: str) -> str:
    url = PATH_SID.sub("", url)
    return url_query_cleaner(url, QUERY_SID_PARAMS, remove=True)


class SessionAwareDupeFilter(RFPDupeFilter):
    def request_fingerprint(self, request):
        return fingerprint(request.replace(url=canonical_url(request.url))).hex()

Version note: In Scrapy 2.7+, RFPDupeFilter.request_fingerprint is deprecated in favor of the REQUEST_FINGERPRINTER_CLASS setting. If you see a deprecation warning, implement a custom fingerprinter instead:

from scrapy.utils.request import fingerprint

class SessionAwareFingerprinter:
    @classmethod
    def from_crawler(cls, crawler):
        return cls()

    def fingerprint(self, request):
        return fingerprint(request.replace(url=canonical_url(request.url)))

settings.py

# Scrapy < 2.7
DUPEFILTER_CLASS = "myproject.dupefilter.SessionAwareDupeFilter"

# Scrapy >= 2.7
REQUEST_FINGERPRINTER_CLASS = "myproject.dupefilter.SessionAwareFingerprinter"

Step 3: Preserve the Referer explicitly

Set the Referer from the canonical (token-free) parent URL, or from a value you control, in the spider callback. Explicitly set headers are not overwritten by RefererMiddleware.

import scrapy
from myproject.dupefilter import canonical_url


class SessionSpider(scrapy.Spider):
    name = "session_spider"
    start_urls = ["https://example.com/start"]

    def parse(self, response):
        parent = canonical_url(response.url)
        for href in response.css("a.item::attr(href)").getall():
            yield response.follow(
                href,
                callback=self.parse_item,
                headers={"Referer": parent},
                meta={"parent_url": parent},
            )

    def parse_item(self, response):
        yield {"url": canonical_url(response.url)}

Step 4: Preserve the Referer across redirects

RedirectMiddleware builds a new request for the redirect target. Re-apply the Referer with a downloader middleware placed after it.

myproject/middlewares.py

from myproject.dupefilter import canonical_url


class StickyRefererMiddleware:
    """Re-applies the intended Referer on redirected requests."""

    def process_request(self, request, spider):
        parent = request.meta.get("parent_url")
        if parent and not request.headers.get("Referer"):
            request.headers["Referer"] = parent
        elif request.headers.get("Referer"):
            ref = request.headers["Referer"].decode()
            request.headers["Referer"] = canonical_url(ref)

settings.py

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.StickyRefererMiddleware": 650,
}

Priority reference: RedirectMiddleware is 600, CookiesMiddleware is 700, HttpProxyMiddleware is 750. Placing this at 650 runs process_request after the redirect request is constructed and before cookies are attached. Confirm current defaults with scrapy settings --get DOWNLOADER_MIDDLEWARES_BASE.

Downloader middleware process_request runs in ascending priority order. Redirects are re-scheduled through the engine, so every redirected request passes through this middleware again.

Step 5: Pin one cookie jar per logical session

If the server binds the session to a cookie, concurrent requests sharing the default jar can cause token rotation mid-crawl. Isolate each session.

def start_requests(self):
    for i, url in enumerate(self.start_urls):
        yield scrapy.Request(
            url,
            meta={"cookiejar": i},
            callback=self.parse,
        )

def parse(self, response):
    jar = response.meta["cookiejar"]
    parent = canonical_url(response.url)
    for href in response.css("a.item::attr(href)").getall():
        yield response.follow(
            href,
            meta={"cookiejar": jar, "parent_url": parent},
            headers={"Referer": parent},
            callback=self.parse_item,
        )

cookiejar must be passed explicitly on every follow-up request. It is not inherited automatically.

Step 6: Limit concurrency per session

If the server invalidates tokens when it sees overlapping requests, serialize them.

# settings.py
CONCURRENT_REQUESTS_PER_DOMAIN = 1
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = False

Step 7: Verify

scrapy crawl session_spider -s LOG_LEVEL=DEBUG 2>&1 | grep -E "Referer|Redirecting|Filtered duplicate"

Expected: no Filtered duplicate lines for pages differing only in session token, and each request line shows a token-free Referer.

Why This Fails for Some Users

• Server validates Referer against the session. Some servers store the last URL served to a session and compare it to the incoming Referer. Stripping the token breaks that match. In this case, keep the token in the Referer and instead refresh the token via a login or entry request, then rebuild the URL from the new value.
• Referer header casing or duplication. Scrapy normalizes header names, but a downloader middleware that sets request.headers["referer"] and another that sets "Referer" still resolve to one header. A proxy or custom DEFAULT_REQUEST_HEADERS entry can introduce a conflicting value; check DEFAULT_REQUEST_HEADERS in settings.py.
• Middleware ordering conflicts. Third-party middlewares (e.g., scrapy-splash, scrapy-playwright, scrapy-rotating-proxies) may issue requests that bypass your priority-650 middleware or re-create requests without meta. Inspect with scrapy settings --get DOWNLOADER_MIDDLEWARES and adjust priorities.
• Browser-rendered flows (Playwright/Splash). The browser sets its own Referer and may mint a new session ID via JavaScript. Scrapy-level header overrides do not apply to sub-resource requests made by the browser. Set headers through the browser context (extra_http_headers) instead.
• Retries lose headers or meta. RetryMiddleware copies the request, preserving headers and meta, but a custom retry that builds scrapy.Request(url) fresh drops them. Use request.replace() or request.copy().
• dont_filter=True masking duplicates. Setting it globally on follow-ups bypasses the dupefilter entirely, so the canonicalization in Step 2 has no effect and the crawl can loop on rotating tokens.
• Cross-domain requests. Under OriginPolicy or same-origin, the Referer is trimmed or omitted by design on cross-origin navigation. An explicit headers={"Referer": ...} overrides this.
• Scrapy version differences. REFERRER_POLICY and REQUEST_FINGERPRINTER_CLASS behavior differs across releases, and the default fingerprinter implementation changed in 2.7. Check scrapy version -v and the release notes for your version.
• Python 3.12+ and w3lib. url_query_cleaner is stable across w3lib versions, but an outdated pinned w3lib can lack parameter-removal behavior. Upgrade with pip install -U w3lib.
• Shared session across distributed workers. With scrapy-redis or similar, the cookie jar lives in each worker process. Requests for one logical session distributed across workers use different jars and trigger token rotation. Route all requests for a session to one worker (e.g., shard queue keys by session ID).
• Server-side IP binding. If the session is bound to the client IP, rotating proxies invalidate it on every IP change. Use sticky proxy sessions or pin one proxy per cookiejar.