Fixing LiteSpeed OpenLiteSpeed Cache Crawler 403 Forbidden Loop

Root Cause Diagnosis

The LiteSpeed Cache crawler is a server-side HTTP client (User-Agent lscache_runner) that requests your public URLs from the origin server’s own IP. When a layer in that path (CDN bot filtering, WAF or ModSecurity, OpenLiteSpeed access control or per-client throttling, a host firewall, or a WordPress security plugin) classifies that source as hostile, it returns 403. A 403 is never stored in LSCache, so each URL stays uncached, is retried on every crawl cycle, and accumulates as a failed or blocklisted entry.

Quick Fix

Run from the server to identify which layer blocks the crawler:

DOMAIN=example.com
SERVER_IP=$(curl -s4 https://ifconfig.me)

# Path 1: normal DNS/CDN route (what the crawler does by default)
curl -s -o /dev/null -w "via DNS/CDN : %{http_code}\n" -A "lscache_runner" https://$DOMAIN/

# Path 2: direct to origin, bypassing the CDN
curl -s -o /dev/null -w "direct      : %{http_code}\n" -A "lscache_runner" \
  --resolve $DOMAIN:443:$SERVER_IP https://$DOMAIN/

Result	Meaning	Immediate fix
via DNS/CDN 403, direct 200	CDN or WAF blocks the crawler	Set Server IP (below)
Both 403	Origin-side block (OLS ACL, ModSecurity, .htaccess, firewall)	Continue to Step 3 onward

To apply the CDN fix: WP Admin → LiteSpeed Cache → Crawler → General Settings → Server IP → enter the origin’s public IPv4 → Save Changes. The crawler then pins the hostname to that IP and skips DNS and CDN entirely.

Step-by-Step Resolution

1. Capture the exact 403 source

curl -sI -A "lscache_runner" --resolve $DOMAIN:443:$SERVER_IP https://$DOMAIN/ \
  | grep -iE "^HTTP/|^server:|cf-mitigated|x-litespeed-cache"

• cf-mitigated: challenge means Cloudflare is challenging the request.
• server: LiteSpeed with no CDN headers means the block is at the origin.

Then correlate with the logs:

tail -f /usr/local/lsws/logs/access.log | grep lscache_runner
tail -f /usr/local/lsws/logs/error.log

Per-vhost access logs are typically under /usr/local/lsws/<vhost>/logs/ or /home/<user>/logs/, depending on the panel.

2. Set the crawler Server IP

WP Admin → LiteSpeed Cache → Crawler → General Settings → Server IP

Enter the IP the origin actually listens on (see the cloud/NAT edge case below).

3. Allowlist the server in OpenLiteSpeed access control

WebAdmin (https://SERVER_IP:7080):

• Server Configuration → Security → Access Control
• Virtual Hosts → your vhost → Security → Access Control

Set Allowed List (a non-empty list restricts everyone not on it):

127.0.0.1T, ::1T, 203.0.113.10T, ALL

The T suffix marks a trusted IP that is exempt from per-client throttling and auto-ban. Make sure the server’s own IP is not in Denied List.

Equivalent in /usr/local/lsws/conf/httpd_config.conf or /usr/local/lsws/conf/vhosts/<vhost>/vhost.conf:

accessControl  {
  allow                   127.0.0.1T, ::1T, 203.0.113.10T, ALL
}

4. Check per-client throttling

Server Configuration → Security → Per Client Throttling

The crawler opens several concurrent connections from one IP. If Connection Soft/Hard Limit or the request rates are low, OLS can throttle or ban the IP for the Banned Period. Trusting the IP (Step 3) is the cleanest fix. Alternatively, raise the limits:

perClientConnLimit  {
  staticReqPerSec         0
  dynReqPerSec            0
  outBandwidth            0
  inBandwidth             0
  softLimit               10000
  hardLimit               10000
  gracePeriod             15
  banPeriod               300
}

Restart after editing:

/usr/local/lsws/bin/lswsctrl restart

5. Cloudflare (only if Step 1 showed cf-mitigated)

Preferred: use Server IP so the crawler never touches Cloudflare.

Otherwise, create a rule at Cloudflare → Security → WAF → Custom rules with action Skip (skip all remaining custom rules, Managed Rules, and rate limiting):

(ip.src eq 203.0.113.10)

Match on source IP, not http.user_agent contains "lscache_runner", because the User-Agent is trivially spoofable.

6. Allowlist in the host firewall and security layers

CSF/LFD:

csf -a 203.0.113.10 "lscache crawler"
csf -r

ModSecurity: identify the triggering rule in the audit log (commonly /usr/local/lsws/logs/auditmodsec.log), then disable that rule ID for the vhost or exclude the server IP.

WordPress security plugins (Wordfence, Solid Security, AIOWPS): add the server IP to their allowlist. Also disable their rate limiting or “block fake crawlers” options for that IP.

7. Remove .htaccess blocks

grep -niE "HTTP_USER_AGENT|Require all denied|deny from|\[F" /path/to/docroot/.htaccess

Remove or scope any rewrite rule that returns [F] for empty, bot-like, or unknown User-Agents, or that denies the server’s own IP.

8. Reset crawler state

• LiteSpeed Cache → Crawler → Blocklist → clear/empty the blocklist (labelled Blacklist in older versions).
• LiteSpeed Cache → Crawler → Map (or Summary) → refresh the crawler map, reset position, then run the crawler manually.

9. Verify

curl -sI https://$DOMAIN/ | grep -i x-litespeed-cache

Run it twice. The second response should show x-litespeed-cache: hit. In access.log, lscache_runner requests should now return 200 instead of 403.

Why This Fails for Some Users

• Cloud NAT (AWS, GCP, Oracle, Azure): the public IP from ifconfig.me is not bound to the instance NIC. OLS may see the private IP (ip -4 addr) as the source of self-requests. Allowlist both the private and public IPs, and use the address that actually reaches OLS as Server IP.
• Reverse proxy in front of OLS (Nginx, HAProxy, Cloudflare Tunnel): OLS sees the proxy’s IP, not the client’s. Configure Server Configuration → General → Use Client IP in Header (with Trusted IP Only), and allowlist the proxy IP in the ACL.
• Cloudflare free plan: Bot Fight Mode cannot be bypassed by custom Skip rules. Only Server IP (direct-to-origin) or disabling Bot Fight Mode avoids it.
• Origin not publicly reachable (Tunnel or private origin): direct-to-IP crawling can fail on hairpin/NAT loopback with timeouts rather than 403. Test with the --resolve command from Quick Fix.
• IPv4/IPv6 mismatch: if the hostname has an AAAA record, the crawler may connect over IPv6 while only the IPv4 address is allowlisted. Add the ::1 and public IPv6 addresses too.
• wp-cron.php blocked: a 403 on /wp-cron.php (by a WAF, .htaccess, or security plugin) prevents the crawler from starting at all. Trigger it from real cron and bypass the CDN:* * * * * curl -s -o /dev/null --resolve example.com:443:203.0.113.10 https://example.com/wp-cron.php?doing_wp_cron
• Basic auth or staging locks: these return 401, not 403, but stall the crawler the same way. Exclude the crawler or lift the lock.
• Multiple sites on different IPs (multisite, domain mapping): Server IP is one global value. Sites bound to other IPs will still fail.
• UI variations: menu labels (Blocklist vs. Blacklist, location of Server IP, Map vs. Summary) differ across plugin versions. Update the plugin before assuming the option is missing.