Resolving WordPress transient cache buildup causing 504 Gateway Timeout on admin-ajax.php

by

Without a persistent object cache, WordPress stores every transient as two rows in wp_options, and an expired row is deleted only when get_transient() requests that exact key or when the daily delete_expired_transients cron event runs. Plugins that write per-request or per-user keys leave hundreds of thousands of rows that are never read again, and the daily cleanup then runs one unbatched multi-table DELETE that holds locks long enough for concurrent admin-ajax.php calls (Heartbeat, autosave, plugin AJAX) to queue. Once every PHP-FPM worker is busy, or a request exceeds request_terminate_timeout, fastcgi_read_timeout, or proxy_read_timeout, the proxy returns 504.

Root Cause Diagnosis

  1. Row accumulation. set_transient() writes _transient_{key} and _transient_timeout_{key} to wp_options. Keys that embed a session ID, URL, or user ID are written once and never read again, so they never expire on access.
  2. Autoload bloat. Transients set without an expiry are written with autoload = yes and are loaded into memory on every request through the alloptions query. Transients with an expiry are normally non-autoloaded, so autoload bloat points to no-expiry transients or non-transient options.
  3. Cleanup contention. The core cleanup runs a single DELETE a, b FROM wp_options a, wp_options b ... join. On a table with hundreds of thousands of transient rows it can run for tens of seconds, blocking writers such as update_option() and set_transient() that Heartbeat and plugin AJAX handlers depend on.
  4. Worker exhaustion. Blocked requests occupy PHP-FPM workers until pm.max_children is reached; new admin-ajax.php requests then queue at the proxy until it times out.

Quick Fix

Delete expired transients with WP-CLI, then flush the object cache:

wp transient delete --expired --allow-root
wp transient delete --expired --network --allow-root   # multisite only
wp cache flush --allow-root

If the table holds more than roughly 100,000 transient rows, skip this and use the batched delete in step 4. A single large DELETE can itself trigger the 504.

Step-by-Step Resolution

1. Back up the database

wp db export /root/pre-transient-cleanup-$(date +%F).sql --allow-root

2. Confirm the diagnosis

Count transient rows and measure autoloaded data:

P=$(wp db prefix --allow-root)
wp db query "SELECT COUNT(*) AS transient_rows FROM ${P}options WHERE option_name LIKE '\\_transient\\_%' OR option_name LIKE '\\_site\\_transient\\_%';" --allow-root
wp db query "SELECT ROUND(SUM(LENGTH(option_value))/1024/1024,2) AS autoload_mb FROM ${P}options WHERE autoload IN ('yes','on','auto','auto-on');" --allow-root

Tens of thousands of transient rows, or autoload data above roughly 1 MB, confirms the diagnosis. List the largest autoloaded rows, which are often not transients:

SELECT option_name, LENGTH(option_value) AS bytes
FROM wp_options
WHERE autoload IN ('yes','on','auto','auto-on')
ORDER BY bytes DESC
LIMIT 20;

Capture where requests actually stall by enabling the PHP-FPM slow log in /etc/php/8.x/fpm/pool.d/www.conf (replace 8.x with your version):

request_slowlog_timeout = 10s
slowlog = /var/log/php8.x-fpm.slow.log

systemctl reload php8.x-fpm
tail -f /var/log/php8.x-fpm.slow.log

Stack traces ending in wpdb::query, set_transient, or delete_expired_transients point to this issue. Traces ending in curl_exec or file_get_contents point to outbound HTTP calls, which are a different cause.

See also  Selenium Chromedriver "session not created: DevToolsActivePort file does not exist" Fix

3. Identify the plugin generating the keys

SELECT SUBSTRING_INDEX(option_name, '_', 4) AS prefix, COUNT(*) AS n
FROM wp_options
WHERE option_name LIKE '\_transient\_%'
GROUP BY prefix
ORDER BY n DESC
LIMIT 15;

The dominant prefix usually matches a plugin slug. Update, reconfigure, or replace that plugin; otherwise the table refills.

4. Delete in batches

Expired-only is the safe default. Deleting all transients is also safe for most sites because they regenerate, but a few plugins store state (OAuth state, one-time notices, maintenance flags) in transients.

P=$(wp db prefix --allow-root)
while :; do
  n=$(wp db query "DELETE FROM ${P}options WHERE option_name LIKE '\\_transient\\_%' OR option_name LIKE '\\_site\\_transient\\_%' LIMIT 5000; SELECT ROW_COUNT();" --skip-column-names --allow-root)
  [ "${n:-0}" -eq 0 ] && break
  sleep 1
done

The sleep 1 gives concurrent queries a window between batches and limits replica lag on replicated setups.

5. Stop autoloading transients

'no' is valid on all WordPress versions, including 6.6+, which introduced additional autoload values.

UPDATE wp_options
SET autoload = 'no'
WHERE (option_name LIKE '\_transient\_%' OR option_name LIKE '\_site\_transient\_%')
  AND autoload <> 'no';

6. Move transients to a persistent object cache

With a persistent object cache, set_transient() writes to the cache backend instead of wp_options, which removes the buildup at its source.

apt install redis-server php-redis
systemctl enable --now redis-server

Add to wp-config.php, above the line /* That's all, stop editing! */:

define('WP_REDIS_HOST', '127.0.0.1');
define('WP_REDIS_PORT', 6379);
define('WP_REDIS_TIMEOUT', 1);
define('WP_REDIS_READ_TIMEOUT', 1);
define('WP_CACHE_KEY_SALT', 'example.com:');

The 1-second timeouts prevent an unreachable Redis server from stalling every request. Install the drop-in and verify:

wp plugin install redis-cache --activate --allow-root
wp redis enable --allow-root
wp redis status --allow-root

Confirm wp-content/object-cache.php exists and the status reads Connected. Set a memory cap in /etc/redis/redis.conf:

maxmemory 256mb
maxmemory-policy allkeys-lru

Rows already in wp_options stay there after the switch; remove them with step 4.

7. Move WP-Cron to system cron

Add to wp-config.php:

define('DISABLE_WP_CRON', true);

Create /etc/cron.d/wordpress:

*/5 * * * * www-data /usr/local/bin/wp --path=/var/www/html cron event run --due-now --quiet
15 3 * * * www-data /usr/local/bin/wp --path=/var/www/html transient delete --expired --quiet

Adjust the user, WP-CLI path, and site path. The second line is unnecessary once a persistent object cache is active.

See also  How to capture network response body in Playwright without blocking navigation

8. Reduce Heartbeat load

Create wp-content/mu-plugins/heartbeat.php:

<?php
add_filter('heartbeat_settings', function ($settings) {
    $settings['interval'] = 60; // allowed range: 15-120 seconds
    return $settings;
});

add_action('admin_enqueue_scripts', function ($hook) {
    if ($hook === 'index.php') { // dashboard only; keep it on post editors for locking
        wp_deregister_script('heartbeat');
    }
});

9. Size PHP-FPM and timeouts correctly

Measure average worker memory and derive pm.max_children:

ps --no-headers -o rss -C php-fpm8.x | awk '{s+=$1;n++} END {printf "%.0f MB avg per worker\n", s/n/1024}'
free -m

pm.max_children ≈ (RAM available to PHP in MB) / (average worker MB)

Set it in /etc/php/8.x/fpm/pool.d/www.conf. Raise timeouts only as a stopgap during cleanup:

request_terminate_timeout = 120

Nginx, in the location ~ \.php$ block:

fastcgi_read_timeout 120s;

Apache with mod_proxy_fcgi:

ProxyTimeout 120

nginx -t && systemctl reload php8.x-fpm nginx

10. Verify

wp db query "SELECT COUNT(*) FROM $(wp db prefix --allow-root)options WHERE option_name LIKE '\\_transient\\_%';" --allow-root
curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" -X POST https://example.com/wp-admin/admin-ajax.php -d "action=heartbeat"
grep -c "reached pm.max_children" /var/log/php8.x-fpm.log

Expect a 200 response well under one second, a stable transient row count, and no new pm.max_children warnings.

Why This Fails for Some Users

  • The source plugin is still active. Rows return within hours or days. Step 3 is mandatory; deletion alone is temporary.
  • wp_options is not the bottleneck. Slow queries on wp_postmeta, wp_usermeta, Action Scheduler tables (wp_actionscheduler_*), or WooCommerce session tables produce the same 504. Confirm with the slow query log:SET GLOBAL slow_query_log = 1; SET GLOBAL long_query_time = 2; SHOW VARIABLES LIKE 'slow_query_log_file';
  • Autoload bloat is not from transients. If step 2 shows large non-transient rows, the cause is a plugin storing large settings in autoloaded options. Set those to non-autoload individually with wp option autoload set <name> no (WP-CLI 2.10+ with WordPress 6.4+), or with an UPDATE.
  • PHP-FPM worker exhaustion without database slowness. Long outbound HTTP calls (license checks, update checks, remote APIs) also hold workers. The slow log from step 2 distinguishes them from database stalls.
  • Multiple timeout layers. Nginx, Apache, PHP-FPM, a load balancer, and a CDN each have independent limits. Cloudflare returns 524 after 100 seconds on non-Enterprise plans regardless of origin settings. Test against the origin directly:curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" --resolve example.com:443:ORIGIN_IP -X POST https://example.com/wp-admin/admin-ajax.php -d "action=heartbeat"
  • Memcached item size limit. The alloptions array is cached as one item. If it exceeds Memcached’s default 1 MB item limit, the write silently fails and WordPress queries the database on every request. Start Memcached with -I 4m or shrink autoload data.
  • Redis misconfiguration. Without maxmemory and an eviction policy, Redis can exhaust RAM. A stale object-cache.php from a removed plugin can break all requests; delete it from wp-content/. When a persistent cache is active, wp transient delete operates on the cache, so leftover database rows must be removed with SQL (step 4).
  • Shared hosting. WP-CLI, Redis, and system cron are often unavailable. Use phpMyAdmin with the same DELETE ... LIMIT statement, repeated until zero rows are affected, and ask the provider for an object cache.
  • Multisite. Site transients live in wp_sitemeta, and each subsite has its own wp_{blog_id}_options table. Run cleanup per site:wp site list --field=url --allow-root | xargs -I % wp --url=% transient delete --expired --allow-root
  • MyISAM tables. MyISAM locks the whole table during DELETE, blocking all reads. Convert to InnoDB, which also rebuilds the table:ALTER TABLE wp_options ENGINE=InnoDB;
  • Fragmentation after mass deletion. Space and performance do not fully recover until the table is rebuilt:OPTIMIZE TABLE wp_options; On InnoDB this uses online DDL with brief locks at the start and end, and it needs free disk space roughly equal to the table size. Run it off-peak.
  • Replication lag. Large deletes on a primary delay replicas, and read-split setups can serve stale alloptions data. Keep batch sizes at or below 5,000 rows and watch Seconds_Behind_Source.
  • Low-memory VPS (1 GB or less). An oversized innodb_buffer_pool_size or Redis maxmemory causes swapping, which looks like a timeout problem. Size the buffer pool to roughly 50-60% of RAM left after PHP-FPM, in /etc/mysql/mysql.conf.d/mysqld.cnf, and check free -m and vmstat 1.
  • WAF or ModSecurity rules. Body inspection on admin-ajax.php POST requests can add latency independent of the database. Check /var/log/apache2/modsec_audit.log or the security plugin’s firewall log.
  • Deleting all transients removes plugin state. If a plugin stores non-regenerable data in transients, restrict step 4 to expired rows by adding option_value < UNIX_TIMESTAMP() on the matching _transient_timeout_ rows, or use the join form from the core cleanup.
See also  Handling sticky dynamic session IDs in Python Scrapy without losing referer headers

Leave a Comment