Fix: Apache graceful restart failing due to lingering defunct workers

Root Cause Diagnosis

A defunct (Z state) process is a child that has already exited but has not been reaped by its parent via wait(), so kill -9 has no effect on it. On apachectl graceful (SIGUSR1), the parent httpd retires the old generation and expects to reap each old child as it exits. If the parent is blocked (I/O hang, hung piped logger) or the process that forked the children (a worker running CGI or PHP proc_open, or a non-init PID 1 in a container) never calls wait(), the zombies stay in the process table. They keep occupying scoreboard slots and PID/task quota, so the new generation cannot spawn and the reload stalls or fails with AH00159: fork: Unable to fork new process or scoreboard is full, not at MaxRequestWorkers.

Quick Fix

Find out who the zombies’ parent is:

ps -eo pid,ppid,stat,etime,cmd | awk '$3 ~ /^Z/ {print $2}' | sort | uniq -c

Then act on that PPID:

# PPID is a worker (child of the main httpd): terminate it; zombies are reparented and reaped
kill -TERM <PPID>

# PPID is the main httpd (the PID in the pidfile): perform a full restart, not a graceful one
systemctl restart httpd        # Debian/Ubuntu: systemctl restart apache2

If restart hangs on a process in D state, see the NFS/storage case under “Why This Fails for Some Users”.

Step-by-Step Resolution

1. Identify the main process and the pidfile

apachectl -t -D DUMP_RUN_CFG | grep -i pid
cat /run/httpd/httpd.pid          # RHEL family
cat /run/apache2/apache2.pid      # Debian family

2. Inspect the process tree and states

ps -eo pid,ppid,stat,etime,wchan:24,cmd --forest | grep -E '[h]ttpd|[a]pache2|defunct'

• A Z entry whose PPID is the main PID means the parent is not reaping.
• A Z entry whose PPID is a worker means the worker’s CGI/PHP subprocess was never waited on.
• Workers in D state point to blocked I/O, not an Apache bug.

3. Check whether the parent is blocked

MAIN=$(cat /run/httpd/httpd.pid)
ps -o pid,stat,wchan:32,cmd -p "$MAIN"
cat /proc/$MAIN/stack          # root only
strace -p "$MAIN" -f -e trace=wait4,waitid,write   # look for a stuck write() to a pipe

4. Check piped loggers

A hung logger fills its pipe, so workers block in write() and never finish, which stalls the graceful cycle.

grep -RnE 'CustomLog|ErrorLog' /etc/httpd /etc/apache2 2>/dev/null | grep '|'
ps -C rotatelogs -o pid,ppid,stat,cmd

Replace piped logging with file logging plus logrotate:

CustomLog "logs/access_log" combined

# /etc/logrotate.d/httpd
/var/log/httpd/*log {
    weekly
    missingok
    sharedscripts
    postrotate
        /usr/bin/systemctl reload httpd.service > /dev/null 2>&1 || true
    endscript
}

5. Give the scoreboard headroom for the old generation

During a graceful restart, old and new generations coexist. Set ServerLimit above the steady-state process count.

/etc/httpd/conf.modules.d/00-mpm.conf (Debian: /etc/apache2/mods-available/mpm_event.conf)

<IfModule mpm_event_module>
    ServerLimit          32
    ThreadsPerChild      25
    MaxRequestWorkers   600      # 24 active processes; 8 slots spare for the retiring generation
    KeepAliveTimeout      5
</IfModule>

Requirement: MaxRequestWorkers <= ServerLimit * ThreadsPerChild. Lower KeepAliveTimeout and Timeout so old workers drain faster.

6. Raise systemd process/task limits if forks fail

systemctl edit httpd

[Service]
TasksMax=4096
LimitNPROC=4096

systemctl daemon-reload

7. Fix the zombie source in application code

• PHP (mod_php): every proc_open() needs a matching proc_close(); every popen() needs pclose(). Otherwise the zombie persists until that worker exits.
• CGI scripts that background a process must redirect stdin/stdout/stderr (cmd >/dev/null 2>&1 &), or the worker stays busy.
• Cap worker lifetime so leaked zombies are cleared when the worker exits:

MaxConnectionsPerChild 10000

8. Containers: run an init as PID 1

docker run --init ...

# docker-compose.yml
services:
  web:
    init: true

Or use tini as the ENTRYPOINT. Orphaned grandchildren reparent to PID 1; if PID 1 is httpd, it does not reap children it did not spawn itself.

9. Validate and reload

apachectl configtest && systemctl reload httpd
apachectl fullstatus | grep -iE 'generation|requests currently'   # requires mod_status
ps -eo stat,cmd | grep -c '[d]efunct'                              # expect 0

Why This Fails for Some Users

• Parent in D state (NFS/CIFS DocumentRoot or log path, failing disk): Signals are not delivered until the I/O returns, so restart hangs too. Fix the mount, or as a last resort umount -l <path>, then systemctl kill -s SIGKILL httpd && systemctl start httpd. Use soft,timeo= mount options for non-critical paths.
• GracefulShutdownTimeout has no effect: It applies only to graceful-stop, not to graceful restart. Bound drain time with Timeout, KeepAliveTimeout, and (with mod_http2, where supported) H2GracefulShutdownTimeout.
• Long-lived connections (WebSocket via mod_proxy_wstunnel, streaming, HTTP/2): Old workers stay in G (gracefully finishing) state indefinitely. These are not defunct, but they produce the same scoreboard exhaustion. Verify with apachectl fullstatus.
• Zombies accumulate under a non-Apache parent: If PPID is php-fpm, fcgid, or a supervisor rather than httpd, restarting Apache will not clear them. Restart or fix that service.
• cgroup and rlimit ceilings: Zombies count toward pids.max (TasksMax), RLIMIT_NPROC for the apache/www-data user, and kernel.pid_max. Check with systemctl show httpd -p TasksMax,TasksCurrent and ulimit -u for the service user.
• MPM differences: prefork ties one process to one connection, so the slot count is large and exhaustion is fast. event and worker retire processes with active threads more slowly, so the headroom formula in step 5 matters more.
• Debian/Ubuntu logrotate collisions: postrotate runs reload (a graceful restart). If it fires while old workers are still draining, restarts overlap. Stagger rotation away from peak traffic and avoid multiple rotate jobs targeting the same service.

---

I can also put this into a Markdown file for you if you’d like.