Resolving High System Load Average While CPU and RAM Utilization Remain Low (Uninterruptible Sleep)

by

Root Cause Diagnosis

Linux load average counts tasks in the runnable state (R) plus tasks in uninterruptible sleep (D), so processes blocked inside the kernel waiting on I/O inflate the load figure without consuming CPU cycles or memory. The blocked tasks are almost always waiting on a storage device, a network filesystem (NFS/CIFS), a hung or saturated disk queue, a failing drive, or a kernel lock, and they cannot be killed by signals until the I/O completes or times out. The fix is to identify which device, mount, or driver the D state tasks are blocked on, then resolve that dependency.

Quick Fix

List the tasks in D state and the kernel function they are blocked in:

ps -eo state,pid,ppid,user,wchan:32,cmd | awk '$1=="D"'

Count them and watch device saturation at the same time:

ps -eo state | grep -c '^D'
iostat -xz 1 5

If the wchan column shows nfs_*, rpc_*, or io_schedule, the blocker is a stalled mount or a saturated disk. For a hung NFS mount, lazily detach it:

umount -f -l /mnt/nfs_share

Step-by-Step Resolution

1. Confirm the load is I/O wait and not CPU

uptime
vmstat 1 5

In vmstat, a high b column (blocked processes) and a high wa value, with low us and sy, confirm uninterruptible sleep. Compare with r (runnable); if b is large and r is small, the load is I/O driven.

cat /proc/pressure/io
cat /proc/pressure/cpu
cat /proc/pressure/memory

On kernels 4.20 or later with PSI enabled, some avg10 in /proc/pressure/io above roughly 10 confirms I/O stall. If the file is missing, add psi=1 to the kernel command line.

2. Identify the blocked processes and their kernel stacks

# Per-process state and wait channel
ps -eo pid,stat,wchan:32,cmd | awk '$2 ~ /^D/'

# Full kernel stack of a specific blocked PID (requires root)
cat /proc/<PID>/stack

# Dump all blocked task stacks to the kernel log
echo w > /proc/sysrq-trigger
dmesg | tail -n 200

Enable SysRq first if it is disabled:

sysctl -w kernel.sysrq=1

Look for frames such as nfs_wait_bit_killable, jbd2_log_wait_commit, blk_mq_get_tag, wait_on_page_bit, or scsi_*. The top frame identifies the subsystem.

3. Map the block to a device or mount

# Which files are open by the blocked PID
ls -l /proc/<PID>/fd

# Which process holds a mount busy
fuser -vm /mnt/nfs_share
lsof +f -- /mnt/nfs_share

lsof and fuser can themselves hang on a dead NFS mount. Use a timeout:

timeout 5 stat /mnt/nfs_share || echo "mount unresponsive"

4. Check disk health and queue saturation

iostat -xz 1 10

Check these columns: %util near 100, await (or r_await/w_await) above 20 ms on SSDs or 50 ms on HDDs, and aqu-sz (average queue size) greater than the device queue depth. High await with low throughput indicates a failing or throttled device.

# SMART health
smartctl -a /dev/sda
smartctl -H /dev/nvme0n1

# Kernel I/O errors and resets
dmesg -T | grep -Ei 'i/o error|blk_update_request|task abort|reset|timeout|nvme|ata[0-9]'

A non-zero Reallocated_Sector_Ct, Current_Pending_Sector, or repeated ata link resets point to a failing drive. Replace it, since software tuning will not fix it.

See also  How to remove stubborn immutable Linux file when rm -f fails with Operation not permitted

5. Find the process generating the I/O

iotop -oPa
pidstat -d 1 10

Without iotop, use:

cat /proc/<PID>/io

Rising write_bytes or read_bytes identifies the source. Throttle or reschedule the offender:

ionice -c3 -p <PID>       # idle I/O class
renice +10 -p <PID>

ionice classes only take effect with the bfq or cfq scheduler, not none or mq-deadline.

6. Fix NFS and network filesystem stalls

Check current mount options:

nfsstat -m
mount | grep nfs

For a hard mount on an unreliable network, tasks stay in D state indefinitely. Use options that bound the wait in /etc/fstab:

nfs-server:/export  /mnt/nfs_share  nfs  soft,timeo=100,retrans=3,bg,nofail,_netdev,vers=4.2  0 0

timeo is in tenths of a second, so timeo=100 is 10 seconds. soft returns an I/O error to the application instead of blocking, which risks data corruption for writes. Use hard,intr (or hard alone on kernels 2.6.25 and later, where intr is ignored) for data you cannot lose, and fix the network or server instead. Confirm server-side:

rpcinfo -p nfs-server
showmount -e nfs-server
ss -tn state established '( dport = :2049 )'

For CIFS, set echo_interval=10 and soft in the mount options, and check dmesg for CIFS: server not responding.

7. Check the I/O scheduler and queue depth

cat /sys/block/sda/queue/scheduler
cat /sys/block/sda/queue/nr_requests
cat /sys/block/sda/device/queue_depth

Set a suitable scheduler persistently with a udev rule in /etc/udev/rules.d/60-ioscheduler.rules:

ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"

udevadm control --reload && udevadm trigger

8. Check writeback and dirty page settings

Large dirty-page buildup followed by synchronous flushes can block writers in D state.

sysctl vm.dirty_ratio vm.dirty_background_ratio vm.dirty_bytes vm.dirty_background_bytes
grep -E 'Dirty|Writeback' /proc/meminfo

On systems with large RAM and slow disks, use byte limits in /etc/sysctl.d/99-writeback.conf:

vm.dirty_background_bytes = 268435456
vm.dirty_bytes = 1073741824

sysctl --system

Setting the _bytes variants zeroes the matching _ratio setting.

See also  Docker Container Time Drift: Syncing Container Clock with Host RTC

9. Check filesystem, journal, and RAID/LVM layers

# ext4 journal commit stalls
dmesg -T | grep -Ei 'jbd2|ext4|xfs|btrfs'

# Software RAID resync consuming bandwidth
cat /proc/mdstat

# LVM snapshots slowing writes
lvs -a -o +devices

During an md resync, cap the speed in /etc/sysctl.d/99-md.conf:

dev.raid.speed_limit_max = 50000
dev.raid.speed_limit_min = 1000

Large LVM snapshots degrade write performance because of copy-on-write, so remove stale snapshots with lvremove.

10. Configure a hung task watchdog for visibility

sysctl kernel.hung_task_timeout_secs
sysctl -w kernel.hung_task_timeout_secs=60

Tasks in D state beyond this period are logged to dmesg with a stack trace. Set kernel.hung_task_panic=0 (default) unless you want a deliberate crash for a vmcore.

Why This Fails for Some Users

  • Load average on Linux includes D state by design: Unlike BSD and most other Unix systems, Linux counts uninterruptible tasks, so a high number is not necessarily a fault. A fixed set of long-lived D tasks (for example an nfsd thread or a kernel worker) can add 1 to the load permanently. Verify that the processes in D state are new or growing in number before treating the load as a problem.
  • Kernel threads in D state are normal: Idle kernel threads such as kworker, jbd2/*, or nfsd sometimes show D transiently. Use PSI (/proc/pressure/io) as the authoritative metric, because it measures actual stall time.
  • Containers hide the cause: Inside a container, ps shows only that namespace’s processes, and iostat reports the host devices through cgroup-agnostic counters. Run diagnostics on the host, and check cgroup I/O pressure at /sys/fs/cgroup/<group>/io.pressure (cgroup v2).
  • Cloud and virtual disks are throttled invisibly: AWS EBS gp2 burst credits, gp3 provisioned IOPS/throughput, Azure managed disk caps, and GCP persistent disk quotas throttle I/O at the hypervisor without any error in dmesg. The symptom is high await with a constant IOPS or throughput ceiling in iostat. Check the provider’s volume metrics (BurstBalance, VolumeQueueLength) and raise the provisioned limits.
  • Steal time and noisy neighbors: On shared VMs, %st in vmstat or top inflates stalls and can mimic I/O wait. Move to a dedicated instance if steal is consistently above 5%.
  • Swap activity with low reported RAM use: Check vmstat columns si and so. Heavy swapping on a slow device produces D tasks even when free appears healthy, which happens when cgroup memory limits (not host RAM) trigger reclaim. Inspect memory.events and memory.pressure in the cgroup.
  • soft NFS mounts can corrupt data: The soft option makes tasks fail rather than hang, but writes may be silently lost on timeout. Use it only for read-mostly or disposable data, and prefer fixing network or server availability.
  • Hung D processes cannot be killed: kill -9 has no effect until the blocking call returns. If the I/O never completes (dead NFS server, failed disk, hung HBA), the only options are restoring the backend, forcing the mount off with umount -f -l, resetting the device through /sys/block/sdX/device/delete or echo 1 > /sys/block/sdX/device/rescan, or rebooting.
  • Old kernels and drivers: RHEL/CentOS 6 and 7 (kernels 2.6.32 and 3.10), and drivers with known blk-mq or NVMe timeout bugs, can leave requests stuck indefinitely. Update the kernel and storage firmware, and check nvme_core.default_ps_max_latency_us=0 and pcie_aspm=off on the kernel command line for NVMe drives that drop off the bus.
  • Multipath failover delays: With multipath-tools, queued I/O during path failure (queue_if_no_path or no_path_retry queue) holds tasks in D state indefinitely. Set no_path_retry 5 (or a finite count) in /etc/multipath.conf so failures surface instead of hanging.
  • FUSE and userspace filesystems: A stuck FUSE daemon (sshfs, s3fs, rclone mount) puts every accessing process in D state. Check /sys/fs/fuse/connections/*/ and abort a hung connection with echo 1 > /sys/fs/fuse/connections/<N>/abort.

Leave a Comment