How to clear dead Docker overlay2 storage directory safely without breaking daemon

by

Root Cause Diagnosis

/var/lib/docker/overlay2/ accumulates directories named by content-addressable hash for every image layer, stopped container’s writable layer, and build-cache intermediate that Docker has ever created; the daemon does not garbage-collect them automatically. Directories become “dead” (safe to remove) when their owning image, container, or build cache entry no longer exists in Docker’s internal reference graph, but a crashed daemon, a killed docker build, or manual interference with /var/lib/docker can also leave orphaned directories with no corresponding metadata, or conversely leave metadata pointing at a directory you deleted by hand. Manually deleting anything under overlay2/ without going through the Docker API breaks the mount references Docker tracks internally (in /var/lib/docker/image/overlay2/layerdb/ and the container/image JSON stores), which can corrupt running containers, fail future container starts with “no such file or directory”, or leave the daemon unable to start at all.

Quick Fix

Never rm -rf inside overlay2/ directly. Use Docker’s own pruning, which cross-references the reference graph before deleting anything:

docker system df -v
docker system prune -a --volumes

-a removes all unused images (not just dangling ones); --volumes removes unused volumes. Review docker system df -v output first if the host has data you don’t want to lose, since this command is destructive and irreversible.

Step-by-Step Resolution

1. Inventory current usage before touching anything

docker system df -v
du -sh /var/lib/docker/overlay2/* 2>/dev/null | sort -rh | head -20
df -h /var/lib/docker

docker system df -v shows reclaimable space per image, container, and volume, mapped through Docker’s own accounting. The raw du pass tells you which individual overlay2 directories are largest, useful for identifying whether the bloat is images, container logs, or build cache before you decide what to prune.

2. Remove unused containers first, then images, then build cache

Prune in this order — containers depend on images, and both depend on overlay2 layers, so top-down avoids the daemon holding references you’re trying to clear.

docker container prune -f
docker image prune -a -f
docker builder prune -a -f
docker volume prune -f

Check space reclaimed at each step:

docker system df

3. Identify genuinely orphaned overlay2 directories

After a full prune, cross-reference remaining overlay2/ directories against what Docker still knows about:

comm -23 \
  <(ls /var/lib/docker/overlay2/ | sort) \
  <(docker system df -v --format '{{json .}}' 2>/dev/null; \
    find /var/lib/docker/image/overlay2/layerdb/sha256 -maxdepth 1 -type d -exec basename {} \; | sort) \
  > /tmp/possible_orphans.txt

A more reliable method on modern Docker is to stop the daemon and use the built-in check rather than hand-rolled diffing, since layerdb references are stored as symlinked cache-id files, not directory names directly:

sudo systemctl stop docker
sudo systemctl stop docker.socket containerd

With the daemon stopped, no new references can appear mid-comparison. List directories under overlay2/ and cross-check each against cache-id files:

for d in /var/lib/docker/overlay2/*/; do
  id=$(basename "$d")
  grep -rl "$id" /var/lib/docker/image/overlay2/layerdb/sha256/*/cache-id 2>/dev/null || echo "ORPHAN: $id"
done > /tmp/overlay2_orphans.txt

wc -l /tmp/overlay2_orphans.txt

Review the list manually. Entries flagged ORPHAN have no cache-id file pointing at them and are candidates for removal, but confirm none are -init suffixed companions to a layer still in use (init layers share a parent hash prefix with their container layer).

See also  Resolving Linux swap partition not mounting automatically on system reboot

4. Remove confirmed orphans with the daemon stopped

sudo systemctl status docker   # confirm it is stopped, not just the socket
sudo tar -czf /root/overlay2-backup-$(date +%F).tar.gz \
  -C /var/lib/docker overlay2 image/overlay2/layerdb

while read -r orphan; do
  id="${orphan#ORPHAN: }"
  sudo rm -rf "/var/lib/docker/overlay2/${id:?}"
done < /tmp/overlay2_orphans.txt

The ${id:?} guard aborts the loop if a line parses to an empty string, preventing an accidental rm -rf /var/lib/docker/overlay2/. The tar backup lets you restore if a directory is removed incorrectly; delete it once the daemon starts cleanly and containers run normally.

5. Restart Docker and validate

sudo systemctl start containerd
sudo systemctl start docker

sudo systemctl status docker
docker ps -a
docker images
docker run --rm hello-world

If any existing container fails to start with an error referencing a missing merged, diff, or lower path under overlay2/, that container’s writable layer was affected. Recreate the container from its image (docker compose up -d or docker run with the original parameters); the container’s data is not recoverable if its layer was deleted, only stateless recreation is possible.

6. Prevent recurrence

Enable log rotation and periodic pruning rather than manual cleanup:

/etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "storage-driver": "overlay2"
}

Add a scheduled prune (adjust retention to your build cadence):

sudo crontab -e

0 3 * * 0 docker system prune -af --filter "until=168h" >> /var/log/docker-prune.log 2>&1

Why This Fails for Some Users

  • Daemon won’t start after manual deletion: If a directory referenced by a cache-id file was removed, dockerd logs error mounting overlay: no such file or directory or layer does not exist on startup and refuses to serve requests for affected images. Restoring from the tar backup (Step 4) or fully reinstalling affected images (docker pull/docker build again) is the only recovery.
  • overlay2 vs overlay2-fuse or zfs/btrfs drivers: This procedure applies only to the overlay2 storage driver. Confirm with docker info --format '{{.Driver}}' before following layerdb-based orphan detection; other drivers use different metadata layouts (zfs datasets, btrfs subvolumes) and the layerdb/sha256/*/cache-id cross-reference does not apply.
  • Kernel overlay filesystem limits: Some older kernels (pre-4.x) or specific distributions impose a maximum overlay stacking depth; large images with many layers can hit too many levels of symbolic links independent of disk space, which pruning does not fix. This requires reducing image layer count (multi-stage builds, --squash) rather than storage cleanup.
  • Containers using --rm with orphaned state: Containers started with --rm that crash mid-write can leave a writable layer directory behind if the daemon itself crashes before cleanup completes; docker container prune won’t find them since no container object exists, but docker system prune catches them via dangling layer detection in most cases. Reboot and re-run prune if directories persist.
  • Build cache from BuildKit vs legacy builder: docker builder prune only clears the currently active builder’s cache. If BuildKit was enabled (DOCKER_BUILDKIT=1) after builds ran on the legacy builder, or vice versa, orphaned cache from the other builder can remain. Check both: docker buildx du for BuildKit, docker system df for legacy.
  • Overlay2 directories held open by a zombie process: rm on a directory with an active bind mount or a process holding an open file descriptor fails silently or leaves .wh. whiteout files behind. Check with lsof +D /var/lib/docker/overlay2/<id> and mount | grep <id> before forcing removal; unmount first with umount /var/lib/docker/overlay2/<id>/merged if it appears mounted.
  • Rootless Docker: Paths differ entirely — data lives under ~/.local/share/docker/overlay2/ and the daemon runs as the invoking user, not root. Adjust all paths and drop sudo from commands, or they will silently operate on the wrong (nonexistent) root-owned path.
  • containerd managed separately from Docker (Kubernetes nodes running both): On nodes where containerd also serves a Kubernetes CRI socket independent of the Docker daemon, overlay2 directories under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/ are a separate, parallel structure from Docker’s own /var/lib/docker/overlay2/. Pruning Docker’s does not affect containerd’s CRI-managed layers, and vice versa; confirm which subsystem owns the space you’re trying to reclaim with crictl vs docker.

Leave a Comment