Fix: Docker Exited with Code 137 on Low-Memory ARM Instances

by

Root Cause Diagnosis

Exit code 137 is 128 + 9, indicating the container process received SIGKILL — on low-memory ARM instances this is almost always the Linux kernel OOM killer terminating the container because it exceeded available memory, not a Docker-internal failure. ARM instances (particularly AWS Graviton, Raspberry Pi, and budget ARM VPS tiers) compound this because many base images and pre-built binaries default to memory allocation patterns tuned for x86 JIT/runtime behavior, and ARM builds of runtimes like Node.js, Java, and Python often carry higher baseline memory overhead per process on identical workloads, pushing containers over unset or misconfigured memory limits faster than on equivalent x86 hardware.

Quick Fix (1-Minute Override)

Check if this was actually an OOM kill (not a crash) and confirm immediately:

docker inspect <container_id> --format='{{.State.OOMKilled}}'

If true, raise the container’s memory limit as an immediate stop-gap:

docker update --memory=512m --memory-swap=1g <container_id>

This buys headroom while you diagnose the actual memory consumer in Step 2 below — it is not a permanent fix if the underlying process has a real memory leak.

Step-by-Step Resolution

Step 1: Confirm OOM Kill via System Logs

Cross-check the kernel log to confirm the OOM killer specifically targeted this container’s process (not an unrelated system process, which would point to system-wide memory pressure instead):

dmesg -T | grep -i "killed process"

Look for oom-kill entries with a pid matching the container’s process:

docker inspect <container_id> --format='{{.State.Pid}}'

Step 2: Check Actual Memory Usage Pattern

Before setting arbitrary limits, profile actual consumption to size limits correctly:

docker stats <container_id> --no-stream

For a fuller picture over time (catches spikes that --no-stream misses):

docker stats <container_id>

Watch MEM USAGE / LIMIT during the workload that triggers the crash — if usage climbs steadily rather than spiking once, this indicates a memory leak in the application rather than an undersized limit.

See also  Docker Container Time Drift: Syncing Container Clock with Host RTC

Step 3: Set Explicit Memory Limits in docker-compose.yml

Avoid relying on defaults (no limit = container can consume all host memory, triggering system-wide OOM behavior that kills unpredictably). Set explicit bounds:

services:
  app:
    image: your-app:latest
    deploy:
      resources:
        limits:
          memory: 384M
        reservations:
          memory: 128M
    mem_swappiness: 0

For plain docker run (non-compose):

docker run --memory=384m --memory-swap=384m --oom-kill-disable=false your-app:latest

Note: --memory-swap equal to --memory disables swap for the container entirely — appropriate on ARM instances with no swap configured (common default on cloud ARM tiers).

Step 4: Configure or Verify Host-Level Swap

Many ARM cloud instances (particularly AWS Graviton t4g/a1 tiers) ship with zero swap by default. Without swap, any memory spike beyond the physical RAM ceiling triggers immediate OOM kill rather than degraded performance:

free -h

If swap shows 0 across the board, add a swap file:

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Verify:

swapon --show

Step 5: Tune Application-Level Memory Ceilings to Match Container Limits

If the container runs a JVM, Node.js, or similar runtime with its own heap management, an unset runtime memory ceiling can exceed the Docker container limit before the runtime’s own GC triggers, causing the kernel OOM killer to act first. Explicitly cap the runtime below the container limit:

Node.js (Dockerfile or entrypoint):

CMD ["node", "--max-old-space-size=256", "server.js"]

Java:

ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75.0", "-jar", "app.jar"]

Set the runtime ceiling to roughly 75-80% of the container’s --memory limit, leaving headroom for non-heap overhead (thread stacks, native buffers).

Step 6: Adjust Kernel OOM Score to Deprioritize Non-Critical Containers

On multi-container hosts where a low-priority container should be killed before a critical one under memory pressure, set oom_score_adj per container:

services:
  low-priority-worker:
    image: worker:latest
    oom_score_adj: 500
  critical-api:
    image: api:latest
    oom_score_adj: -500

Lower values (down to -1000) make a container less likely to be killed; higher values (up to 1000) make it a preferred kill target.

See also  Fixing rsync error: some files/attrs were not transferred (see previous errors) (code 23)

Why This Fails for Some Users

  • cgroup v1 vs v2 memory accounting differences: Some older ARM-based distributions (Raspberry Pi OS pre-Bullseye, older Ubuntu ARM builds) default to cgroup v1, which handles memory limit enforcement differently than cgroup v2 — specifically, v1 can undercount memory used by kernel-side caches attributed to the container, causing limits to be hit earlier than the configured value suggests. Check with:mount | grep cgroup Migrating to cgroup v2 (default on newer kernels, 5.8+) resolves the discrepancy but requires a full OS/kernel upgrade path, not a Docker-level fix.
  • QEMU emulation overhead on cross-architecture builds: If the image being run was built for x86_64 and is running under QEMU emulation on an ARM host (rather than a native ARM image), emulation overhead roughly doubles effective memory usage per process, causing containers that fit comfortably on native ARM to OOM under emulation. Confirm architecture match with:docker inspect <image_name> --format='{{.Architecture}}' uname -m These must match, or the image must be an explicit multi-arch manifest with a native arm64 variant.
  • Raspberry Pi GPU memory split reducing available system RAM: On Raspberry Pi-based Docker hosts, the gpu_mem setting in /boot/config.txt reserves RAM for GPU use, which is invisible to free -h as available memory but still reduces the pool Docker containers can draw from. On headless server deployments, reduce this to the minimum:gpu_mem=16
  • Missing swap accounting kernel parameter on ARM64 kernels: Some ARM64 kernel builds ship without swapaccount=1 enabled at boot, which silently disables Docker’s ability to enforce --memory-swap limits correctly, causing containers to consume unconstrained swap before an OOM kill occurs inconsistently. Check and add to boot parameters if using GRUB (uncommon on ARM but present on some ARM server boards):cat /proc/cmdline | grep swapaccount
  • Docker Desktop resource limits on ARM Macs (Apple Silicon) misapplied to remote ARM deployments: Developers testing locally on M1/M2 Macs sometimes assume local success guarantees production behavior on ARM cloud instances, but Docker Desktop’s VM-level memory allocation (configurable in Docker Desktop settings, often generously set to 4GB+ by default) masks limits that a production ARM instance with genuinely constrained physical RAM (512MB-1GB tiers) will hit immediately — always test against a resource-limited container locally before deploying to a low-memory ARM target.
See also  Linux out of memory killer (oom-killer) picking wrong high-priority daemon: Tuning oom_score_adj

Leave a Comment