Linux systemd service status: main process exited, code=killed, status=9/KILL

Root Cause Diagnosis

code=killed, status=9/KILL means the service’s main PID was terminated by SIGKILL, which cannot be caught or handled. It was not a clean exit and not an application crash. systemd reports the signal but not the sender, and the sender is one of four things: the kernel OOM killer (global memory exhaustion, or a cgroup MemoryMax= limit), systemd-oomd acting on memory pressure, systemd’s own timeout escalation (SIGTERM, then SIGKILL after TimeoutStopSec=/TimeoutStartSec=), or an external kill -9 from a script, monitor, or container runtime. The Result field in systemctl show and the kernel log narrow this down in seconds.

Quick Fix

Identify the killer first:

UNIT=myapp.service
systemctl show "$UNIT" -p Result,ExecMainCode,ExecMainStatus,MemoryMax,OOMPolicy,TimeoutStopUSec
journalctl -k -b | grep -iE 'out of memory|oom-kill|killed process'
journalctl -u "$UNIT" -b --no-pager | tail -40

Result= / log evidence	Cause	Go to
oom-kill, or kernel Killed process line	Kernel or cgroup OOM	Steps 2, 3
signal, and journalctl -u systemd-oomd shows a kill	systemd-oomd	Step 4
timeout, or State 'stop-sigterm' timed out. Killing.	systemd timeout escalation	Step 5
signal, and nothing in kernel log or oomd	External kill -9	Step 6

If it is memory, raise the limit at runtime and add swap. This does not survive reboot or daemon-reload, so make it permanent with step 3:

sudo systemctl set-property --runtime "$UNIT" MemoryMax=2G
sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile
sudo systemctl restart "$UNIT"

set-property only helps if a cgroup limit caused the kill. If the kernel line says constraint=CONSTRAINT_NONE, the whole host ran out of memory and only swap or a smaller workload helps.

Step-by-Step Resolution

1. Read the exact termination record

systemctl status "$UNIT" --no-pager -l
systemctl show "$UNIT" -p Result,ExecMainCode,ExecMainStatus,NRestarts,ActiveEnterTimestamp,InactiveEnterTimestamp

ExecMainCode=2 (CLD_KILLED) with ExecMainStatus=9 confirms SIGKILL. To inspect an earlier boot, the journal must be persistent (/var/log/journal/ exists, or Storage=persistent in /etc/systemd/journald.conf):

sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald
journalctl -k -b -1 | grep -iE 'oom|killed process'

2. Read the kernel OOM record

sudo dmesg -T | grep -iE 'out of memory|oom-kill|killed process'

Two shapes:

oom-kill:constraint=CONSTRAINT_NONE,...                      # global: host out of memory
Out of memory: Killed process 2211 (mysqld) total-vm:... anon-rss:...

oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/system.slice/myapp.service   # cgroup limit hit
Memory cgroup out of memory: Killed process 2211 (mysqld) ...

• CONSTRAINT_NONE: the victim is chosen by oom_score, so the killed service may not be the one consuming the most memory. Read the Tasks state table above the kill line for the real consumer.
• CONSTRAINT_MEMCG and oom_memcg=: the unit, or a parent slice, hit its own limit.

Inspect the cgroup counters (cgroup v2):

systemctl show "$UNIT" -p MemoryMax,MemoryHigh,MemorySwapMax,MemoryCurrent
cat /sys/fs/cgroup/system.slice/$UNIT/memory.events     # oom_kill > 0 confirms cgroup OOM
cat /sys/fs/cgroup/system.slice/$UNIT/memory.max
cat /sys/fs/cgroup/system.slice/$UNIT/memory.peak       # kernel 5.19+
systemd-cgtop -m -n 1

Also check parent slices (system.slice, user-1000.slice), because a slice limit kills its children.

3. Fix memory limits with a drop-in

sudo systemctl edit myapp.service

/etc/systemd/system/myapp.service.d/override.conf

[Service]
MemoryHigh=1536M
MemoryMax=2G
MemorySwapMax=1G
OOMScoreAdjust=-500
Restart=on-failure
RestartSec=5

• MemoryHigh below MemoryMax throttles and reclaims before the hard limit kills.
• OOMScoreAdjust=-500 lowers the service’s odds of being picked in a global OOM. It does not protect against a cgroup MemoryMax= kill or systemd-oomd.
• Restart=on-failure restarts after SIGKILL, because SIGKILL counts as an unclean signal. If it loops, tune StartLimitIntervalSec= and StartLimitBurst= under [Unit].
• OOMPolicy=stop (default) stops the whole unit when any of its processes is OOM-killed, and the result becomes oom-kill. OOMPolicy=continue keeps the unit running after a child is killed.

Reduce the demand as well as raising the limit:

# MariaDB/MySQL: /etc/mysql/mariadb.conf.d/50-server.cnf, or /etc/my.cnf.d/server.cnf
innodb_buffer_pool_size = 512M
max_connections = 100

# PHP-FPM pool: /etc/php/8.3/fpm/pool.d/www.conf
pm.max_children = 10
pm.max_requests = 500

Java: set -Xmx below MemoryMax=, leaving 25% or more for metaspace and native memory.

4. Rule out systemd-oomd

oomctl
journalctl -u systemd-oomd -b --no-pager

A kill looks like Killed /system.slice/myapp.service due to memory pressure for /system.slice being 65.30% > 50.00% for > 20s. Opt the unit out:

/etc/systemd/system/myapp.service.d/oomd.conf

[Service]
ManagedOOMPreference=omit

Or loosen the global thresholds:

/etc/systemd/oomd.conf

[OOM]
SwapUsedLimit=90%
DefaultMemoryPressureLimit=80%
DefaultMemoryPressureDurationSec=30s

sudo systemctl daemon-reload && sudo systemctl restart systemd-oomd

systemd-oomd is enabled by default on Fedora 34+ and Ubuntu desktop 22.04+. Ubuntu Server ships it installed in some releases and not in others. systemctl status systemd-oomd tells you.

5. Fix timeout-driven SIGKILL

Signature in the journal:

myapp.service: State 'stop-sigterm' timed out. Killing.
myapp.service: Killing process 1234 (myapp) with signal SIGKILL.

The service ignored SIGTERM for longer than TimeoutStopSec= (default 90s, set by DefaultTimeoutStopSec= in /etc/systemd/system.conf). Causes:

• The app does not handle SIGTERM, or drains connections slowly.
• ExecStart= runs through a shell wrapper that does not forward signals. Use exec:

ExecStart=/bin/sh -c 'exec /usr/local/bin/myapp --config /etc/myapp.conf'

• On start, Type= does not match the daemon’s behavior, so systemd waits for a readiness signal that never comes, times out, and kills it:

# Type=simple   : process stays in the foreground (default)
# Type=forking  : process daemonizes; requires PIDFile=
# Type=notify   : process calls sd_notify(READY=1); required for the setting to work
# Type=exec     : like simple, but waits for exec() to succeed

Extend the stop window if the drain is legitimate:

[Service]
TimeoutStopSec=300
TimeoutStartSec=300

6. Trace an external SIGKILL

If steps 2 through 5 show nothing, find the sender. auditd:

sudo auditctl -a always,exit -F arch=b64 -S kill -F a1=9 -k sigkill
sudo auditctl -a always,exit -F arch=b64 -S tgkill -F a2=9 -k sigkill
sudo ausearch -k sigkill -i --start recent

Or bpftrace (comm/pid are the sender, args-> is the target):

sudo bpftrace -e 'tracepoint:signal:signal_generate /args->sig == 9/ {
  printf("%s pid=%d -> %s pid=%d\n", comm, pid, args->comm, args->pid); }'

Common senders:

systemctl status earlyoom monit                          # userspace OOM daemons and supervisors
sudo grep -rnE 'kill -9|pkill|killall' /etc/cron* /var/spool/cron /usr/local/bin /opt 2>/dev/null

Remove or fix the offending job. Cleanup scripts that pkill by name often match unrelated services.

7. Check the user-session kill case

A service started from an SSH or user session (systemctl --user, or nohup in a login shell) is killed at logout if KillUserProcesses=yes in /etc/systemd/logind.conf. The result is SIGTERM, then SIGKILL after the timeout. Keep user services alive:

sudo loginctl enable-linger <username>

8. Verify

sudo systemctl daemon-reload
sudo systemctl restart "$UNIT"
systemctl show "$UNIT" -p Result,NRestarts,MemoryMax,MemoryCurrent
cat /sys/fs/cgroup/system.slice/$UNIT/memory.events
journalctl -u "$UNIT" -f

oom_kill in memory.events must stop increasing, and NRestarts must stay flat.

Why This Fails for Some Users

• No trace after reboot: With a volatile journal, the OOM record is lost on reboot. The service log just stops mid-line, since SIGKILL gives the process no chance to log. Make the journal persistent (step 1) so the next occurrence is diagnosable.
• dmesg returns nothing: kernel.dmesg_restrict=1 blocks non-root users. Use sudo. The ring buffer can also wrap on a busy host, so use journalctl -k instead.
• cgroup v1 (CentOS/RHEL 7, systemd 219): MemoryMax=, MemoryHigh=, and OOMPolicy= do not exist. Use MemoryLimit=. Counters are in /sys/fs/cgroup/memory/system.slice/myapp.service/memory.failcnt and memory.oom_control, not memory.events. The Result=oom-kill value is absent, so an OOM kill shows as plain signal.
• Non-reclaimable memory counts toward the limit: /tmp or /dev/shm on tmpfs, and PrivateTmp= files on tmpfs, are charged to the cgroup that wrote them. They cannot be reclaimed without swap, so a service that stages large files can hit MemoryMax= with a small resident set. Check df -h /tmp /dev/shm.
• Wrapper process is “main”: With ExecStart=/bin/sh -c '...' and no exec, the shell is the main PID. If a child is OOM-killed and the shell exits with a different code, the status will not say 9/KILL. Look for the kill line in the kernel log regardless of what systemd reports.
• Containers (Docker, LXC, Kubernetes): The OOM decision is made by the host cgroup, and dmesg inside the container often shows nothing. Docker reports exit code 137 (128 + 9), and Kubernetes reports OOMKilled. Check the host’s kernel log and the container memory limit, not the inner systemd.
• WSL2: The VM memory cap comes from memory= in %UserProfile%\.wslconfig (default is 50% of host RAM). Raise it and run wsl --shutdown.
• Cloud VMs with 1 GB RAM and no swap: Most cloud images ship without swap, so a modest spike triggers a global OOM. Add swap as in the Quick Fix. On Btrfs, fallocate swap files require chattr +C and no compression, or btrfs filesystem mkswapfile on newer btrfs-progs. Use dd on filesystems where fallocate produces holes.
• vm.overcommit_memory=2: Strict overcommit turns allocation failures into ENOMEM at malloc() instead of OOM kills. The app then aborts itself, usually with SIGABRT (status=6/ABRT), not KILL.
• WatchdogSec= escalation: A missed watchdog ping sends SIGABRT first, then SIGKILL after TimeoutAbortSec= if the process does not die. This shows up as Result=watchdog. Either fix the notify loop or remove WatchdogSec=.
• Multiple concurrent limits: A service can be under its own MemoryMax= while its parent slice or user.slice hits a limit. Walk up the tree: systemctl show system.slice -p MemoryMax and systemd-cgls.
• OOMScoreAdjust= set for the wrong scope: It only affects the kernel’s global OOM choice. It cannot prevent cgroup-limit kills or systemd-oomd, and negative values require the unit to run with CAP_SYS_RESOURCE (root system services have it by default).

---