Root Cause Diagnosis Containers on a user-defined bridge network send all DNS queries to Docker’s embedded resolver at 127.0.0.11, which answers container names itself and forwards everything else to upstream servers taken from the host’s /etc/resolv.conf. The lookup fails when that forwarding path breaks: the host lists only a loopback stub resolver Docker cannot use … Read more
A script that works in an interactive terminal but fails silently under cron runs in a different execution environment: cron starts jobs with a minimal environment (PATH=/usr/bin:/bin, SHELL=/bin/sh, no login or profile files sourced, no TTY, HOME set but working directory defaulting to $HOME), and it discards stdout and stderr unless a mailer is configured, … Read more
iptables rules disappear after a reboot because they exist only in the kernel’s in-memory netfilter tables, and nothing reloads them at boot unless a service restores a saved ruleset from disk. Root Cause Diagnosis The iptables command modifies live kernel state. That state is discarded at shutdown, and the kernel boots with empty chains and … Read more
Root Cause Diagnosis rm: cannot remove ‘file’: Operation not permitted (errno EPERM) appearing even for root means the inode carries a protection that overrides normal permission checks, most commonly the ext2/3/4 immutable attribute (i) or append-only attribute (a). With either flag set, the kernel refuses unlink, rename, truncate, write, and hard-link operations until the flag … Read more
Root Cause Diagnosis Linux load average counts tasks in the runnable state (R) plus tasks in uninterruptible sleep (D), so processes blocked inside the kernel waiting on I/O inflate the load figure without consuming CPU cycles or memory. The blocked tasks are almost always waiting on a storage device, a network filesystem (NFS/CIFS), a hung … Read more
rsync exits with code 23 when it completes the transfer but fails to read, write, or set attributes on one or more files, and the “(see previous errors)” suffix means the actual cause is in an earlier line of the output, not in the final summary message. Root Cause Diagnosis Code 23 is a partial-transfer … Read more
Root Cause Diagnosis Containers do not have their own clock. They share the host kernel’s system clock (CLOCK_REALTIME), so a container cannot drift independently on a native Linux host. If container time is wrong, either the host clock is wrong (no NTP sync, or RTC and system time disagree), or the container runs inside a … Read more
Root Cause Diagnosis systemd-journald stores logs in /var/log/journal/ when persistent storage is enabled, and its default size cap is 10% of the filesystem, bounded at 4 GiB by SystemMaxUse. On a small root partition, or when a service floods the journal (crash loops, debug logging, failed-auth spam), 10% of the filesystem can still be a … Read more
Root Cause Diagnosis OpenSSL 3.x returns error:0A00010B (SSL_R_WRONG_VERSION_NUMBER) when the first bytes received after the TLS ClientHello are not a valid TLS record. The server on that port is almost always speaking plaintext (HTTP, or a non-TLS protocol) or a different TLS framing, or an intermediary (proxy, load balancer) answered the handshake with a non-TLS … Read more
Root Cause Diagnosis The kernel OOM killer selects the process with the highest oom_score, computed as the process’s share of total memory (RSS + swap + page tables) scaled to 0-1000, plus oom_score_adj (range -1000 to +1000). A large, critical daemon such as a database, with a default oom_score_adj of 0, outscores many small processes … Read more