Python script crashing with Segmentation Fault (core dumped) inside Docker Alpine

by

Alpine Linux uses musl libc instead of glibc, and a Python process segfaults on it when a C extension built for glibc, a glibc-only binary wheel, or code that relies on glibc’s default thread stack size runs against musl’s different ABI and limits. The crash happens in native code, not in Python, so no Python traceback appears.

Root Cause Diagnosis

Alpine’s musl libc differs from glibc in its ABI, its dynamic loader, and its default pthread stack size (128 KiB, versus 8 MB on glibc). Packages that ship only manylinux (glibc) wheels either fail to install or, if forced in through gcompat, call glibc symbols that behave differently under musl and corrupt memory. Native extensions that spawn threads or recurse deeply (NumPy/OpenBLAS, gRPC, lxml, Pillow, cryptography) can also overflow musl’s small thread stack, which the kernel reports as Segmentation fault (core dumped).

Quick Fix

Rebase the image on a glibc-based Python image:

# Before
FROM python:3.12-alpine

# After
FROM python:3.12-slim

Rebuild without cache:

docker build --no-cache -t myapp .
docker run --rm myapp

If the crash disappears, musl was the cause. If you must stay on Alpine, follow the steps below.

Step-by-Step Resolution

1. Get the faulting frame from Python

python -X faulthandler script.py
# or
export PYTHONFAULTHANDLER=1

The Fatal Python error: Segmentation fault output lists the Python call stack at the moment of the crash. The last frame inside a C-backed module (for example numpy/core/_multiarray_umath) identifies the extension at fault.

2. Remove gcompat if present

apk del gcompat

gcompat emulates glibc symbols and is a common source of silent memory corruption. If your build now fails with Error loading shared library ld-linux-x86-64.so.2, a glibc-only binary is being loaded, so go to step 3.

See also  How to capture network response body in Playwright without blocking navigation

3. Install musl-compatible wheels or build from source

Upgrade pip (musllinux wheel support needs pip ≥ 21.0) and check what tags it accepts:

pip install --upgrade pip
pip debug --verbose | grep -i musllinux

Install the build toolchain and force a source build for the faulting package:

RUN apk add --no-cache build-base python3-dev libffi-dev openssl-dev \
    musl-dev linux-headers
RUN pip install --no-cache-dir --no-binary=<package> <package>

Common extra headers: libxml2-dev libxslt-dev (lxml), jpeg-dev zlib-dev (Pillow), postgresql-dev (psycopg2), openblas-dev (NumPy/SciPy).

4. Get a native backtrace

RUN apk add --no-cache gdb musl-dbg

gdb -batch -ex run -ex bt --args python script.py

To capture a core file, set the limit on the container and check where the host writes cores:

docker run --ulimit core=-1 --cap-add=SYS_PTRACE myapp
cat /proc/sys/kernel/core_pattern   # host-level setting, not per-container

5. Rule out thread-stack overflow

Test with threading disabled in native libraries:

export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1

If the crash stops, raise the stack size for threads created from Python:

import threading
threading.stack_size(8 * 1024 * 1024)  # call before creating any thread

Threads created inside C libraries are not affected by this call. For those, use a glibc base image.

6. Avoid mixing package sources

Do not combine apk-installed Python packages (py3-numpy) with pip packages of the same name in one environment. Use one source per package and create a virtualenv if you need isolation:

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

7. Fall back to Debian-based images

If a dependency publishes no musllinux wheels and has a heavy native build (TensorFlow, PyTorch, ONNX Runtime, some gRPC versions), use python:3.x-slim. A multi-stage build keeps the image small without musl:

FROM python:3.12-slim AS build
RUN pip wheel --no-cache-dir -r requirements.txt -w /wheels

FROM python:3.12-slim
COPY --from=build /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/*

Why This Fails for Some Users

  • Emulated architecture: Running an amd64 image on Apple Silicon or ARM hosts through QEMU can segfault independently of musl, especially with extensions using AVX/SSE. Run a native image with docker run --platform linux/arm64 ..., or build for your host architecture.
  • Wheel tag mismatch: A pinned old version of a package may have no musllinux wheel even when newer versions do. Pip then builds from source, or you copied a venv built on a glibc host into the image.
  • Prebuilt binaries copied into the image: PyInstaller outputs, vendor .so files, and binaries from other distros will crash if built against glibc.
  • Library version drift: Alpine upgrades musl and OpenSSL between releases (3.17 → 3.19+). An extension compiled in one release and reused in another can crash. Rebuild extensions after changing the Alpine tag.
  • Memory limits: An OOM kill from a cgroup limit exits with code 137 (SIGKILL), not a segfault. If you see 137, check docker stats and raise --memory instead of changing the base image.
  • Stack limits: ulimit -s and docker run --ulimit stack= affect the main thread only. musl’s thread stack size is set at thread creation and ignores them.
  • Stripped binaries: Without musl-dbg and debug symbols, gdb backtraces show ?? frames. Install the debug packages before drawing conclusions from a trace.
See also  Fixing Selenium target window already closed error during fast alert handling

Leave a Comment