Exit code 137 and OOM killed: what they mean, and how to get warned first
Exit code 137 means the process received SIGKILL and stopped at once: 128 plus signal 9. Most of the time the sender is the Linux out-of-memory (OOM) killer, acting because the container went over its memory limit. The other common cause is a stop that ran out of time. The platform sent SIGTERM, the process did not exit within the grace period, and it was killed.
The kill gives the process no chance to log anything. That is why the last log line is often just Killed, and why the memory graph can look calm right up to the crash.
Is it out of memory, or something else?
Check where the kill was recorded before you change anything.
| Where it runs | How to check | What OOM looks like |
|---|---|---|
| Docker | docker inspect <container> --format '{{.State.OOMKilled}} {{.State.ExitCode}}' | true 137 |
| Kubernetes | kubectl describe pod <pod>, under Last State | Reason: OOMKilled, Exit Code: 137 |
| Amazon ECS | The container’s Reason in the task details, or containers[].reason in aws ecs describe-tasks | OutOfMemoryError: Container killed due to memory usage |
| Any Linux host | journalctl -k | grep -i "killed process" or dmesg -T | grep -i oom | Out of memory: Killed process 1234 (node) or Memory cgroup out of memory |
| cgroup v2 | cat /sys/fs/cgroup/memory.events inside the container | oom_kill above 0, counted since the container last started: a worker killed while the main process survived |
If none of these shows an OOM kill, look at how the process was stopped. A redeploy, a scale-down or a failed health check sends SIGTERM first. An app that ignores SIGTERM is killed after the grace period: 10 seconds for docker stop, and 30 seconds by default in Kubernetes. That also ends with 137.
Exit codes you will see next to 137
| Code | Signal | Usually means |
|---|---|---|
| 137 | SIGKILL (9) | An OOM kill, or a stop that ran past its grace period |
| 143 | SIGTERM (15) | Stopped by SIGTERM: a normal stop that ended within the grace period |
| 139 | SIGSEGV (11) | A crash in native code |
| 134 | SIGABRT (6) | The runtime aborted itself, for example Node’s “JavaScript heap out of memory” |
| 1 | None | The app exited with an error of its own |
Node’s heap error and a 137 are different problems. With 134, Node hit its own heap limit. With 137, the container hit its limit first.
Why containers run out of memory
- The limit is below the real peak. Average memory looks fine, but a batch job, an image resize or a large JSON payload pushes the peak past the limit.
- A leak. Memory climbs over hours or days and never comes back down. On a graph it looks like a sawtooth: a slow climb, a kill, a restart, and the same climb again.
- The runtime does not know the container limit. Node’s
--max-old-space-sizeset above the container limit, or a JVM sized from the host’s RAM, lets the heap grow past what the container allows. - More processes than you counted. Every worker, cluster process or Gunicorn worker holds its own copy of the app.
- No limit at all on a shared server. Without a limit, a container grows until the server runs out. The kernel then kills the process with the highest OOM score, which can be your database rather than the app that leaked.
How to fix it
- Measure the peak, then size for it. Set the limit to the observed peak plus 20 to 30 percent headroom.
- Cap the runtime below the container. For Node, set
--max-old-space-sizeto about 75 percent of the container limit. For the JVM, use-XX:MaxRAMPercentage=75. - Cut concurrency. Run fewer workers per container, or more replicas with fewer workers each.
- Stream instead of buffering. Process uploads, exports and query results in chunks.
- Find the leak. Compare two heap snapshots taken an hour apart, and look for caches and listeners that only grow.
- Give every container a limit on your own servers, so one service cannot take the whole host down with it.
- Handle SIGTERM. Close connections and exit within the grace period, so a normal stop ends cleanly (0, or 143) instead of 137.
Docker on your own server
Docker sets no memory limit unless you pass --memory, or mem_limit in Compose. On a VPS running several containers, the first sign of a leak is often an unrelated container dying. Set a limit per container, and check the server too:
docker stats --no-stream
free -h
journalctl -k | grep -i "killed process"Watch the server’s own memory as well as each container’s, because the whole host is what the OOM killer acts on.
Kubernetes: OOMKilled
A pod is OOMKilled when a container goes over resources.limits.memory. The kubelet restarts it according to the pod’s restart policy. Repeated kills show up as CrashLoopBackOff. Eviction is a different event. It removes pods when the whole node runs low on memory, and the pod shows Evicted rather than OOMKilled.
Amazon ECS: exit code 137
ECS stops a container with exit code 137 and the reason OutOfMemoryError when it goes over its hard memory limit. A soft limit (memoryReservation) only reserves memory. Without a container memory limit, the cap is the task’s memory if one is set (always on Fargate), otherwise the instance’s. If the task stopped with 137 and no OOM reason, the usual cause is a stop timeout during a deployment.
CI jobs: GitLab CI and GitHub Actions
A build or test step that ends with 137 has usually hit the runner’s memory. Typical causes are a bundler, a TypeScript build or a test runner with too many workers. Lower the worker count, or move the job to a larger runner.
How to see it coming
An OOM kill is the last step of something you can usually watch happen.
- Alert on memory as a share of the limit, held for a few minutes. A sustained threshold catches leaks and slow climbs with hours to spare, and does not fire on a single busy request.
- Alert on the crash and the restart too. A spike that lasts a few seconds falls between metric samples and shows on no graph. The restart is the only signal it leaves, so treat repeated restarts as a memory problem until you prove otherwise.
- Keep the history. A leak is obvious across a week and invisible across an hour.