Opyntel

OOM killed on Fly.io: what “Out of memory: Killed process” means, and how to see it coming

When a Fly.io log shows Out of memory: Killed process 257 (node), the kernel inside your Machine ran out of memory and killed the app. Each Machine is a small VM with a fixed amount of memory, and nothing outside it gives way. Fly.io then restarts the Machine according to its restart policy, and may email you that the app ran out of memory. For the general picture, see exit code 137 and OOM killed.

Confirm it

  • fly logs shows the kernel’s Out of memory: Killed process line, with the process name and its memory.
  • In Fly.io’s metrics, the fly_instance_exit_oom series is 1 when a Machine was killed for exceeding its memory.
  • The memory panels in Fly.io’s managed Grafana show fly_instance_memory_mem_available falling towards zero before the kill.

How to fix it

  1. Give the Machine more memory: fly scale memory 1024. The smallest presets leave little room for Node, Python or Ruby.
  2. Add swap with swap_size_mb in fly.toml. It is a cushion for short peaks, not a fix for a leak.
  3. Cap the runtime. For Node, set --max-old-space-size to about 75 percent of the Machine’s memory.
  4. Size the release command. Migrations and asset builds often need more memory than the running app, and a kill there fails the deploy. Give the release Machine more with [deploy.release_command_vm] in fly.toml.
  5. Split process groups, so a worker that needs memory runs on its own Machine size.

What Fly.io warns you about

Fly.io may email you after an OOM kill. It has no built-in alerting on metrics, and its managed Grafana has no alerting either. To be warned while memory is climbing, you have to run your own Grafana or Prometheus alerting against Fly.io’s metrics.

How Opyntel warns you first

Opyntel reads your Fly.io organization’s metrics with a read-only token, and shows each Machine’s memory as a share of its size, next to the rest of your services. It alerts when a Machine’s memory passes your threshold, by email, Slack or SMS depending on your plan. OOM exits and restarts land on one activity timeline, so a Machine that keeps getting killed stands out across all your apps.

Fly.io monitoring in Opyntel
FAQ

Fly.io out-of-memory questions, answered

Deploy-time kills, swap and alerts.

The release command runs on a temporary Machine sized like the largest Machine in your default process group. Migrations and asset compilation there are a common peak; give it more with [deploy.release_command_vm] in fly.toml.

It moves the kill later. It helps with short peaks, but a leak fills swap too, and a swapping app slows down first.

Not from Fly.io’s own dashboard. You need alerting on its Prometheus metrics, from your own Grafana or from Opyntel.

The alerts your platform doesn’t send.

One dashboard for everything you run, with the alerts your platforms leave out. Read-only, nothing to install in your apps.

See how it works