Opyntel

Check memory usage on Linux, and what swap usage means

To check memory usage on Linux, run free -h and read the available column: that is how much memory programs can still get. The free column looks alarming on any busy server because Linux fills spare memory with disk cache and hands it back when a program asks. Then use top or ps to see which process holds the memory, and docker stats for containers. Swap in use is normal; swap that keeps moving is the warning sign.

Check memory usage with free

bash
free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       2.1Gi       180Mi        12Mi       1.5Gi       1.4Gi
Swap:          2.0Gi       310Mi       1.7Gi
ColumnWhat it means
usedMemory programs hold, without the cache
freeMemory nothing uses, not even the cache; low is normal
buff/cacheDisk cache the kernel gives back when a program needs it
availableWhat a program can still get without swapping: the number to watch

cat /proc/meminfo shows the same figures in more detail, and vmstat -s prints them as a list.

Find which process uses the memory

bash
top -o %MEM                          # live, sorted by memory; press M in top to switch
ps aux --sort=-rss | head -n 11       # the ten biggest, by resident memory
sudo smem -tk                         # proportional share, if smem is installed

RSS is the memory a process holds in RAM right now. Shared libraries count towards every process that maps them, so the column adds up to more than the machine has; smem splits shared pages fairly. A process whose RSS only grows over days, under the same load, is leaking.

Memory per Docker container

bash
docker stats --no-stream --format 'table {{.Name}}	{{.MemUsage}}	{{.MemPerc}}'

MemUsage shows each container against its limit, or against the whole machine when it has none. A container near its limit is the likeliest to be killed, with exit code 137 (OOM killed). Set a limit with --memory or mem_limit in Compose so one container cannot starve the rest.

What swap usage means

Swap is disk space the kernel moves idle memory pages to. Some swap in use only means the kernel parked pages nothing touched for a while, which frees RAM for the cache. It turns into a problem when pages move in and out all the time, because every move waits on the disk.

bash
swapon --show          # swap devices and how full they are
vmstat 1 5             # si and so: memory swapped in and out per second (KiB)

Steady non-zero si and so mean the server is short of memory: add RAM, cap the process that grows, or move a service off the box. vm.swappiness only changes how early the kernel swaps, not how much memory you have.

When memory runs out

With no memory left and swap full or absent, the kernel’s OOM killer picks a process and kills it. sudo dmesg -T | grep -i "killed process" shows which one and when. In a container it shows as exit code 137; a container that is killed and started again in a loop is covered in why a container keeps restarting.

How Opyntel watches memory on every server

The Opyntel agent runs on each server with one docker run command and reports memory, swap and memory per container from inside the server, using outbound HTTPS only. Opyntel keeps that history and alerts when memory passes your threshold, by email, Slack or SMS depending on your plan. Every server sits in one dashboard with its disk, load and containers, next to the other platforms you run, so you see which server is short of memory without logging in to run free.

FAQ

Linux memory usage questions, answered

Low free memory, swap, memory per process and how much a server needs.

Linux uses spare memory as disk cache and gives it back when a program needs it. Read the available column of free -h instead.

Not on its own. It is a problem when vmstat shows pages swapping in and out all the time, which means the server is short of memory.

ps aux --sort=-rss | head shows the biggest processes, and top sorts live by memory when you press M.

docker stats --no-stream shows each container’s memory against its limit.

The alerts your platform doesn’t send.

One dashboard for everything you run, with the alerts your platforms leave out. Read-only, nothing to install in your apps.

See how it works