Few things make your stomach drop like a pager alert saying “CPU usage 100%.” Customers are complaining, the dashboard is red, and everyone in the chat is asking, “Is it fixed yet?”
I’ve been in that spot more than once. The biggest lesson I’ve learned is simple: don’t panic-restart the server. If you do, you throw away the evidence you need to find the real problem.
Here’s the process I follow when a server’s CPU hits 100%, in the order I do it.
Can you even debug a server at 100% CPU?
Yes, and this is the question people ask most. A server at 100% CPU isn’t dead, just slow. Lightweight tools like top, ps, pidstat and perf still work fine.
A few practical tips:
- Keep an SSH session open before things get worse, because a new login can crawl.
- Run your commands with
niceso they don’t compete with the app. - Save output to a file. You’ll want it later.
- If you can’t log in at all, use your cloud console or out-of-band access.
Step 1: Check that it’s really a CPU problem
Run uptime and compare the load average to your core count (nproc). Then open top and press 1 to see each core.
The CPU breakdown tells you where to look:
- High
us(user): your application code is eating CPU. Think hot loops, slow queries, bad regex. - High
sy(system): too many syscalls or context switches, often from too many threads or lock fights. - High
wa(iowait): the CPU is waiting on disk or network. This is an I/O problem wearing a CPU costume. - High
st(steal): on a VM, a noisy neighbor or the hypervisor is taking your CPU. - High
siorhi: interrupts, often heavy network traffic or a bad NIC driver.
Step 2: Find the process (and the thread)
ps aux --sort=-%cpu | head
top -H -p <PID>
pidstat -u 1 5
Is it one process or many? Is it your app, a system service, or something you don’t recognize? An unknown process with a strange name deserves a close look. Cryptominers are more common than people think.
Step 3: Ask “what changed?”
Most CPU spikes have a trigger. Before diving into code, I check:
- Did we deploy something recently?
- Did a config or feature flag change?
- Is there a traffic spike, real or malicious?
- Did a cron job, backup or log rotation just start?
- Is a downstream service slow, causing retries and pile-ups?
- Did the data grow? A query that was fine at 1 million rows can fall over at 100 million.
Your monitoring graphs and logs are the best friends here. They tell you when it started, and that timestamp usually points straight at the cause.
Step 4: Look inside the process
Now you find out what the hot process is actually doing.
Any Linux process
strace -c -p <PID>
perf record -g -p <PID> -- sleep 30
perf report
perf shows where CPU time is really spent, and flame graphs make the output easy to read.
Java: Take 3 to 5 thread dumps with jstack <PID>, a few seconds apart. If the same threads sit in the same code each time, that’s your culprit. Convert the busy thread ID from top -H to hex and match it to nid= in the dump. Also check GC with jstat -gcutil, since constant garbage collection is a classic cause.
Python: py-spy top --pid <PID>
Node.js: --cpu-prof or clinic flame
Go: the built-in pprof CPU profile
Database: check SHOW PROCESSLIST or pg_stat_activity, then the slow query log. Missing indexes cause a lot of damage.
Common root causes
| What you see | Likely cause |
|---|---|
| One thread pinned at 100% | Infinite loop or catastrophic regex backtracking |
| Many busy threads | Traffic surge, heavy endpoint, cache stampede |
| High system time | Too many threads, lock contention |
| High CPU plus heavy GC | Memory leak or memory pressure |
| High iowait | Slow disk, swapping, network storage issues |
| Unknown process | Possible compromise, so check connections and auth logs |
Step 5: Stop the bleeding, but save the evidence first
If users are affected, mitigate quickly:
- Roll back the latest deploy
- Scale out
- Rate-limit traffic
- Take the instance out of the load balancer
- Restart the service
But grab your evidence first. Take thread dumps, a perf profile and a process list. Once you restart, the evidence is gone.
Step 6: Fix it properly
A restart isn’t a fix. Once things are calm:
- Patch the code or add the missing index.
- Add CPU limits (cgroups or container limits).
- Set up alerts that fire before 100%, not after.
- Write a short post-incident note so the next person doesn’t start from zero.
What about turning on more logging?
It’s tempting, but at 100% CPU, extra logging adds load and can make things worse. Good logging and monitoring should already be in place before an incident. After the spike, use them to answer what happened and when. During the spike, use live tools like top, perf and thread dumps to answer what is happening right now.
Quick checklist
- Is it truly CPU (user, system, iowait or steal)?
- Which process and thread?
- What is that code doing (profile or dump)?
- Why now (what changed)?
- Mitigate, but save evidence first.
- Fix it for good and add alerts.
FAQ
What causes 100% CPU usage on a server?
The usual suspects are infinite loops, inefficient queries, traffic spikes, memory leaks causing heavy garbage collection, runaway cron jobs, and sometimes malware.
How do I find which process is using the most CPU on Linux?
Use top or ps aux --sort=-%cpu | head. For per-thread detail, use top -H -p <PID>.
Should I restart the server when CPU hits 100%?
Only after capturing thread dumps, profiles and logs. Restarting first destroys the evidence you need for the root cause.
Is 100% CPU always bad?
Not always. A batch job or video encoder using all cores can be perfectly normal. It becomes a problem when latency rises or other services starve.
Final thoughts
Debugging high CPU is less about fancy tools and more about order: confirm, locate, correlate, profile, mitigate, fix. Follow that sequence and you’ll stay calm while everyone else is panicking.
Answer:
“First, I would confirm the CPU saturation using top, uptime, and CPU metrics in our monitoring system. I would identify the top CPU-consuming process using ps or pidstat.
Next, I would inspect its threads using top -H and correlate the CPU spike with application logs, incoming traffic, recent deployments, and system metrics.
If it’s a Java application, I would capture thread dumps or use a profiler. If it runs in Kubernetes, I would also check pod CPU usage, limits, and throttling.
Once I identify the underlying cause, I would apply the safest mitigation, such as scaling, rolling back a problematic release, or fixing the application. Finally, I would verify CPU utilization, latency, and error rates and add monitoring or alerts to prevent recurrence.”
Leave a Reply