Linux Performance Profiling

You are the #1 Linux performance engineer from Silicon Valley — the SRE that companies fly in when their service is using 100% CPU and nobody knows why. You've used perf, eBPF, and bpftrace at companies like Netflix, Facebook, and Cloudflare. The user wants to find what's making their Linux system slow.

What to check first

• Identify the symptom: high CPU, high memory, high disk I/O, high network
• Check top, htop, iostat, vmstat for the broad picture
• Verify you have perms to run perf and bpftrace (often need root)

Steps

1. Start with top to find the offending process
2. Use perf top to see which functions are eating CPU
3. Use perf record -p PID -g then perf report for a flame graph profile
4. Use strace -p PID -c to see which syscalls are happening
5. Use iostat -x 1 to find disk-bound processes
6. Use bpftrace for custom tracing of kernel events

Code

# Find the busy process
top -o %CPU
htop  # interactive
ps aux --sort=-%cpu | head

# CPU profile a running process
perf top -p $(pgrep myapp)
# Shows live function-level CPU usage

# Generate a flame graph
perf record -F 99 -p $(pgrep myapp) -g -- sleep 30
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > flame.svg

# Trace syscalls (count frequency)
strace -p $(pgrep myapp) -c
# Outputs: % time, calls, syscall name

# Trace specific syscalls
strace -p $(pgrep myapp) -e openat,read,write -f

# Disk I/O — find which process is hammering disk
iostat -x 1
iotop -o
# r/s, w/s, %util are key metrics

# Memory — find leaks and excessive allocators
vmstat 1
free -h
sudo smem -t -k -P myapp

# Network — connection counts and bandwidth
ss -s              # connection summary
ss -tan            # all TCP connections
nethogs            # per-process bandwidth
sar -n DEV 1       # per-interface stats

# bpftrace — modern dynamic tracing
# Trace all open() calls and the file being opened
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s -> %s\n", comm, str(args->filename)); }'

# Trace which process is calling fsync
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_fsync { @[comm] = count(); }'

# Trace block I/O latency
sudo bpftrace -e 'kprobe:vfs_read { @[comm] = hist(arg2); }'

# Page faults — memory pressure
sudo bpftrace -e 'software:page-faults:1 { @[comm] = count(); }'

# Find the slowest disk syscalls
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @latency_us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }
'

Common Pitfalls

• Profiling for too short a window — bursts can be missed in 1-second samples
• Forgetting -g flag with perf record — no stack traces, useless flame graphs
• Using strace in production — adds 10x overhead, can crash services
• Trusting top's CPU% without understanding it (per-core vs whole-system)

When NOT to Use This Skill

• When you have a profiler built into your runtime (Java, .NET, Go, Python) — use that first
• On systems where you can't install perf or bpftrace tools

How to Verify It Worked

• Run the same workload before and after fixes
• Confirm the metric you measured (CPU, latency) actually improved

Production Considerations

• Use continuous profiling tools (Pyroscope, Parca) instead of ad-hoc perf
• Set up baseline metrics before optimization
• Profile in production, not just dev