200.1 Measure and Troubleshoot Resource Usage

Introduction

Performance troubleshooting is not about finding one magic command. It is about measuring the system, deciding which resource is saturated, and proving the cause before changing anything.

What you should be able to do after this lesson:

• Read CPU, memory, load, disk, and network indicators.
• Distinguish demand from an actual bottleneck.
• Find the processes and clients responsible for resource use.
• Correlate metrics collected at the same time.
• Select the next diagnostic command from the observed symptom.

Big Idea: Utilization Is Not the Same as Saturation

Performance work becomes much easier when every resource is examined through three questions:

1. Utilization: How busy is the resource?
2. Saturation: Is work waiting in a queue?
3. Errors: Is the resource or its path reporting failures?

A CPU can be 100 percent busy without hurting users if work still completes within its target time. A disk at lower utilization can still be the bottleneck when requests wait for slow storage. Always connect system metrics to a user-visible symptom such as request latency, failed jobs, or reduced throughput.

Start with a Baseline

A number is useful only when you know what normal looks like. Record measurements during ordinary operation and compare them with the incident period.

Useful first commands:

uptime
free -h
vmstat 1
top

Avoid making several tuning changes at once. A change without a before-and-after measurement does not prove an improvement.

CPU and Load

uptime shows load averages for approximately 1, 5, and 15 minutes. Load includes runnable tasks and tasks blocked in uninterruptible sleep, commonly storage I/O. It is not a direct CPU percentage.

uptime
ps -eo pid,comm,%cpu,%mem,state --sort=-%cpu | head
top

Interpretation:

• High CPU utilization with a long run queue suggests CPU pressure.
• High load with idle CPU can indicate tasks blocked on I/O.
• One busy core can limit a single-threaded process even when total CPU is below 100 percent.
• A high steal value in a virtual machine indicates time taken by the hypervisor.

vmstat 1 is useful for correlation. Important columns include r for runnable tasks, b for blocked tasks, si and so for swapping, bi and bo for blocks received from and sent to block devices, wa for I/O wait, and us/sy/id for CPU time.

Memory and Swap

Linux uses free memory for cache, so a small free value alone is not a problem. Focus on available memory, swap activity, and reclaim pressure.

free -h
vmstat 1
ps -eo pid,comm,rss,vsz,%mem --sort=-rss | head

Warning signs:

• sustained si and so activity
• growing resident memory for one process
• the OOM killer in kernel logs
• latency that appears while pages are repeatedly moved between RAM and swap

Check kernel messages when an application disappears unexpectedly:

journalctl -k | grep -i -E 'oom|out of memory|killed process'

Disk I/O

iostat separates device utilization and latency from process-level observations.

iostat -xz 1
iotop
pidstat -d 1

Useful ideas:

• throughput is the amount of data transferred
• IOPS is the number of operations
• latency is the time each operation waits
• queue depth shows outstanding work

A device near saturation with rising await time is more convincing evidence than a single high throughput number. Use iotop or pidstat to connect device pressure to processes.

Network I/O

Inspect interface counters, sockets, and traffic separately.

ip -s link
ss -s
ss -tulpn
sar -n DEV 1

Look for dropped packets, errors, retransmissions, an unexpected number of connections, or one interface nearing its link capacity. Tools such as iftop, nload, or iptraf-ng can help map bandwidth use to peers. Packet capture is useful when counters show a network symptom but not its cause:

tcpdump -ni eth0

When the system routes or filters traffic, inspect that path too:

iptables -L -n -v
iptables -t nat -L -n -v
sar -n DEV,TCP,ETCP 1

Rule counters show which firewall paths receive packets. Compare interface throughput before and after the router to distinguish client demand, packet loss, and filtering. iptraf-ng or iftop can group traffic by host so one client consuming most of the bandwidth becomes visible.

Processes, Files, and Connections

Connect resource use to an owner:

pstree -p
ps aux --sort=-%cpu | head
lsof -p <pid>
lsof -i
w

lsof can reveal which process holds a busy file, deleted log, or listening socket. w shows logged-in users and their activity.

A Reliable Investigation Order

1. Record the exact symptom and time.
2. Check load, CPU, available memory, swap, disk, and network.
3. Identify the saturated resource.
4. Find the process, device, or client producing the demand.
5. Compare application, kernel, and service logs for the same period.
6. Make one controlled change.
7. Measure again and document the result.

Guided Practice: Classify a Bottleneck

Open three terminals on a test system. Do not create artificial load on a production host.

In the first terminal, capture the broad picture:

vmstat 1

In the second, inspect disks:

iostat -xz 1

In the third, inspect processes and sockets:

pidstat -dur 1
ss -s

Observe for at least one minute and answer:

• Does r remain above the available CPU count?
• Does b increase while CPU idle time remains high?
• Are si and so repeatedly nonzero?
• Does one disk show a sustained queue and rising await?
• Which process consumes the corresponding CPU, memory, or I/O?

The goal is not to find a problem on an idle lab machine. It is to practice correlating the same time window across tools. Save the output with timestamps so a later incident can be compared with this baseline.

Troubleshooting Scenario

A web server has a load average of 18 on an eight-core VM, but CPU idle time is 70 percent. vmstat shows several blocked tasks, iostat shows high await on one volume, and iotop points to a backup process.

The evidence indicates storage saturation rather than CPU shortage. Pausing or rescheduling the backup is a controlled test. Adding CPU would not address the queue. After the change, verify that blocked tasks, disk latency, and application response time all return toward baseline.

Exam Focus

• Know that load average includes runnable tasks and tasks in uninterruptible sleep.
• Connect vmstat columns to CPU, swapping, and block I/O.
• Use iostat and iotop for device and process perspectives.
• Use ss, netstat, iptraf, and interface counters for network demand.
• A defensible diagnosis correlates symptoms, queues, owners, and timestamps.

Common Mistakes

• Treating load average as CPU percentage.
• Treating cached memory as wasted memory.
• Looking only at averages that hide short spikes.
• Restarting a service before preserving evidence.
• Assuming high utilization is bad when latency and queues remain healthy.

Recap

• Diagnose from evidence, not from a single metric.
• vmstat, iostat, sar, top, ps, ss, and lsof answer different questions.
• Correlation in time is what turns several measurements into a cause.
• Always compare with a known baseline and verify after a change.