**Performance Diagnostics for Linux in 60 Seconds**
When you log in to a Linux server to troubleshoot a performance issue, what should you check in the first minute? At Netflix, we operate a massive EC2 Linux cloud and use various performance analysis tools like Atlas for cloud monitoring and Vector for on-demand instance analysis. While these tools are powerful, there are times when you need to log in directly and run standard Linux commands.
In this article, the Netflix Performance Engineering team will walk you through the best practices for performing performance analysis on the command line within the first 60 seconds using standard Linux tools.
**First 60 Seconds: Overview**
By running the following ten commands, you can quickly get an overview of the system’s processes and resource usage. By analyzing the output—especially error messages and signs of saturation—you can identify potential bottlenecks and optimize your resources. Saturation refers to when a resource is overloaded beyond its capacity, often reflected in longer request queues.
The commands include:
- `uptime`
- `dmesg | tail`
- `vmstat 1`
- `mpstat -P ALL 1`
- `pidstat 1`
- `iostat -xz 1`
- `free -m`
- `sar -n DEV 1`
- `sar -n TCP,ETCP 1`
- `top`
Some of these commands require the sysstat package to be installed. The data from these commands helps implement the USE method (Usage, Saturation, and Errors), which is useful for identifying performance issues across different resources such as CPU, memory, and disk. These commands also help eliminate possibilities and narrow down the scope of investigation.
**Command Breakdown**
**1. uptime**
This gives a quick view of the system's average load over the last 1, 5, and 15 minutes. It shows how many processes are waiting to run or are currently running. A rising load over time may indicate a problem that needs further investigation.
**2. dmesg | tail**
This displays the last few kernel messages. Look for errors such as OOM-killer events or dropped TCP packets. These can be key indicators of performance issues.
**3. vmstat 1**
Vmstat provides a statistical summary of system activity every second. Key columns to watch include:
- `r`: Number of processes waiting for or running on the CPU.
- `si`/`so`: Swap-in and swap-out rates.
- `us`, `sy`, `id`, `wa`: CPU usage breakdown (user, system, idle, I/O wait).
High values in `r` may indicate CPU saturation, while high `wa` suggests a disk bottleneck.
**4. mpstat -P ALL 1**
This command provides per-CPU statistics. It helps identify if some CPUs are overutilized while others are underused, which might point to unbalanced workloads or single-threaded applications.
**5. pidstat 1**
Pidstat monitors process-level resource usage. It shows CPU and memory consumption per process, helping identify which processes are consuming excessive resources.
**6. iostat -xz 1**
Iostat provides insights into disk I/O performance. Key metrics include:
- `r/s`, `w/s`: Read and write operations per second.
- `await`: Average I/O latency.
- `%util`: Device utilization.
High `%util` and `await` values may indicate disk saturation.
**7. free -m**
This command shows memory usage, including buffers and cache. Linux uses unused memory as a cache, so it's important to understand the `-/+ buffers/cache` line for more accurate memory usage.
**8. sar -n DEV 1**
Sar reports network interface throughput, showing received and transmitted data. It also includes utilization metrics, which can help identify network bottlenecks.
**9. sar -n TCP,ETCP 1**
This command provides TCP metrics such as active and passive connections and retransmissions. High retransmission rates may indicate network instability or server overload.
**10. top**
Top gives a real-time view of system processes and resource usage. While it’s useful for quick insights, it lacks historical data tracking, making it less ideal for long-term analysis compared to other tools.
**Next Steps**
For deeper analysis, consider exploring additional tools like `nicstat`, `bpftrace`, or `perf`. Brendan Gregg’s *Linux Performance Tools* tutorial at Velocity 2015 covers over 40 commands for observability, benchmarking, tuning, and performance analysis.
By combining these commands with experience, you can quickly diagnose and resolve most performance issues on Linux systems.
ftth drop cable,ftth cable,drop cable ftth,optical fiber drop cable,outdoor ftth drop cable
Guangzhou Jiqian Fiber Optic Cable Co.,ltd , https://www.jqopticcable.com