[Horwitz02] Chapter 11. Performance Tuning and Capacity Planning

Chapter 11. Performance Tuning and Capacity Planning

You will learn about the following in this chapter:

• How to measure and assess CPU performance and capacity

• Techniques for optimizing CPU performance

• How to plan for future CPU resource needs

• Important concepts in system storage

• Gauging storage performance and capacity

• How to tune disk and file system performance

• Guidelines for planning to accommodate future storage needs

• How to assess memory performance and capacity

• Techniques for tuning memory and swap performance

• Planning for future memory and swap capacity

• Gauging network performance and capacity

• How network errors occur and how to measure them

• How to optimize network performance

• Methods for projecting future network capacity needs

Afterthe data center is built out—every server deployed and all servicesrunning smoothly—systems staff can go into “maintenance mode,”concentrating on keeping things up and running and dealing withproblems as they come. This relaxing time doesn't last very long,however. Businesses grow, customer demands increase, disks run out ofspace, and developers demand more horsepower for applications.Performance tuning and capacity planning go hand in hand; thesefunctions help Unix system administrators get the most out of existinghardware and software and plan adequately for future needs.

Thischapter discusses four major subsystems—CPUs, storage, memory and swap,and networking—in its coverage of best practices for gauging andoptimizing current system performance, and planning for future growth.

Gauging CPU Performance and Capacity

The CPU is the heartof any system and as such plays a central role in the system's overallperformance. Many aspects of a system can affect its processors, so CPUtuning and capacity planning can be a very complex task. The next fewsections detail the nature of Unix CPU usage and how to use availableUnix tools to diagnose CPU performance problems and plan for the future.

Interpreting Load Averages

Many users and administratorsassociate a high load average with overworked CPUs, yet they don't knowwhat a load average really is. A load average is the average number of runnableprocesses active on the system over a set period of time. Runnableprocesses include currently running processes and those on the runqueue. Run queues are Unix kernel structures that hold pointers to processes that are able to run but are not currently running on any CPU.

Not All Processes Are Included in a Load Average

The run queue used to calculate a load average does not include queues for processes that are blocking (usually waiting for I/O) or swapped out to disk. The existence of these processes is not related to a CPU's performance.

Theuptime command is universal to all Unix flavors and can be used to examine the load averages for a machine, as follows:

bash$ ssh chills uptime
  7:02pm  up 143 day(s), 13:44,  0 users,  load average: 1.87, 0.92, 0.81

An Alternate Way to Report Load Averages

Load averages are also reported in the first line of the output of the w command.

The three averages reported inthe example represent the load over a period of 1, 5, and 15 minutesrespectively (when read from left to right). The 1-minute load averageis representative of the short-term processing load, while the15-minute average is a long-term load measurement. During short burstsof CPU activity, high 1-minute load averages alongside relatively low15-minute load averages can occur.

Aload average is significant only in terms of the number of processorson a machine. In general, a load average that is less than the numberof installed processors means that a system has adequate processorpower. A four-processor machine with a load average of 3.5 is notoverloaded—it's fully utilizing its resources.

Anotherrule of thumb states that you should add more processors to a machineif the size of the run queue is consistently four or more times thenumber of processors. We can use thevmstat command to determine the size of the run queue on a particular machine over time;vmstat is used to poll various performance statistics such as virtual memory usage, paging activity, and CPU activity. Its usage is as follows:

vmstat [options] [interval] [number]

The interval parameter specifies the number of seconds between polls, and the number parameter specifies how many polls to make (infinite by default). The size of the run queue is reported in ther column ofvmstat's output (the following output is from a four-processor Solaris 8 system):

bash$ vmstat 5
procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 m1 m1 m1   in   sy   cs us sy id
 0 0 8  8846112 684752 0  0  0  0  0  0  0  0  1  1  1 4294967196 0 0 -10 -2 -126
 4 0 19 9164400 961872 70 80 675 3 3  0  0  0  0  0  0  823 33765 1124 93 5  2
 4 0 19 9164400 960336 57 12 585 0 0  0  0  0  0  0  0  804 34366 1070 93 7  0
 4 0 19 9164400 958568 43 12 672 1 1  0  0  0  0  0  0  848 34423 1137 94 6  0
 3 0 19 9164400 956888 49 13 732 0 0  0  0  0  0  0  0  816 33288 1161 90 8  2
 5 0 19 9164400 962064 61 12 771 0 0  0  0  0  0  0  0  896 32233 1300 90 10 0
 4 0 19 9164400 976224 19 12 1035 0 0 0  0  0  1  1  1  874 32232 1216 88 9  3


  

Here,the size of the run queue hovers consistently around four; since thisis a four-processor system, it certainly does not need more processorsbased on this data. Of course, this measurement should be made over alonger period of time; measuring load averages over a few hours during peak usage would be appropriate.

Ignore the First Line ofvmstat's Output

The first line of output from vmstat contains averages of all statistics since the system was booted and should be thrown out, no matter how valid it looks. Subsequent lines contain more current information.

Thefirst thing most administrators try to do when they discover that asystem is overloaded is to identify just what is using all of thatprocessing power. They must first determine whether a user process orthe kernel is responsible for the usage. The following sections will explain this process in detail.

Tracking User Versus System Processing

A CPU is notjust responsible for running processes—the kernel also needs CPUresources, often more so than the processes it spawns. Some of thetasks performed by the kernel that can consume CPU include:

• Excessive paging and swapping

• Software RAID parity calculations

• Allocating and freeing large amounts of memory

• Network traffic filtering

Fortunately,you can determine how much of a processor's time is spent in the kerneland how much is spent executing processes in user space. Theus andsy columns ofvmstat output show the percentage of time spent in user space and system (kernel) space respectively.

bash$ vmstat 5
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 m1 m1 m1   in   sy   cs us sy id
 0 0 8 8846072 684728 0   0  0  0  0  0  0  0  1  1  1 4294967196 0 0 -10 -2 -126
 7 0 19 9154192 843984 28 15 563 0 0  0  0  0  0  0  0  785 34121 1099 94 6  0
 6 0 19 9154192 843960 25 12 683 0 0  0  0  0  0  0  0  741 33823 1004 94 6  0
 6 0 19 9154192 843960 26 12 654 1 1  0  0  0  0  0  0  860 33408 1150 95 5  0
 6 0 19 9154192 843968 27 11 529 0 0  0  0  0  0  0  0  789 33971 1138 94 6  0
 6 0 19 9154192 843960 25 11 577 0 0  0  0  0  0  0  0  758 34772 1026 94 6  0


  

After discarding the first line(as you learned in the earlier example), you can see that 94% of theCPU's processing resources are dedicated to user processes, and only 6%handle kernel code. This distribution of processing power is what youwant from a system; it should spend its time running application code,not stuck in the kernel. The last column of this output,id, is the percentage of time that the CPU is idle (zero, in this case).

Another column to pay attention to iscs, which stands for context switches.A context switch is the transition from user-level code execution tokernel code execution and vice versa. Noting context switches isimportant, because the system incurs a significant performance penaltywith each context switch (such as when an Intel x86 processor mustswitch back and forth from protected mode). Typically, two eventsinvoke a context switch on a Unix system:

• System calls are the entry points into the kernel from user space. When a process needs to open a file, write data to disk, or reboot the system, it uses a system call to tell the kernel to perform these tasks on its behalf. After the task is complete, successful or not, control is passed back to the process in user space.

• Interrupts are more abrupt than system calls. The system generates an interrupt when traffic arrives on a network interface, a key is hit on the keyboard, or a timer expires. The interrupt forces the system into the kernel to handle the interrupt. The interrupt preempts the execution of all processes running on the system at that time—no matter what the processes might be—so that the kernel can process the event that caused the interrupt.

Switching back and forth betweenuser and kernel space can put a strain on any CPU, so context switchesshould be a suspect when your system is sluggish and there is novisible load. Unlike user, system, and idle time, context switches arereported as context switches per interval. The number of contextswitches in each 5-second interval appears in thecs column ofvmstat 5 output.

Compare Like Data

When running performance comparisons, be sure to use the same sampling interval so you can quickly compare the resulting numbers without having to convert them.

No hard and fast rule dictatesthe maximum rate of context switching; the variety of processors,hardware, and operating systems in use today makes this rate impossibleto calculate. To monitor your system's context-switching rate, firstdetermine what rate is normal for your system, then watch forvariations from that rate. If a system usually has 500 to 1,000 contextswitches per second, you know there's a problem when that number shootsup to 3,000. In that case, you could trace the running process to seeexactly what the process is trying to do (you learn more about tracingin “Tracing Processes,” later in this chapter).

Afteryou have determined that user space processes are responsible for yourhigh load, it's not hard to figure out what processes are causing thatload. All Unix systems have commands to help you analyze processes—twoof the more useful ones areps andtop.

Usingps to Monitor CPU Usage by Process

Theps command is available on all Unix platforms, but few people use it to measure CPU usage. Becauseps is available everywhere, however, you can use it to analyze process CPU usage on any Unix version. The output ofps can be verbose, but the only column of interest for this measurement isTIME, which reports the amount of time a process has spent running on a CPU. While individualTIME statistics tell you how busy a process has been up to that point, running repeatedps commands can tell you if a process's CPU time is increasing.

For example, on a heavily loaded Oracle database server,ps gave the following output for one of the Oracle processes over several seconds:

PID TTY      TIME CMD
121 ?       100:45 oracle
121 ?       100:46 oracle
121 ?       100:46 oracle
121 ?       100:46 oracle
121 ?       100:47 oracle
121 ?       100:47 oracle
121 ?       100:47 oracle
121 ?       100:47 oracle
121 ?       100:47 oracle
121 ?       100:48 oracle

The Oracle process, having alreadyused over 100 minutes of CPU time, increases slowly and consumes anadditional second of CPU time over a 4-second interval. This equates toabout 25% of a CPU's time, so this process should definitely beinvestigated further.

Processes Use Little CPU Time

Most processes do not use much CPU time at all, and won't register even a second of CPU time during their lifetime.

Usingtopto Monitor Process CPU Usage

Usingps to look for CPU-intensive processes can be a cumbersome task when there are hundredsof processes on the system in the first place. In such circumstances,it would be nice to let the computer do the work for you, and thetop command makes that possible.

topis a third-party application that usually isn't bundled with anoperating system (though several versions of Linux do provide it foryou). By default,top displays the top 15 processes by CPU usage over 5-second intervals, updating its display after each interval:

load averages:  0.42,  0.11,  0.07                                     10:37:13
85 processes:  83 sleeping, 2 on cpu
CPU states: 49.3% idle, 49.6% user,  0.6% kernel,  0.5% iowait,  0.0% swap
Memory: 2048M real, 1127M free, 656M swap in use, 5025M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
 2846 oracle     1  30    0  593M  569M cpu0  136:08 37.86% oracle
10277 root       1  58    0 2888K 2096K sleep   0:00  0.56% sshd
10283 jhorwitz   1  58    0 1592K 1328K cpu2    0:00  0.17% top
 2841 oracle     5  52    2   11M 7536K sleep   1:14  0.14% ora_db_check.pl
  390 oracle     1  58    0  592M  563M sleep   6:29  0.03% oracle
 1314 oracle     1  58    0  592M  567M sleep   0:01  0.02% oracle
 1312 oracle     1  58    0  592M  567M sleep   0:01  0.02% oracle
10279 jhorwitz   1  20    0 2472K 1952K sleep   0:00  0.02% bash
  352 root       1  56    0 2648K 1424K sleep   5:11  0.01% sshd
 7127 oracle     1  58    0  593M  570M sleep   0:29  0.01% oracle
  400 oracle     1  58    0 9552K 4352K sleep   1:54  0.00% tnslsnr
23323 oracle     1  58    0  592M  568M sleep   0:16  0.00% oracle
  388 oracle     1  58    0  592M  563M sleep   0:37  0.00% oracle
  386 oracle     1  58    0  594M  564M sleep   0:23  0.00% oracle
  235 root       9  53    0   14M   13M sleep   0:21  0.00% nscd


  

Right away, you can see thatin this output the PID 2846, an Oracle process, is consuming almost 38%of the total CPU resources on the machine. The results even show onwhich CPU the process is currently running.

Check Usage on Dual-Processor Systems

Don't forget that the CPU column in top output reports total CPU usage across all of the processors. A process that is consuming 50% of CPU time on a dual-processor machine may not be as dormant as you might think—it may actually be consuming 100% of a single CPU's resources.

On a multi-CPU machine with n processors, a single threaded process can never consume more than the fraction 1/n of CPU time.

In the previous example, most ofthe processes were sleeping—a state in which the processes areblocking, waiting for input or for a resource to become available.Processes consume no CPU time while they are sleeping. Thetopoutput identifies currently executing processes by listing the CPU onwhich they are executing. The following is a list of common processstates reported by thetop command on Solaris:

The Linux version oftop reports process states a bit differently. The most common states you will see are the following:

Useprstat on Solaris

Solaris 8 comes with its own top-like tool called prstat. This CPU-usage by process monitor is Solaris-specific, incurs less processing overhead than top, and can report additional details, such as how many threads exist for a particular process.

On a more heavily loaded server, you can see processes in the run state. This means that the processes are in the run queue waiting for a CPU to become available. Here is thetop output for a very busy dual-CPU Oracle server:

load averages:  4.51,  4.25,  3.97                                     11:00:22
149 processes: 143 sleeping, 4 running, 2 on cpu
CPU states:  0.0% idle, 90.8% user,  9.2% kernel,  0.0% iowait,  0.0% swap
Memory: 2048M real, 860M free, 688M swap in use, 8955M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
 5726 oracle     1  40    0  453M  427M run    40:01 22.95% oracle
 6056 oracle     1  40    0  453M  427M run    10:11 22.16% oracle
 5480 oracle     1  50    0  453M  427M run    60:04 21.27% oracle
 6189 oracle     1  25    2 1280K  936K cpu2    0:08 12.25% gzip
 2222 oracle     1  53    0  453M  424M sleep  33.6H  5.07% oracle
 2220 oracle     4  55    0   11M 5472K run   669:01  4.08% load_NAM.pl
 5713 oracle     1  58    0  454M  430M sleep   0:41  2.20% oracle
18626 oracle     1  58    0  452M  418M sleep  98:49  1.02% oracle
 6190 root       1  58    0 2888K 2152K sleep   0:00  0.56% sshd
18624 oracle     1  58    0  454M  420M sleep 130:28  0.53% oracle
19882 oracle     1  60    0  454M  425M sleep   6:09  0.31% oracle
 6198 jhorwitz   1  58    0 1624K 1336K cpu1    0:00  0.26% top
 6045 oracle     1  59    0  453M  429M sleep   0:08  0.24% oracle
 6164 oracle     4  25    2   11M 7632K sleep   0:00  0.22% arch_maint.pl
28600 oracle     1  54    0  455M  427M sleep   3:25  0.19% oracle


  

Note how there are two processes executingon the two CPUs while four are waiting in the run state. This state isreflected in the load averages as well; with four processes in the runqueue, you would expect a load average above 4, and indeed, the 1- and5-minute load averages are 4.51 and 4.25 respectively.

Usingtime to Monitor Process CPU Usage

Thetime command runsa command specified on the command line and reports back the real,user, and system time used by that command (real time is actual clocktime). Although not terribly useful for already active processes,timecan help you test programs before they run to gauge their CPU usage.You also can insert the command into startup scripts to automaticallyprofile applications as they start. To profile the CPU usage ofcalculating the MD5 checksum for the Linux kernel source, you can do the following:

bash$ time md5sum linux-2.4.17.tar.bz2
0628d142eeb6fc31856d7a60d3f30a82  linux-2.4.17.tar.bz2

real    0m0.356s
user    0m0.280s
sys     0m0.070s

Here, you see that most ofmd5sum'stime is spent in user space doing the actual calculations. The 0.07second in system space is most likely time spent opening, closing, andreading the file from disk.

Useptime for Command Children

The time command will not calculate times for any children of the command you run. The Solaris command ptime can do this for you.

Analyzing Historical CPU Performance

HistoricalCPU usage can be just as useful as real-time data. You can see trendsover time, and can look for the causes of performance problems afterthe fact. Both Solaris and Red Hat Linux come packaged withsar, the system activity reporter.sarreports historical performance data collected over time, and it can beused to look at everything from the CPU usage at 8:00 a.m. this morningto disk activity from three months ago. Several examples in thischapter usesar, but this section focuses on its CPU reporting capabilities.

Usesar for Real-Time Reporting

You can use sar for real-time reporting by specifying an interval in seconds and a count as the last two arguments to the command. sar -u 10 6 reports CPU utilization six times at 10-second intervals.

In its default configuration,sar-u reports CPU utilization for the current day. By default,sarkeeps a separate data file for each day, and you can view data fromdifferent days by supplying the patch to one of those files using the-fflag. The data reported includes the percentage of time spent in usermode, system mode, and waiting for a process blocking for I/O, as wellas the portion of the CPUs that were idle, as follows:

bash$ sar -u

SunOS chills 5.8 Generic_108528-09 sun4u    03/15/02

00:00:00    %usr    %sys    %wio   %idle
00:05:00      39       2       0      58
00:10:00      44       2       0      54
00:15:01      41       2       0      56
00:20:00      42       3       0      55
00:25:00      45       2       0      53
00:30:00      42       2       0      56
00:35:01      41       3       0      56
00:40:00      46       4       0      50
00:45:00      42       3       0      55
00:50:00      52       3       0      45
00:55:00      47       2       0      50
01:00:00      44       2       0      54

In the preceding example,sar collected samples every 5 minutes; you can change the timing by altering thecron job that collects the data. Although it may not be the same on all systems, on the Solaris 8 system thecrontab command looks like this:

# crontab –l sys
# Collect system activity data
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/lib/sa/sa1

System Activity Log Repository

On Solaris, system activity logs are stored in /var/adm/sa. On Red Hat Linux, they are stored in /var/log/sa. Each file is named sa NN, where NN is the day of the month the logs were collected. The command sar -u -f /var/adm/sa/sa26 would display CPU usage from the 26th day of the month.

• 
• Create Bookmark (Key: b)Create Bookmark
• Create Note or Tag (Key: t)Create Note or Tag
• Download (Key: d)Download
• Email This Page (Key: e)Email This Page
• PrintPrint

• Html View (Key: h)Html View
• Zoom Out (Key: -)Zoom Out
• Zoom In (Key: +)Zoom In
• Toggle to Full Screen (Key: f)
• Previous (Key: p)Previous
• Next (Key: n)Next

Related Content

System performance checkup
From: UNIX® System Administration Handbook, Third Edition

STEP 4. Monitoring CPUs
From: Configuring and Tuning Databases on the Solaris™ Platform

Linux Performance Tools: CPU
From: Optimizing Linux® Performance: A Hands-On Guide to Linux® Performance Tools

CPU, Memory, and I/O with top(1)
From: Absolute FreeBSD, 2nd Edition

Analyzing CPU Performance
From: Pro Ubuntu Server Administration

CPU-bound system
From: IBM eServer Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

Reporting Virtual Memory Utilization (vmstat)
From: Solaris™ Application Programming

CPU Diagnostic Tools
From: HP-UX 11i Tuning and Performance

CPU Tuning
From: Pro Ubuntu Server Administration

System performance checkup
From: Linux® Administration Handbook, Second Edition