[Cockcroft98] Chapter 3. Performance Measurement

Chapter 3. Performance Measurement

Insome cases, it is obvious that there is a performance problem; when itis fixed, there may be a noticeable improvement and your job isfinished. It is much more common for performance tuning to consist ofmany small cumulative steps. Some of the steps may reduce performanceoverall or make one part of the system faster at the expense of anotherpart of the system.

Problemscan arise when it is necessary to quantify the changes or to measureminor performance changes in complex systems. If the aim of the work isto improve the performance in a well-defined benchmark, such as one ofthe SPEC or TPC measures, then the measurement can be well defined. Inreal life, it may be necessary to design a controlled set ofexperiments to measure performance on the system being worked on.

Theanalysis of test results can be used to produce sizing or performancemodels. A method for producing a configuration and sizing guide formultiuser systems is described at the end of this chapter.

The normal situation, illustrated in Figure 3-1,is that you have a computer system with a particular configuration andan application workload running on that system. You then measure theperformance, analyze the results, and make changes to the configuration.

Inmany cases, no measurements at all are taken until there is some signof a problem. It is important to collect a background level ofinformation at all times so you can see what changed when a problemoccurs. If you just collect performance measurements, you still havethree unknown quantities. You need to keep careful records ofconfiguration changes and to monitor the workload levels as well. Thatleaves the random fluctuations, and they can be averaged out over timeor correlated with time of day and the calendar. This averaging is whyyou need to collect over a long period.

The Workload

Theworkload is actually one of the hardest things to deal with. In somecases, you can have total control, for example, when running a singleCPU-bound application with fixed input data. In other cases, theworkload is totally beyond your control, for example, on a timesharingserver with many users running different applications.

Theworkload selection is also your weakest point if someone wants tochallenge the results you are claiming credit for. If you didn’t spenda lot of time researching your workload, then you have no comeback to acritic who says, “That’s all very well, but in real life theapplications aren’t used like that, and you have measured and tuned thewrong things.” It is much harder for someone to dismiss a detailed,written justification of the workload you have used. For this reason,it is worth writing up the workload and circulating the workloadjustification to interested parties before you start to collect data.You are likely to get valuable review feedback that will help youtarget your efforts more precisely.

Fixed Workloads

Thebiggest problem with a fixed workload is that because you have totalcontrol, you yourself have to make decisions that can affect theresults you get. In real situations, the same program is usually runwith different input data each time it is used. You must make sure thatthe input data you use is somehow representative of the kinds of datafor which you want to optimize the system performance. It is worthspending a lot of time investigating the typical usage of anapplication by your user base. If you don’t investigate, then yourcarefully tuned optimizations may not have the desired effect.

Random Workloads

Randomworkloads are in some ways easier to deal with. If you can’t controlthe workload, you don’t have to spend so much time investigating it.Instead, you have to spend more time analyzing the measured results toseparate out the random fluctuations from the effect of your tuning.Some fluctuations will occur, depending on the time of day or whether adepartmental staff meeting is in progress or whether a particularproject team is trying to meet a delivery deadline.

Insome cases where you have a very large number of users, the averagebehavior of the users is very predictable. This is the case forInternet web servers, where the daily load levels follow a regularpattern. The pattern is disturbed only by events that reach a largenumber of people, such as new content on the web server, a major newsevent, or a popular TV program!

Managed Workloads

Anincreasing number of tools can be used to capture, encapsulate, anddrive a workload into your test system from another computer. Theworkload can either be captured from a live session or constructedmanually. These tools can be divided into two categories. Journalling-basedtools capture the user input only and simply attempt to replay it at afixed speed. They blindly continue even if something unexpected happensor the system responds more slowly than during capture, and they cannotbe programmed to react appropriately to variable responses from thesystem being tested. Emulator-basedtools pattern-match the received responses to ensure that the system isstill on track, measure response times, and decide when the next inputis due. These are sophisticated and expensive tools, so they are notfor the casual user. They allow complex workloads to be run reliablyand repeatedly and are normally used to produce system sizinginformation or capacity planning guides. Traditionally, these toolshave worked with ASCII text applications on directly connected lines ortelnet over Ethernet, but they have been extended to work withclient/server database protocols and HTML. The main vendors arePurePerformix Inc., with Empower™; Performance Awareness Corp, withPreVue™; and Mercury, with LoadRunner™. These tools are often known asRTE systems, for Remote Terminal Emulator. In “Emulating NC-Based Users” on page 112, I describe how to emulate Java applications that use arbitrary client/server protocols.

Workload Characterization

Ifyour workload consists of multiple users or individual components allrunning together, then it can be very useful to run each componentseparately (if possible) and measure the utilization profile over time.A plot of the CPU, disk, and network activity will show you when thepeak load occurs and whether the load is even or peaks with idle timein between. Capacity planning tools, like the Best/1™ product from BGSInc., sample the resource usage of each process on the system tomeasure the profile over time. Best/1 helps you create user classes andtransaction classes, then combines them into workloads. Thisinformation is fed into a queuing model that can be used to predict thebehavior of the system under various conditions. The most common usesare to see what would happen to response times if more users were addedto the system and to predict the effect of hardware upgrades.

Configuration Changes

Thereis a tendency to make a bunch of small changes at one time, in the hopethat they will have a big effect on the system. What is needed insteadis a systematic approach that gives you the results you need to decidewhich changes are making a significant difference and which changes canbe ignored.

There are always too many possibilities. To illustrate, in Table 3-1I tabulated some possible configuration factors from the subsequentchapters of this book. The important thing is to clearly identify thefactors you wish to vary and then to pick a small number of levels thatspan the usable range for the factor.

Table 3-1. Example Configuration Factors and Levels

Thebest way to approach a large-scale set of tests is to perform aninitial small set of tests with just two levels for each factor. RajJain, in The Art of Computer Systems Performance Analysis,describes a sophisticated statistical technique that drasticallyreduces the number of tests required to work out the factors that makea significant difference and those that can be ignored. To fullymeasure every combination of six different factors with four levels foreach factor would take 4⁶ = 4096 separate measurements. Reducing to two levels for each factor brings this number down to 2⁶= 64 separate measurements. Using the statistical methods to analyzethe results, you carefully pick certain combinations of levels tomeasure and do the analysis on only seven measurements to find outwhich factors can be ignored for a more complete set of tests.

Oneextra benefit of doing a small preliminary test is that you startanalyzing the results sooner than you would if you did all themeasurements in one session. Since you will often find some bug in yourmeasurement methods, you have a chance to fix it before you spend a lotlonger taking bad measurements. It is common for a large proportion ofthe measured data to be useless for some reason, even in thebest-designed experiments.

Measurement Types and Relationships

Thefundamental types are throughput, response time, queue length, andutilization. Nearly all the measurements you can make fit into one ofthese categories. You can perform a simple mental check whenever youare collecting a set of measurements to make sure that you arecollecting at least one of each type. You do not have the completepicture and can easily be misled if you don’t have a complete set ofmeasurements available to you. In some cases, given some of thesemeasures or some related measures, you can calculate the complete set.

It is important to remember that it is the steady state averagethroughput, response time, queue length, and utilization that are beingmeasured. The most common source of confusion when dealing with themathematics of queueing comes from forgetting that everything is alwaysworked out as an average.

Fundamental Types

Throughputis the amount of work that is completed in a given amount of time. Thetransactions per second (TPS) measures quoted for databases are oneexample. Do not confuse throughput with bandwidth. Bandwidth is typically the peak, possible speed ignoring any overhead. Maximum throughput is how much work you can actually get done at 100 percent utilization.

• A Fast SCSI bus rated at 10.0 Mbytes/s bandwidth has a maximum throughput of about 7.5 Mbytes/s when you take the SCSI protocol overhead into account. A Fast Wide SCSI bus rated at 20.0 Mbytes/s has a maximum throughput of about 16 Mbytes/s.

• The MBus used on the SPARCstation 10 and the XDBus™ used on the SPARCserver 1000 both have the same bandwidth of 320 Mbytes/s. Assuming a typical mix of transaction types, the throughput of the MBus is about 100 Mbytes/s, and the throughput of the XDBus is about 250 Mbytes/s. The large variation is due to a totally different protocol on each bus. The MBus is not good for throughput, but it does have substantially less latency than that of the XDBus.

• The throughput of a disk is reported as both reads and writes per second, and Kbytes read and written per second by iostat.

Response timeis a measure of the time that the user has to wait for some work tocomplete. When measured by an RTE system, this can be as fine as thetime taken to echo characters being typed, or it can be the time takenfor a database transaction to complete for that particular user.Latency is the same thing, but this term is more often used when aprotocol is being discussed. The BGS Best/1 performance modeling toolmeasures the CPU consumption of a process over time. The elapsed timetaken to consume a fixed amount of CPU time can be defined as theresponse time. If the system is overloaded, that process will slow downand accumulate CPU time more slowly, and Best/1 reports that increasein response time.

Think timeis a measure of how long the user waits before starting anotheroperation. A high think time reduces throughput and utilization.

Service timeis the time taken to process the request once it has reached the frontof the queue. At low utilization levels, the queue is empty, so theresponse time is the same as the service time. Service time can becalculated as the utilization divided by the throughput. The measurethatiostat callssvc_t is actually the response time of the disk, not the service time, if the right terminology is used.

Queue lengthis the number of requests that are waiting for service. A long queueincreases the response time but does not affect the service time. Queuelength can be calculated as the throughput times the response time.

Utilizationis a measure of how much of the computer’s resources were used to dothe work. It is the busy time as a proportion of the total time.

To illustrate these measurements, we can use some sampleiostat output, shown in Figure 3-2, to check the formulas and derive what is missing.

Figure 3-2. Exampleiostat -x Disk Statistics

extended disk statistics 
disk      r/s  w/s   Kr/s   Kw/s wait actv svc_t %w %b 
sd1       0.6 14.6    2.0  220.8  0.0  0.7  43.9  0 16

Forsd1, the throughput isr/s+w/s, which is 15.2. The total queue length iswait+actv, which is 0.7; disk utilization is%b, which is 16% or 0.16.

Thequeue length needs to be reported more precisely than 0.7 to get anexact result. Working backward, the queue length should be 0.67 if thethroughput and response times given are accurate. It is clear thatiostat is reporting the response time of an I/O request assvc_t. The service time of the disk itself (which is what the disk makers quote) is 11 ms in this case.

Rememberingthat these times are averages and that the mathematics assumes anexponential distribution of service times, we can estimate responsepercentiles.

• 80 percent of response times occur in less than the average response x 5 / 3.

• 90 percent of response times occur in less than the average response x 7 / 3.

• 95 percent of response times occur in less than the average response x 9 / 3 (or x 3).

So,if you are dealing with a requirement that involves the 95th percentileresponse time, you can estimate that this is about three times theaverage response time.

Multiuser System Response Time

Ifa particular configuration of a multiuser system is tested with an RTE,then the test can be repeated under controlled conditions withdifferent load levels or user counts. The throughput, response time,and utilization can be measured at each load level and plotted. Theresulting set of plots normally takes one of the forms shown in Figure 3-3.

Ifseveral users run a job at once, then the utilization level will behigher, the response time will be similar, and the throughput will haveincreased. At some point as users are added, the utilization approaches100 percent and the user jobs must wait in a queue before they cancontinue. Utilization is a generic term. In practice, the CPU usagelevel is the most common utilization measure, although it is alsocommon for the initial bottleneck in the system to be an overloadeddisk drive with high utilization.

As shown in Figure 3-3,once we reach 100 percent utilization, throughput stops increasing andthe response time starts to climb. This is the saturation point of thesystem. If still more users are added, the overhead of managing longqueues and switching between users takes an increasingly largeproportion of the CPU, the system CPU time climbs, the user CPU timedrops, and the throughput drops with it.

I have shown two generic curves for response time. They represent two different cases. The open systemresponse time curve occurs when you have a very large number ofpotential users, such as on an Internet-connected web server. The closed systemresponse time curve occurs when you have a finite number of users, suchas a database order entry system, and the users have a think time. Thequeuing equations are shown in Figure 3-4.:

Inan open system, if you consider a web server running SPECweb96, theresponse time at low load levels and low utilization over a LAN isabout 5 ms, so that value provides an estimate of the base-levelservice time for one operation when there is no queue to delay it. AsCPU utilization reaches 50% (0.5 in the equation), the response timeshould double to 10 ms. At 90% busy, the response time will be 50 ms.If you ever reach 100% busy, you will be getting more incoming requeststhan you can complete, so the queue grows to be infinitely long and thesystem responds infinitely slowly. In the real world, you get a TCPconnection time-out after many seconds of waiting.

Ina closed system, users are either thinking or waiting. As the systemsaturates, more users are waiting and fewer are thinking. Ultimately,they are all waiting, so there are no more users left to increase theload. This is why response time increases gradually and the system doesnot come to a complete halt under a heavy load. For example, with oneuser completing a transaction every ten seconds (throughput = 0.1) andthinking for nine seconds, each transaction must have a response timeof one second. One hundred users with a throughput of eighttransactions per second would give a response time of 4.5 seconds.

Theideal situation is to always run under the 90 percent utilizationlevel, with a very stable workload. In practice, most machines have awidely fluctuating workload and spend most of the time at a lowutilization level. When a peak load occurs, it may exceed thesaturation point; what happens at saturation can make a big differenceto the perceived performance of the system.

Atsaturation, the response time increases. The goal should be to have asteady increase over a wide load range; that way, users find that thesystem gradually seems to slow down, and they tend to reduce theirdemand on the system. If the response time increases sharply over ashort load range, users find that the system suddenly stops respondingwith little advance warning, and they will complain about poorperformance.

Whenyou are installing a new system or tuning an existing one, deliberatelydrive it into a saturated state. If you then carefully tune theperformance under excessive load, you may be able to get a gradualresponse degradation curve. This result may mean that the system beginsto slow down a little earlier than before, which acts as a warning tothe users. Be aware that if a system is tuned so that every resourcehits 100 percent utilization at the same time, you will have a veryfast degradation at the saturation point.

Collecting Measurements

Accordingto Heisenberg’s Principle, one cannot measure something without alsoaffecting it in some way. When you measure the performance of a system,be aware of the effect that your measurement tools may have.Fortunately, most of the data collection utilities have a negligibleimpact on the system. Asar,vmstat, oriostatcollecting data at 5-second intervals is not going to make a noticeabledifference. Collecting a system call trace of an active process is anexample of the kind of high-speed data collection that should be usedwith care.

Using Accounting to Monitor the Workload

Ifyou have access to a group of end users over a long period of time,then enable the Unix system accounting logs. The logs can be useful ona network of workstations as well as on a single time-shared server.From them, you can identify how often programs run, how much CPU time,disk I/O, and memory each program uses, and what the work patternsthroughout the week look like. To enable accounting to startimmediately, enter the three commands shown at the start of Figure 3-5.

Figure 3-5. How to Start Up System Accounting in Solaris 2

# ln /etc/init.d/acct /etc/rc0.d/K22acct 
# ln /etc/init.d/acct /etc/rc2.d/S22acct 
# /etc/init.d/acct start 
Starting process accounting 
# crontab -l adm 
#ident  "@(#)adm        1.5     92/07/14 SMI"   /* SVr4.0 1.2   */ 
#min    hour    day     month   weekday 
0       *       *       *       *       /usr/lib/acct/ckpacct 
30      2       *       *       *       /usr/lib/acct/runacct 2> \ 
 /var/adm/acct/nite/fd2log 
30      9       *       *       5       /usr/lib/acct/monacct

Check out the section “Administering Security, Performance, and Accounting in Solaris 2” in the Solaris System Administration AnswerBook and see the acctcom command. Somecrontabentries must also be added to summarize and checkpoint the accountinglogs. Collecting and checkpointing the accounting data itself puts no additional loadonto the system. Accounting data is always collected by the kernel, sothe only decision is whether to throw it away when the process exits orto write a 40-byte record to the accounting file. The summary scriptsthat run once a day or once a week can have a noticeable effect, sothey should be scheduled to run out-of-hours. The most useful summarycan be found in/var/adm/acct/sum/rprtMMDD and in/var/adm/acct/fiscal/fiscrptMM, whereMM is the month andDD is the day. This is the data provided for each user:

LOGIN    CPU (MINS)    KCORE-MINS    CONNECT (MINS)  DISK     # OF    # OF# DISK    FEE 
UID   NAME     PRIME NPRIME  PRIME   NPRIME  PRIME   NPRIME  BLOCKS  PROCS    SESSSAMPLES 
0     TOTAL   83     235     865222  2446566 3994    10638   0       1062     14  0        0 
0     root    58     165     687100  1945493 0       0       0       63       0   0        0 
9506  adrianc 6      18      134523  381188  3994    10638   0       366      14  0        0

    

The data provided for each command includes some indication of process size as well as the CPU usage.

TOTAL COMMAND SUMMARY 
COMMAND   NUMBER     TOTAL     TOTAL     TOTAL   MEAN   MEAN    HOG    CHARS        BLOCKS 
NAME        CMDS    KCOREMIN    CPU-MIN  REAL-MIN  SIZE-K CPU-MIN  FACTOR TRNSFD     READ 
TOTALS     26800   177210.42    37.02    76549.92 4786.86  0.00  0.00   3120715200 4633 
netscape       3    91875.33     6.33    10471.48 14523.83  2.11  0.00    683552768 6 
mailtool       7    29960.85     2.99    19144.39 10018.68  0.43  0.00    525712736 15 
symon          5    13687.47     1.26       52.80 10861.63  0.25  0.02     19416320 6 
sh          3947     7840.11     4.90     5650.93 1599.53  0.00  0.00      3404445  9

    

Collecting Long-Term System Utilization Data

Collectoverall system utilization data on all the machines you deal with as amatter of course. For Solaris 2, this process is already set up andjust needs to be uncommented from thecrontab file for thesys user. Figure 3-6 illustrates a crontab collection for long-term use.

Figure 3-6.crontab Entry for Long-Termsar Data Collection

# crontab -l sys 
#ident"@(#)sys1.592/07/14 SMI"/* SVr4.0 1.2*/ 
# 
# The sys crontab should be used to do performance collection. See cron 
# and performance manual pages for details on startup. 
# 
0 * * * 0-6 /usr/lib/sa/sa1 
20,40 8-17 * * 1-5/usr/lib/sa/sa1 
5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

The Solaris 2 utilization log consists of asar binary log file taken at 20-minute intervals throughout the day and saved in/var/adm/sa/saXX, whereXX is the day of the month.sar collects a utilization profile for an entire month. Save the monthly records for future comparison.

Whena performance-related problem occurs, it is far easier to identify thesource of the problem if you have measurements from a time when theproblem was not present. Remember that the real-life user workload islikely to increase over time. If you are trying to justify to a managerthe purchase of an upgrade, that manager will find it hard to dismiss aplot showing a long-term utilization trend.

Percollator—A Customizable Data Collector

I wrote an“se” script calledpercollator.se (see “The SE Toolkit Web Server Performance Collator, percollator.se”on page 88), which stands for Performance Collator but is also a Javacoffee reference (it has a Java-based graphical browser). The scriptaims mainly to log the performance of web servers and proxy servers,but it can be customized to log anything. It gathers data from manysources, including application log files and also includes a summary ofthe state of the system, using the same rules as thevirtual_adrian.se script.

Processing and Analyzing the Measurements

Processingand analyzing the measured results of a set of experiments is anopen-ended activity. Collected data is often simply processed intoaverages and tabulated or plotted. I’m sure that even with the best ofintentions, the postprocessing and analysis of results could often bedescribed as cursory. Part of the problem is a lack of tools andtechniques, so I will explore some simple methods and give examples ofthe kind of advanced results that can be obtained by use of a goodstatistics package like SAS/CPE™ or S-PLUS™.

Generating Averages withsar

If you have the automatic utilization log enabled, then the default for asar command is to print out the log for the day with an average at the end. Wheneversar is used to read asar log file, you can specify start and end times, andsar will produce an average.

See “Understanding vmstat and sar Output” on page 320 for more details.sar records all the information that the more familiariostat andvmstatcommands report, as well as many other useful measurements. Its majorflaws are that it does not store any network-related information, so asupplementalnetstat output log may be needed, and its output formats do not show all the collected data.

You can use anawk script^[1] to pick out the information of interest fromsar, and you can combine data from several measurements into a table like the one shown in Figure 3-7.This table shows how the CPU and disk utilization vary with increasingnumber of users on a complex, RTE-driven, database workload that has alarge batch processing component.

^[1] Details of this script are left as an exercise for the reader, but the Solarishead,tail,cut, and paste commands may also be useful.

Using the Results Generated by an Emulator System

Emulatorsystems run a program to emulate each user. This program uses a scriptof commands and responses and includes defined operation sequences(sometimes known as user functions). Each user program writes a log ofevery command and response that has the function start and end pointswith accurate timestamps embedded in it. After a mixture of theseprograms has been run as a workload, special postprocessing commandsextract the named functions and timestamps. Reports can then begenerated for a particular start and end time during the test (whichshould match the start and end time used for thesarutilization averages). The report lists the number of times each namedfunction completed in the specified time interval and the maximum,minimum, mean, and standard deviation of the time taken.

Thenumber of functions completed in the time interval is a throughputmeasure, and the average time taken is a latency measure. This measure,together with thesar-based utilization information, affords a good basis for further analysis.

If a set of results is taken with varying numbers of users, then the measurements will often have the form shown in Figure 3-3 on page 46.

Obtaining Results on a Live System

Ifyou are monitoring a live system with real users, then you will need tofind a way of quantifying the throughput. Response times are very hardto obtain on a live system, but in many cases, they can be inferredfrom the throughput and utilization measures. To find a throughputmeasure, look at the system accounting reports to see if a keyapplication program that is associated with the users uses asignificant amount of CPU or I/O. You can also look at the typicalexecution times for the program, and it may be possible to derive athroughput measure either from the number of times the program executedor from the total amount of CPU time or I/O consumed by that program ina given time. Database applications are somewhat easier because you canusually identify how many transactions of a given type have occurred ina given time period and use this as the throughput measure.

Whereasa managed workload using an RTE may process data over a time periodmeasured in minutes, a real-life workload will often need measurementperiods measured in hours or days. If users log in only once each dayand use a single instance of an application throughout the day, thenthe daily CPU usage for that application could be extracted from theaccounting logs each night, by means ofawk, as a throughput measure.

Analyzing the Results

Theanalysis required depends on the situation. If you are conducting asizing or capacity planning study, then many results for differentconfigurations of the system can be combined into a sizing model. Ifyou are monitoring a live system and trying to identify where the nextbottleneck will occur, then you need to build a model of changes in thesystem over time.

Producing a Sizing Model

Iwill briefly describe a method that has worked well for sizingmultiuser workloads, using the Performix Empower RTE and the StatSciS-PLUS statistical modeling package.

Thetabular data on utilization, throughput, and response time at varyinguser counts was produced for each system configuration that was tested.A system was set up with a particular disk, memory, and CPUconfiguration, and then a series of measurements was made withincreasing numbers of users. At the high user counts, the system waspushed well into an overloaded state for some configurations.Configuration changes were made by varying the number of CPUs^[2], changing the disk setup, and changing the amount of think time in the workload scripts.

^[2] This can be done programmatically; see “CPU Control Commands — psrinfo, psradm, and psrset” on page 229.

Torepeat our experience: First, plot the data to make sure that it looks“clean.” Next, estimate the saturation point for each configuration interms of users. Then, fit a straight line from the origin through thefirst few points of the throughput data. This line represents theregion where the system is underused; the only constraint on throughputis the think time of the users (the rate at which they ask the systemto do work for them). Draw another straight horizontal line from thepoint of peak throughput; the intersection of the two lines is definedas the saturation point and projected down to produce a user count. Figure 3-8 illustrates the technique.

Wefound this measure to be very stable, and it didn’t fluctuate much whenwe repeated the measurements on the same configuration, whereas theposition of the peak throughput varied much more. The measure has theadditional benefit of not requiring measurements of response time, soit can be used in situations where the response time measurement cannotbe obtained.

Youcan characterize each test by its saturation point measured in numberof users, the peak throughput level, the utilization levels of thevarious parts of the system at the measured saturation point, and theconfiguration factors used for the test. The response time at thispoint should always be good, so it does not need to be included.

Store the test results in S-PLUS as a data framefor modeling. S-PLUS can calculate a model that produces a set ofcoefficients relating the configuration factors to the saturationpoint. Each coefficient is used for one level of the factor; forexample, in an ideal world, the CPU coefficients might be 0.5 for twoCPUs, 1.0 for four CPUs, and 2.0 for eight CPUs (see Figure 3-9).When we used the model to predict results, the values predicted by thismodel were within a few user values of the measured results across theentire range. The residual errors were small, and we obtained a goodfit for the model. A similar formula for predicting the peak throughputcan also be obtained.

In the formula shown in Figure 3-9,a coefficient is produced for each value shown. To get the saturationpoint value in users for a particular combination, the coefficients forthat combination are multiplied, for example, Intercept ×=Timeshare ×6CPU ×=MediumThink.

Auseful aspect of this method is that the statistics package maintainsinformation on the variance of each coefficient, and when the formulais used to predict the value of a saturation point, the standarddeviation of that prediction is also calculated. The tests used tocalculate the model represent a sparse subset of the completeconfiguration space (i.e., to save time, not every possible combinationof configuration options was tested). So, if a group of relatedpredictions is calculated to have a large standard deviation, then moretests can be performed in the region of that group, and the model canbe regenerated with reduced variance.

Thestatisticians among you may be curious about the method used to producea multiplicative model equation, when the standard techniques work withadditive models. There are two ways to handle this in S-PLUS. One wayis to take logarithms of the data before calculating the additivemodel, then use exponents to regenerate the coefficients. A moresophisticated method is to use the generalized linear model with aPoisson link function, which sounds complicated but is actually easierto use. (It took me a while to work out that this was the right thingto do, since S-PLUS has hundreds of possible modeling options!). Thismodel has the property that the variance of the result is proportionalto its value, so the error in a 10-user prediction might be 1 user,while for a 200-user prediction, the error might be 20 users.

Looking for Trends

Take many short samples of utilization data (like the hourlysarmeasures) and plot them to produce a distribution curve or histogramover a longer period (perhaps weekly). That way, you count the numberof hours per week that utilization was in a certain range and, as timegoes on, you can compare the weekly distribution plots to see if thereis an increase in the number of hours per week at high utilizationlevels. Do this for CPU run queue size, for each disk’s service time,and for the page scan rate^[3].

^[3] A high scan rate indicates a memory shortage; see “Understanding vmstat and sar ” on page 320.

Further Reading

The Practical Performance Analystby Neil Gunther, McGraw-Hill, 1997, provides an easy-to-follow guide tothe parts of performance modeling and queuing theory that you reallyneed to know.

The Art of Computer Systems Performance Analysisby Raj Jain provides comprehensive coverage of techniques forexperimental design, measurement, simulation, and performance modeling.

Statistical Models In Sby John M. Chambers and Trevor J. Hastie is quite specific to theS-PLUS statistics package, but it gives an excellent insight into therange of techniques available in modern statistics and how thecombination of statistics and numerical modeling can be used to makesense of data.