Red Hat Linux Networking , System Administration (P29) ppsx

The sixyou’ll look at are listed here in alphabetical order: ■■ free— Reports the amount of free and used memory in the system ■■ iostat— Provides detailed CPU and I/O usage information

In this chapter, you learned how to back up and restore your file system Youlearned how to choose which files are important to back up and to choose abackup medium, a backup method, and a tape rotation schedule appropriatefor the needs of your situation You also learned how to use low-level archiv-ing tools such as tar and dump to produce archives and file system data and

to restore corrupted file system data from archives In addition, you learnedhow to configure and use AMANDA, an advanced archiving tool

Performance Monitoring

IN THIS CHAPTER

■■ System Performance Monitoring Tools

■■ Measuring Memory Usage

■■ Viewing Running Tasks

■■ Monitoring I/O Activity

■■ Using sar

This chapter describes some of the tools you can use to monitor the status andperformance of your Fedora Core or RHEL system Utilities like free, top,and ps provide basic information about the status of the system at givenpoints in time For ongoing monitoring, you would use tools like iostat,vmstat, and sar

System-Performance-Monitoring Tools

The first group of tools this chapter discusses enables you to take snapshots ofsystem performance at a given point in time You can use this data to createbaseline metrics of your system’s performance This historical data serves as aguide against which you measure the impact of changes you make You canuse a variety of tools, many more, in fact, than this chapter covers The sixyou’ll look at are listed here in alphabetical order:

■■ free— Reports the amount of free and used memory in the system

■■ iostat— Provides detailed CPU and I/O usage information

■■ sar— Collects, saves, or reports on a comprehensive list of systemactivity data

C H A P T E R

32

■■ slabtop— Reports kernel memory usage

■■ top— Displays a real-time list of running processes

■■ vmstat— Shows virtual memory and I/O system usageOne of the things you will notice is that each utility has some overlap withother utilities For example, free and vmstat both report on virtual memoryusage, although vmstat provides considerably more detail than does free.Likewise, vmstat and iostat can both provide I/O (input/output) usagedata; again, iostat’s I/O analysis is more complete than vmstat’s is Thefollowing sections disregard these areas of overlap and focus on what eachutility does best For instance, you won’t see any discussion of vmstat’s diskI/O-specific features, nor will you read much about iostat’s ability to report

on running processes (an area in which it overlaps with top)

Measuring Memory Usage

Even on systems that seem to have ample physical RAM, it is still a good idea

to know how much memory is in use and how much is available Excessivememory consumption, perhaps due to a memory leak in a running program,can slow a system down and eventually force a reboot to reclaim the “lost”memory At the highest level, you can use free command to show a quickreport of how much memory is in use and how much is free vmstat showsmore detail about memory usage, especially swap usage The slabtop com-mand shows you how the kernel itself is allocating memory

Memory Usage as Seen by Users and Processes

You can use two commands to obtain summary information about the system’smemory usage The free command shows information about the amount ofmemory that is used and unused, including both physical RAM and swapspace vmstat shows the same information in greater detail

Free’s syntax is:

free [-b|-k|-m] [-o] [-s secs] [-t]

Invoked without command line arguments, free’s output looks like the following:

$ free

total used free shared buffers cached Mem: 515800 500652 15148 0 0 255356 -/+ buffers/cache: 245296 270504

If you want the output to be displayed in bytes, rather than kilobytes, usethe -b option; use -m to display the output in megabytes; -k displays the out-put in kilobytes, the default If you’re math challenged, the -t option adds aline to the bottom of the output showing totals values.

The -o option disables the -/+ buffers/cached: line, which shows ments made to the used and free physical RAM These adjustments are neces-sary if you want to know how much RAM is actually in use and how muchRAM the kernel has set aside for its own use The kernel keeps a certainamount of RAM available for I/O and memory buffers to facilitate I/O Theamount of buffer memory varies over time as it is used and released From thepoint of view of the system as a whole, RAM used as buffer memory is always

adjust-“in use,” even if the kernel has not allocated it at a given point in time Fromthe kernel’s point of view, however, unused buffer memory is just that, unused(or free) Without the -o option, you see memory usage from the kernel’s point

of view With the -o option, you can visualize memory consumption from theview of the system as a whole In the free example just shown, just over 263

Mb (270,504 Kb) is “free” from the kernel’s point of view If you use the -ooption, you won’t see the amount of amount of RAM allocated as kernel buffermemory

The other columns of output show the amount of memory allocated asshared memory (System V IPC shared memory, to be precise), additional non-specific buffer memory, and the amount of cached data in memory The sharedmemory column should be disregarded because it is no longer used

The final option that might prove useful is the -s secs option, whichcauses free to redisplay its report every secs seconds The following exam-ple shows free’s output immediately before and during a kernel compilation:

$ free -s5 -m -o

total used free shared buffers cached Mem: 503 496 7 0 0 253 Swap: 1027 0 1027

total used free shared buffers cached Mem: 503 495 8 0 0 253 Swap: 1027 0 1027

total used free shared buffers cached Mem: 503 494 8 0 0 243 Swap: 1027 0 1027

total used free shared buffers cached Mem: 503 489 14 0 0 244 Swap: 1027 0 1027

This example used the -m option to display the output in megabytes, the -ooption to turn off the buffer adjustment, and the -s5 option to refresh the display

every five seconds The kernel compilation started between the first and secondupdates One of the features you’ll notice in the bold-faced section is that theamount of cached data fell when the kernel build process started Presumably, this occurred because data the kernel needed had to be read fromdisk, forcing a certain amount of cached data to be flushed.

vmstat digs deeper into memory usage than free and pays particularattention to virtual memory (swap) usage If your system is constantly swap-ping, disk I/O will slow to a crawl and the system will seem slow to respond

to user input vmstat makes it possible for you to detect the problem You canthen use top or one of the other utilities discussed in this chapter to identifywhat is causing the excessive swapping First, however, vmstat’s syntax,bearing in mind that this discussion ignores options not related to virtualmemory:

vmstat [-S k|K|m|M] [-a] [-n] [secs [cnt]]

vmstat [-S k|K|m|M] -m

vmstat [-S k|K|m|M] -s

To change the display unit, which defaults to bytes, use -S k for units of

1000 bytes, -S K for true kilobytes (1024 bytes), -S m for units of 1,000,000bytes, or -S M for true megabytes (1,048,576 bytes) The examples in the textuse -S K Certain vmstat reports can be refreshed every secs seconds and, ifcnt is specified, will refresh cnt times every secs seconds before vmstatterminates In its simplest usage, vmstat’s output looks like the following:

$ vmstat -S K

procs -memory - -swap -io system

cpu r b swpd f cpu ree buff cache si so bi bo in cs us sy id wa

2 0 852 16132 0 249232 0 0 33 26 10 82 94 2 4 0

This information shows only the average usage since the system wasbooted To get information about current usage, you must request a refreshingdisplay using secs and, if you multiple reports, cnt, as shown in the follow-ing example:

$ vmstat -S K 5 5

procs -memory - -swap -io system

cpu r b swpd f cpu ree buff cache si so bi bo in cs us sy id wa

3 0 852 4880 0 250120 0 0 33 26 15 135 93 3 4 0

2 0 852 14628 0 250308 0 0 0 30 1132 405 94 6 0 0

3 0 852 14168 0 250444 0 0 0 31 1131 418 93 7 0 0

4 0 852 6780 0 250528 0 0 0 35 1130 375 94 6 0 0

What information is shown? In the procs section, the r column shows thenumber of processes that are ready to run and waiting for their turn to run onthe CPU and the b column shows the number of processes that are blocked, orsleeping, and thus not ready to run In the first example, therefore, two processesare ready to run and waiting for CPU time and no processes are blocked.

The four columns under the memory heading, show the following information:

■■ swpd— The amount of virtual memory in use

■■ free— The amount of physical RAM not in use

■■ buff— The amount of physical RAM used as buffers

■■ cache— The amount of physical RAM used as cache

As you can see in the second example, created during a kernel compile, theamount of free and cache memory fluctuates constantly; the more active thesystem, the greater the fluctuation

If you specify -a, inact and active replace the buff and cache columnsunder the memory heading inact displays the amount of inactive memory

and active displays the amount of active memory Inactive memory is the

buffer memory the free command shows as free (unused) when buffer

adjustments are enabled; active memory is memory that is both allocated and in

use and maps to the used buffer memory reported by the free command Thefollowing vmstat example shows the effect of the -a option:

$ vmstat -S K -a 5 5

procs -memory - -swap -io system

cpu r b swpd f cpu ree inact active si so bi bo in cs us sy id wa

In the io section, the columns bi and bo show the number of disk blocks (inunits of 1024 bytes) read from and written to, respectively, the system’s blockdevices Under the system heading, in lists the number of interrupts receivedper second and cs shows the number of context switches per second Valuesunder the cpu heading, finally, show the disposition of CPU usage, with eachcolumn expressed as a percentage of total CPU time (due to rounding, the val-ues might not add to 100 percent) The specific columns are:

■■ us— The percentage of CPU time spent running user, or nonkernel, code

■■ sy— The percentage of time spent executing system, or kernel, code

■■ id— The percentage of CPU time that the CPU is idle

■■ wa— The percentage of CPU time spent waiting for I/O to complete

Examining Kernel Memory Usage

The memory usage information discussed so far examined memory from thepoint of view of the user or running processes You haven’t seen with anyamount of detail how the kernel itself is using memory The last vmstat option,-m, gives you a window into the kernel’s internal memory usage The -m option

causes vmstat to display kernel slab usage Slabs are caches of frequently used

kernel memory objects, such as inodes, directory entries, file pointers, and dom blocks of memory of specific sizes, such as 8192 bytes, 4096 bytes, and so

ran-on Rather than use vmstat to view slab usage, however, you should useslabtop, which does for slabs what the top command does for processes,namely, show slab usage in a real-time updated format Slabtop’s syntax is:

slabtop [-d secs] [-s sort] [-o]

-d secsspecifies the number of seconds to pause between updates -o tellsslabtopto display its output once and then exit The -s sort option sets thesort order, which defaults to the number of slab objects descending order, forthe displayed slabs to sort sort can be one of the values listed in Table 32-1

Table 32-1 slabtop Sorting Criteria

CRITERIA ORDER DESCRIPTION

a Ascending Sort by the number of active objects

b Ascending Sort by the number of objects per slab

c Descending Sort by cache size

l Descending Sort by the number of slabs

v Descending Sort by the number of active slabs

n Ascending Sort by the slab name

o Descending Sort by the number of objects (this the default sort

order)

p Descending Sort by the number of pages per slab

s Descending Sort by the object size

u Descending Sort by cache utilization

Figure 32-1 Viewing slabtop’s default output.

Invoked with no options, slabtop’s output resembles Figure 32-1

The slab cache listing, updated every three seconds by default, showsdetailed slab cache information The top five lines show summary informationfor the number of individual objects, the total number of slabs containingobjects, the number of slab caches, and slab size statistics The bottom portion

of the display shows the specifics for each type of slab cache sorted in ing order by the object type You can change the sort order at runtime by press-ing the key associated with the sort criteria that interests you (see Table 32-1)

descend-T I P The p sort option for sorting slabtop’s output by the number of pages per slab does not appear to function in slabtop version 3.2.3 However, you can view this information using the following sort invocation:

$ sort -k6,6 -nr < /proc/slabinfo | cut -f1 -d:

size-131072(DMA) 0 0 131072 1 32 size-131072 0 0 131072 1 32 size-65536(DMA) 0 0 65536 1 16 size-65536 4 4 65536 1 16 size-32768(DMA) 0 0 32768 1 8 size-32768 49 49 32768 1 8 size-16384(DMA) 0 0 16384 1 4 size-16384 3 3 16384 1 4 tcpv6_sock 1 5 1376 5 2 task_struct 120 120 1392 5 2

slabinfo - version anon_vma 3069 3213 32 119 1 arp_cache 2 20 192 20 1 as_arq 0 0 64 61 1 avc_node 12 600 52 75 1 bdev_cache 14 18 608 6 1 bio 287 287 96 41 1 biovec-1 293 452 16 226 1 biovec-16 260 260 192 20 1

Again, the sixth column shows the number of pages per slab.

If you run slabtop on a kernel that was compiled with the configurationoption CONFIG_DEBUG_SLAB enabled, you will see additional slab cache statis-tics The first line of the output will include (statistics) and the real-time displaywill show five additional columns:

■■ The maximum number of active objects in the slab

■■ The number of times objects have been allocated

■■ The number of times new pages have added to the cache (cache growth)

■■ The number of times unused pages have been removed from the cache(cache reaping)

■■ The number of errors allocating new pages to the cache Unless you run a debugging kernel and are actively working on the kernel,you won’t need this additional information Nevertheless, you will at leastknow how to produce this information if someone asks for it

Viewing Running Tasks

In many cases, you will be less concerned about how much memory a process

is using and more concerned about what processes are running, or perhapsmore likely, what processes are running out of control The canonical tools forviewing running processes are ps and top ps gives you a snapshot view ofthe currently active processes, and top gives you a real-time updated display

of running processes

Getting Started with ps

The implementation of ps that is used on Linux systems (from the procpssuite) is a classic example of a Linux command gone horribly wrong It has fartoo many options, a number of which are redundant On the other hand, itlacks built-in email functionality, so ps might yet be salvageable Seriously, ps

is a powerful tool for viewing the current process list Refer to the section titled

“Obtaining Process Information” in Chapter 28 for discussions about tional process management programs

addi-Tables 32-2 through 32-5 borrow the layout of ps’s syntax description fromits manual page and organize each group of options into tables based on theoptions’ purpose ps supports both Unix98 options, which are preceded by ahyphen (-), and BSD options, which lack the initial - Where the functionality

is identical or very similar, the BSD options have been omitted In some cases,apparently identical Unix98 and BSD are listed because the BSD option showsdifferent output from the similarly invoked Unix98 option

The options from this list that you’ll most likely use are -e to select allprocess, r to select only running processes, and -N to reverse the meaning ofthe selection criteria -N helps you express selection criteria for which there are

no command line options For example, if you want to see all processes except

those owned by the users bubba and root (see the -u, -U, and U options inTable 32-3), you might use the following command:

-e Selects all processes

T Selects all processes on the invoking terminal

r Selects only running processes

x Selects processes without controlling TTYs

By default, ps selects all processes with the same effective user ID (euid) as the

current user and shows the PID, TTY, accumulated CPU time, and the programname However, if your ps invocation includes BSD-style options (options thatlack an initial -) the display will also include the process status and show thecomplete command line used, as shown in the previous example If you replacethe BSD-style U option with -u, the output looks slightly different:

dis-Table 32-3 Process Selection

-C command Selects by the command name matching pattern command -G rgid | name Selects by real group ID (RGID) rgid or group name -p pid Selects by PID pid

p pid Selects by PID pid and displays the program executed -U ruid | name Selects by real user ID (RUID) ruid or user name

-u euid | name Selects by effective user ID (EUID) euid or user name

U name Selects processes for user name and the program executed

Table 32-4 Standard Output Formats

l Display long output format

s Displays output in signal format

v Displays output in virtual memory format

The options for selecting output formats make it much easier for you toview the process information that you want Here again, the distinctionbetween the standard Unix98-style options and BSD-style options rears itsugly head Notice in the following example the difference between the Unix98-joption, which displays the selected processes in jobs format, and the BSD joption, which shows the selected processes in job control format:

The difference or lack of difference between Unix98 and BSD ps options iseven more apparent with the -l and l options, both of which display infor-mation in so-called long format:

Table 32-5 Modifying Output Format

OPTION DESCRIPTION

C Uses raw CPU time for %CPU instead of a decaying average

c Shows the true command name

e Shows environment after the command

f Displays process hierarchy as ASCII art (forest)

h Suppresses header lines (repeats header lines in BSD personality) -H Shows process hierarchy (forest)

S Includes some dead child process data (as a sum with the parent) -w Displays output in wide format

w Displays output in wide format

The options for modifying the output format make it possible to do somevery interesting things with ps For example, to see a complete list of process

in a nicely formatted ASCII art tree, try the following command:

high-$ ps axfj

5048 5216 5048 5048 ? -1 S 500 0:00 \_ /bin/sh /usr/lib/fi

5216 5253 5048 5048 ? -1 S 500 0:00 | \_ /bin/sh /usr/li

5253 5258 5048 5048 ? -1 Sl 500 285:27 | \_ /usr/lib/fi

By way of explanation, the columns that address memory usage are:

■■ MAJFL— Shows the number of major faults, which occur when datathe CPU needs isn’t resident in L1 or L2 cache and must be retrievedfrom main menu

■■ TRS— Shows the text resident size, the amount of memory used by aprogram’s text segment, which contains initialized data

■■ DRS— Shows the disk resident size, the amount of physical RAM theprocess has consumed that is currently swapped to disk

■■ RSS— Shows the resident set size, the amount of physical RAM consumed that isn’t swapped to disk

■■ %MEM— Shows the percentage of total physical RAM the process consumes

psis feature-rich, embarrassingly so It probably deserves a chapter of itsown, but we have other utilities to cover Time spent with the manual page andexperimenting with ps, especially comparing the output of similar options, will

be rewarded with a knowledge of ps (and your system’s running processes) thatwill earn you serious geek points at the next Linux installfest you attend

Using top

Although ps is amply capable of showing you what processes are running at a

given point in time, top excels at showing how processes behave over time Its

user interface enables you to sort the output in a variety of ways In addition,

you can use top to send signals to running processes and change their ties If you have a set of top options you always use, you can set these in a con-figuration file that top reads when it starts.

priori-To get started, top’s basic syntax is:

top [-i] [-s] [-S] [-u user] [-d secs] [-n count] [-p pid]

See the top manual page for a complete list of all the command line options.The -d secs option sets the delay in seconds between each top update (thedefault is 3 seconds) top will update continuously in real time unless youspecify -n count, which limits top to count updates before it exits If youdon’t want to see idle processes in the listing, specify the i option The -soption starts top in secure mode This option is primarily used to prevent straykeystrokes from harming a running system It only makes sense to use it whenstarting top as root because mortal users can affect only their own processes Ifyou want to see the CPU time consumed by all processes and their children,specify -S to see the time expressed as a cumulative sum If you are interested

in only the processes owned by a specific user, use the -u user option To itor a specific process or set of processes, use the -p pid option To monitormultiples processes, you can specify multiple pids in a comma-separated list.Figure 32-2 shows the default top display when no options are specified

mon-Figure 32-2 The default top display.