Red Hat Linux 7 Unleashed

Red Hat Linux 7 Unleashed

By William Ball

Analyzing Performance

Performance analysis is the process of identifying performance bottlenecks in your system. It involves a number of steps. The first step is to look at the big picture: Is the problem CPU- or I/O-related? If it is a CPU problem, what is the load average? You should probably check what processes are running and what is causing the problem. If it is an I/O problem, is it paging or normal disk I/O? If it is paging, increasing memory might help. You can also try to isolate the program or the user causing the problem. If it is a disk problem, is the disk activity balanced? If you have only one disk, you might want to install a second.

The next section looks at several tools that you can use to determine the answers to the preceding questions.

Determining CPU Usage with vmstat

CPU usage is the first test on the list. There are many different ways to obtain a snapshot of the current CPU usage. The one I focus on here is vmstat. The vmstat command gives you several pieces of data, including CPU usage. The syntax for the command is

vmstat interval [count]

interval is the number of seconds between reports, and count is the total number of reports to give. If the count is not included, vmstat will run continuously until you stop it with Ctrl+C or kill the process.

Here is an example of the output from vmstat:

[scooby@cartoons]$ vmstat 5 5
 procs                  memory    swap        io    system         cpu
 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id
 0 0 0  1104  1412 10032 36228   0   0   10    8   31   15   7   4  24
 0 0 0  1104  1736 10032 36228   0   0    0    3  111   18   1   1  99
 0 0 0  1104  1816 10032 36228   0   0    0    1  115   23   2   2  96
 0 1 0  1104  1148 10096 36268   8   0    7    4  191  141   4   6  91
 0 0 0  1104  1868  9812 35676   6   0    2   10  148   39  25   4  70

The first line of the CPU Usage report displays the average values for each statistic since boot time. It should be ignored. For determining CPU used, you are interested in the last three columns, as indicated by the cpu heading. The column names are us, sy, and id (see Table 21.2).

Table 21.2. vmstat Fields For CPU Usage

CPU Description
us Percentage of CPU cycles spent on performing user tasks.
sy Percentage of CPU cycles spent as system tasks. These tasks include waiting on I/O, performing general operating system functions, and so on.
id Percentage of CPU cycles not used. This is the amount of time the system was idle.

A high CPU time (or low idle time) is not necessarily indicative of an overall CPU problem. It could be that a number of batch jobs running just need to be rearranged. To determine whether the system actually has a CPU problem, it is important to monitor the CPU percentages for a significant period of time. If the percentages are high over an extended time, then the system definitely has a problem.

Next, look at a different section of the vmstat output. If the problem is not CPU-related, check whether it is a problem with paging or normal disk I/O. To determine whether it is a memory problem, look at the headings memory and swap:

[scooby@cartoons]$ vmstat 5 5
 procs                  memory    swap        io    system         cpu
 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id
 1 0 0  1096  1848  4580 37524   0   0    9    8    8   17   7   3  29
 1 0 0  1096  1424  4580 37980   0   0   92   10  125   24  94   4   3
 2 0 0  1096   864  4536 38408   0   0  112   31  146   42  93   2   5
 2 0 0  1096   732  4360 38480  10   0   98    7  146   48  97   3   1

Included in the vmstat output is a "snapshot" table detailing the status of your system's memory (see Table 21.3).

Table 21.3. vmstat Fields for Memory and Swap Usage

Memory & Swap Description
swpd The amount of virtual memory used (KB)
free The amount of idle memory (KB)
buff The amount of memory used as buffers (KB)
cache The amount of memory left in the cache (KB)
si The amount of memory swapped in from disk (KB/s)
so The amount of memory swapped to disk (KB/s)

The most important of these fields is the swap in column. This column shows paging that has previously been swapped out, even if it was done before the vmstat command was issued.

The io section is used to determine whether the problem is with blocks sent in or out of the device:

[scooby@cartoons]$ vmstat 5 5
 procs                  memory    swap        io    system         cpu
 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id
 1 0 0  1096  1848  4580 37524   0   0    9    8    8   17   7   3  29
 1 0 0  1096  1424  4580 37980   0   0   92   10  125   24  94   4   3
 2 0 0  1096   864  4536 38408   0   0  112   31  146   42  93   2   5
 2 0 0  1096   732  4360 38480  10   0   98    7  146   48  97   3   1

The io section is described in Table 21.4.

Table 21.4. vmstat Fields for IO

IO Description
bi The blocks sent to a block device (blocks/s)
bo The blocks received from a block device (blocks/s)
cs The number of context switches per second

These fields run from several to several hundred (maybe even several thousand). If you are having a lot of in and out block transfers, the problem is probably here. Keep in mind, however, that a single reading is not indicative of the system as a whole; it's just a snapshot of the system at that time. Processes can exist in three states: runtime, uninterrupted sleep, and swapped out (see Table 21.5).

Table 21.5. vmstat Fields for Processor Usage

Processes Description
r The number of processes waiting for runtime
b The number of processes in uninterrupted sleep
w The number of processes swapped out but otherwise able to run

The number of processes waiting for runtime is a good indication that there is a problem. The more processes waiting, the slower the system. More than likely, you won't look at vmstat unless you already know there is a bottleneck somewhere, so the r field doesn't give you much vital information.

Using top to Identify Problems

The top command provides another tool for identifying problems with a Linux system. The top command displays the top CPU processes. More specifically, top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system and can provide an interactive interface for manipulating processes. The default is to update every five seconds. The following is an example of the output from top:

  1:36am  up 16 days,  7:50,  3 users,  load average: 1.41, 1.44, 1.21
60 processes: 58 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: 89.0% user,  8.5% system, 92.4% nice,  3.9% idle
Mem:  63420K av, 62892K used,   528K free, 32756K shrd,  6828K buff
Swap: 33228K av,  1096K used, 32132K free               38052K cached
PID USER     PRI  NI  SIZE  RSS SHARE STATE  LIB %CPU %MEM   TIME COMMAND

Table 21.6 explains the values output when using the top command.

Table 21.6. top Fields

Field Description
up The time the system has been up and the three load averages for the system. The load averages are the average number of processes ready to run during the last 1, 5, and 15 minutes. This line is just like the output of uptime.
processes The total number of processes running at the time of the last update. This is also broken down into the number of tasks that are running, sleeping, stopped, and zombied.
CPU states The percentage of CPU time in user mode, system mode, niced tasks, and idle. (Niced tasks are only those whose nice value is negative.) Time spent in niced tasks is also counted in system and user time, so the total is more than 100 percent.
Mem Statistics on memory usage, including total available memory, free memory, used memory, shared memory, and memory used for buffers.
Swap Statistics on swap space, including total swap space, available swap space, and used swap space. This and Mem are just like the output of free.
PID The process ID of each task.
USER The username of the task's owner.
PRI The priority of the task.
NI The nice value of the task. Negative nice values are lower priority.
SIZE The size of the task's code plus data, plus stack space, in kilobytes.
RSS The total amount of physical memory used by the task, in kilobytes.
SHARE The amount of shared memory used by the task.
STATE The state of the task, either S for sleeping, D for uninterrupted sleep, R for running, Z for zombied, or T for stopped or traced.
TIME Total CPU time the task has used since it started. If cumulative mode is on, this also includes the CPU time used by the process's children that have died. You can set cumulative mode with the S command-line option or toggle it with the interactive command S.
%CPU The task's share of the CPU time since the last screen update, expressed as a percentage of total CPU time.
%MEM The task's share of the physical memory.
COMMAND The task's command name, which is truncated if tasks have only the name of the program in parentheses (for example, (getty)).

As you can probably tell from the server used to obtain the previously displayed top data, there are no current bottlenecks in the system.

Using free to Show Available Memory

free is another good command for showing the amount of used and free memory:

[scooby@cartoons]$ free
             total       used       free     shared    buffers     cached
Mem:         63420      61668       1752      23676      13360      32084
-/+ buffers:            16224      47196
Swap:        33228       1096      32132

Table 21.7 describes the values returned when using the free command.

Table 21.7. free Fields

Field Description
Mem Shows the physical memory.
total Shows the amount of physical memory not used by the kernel, which is usually about a megabyte.
used Shows the amount of memory used.
free Shows the amount of free memory.
shared Shows the amount of memory shared by several processes.
buffers Shows the current size of the disk buffer cache.
cached Shows how much memory has been cached off to disk.
Swap Shows similar information for the swapped spaces. If this line is all zeros, your swap space is not activated.

To activate a swap space, use the swapon command. The swapon command tells the kernel that the swap space can be used. The location of the swap space is given as the argument passed to the command. The following example shows starting a temporary swap file:

[root@cartoons]# swapon /temporary_swap partition

To automatically use swap spaces, list them in the /etc/fstab file. The following example lists two swap files for /etc/fstab:

/dev/hda8 none swap sw 0 0
/swapfile none swap sw 0 0

To remove a swap space, use the swapoff command. Usually, this is necessary only when using a temporary swap space.

CAUTION

If swap space is removed, the system will attempt to move any swapped pages into other swap space or to physical memory. If there isn't enough space, the system will freak out but will eventually come back (this is commonly called thrashing). The system will be unavailable during the time it is trying to figure out what to do with these extra pages. Also keep in mind that, depending upon how much real memory you have, the system may never figure out what to do with these pages.

Using renice to Alter Process Priorities

The renice command is used to alter the priority of running processes.

By default in Red Hat Linux, the nice value is 0. The range of this is –20 to 20. The lower the value, the faster the process runs. The following example shows how you display the nice value by using the nice command. My shell is running at the default value of 0. To check this another way, I issue the ps -l command. The NI column shows the nice value:

[scooby@scooby]$ nice
0
[scooby@cartoons]$ ps -l
 FLAGS   UID   PID  PPID PRI  NI   SIZE  RSS WCHAN      STA TTY TIME COMMAND
   100   759  3138  3137   0   0   1172  636 force_sig  S   p0  0:00 -bash
100000   759  3307  3138  12   0    956  336            R   p0  0:00 ps -l

I can change the nice value by using the renice command. The syntax of the command follows:

renice priority [[-p] pid ...] [[-g] pgrp ...] [[-u] user ...]

In the following example, the shell's nice value is changed to a value of 5. This means that any process with a lower value will have priority on the system:

[scooby@cartoons]$ renice 5 3138
3138: old priority 0, new priority 5
[scooby@cartoons]$ nice
5
[scooby@cartoons]$ ps -l
 FLAGS   UID   PID  PPID PRI  NI   SIZE  RSS WCHAN      STA TTY TIME COMMAND
   100   759  3138  3137   5   5   1172  636 force_sig  S N p0  0:00 -bash
100000   759  3319  3138  14   5    968  368            R N p0  0:00 ps -l

The owner of the process and root have the ability to change the nice value to a higher value. Unfortunately, the reverse is not true:

[scooby@cartoons]$ renice -5 3138
renice: 3138: setpriority: Permission denied

Only root has the ability to lower a nice value. This means that even though I set my shell to a nice value of 5, I cannot lower it even to the default value.

The renice command is a wonderful way of increasing the apparent speed of the system for certain processes. This is a trade-off, however, because the processes that are raised will now run slower.

One other command that may help as you are trying to troubleshoot your system is the lsof command. The lsof command lists open files. The output includes information about the type of file, the device using the file, its node, and its name.

For more info, see the man pages for these commands.

Share ThisShare This

Informit Network