sar in Linux

The sar command under Linux and Unix systems gives the system activity reports. As the system works, a record of activities is kept in certain counters in the kernel. The sar command gives statistics regarding the CPU, I/O, paging, devices, memory, swap space, network, run queue length and load average, interrupts and power management. At the very basic level, it gives the CPU utilization. For example,

$ sar 2 4
Linux 3.0.0-14-generic (hostname) 	Sunday 15 July 2012 	_i686_	(2 CPU)

06:04:43  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle  
06:04:45  IST     all      7.29      0.00      3.02      0.00      0.00     89.70
06:04:47  IST     all      9.73      0.00      3.74      2.00      0.00     84.54
06:04:49  IST     all      6.25      0.00      1.25      0.00      0.00     92.50
06:04:51  IST     all     12.72      0.00      2.99      0.00      0.00     84.29
Average:        all      9.00      0.00      2.75      0.50      0.00     87.75

The above command gives the CPU utilization at an interval of 2 seconds 4 (count) times. If we skip the count, the reports are given continuously. If we skip both the interval and the count, the sar command gives the CPU utilization for the current day.

SYSSTAT

sar is a part of the sysstat package for performance monitoring under Linux. The data collection for generating the reports is done by the sadc program. The first step is to enable the data collection in /etc/default/sysstat.

#
# Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat
# and /etc/cron.daily/sysstat files
#

# Should sadc collect system activity informations? Valid values
# are "true" and "false". Please do not put other values, they
# will be overwritten by debconf!
ENABLED="true"

After changing the ENABLED flag in /etc/default/sysstat, the data collection is started by the command,

$ sudo invoke-rc.d sysstat start 

sadc is called from the shell script, /usr/lib/sysstat/sa1, which in turn is started from the init script /etc/init.d/sysstat. The data is recorded in a binary file, /var/log/sysstat/sadd, where dd is the day of the month for the current day. This way, the data is recorded for each day of the month. Thus, using the relevant data files, sar gives a system administrator the facility to check the various system activity reports for the system's operation in the past.

File options

Using the -o option, sar writes the output data into a file, in addition to the display on the screen. Using the -f option, this file can be read later on get the reports on the screen.

$ sar 2 4 -o sar-data
Linux 3.0.0-14-generic (hostname) 	Monday 16 July 2012 	_i686_	(2 CPU)

10:47:45  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle  
10:47:47  IST     all      6.25      0.00      2.25      0.00      0.00     91.50
10:47:49  IST     all      6.02      0.00      2.26      0.00      0.00     91.73
10:47:51  IST     all      6.73      0.00      1.50      0.00      0.00     91.77
10:47:53  IST     all      6.28      0.00      2.26      0.00      0.00     91.46
Average:        all      6.32      0.00      2.07      0.00      0.00     91.61
$ file sar-data
sar-data: data
$ sar -f sar-data
Linux 3.0.0-14-generic (hostname) 	Monday 16 July 2012 	_i686_	(2 CPU)

10:47:45  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:47:47  IST     all      6.25      0.00      2.25      0.00      0.00     91.50
10:47:49  IST     all      6.02      0.00      2.26      0.00      0.00     91.73
10:47:51  IST     all      6.73      0.00      1.50      0.00      0.00     91.77
10:47:53  IST     all      6.28      0.00      2.26      0.00      0.00     91.46
Average:        all      6.32      0.00      2.07      0.00      0.00     91.61

By using -e [ hh:mm:ss ] in conjunction with -o or -f, it is possible to set an end time for a report. For example,

$ sar -f ./sa14 -e 13:00:00
Linux 3.0.0-14-generic (hostname) 	Saturday 14 July 2012 	_i686_	(2 CPU)

12:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle  
12:05:01  IST     all      3.82      0.00      0.96      0.17      0.00     95.05
12:15:01  IST     all      3.94      0.00      1.03      1.84      0.00     93.18
12:25:01  IST     all      3.83      0.00      0.95      0.09      0.00     95.12
12:35:01  IST     all      3.53      0.00      0.90      0.03      0.00     95.54
...
12:25:02  IST     all      5.98      0.00      1.49      5.43      0.00     87.10
12:35:01  IST     all      5.96      0.00      1.60      3.49      0.00     88.96
12:45:01  IST     all      5.58      0.00      1.39      1.92      0.00     91.11
12:55:01  IST     all      5.96      0.00      1.48      1.81      0.00     90.76
Average:        all      7.17      0.21      1.84      2.36      0.00     88.43

gives a report till 01:00 PM. If -e option is given without specifying any time, a default of 18:00:00 is taken and the report is given till 06:00 PM. Similarly, while reading the sar reports from a file with the -f option, the option -s [hh:mm:ss] can be used to set the start time of the report.

sardd reports

The sa2 command is a script which is called by the script, /etc/cron.daily/sysstat. The latter is scheduled by cron or anacron. sa2 writes daily reports in files named /var/log/sysstat/sardd, where dd is the day of the month.

CPU reports

The -P with the processor numbers(s) gives the CPU statistics. -P ALL gives the statistics for all processors.

$ sar -P 0,1 2 2
Linux 3.0.0-14-generic (hostname) 	Monday 16 July 2012 	_i686_	(2 CPU)  

12:11:45  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:11:47  IST       0      8.50      0.00      1.50      0.00      0.00     90.00
12:11:47  IST       1      3.50      0.00      1.50      0.00      0.00     95.00

12:11:47  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:11:49  IST       0      8.00      0.00      1.00      0.00      0.00     91.00
12:11:49  IST       1      6.00      0.00      2.00      0.00      0.00     92.00

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:          0      8.25      0.00      1.25      0.00      0.00     90.50
Average:          1      4.75      0.00      1.75      0.00      0.00     93.50
$ sar -P ALL 2 2
Linux 3.0.0-14-generic (hostname) 	Monday 16 July 2012 	_i686_	(2 CPU)

12:12:06  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle  
12:12:08  IST     all      7.02      0.00      2.01     48.12      0.00     42.86
12:12:08  IST       0      7.50      0.00      1.50      5.00      0.00     86.00
12:12:08  IST       1      6.50      0.00      2.50     91.00      0.00      0.00

12:12:08  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:12:10  IST     all      6.50      0.00      2.25     45.50      0.00     45.75
12:12:10  IST       0      7.00      0.00      2.50     46.00      0.00     44.50
12:12:10  IST       1      6.03      0.00      2.01     45.23      0.00     46.73

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      6.76      0.00      2.13     46.81      0.00     44.31
Average:          0      7.25      0.00      2.00     25.50      0.00     65.25
Average:          1      6.27      0.00      2.26     68.17      0.00     23.31

The CPU column gives the processor id, %user is the percentage time the processor is working in the user mode, %nice is the percent time processor is working in user mode with nice priority, %system is the percent time the processor is working in the kernel mode, %iowait is the percent time the processor is waiting for the I/O to finish, %steal is the percent time processor is waiting while the hypervisor was servicing another virtual processor and %idle is the percent time processor is idle.

Input/Output

The -b option gives the I/O statistics. For example,

$ sar -b 2 4
Linux 3.0.0-14-generic (hostname) 	Monday 16 July 2012 	_i686_	(2 CPU)  

09:04:02  IST       tps      rtps      wtps   bread/s   bwrtn/s
09:04:04  IST      3.50      0.00      3.50      0.00     44.00
09:04:06  IST      0.00      0.00      0.00      0.00      0.00
09:04:08  IST      4.00      0.00      4.00      0.00     52.00
09:04:10  IST      0.00      0.00      0.00      0.00      0.00
Average:         1.88      0.00      1.88      0.00     24.00

The tps column gives the transfers per second issued to the physical devices. The kernel can combine multiple I/O requests into a single transfer. rtps is the total number of read requests per second issued to the physical devices. Similarly, wtps is the total number of write requests per second issued to the physical devices. breads/s is the total number of blocks (1 block = 512 bytes, here) read per second from the devices. And bwrtn/s is the total number of blocks written to the devices per second.

Paging

The -B option gives the paging statistics.

$ sar -B 2 4
Linux 3.0.0-14-generic (hostname) 	Tuesday 17 July 2012 	_i686_	(2 CPU)

12:27:12  IST  pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff  
12:27:14  IST      0.00      0.00    511.00      0.00   1071.50      0.00      0.00      0.00      0.00
12:27:16  IST      0.00      0.00    505.50      0.00    571.00      0.00      0.00      0.00      0.00
12:27:18  IST      0.00     48.00    500.00      0.00    577.50      0.00      0.00      0.00      0.00
12:27:20  IST      0.00      0.00    499.00      0.00    568.50      0.00      0.00      0.00      0.00
Average:         0.00     12.00    503.88      0.00    697.12      0.00      0.00      0.00      0.00

The pgpgin/s statistic gives the number of kilobytes per second paged in from the hard disk. pgpgout/s is number of kilobytes per second paged out to the hard disk. The fault/s statistic is the major and minor page faults occurring per second in the system. The major page faults result in loading of relevant pages from the hard disk. The minor page faults can be resolved without any loading from the disk. The column majflt/s gives the major page faults per second. pgfree/s is the number of pages put on the free list per second by the system. pgscank/s is the total number of pages scanned per second by the kernel swap daemon, kswapd. pgscand/s is the total number of pages scanned directly per second and pgsteal/s is the total number of pages reclaimed from pagecache and swapcache per second. %vmeff is the ratio of pgsteal and pgscan expressed as a percentage and, if non-zero, is a measure of effectiveness of the virtual memory in the system.

Devices

The -d option gives the statistics for each block device. The device names are printed as dev<major-number>-<minor-number>. In the example below, it is dev8-0. Using the -p option (for pretty printing) along with the -d option gives the familiar sda as the device name.

$ sar -d 2 2
Linux 3.0.0-14-generic (hostname) 	Tuesday 17 July 2012 	_i686_	(2 CPU)

08:04:47  IST       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
08:04:49  IST    dev8-0      0.50    132.00      0.00    264.00      0.01     16.00     16.00      0.80

08:04:49  IST       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util  
08:04:51  IST    dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:       dev8-0      0.25     66.00      0.00    264.00      0.00     16.00     16.00      0.40
$ sar -d -p 2 2
Linux 3.0.0-14-generic (hostname) 	Tuesday 17 July 2012 	_i686_	(2 CPU)

08:05:13  IST       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
08:05:15  IST       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

08:05:15  IST       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
08:05:17  IST       sda      7.50    144.00     40.00     24.53      0.04      5.87      3.73      2.80

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:          sda      3.75     72.00     20.00     24.53      0.02      5.87      3.73      1.40

tps is the total number of transfers per second passed on to the device. rd_sec/s is the number of sectors (512 bytes) read per second. wr_sec/s is the number of sectors written per second. avgrq-sz is the average size of requests sent to the device. avgqu-sz is the average length of queue of requests sent to the device. await is the average total time in milliseconds for the requests to the device to get serviced. This includes the waiting time in the queue and the time spent in servicing the requests. svctm is the average time in milliseconds for the I/O requests to get serviced. %util is the percentage of CPU time spent in issuing I/O requests to the device.

Memory

The -R option gives the memory statistics.

$ sar -R 2 4
Linux 3.0.0-14-generic (hostname) 	Wednesday 18 July 2012 	_i686_	(2 CPU)

03:59:52  IST   frmpg/s   bufpg/s   campg/s
03:59:54  IST     -2.00      0.00      0.00
03:59:56  IST      0.00      0.00      0.00
03:59:58  IST      0.00      1.00     -1.00
04:00:00  IST      0.00      0.00      1.00
Average:        -0.50      0.25      0.00

frmpg/s is the number of memory pages freed by the system per second. A negative value, here, means that the pages have actually been allocated by the system. bufpg/s is the number of additional pages used as buffers by the system per second and campg/s is the number of additional memory pages cached by the system per second.

Swap space

The -S option gives the swap space utilization report.

$ sar -S 2 4
Linux 3.0.0-14-generic (hostname) 	Wednesday 18 July 2012 	_i686_	(2 CPU)  

04:37:46  IST kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
04:37:48  IST   2658020    338096     11.28     30140      8.91
04:37:50  IST   2658020    338096     11.28     30140      8.91
04:37:52  IST   2658020    338096     11.28     30140      8.91
04:37:54  IST   2658020    338096     11.28     30140      8.91
Average:      2658020    338096     11.28     30140      8.91

kbswpfree is the kilobytes of free swap space in the system. Similarly, kbswpused is the kilobytes of used swap space in the system. %swpused is the swap space used, expressed as a percentage of the total swap space. kbswpcad is the cached swapped memory in kilobytes. That is, this memory is in the swap area and is also cached by the system. If more memory is required by the system, this can be used straightaway without the need for swapping as it is already in the swap area. %swpcad is the swap memory cached expressed as a percentage of total swap memory used.

Network statistics

It is possible to get elaborate network statistics with sar. The syntax for the network option is,

    -n { keyword [,...] | ALL }

The keywords are DEV, EDEV, NFS, NFSD, SOCK, IP, EIP, ICMP, EICMP, TCP, ETCP, UDP, SOCK6, IP6, EIP6, ICMP6 and UDP6. One can get the network statistics based on a list of keywords or, by using the option -n ALL, for all the keywords. For example, the statistics for network interfaces is obtained using the keyword DEV.

$ sar -n DEV 2 2
Linux 3.0.0-17-generic (hostname) 	Wednesday 18 July 2012 	_i686_	(4 CPU)

11:39:43  IST     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
11:39:45  IST        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:39:45  IST      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:39:45  IST     wlan0    220.00    191.50    244.69     47.15      0.00      0.00      0.00

11:39:45  IST     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s  
11:39:47  IST        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:39:47  IST      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:39:47  IST     wlan0    249.00    224.00    242.78     47.98      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:        wlan0    234.50    207.75    243.74     47.56      0.00      0.00      0.00

The above table gives a row of output for each interface. The column rxpck/s gives total number of packets received per second. txpck/s gives the total number of packets transmitted per second. rxkB/s is the total number of kilobytes received per second and txkB/s is the total number of kilobytes transmitted per second. rxcmp/s and txcmp/s are the total number of compressed packets received and transmitted per second respectively. rxmcst/s is the total number of multicast packets received per second.

Next, let us look at the IP statistics. For IPV4 statistics to be collected, sadc should have been started with the -S SNMP option.

$ sar -n IP 2 2
Linux 2.6.38-8-virtual (hostname) 	07/18/2012 	_i686_	(1 CPU)

06:11:14 AM    irec/s  fwddgm/s    idel/s     orq/s   asmrq/s   asmok/s  fragok/s fragcrt/s  
06:11:16 AM      2.38      0.00      2.38      1.36      0.00      0.00      0.00      0.00
06:11:18 AM      2.78      0.00      2.78      2.02      0.00      0.00      0.00      0.00
Average:         2.61      0.00      2.61      1.74      0.00      0.00      0.00      0.00

irec/s is the total number of input datagrams received on the interfaces per second (ipInReceives). This includes datagrams received due to errors. fwddgm/s is the number of datagrams received per second for which this host was not the final destination and, therefore, an attempt was made to find a route and forward these datagrams to the final destination (ipForwDatagrams). idel/s is the number of input datagrams delivered per second to upstream protocols (ipInDelivers). orq/s is the number of datagrams received from upstream protocols per second for transmission (ipOutRequests). asmrq/s is the number of IP fragments received per second which need to be re-assembled (ipReasmReqds). asmok/s is the number of IP datagrams re-assembled successfully per second (ipReasmOKs). fragok/s is number of IP datagrams that have been fragmented successfully per second here (ipFragOKs). fragcrt/s is the number of IP datagram fragments generated per second as a consequence of fragmentation (ipFragCreates).

As another example, we look at the TCP statistics. For sadc to collect TCPv4 statistics, sadc should have been started with the option -S SNMP.

$ sar -n TCP 2 2
Linux 2.6.38-8-virtual (hostname) 	07/18/2012 	_i686_	(1 CPU)  

06:09:05 AM  active/s passive/s    iseg/s    oseg/s
06:09:07 AM      0.00      0.00      6.28      8.47
06:09:09 AM      0.00      1.25     19.05     20.30
Average:         0.00      0.65     12.94     14.64

active/s is the number of times TCP connections have made a direct transition from the CLOSED state to the SYN-SENT state per second (tcpActiveOpens). passive/s is the number of times TCP connections have made a direct transition from the LISTEN state to the SYN-RECEIVED state per second (tcpPassiveOpens). iseg/s is the total number of segments received per second including those received due to error (tcpInSegs). oseg/s is the total number of segments sent per second excluding the ones that contain octets for retransmission only (tcpOutSegs).

The last example for network statistics uses the keyword, SOCK. This gives statistics for sockets being used in the system.

$ sar -n SOCK 2 2
Linux 3.0.0-14-generic (hostname) 	Wednesday 18 July 2012 	_i686_	(2 CPU)

08:50:04  IST    totsck    tcpsck    udpsck    rawsck   ip-frag    tcp-tw
08:50:06  IST       717        20         8         0         0         0
08:50:08  IST       717        20         8         0         0         0
Average:          717        20         8         0         0         0

The first column gives the total sockets in the system. The next two columns give the TCP and UDP sockets which are in use. The last three columns give raw sockets in use, IP fragments in queue and TCP sockets in TIME_WAIT state respectively. The last five columns do not add up to total sockets. This should not be a worry since there are many more types of sockets in the system, which are not reported by sar.

Besides above examples, EDEV keyword gives statistics on errors in the network devices, EIP gives IPv4 network error reports, EICMP gives ICMPv4 error reports and ETCP gives TCPv4 error reports. Similar to IPv4, there are IPv6 keywords.

Run queue length and load average

With the -q option, the run queue length and load averages are obtained.

$ sar -q 2 2
Linux 3.0.0-14-generic (hostname) 	Thursday 19 July 2012 	_i686_	(2 CPU)  

10:15:05  IST   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:15:07  IST         1       446      0.30      0.15      0.09         1
10:15:09  IST         2       445      0.28      0.15      0.09         2
Average:            2       446      0.29      0.15      0.09         2

runq-sz is the number of tasks (processes and threads) on the run queue, waiting for the processor. plist-sz is the total number of currently live tasks. ldavg-1, ldavg-5 and ldavg-15 are the load averages for the last one, five and fifteen minutes respectively. blocked is the number of tasks blocked, waiting for the I/O to be completed.

Interrupts

The syntax for the option to get interrupt statistics is,

    -I {int, [,...] | SUM | ALL | XALL }

For data collection for interrupt statistics, sadc should have been started with the option, -S INT. A list of interrupt numbers needs to be passed with the -I option for generating the interrupt statistics report. The keyword SUM gives the total number of interrupts received per second. ALL gives a report on first sixteen interrupts whereas XALL gives the report on all interrupts received per second.

$ sar -I SUM 2 2
Linux 3.0.0-14-generic (hostname) 	Thursday 19 July 2012 	_i686_	(2 CPU)  

12:30:57  IST      INTR    intr/s
12:30:59  IST       sum   1016.50
12:31:01  IST       sum    964.50
Average:          sum    990.50

Power management

The -m option gives the power management statistics. For data collection for power management reports, it is required that sadc should be started with the option -S POWER. The syntax for the option parameter is,

    -m { keyword [, ...] | ALL }

The keywords are CPU, FAN, FREQ, IN, TEMP and USB. With the CPU keyword, instantaneous CPU clock frequency in MHz is reported. The FAN keyword reports the fan speed in rpm, the difference between the current fan speed and its low limit in rpm and the sensor device name. The FREQ keyword reports the weighted average CPU clock frequency in MHz. The IN keyword reports details about voltage inputs. It gives input voltage in volts, relative input voltage value between high and low limits, and the sensor device name. The TEMP keyword gives the report on devices temperature. It gives the device temperature in degrees Celcius, relative device temperature and the sensor device name. The USB keyword gives a report on all the USB devices plugged in to the system. It gives the root hub number of the USB device, the vendor id number, product id number, maximum power consumption of the device, the manufacturer's name and the product name.

$ sar -m USB 1 1
Linux 3.0.0-17-generic (hostname) 	Thursday 19 July 2012 	_i686_	(4 CPU)

01:25:41  IST     BUS  idvendor    idprod  maxpower                manufact                                         product
01:25:42  IST       1      8087        20         0                                                                        
01:25:42  IST       2      8087        20         0                                                                        
01:25:42  IST       1       c45      6409      1000    USB 2.0 Camera_FLV14                                  USB 2.0 Camera  
01:25:42  IST       1       489      e00f         0           Broadcom Corp                       Broadcom Bluetooth Device
01:25:42  IST       2       3f0      4817       196         Hewlett-Packard                               HP LaserJet P1007

Summary         BUS  idvendor    idprod  maxpower                manufact                                         product
Summary           1      8087        20         0                                                                        
Summary           2      8087        20         0                                                                        
Summary           1       c45      6409      1000    USB 2.0 Camera_FLV14                                  USB 2.0 Camera
Summary           1       489      e00f         0           Broadcom Corp                       Broadcom Bluetooth Device
Summary           2       3f0      4817       196         Hewlett-Packard                               HP LaserJet P1007