pidstat

The best way to use pidstat is to pass a list of process ids with the -p parameter. The -u option reports CPU utilization. The -l parameter displays the process command name and its arguments. And we can optionally put time interval in seconds for the interval between reports and again, optionally, a count for number of reports.

$ pidstat -p 3947,3998  -l -u 2 3
Linux 3.0.0-14-generic (hostname) 	Wednesday 27 June 2012 	_i686_	(2 CPU)

02:56:34  IST       PID    %usr %system  %guest    %CPU   CPU  Command
02:56:36  IST      3947    5.47    0.50    0.00    5.97     1  /usr/lib/firefox-11.0/firefox 
02:56:36  IST      3998    0.00    1.00    0.00    1.00     0  /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja 

02:56:36  IST       PID    %usr %system  %guest    %CPU   CPU  Command
02:56:38  IST      3947    3.50    0.50    0.00    4.00     0  /usr/lib/firefox-11.0/firefox 
02:56:38  IST      3998    0.50    0.50    0.00    1.00     1  /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja  

02:56:38  IST       PID    %usr %system  %guest    %CPU   CPU  Command
02:56:40  IST      3947    3.00    0.50    0.00    3.50     1  /usr/lib/firefox-11.0/firefox 
02:56:40  IST      3998    0.50    1.00    0.00    1.50     0  /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja 

Average:          PID    %usr %system  %guest    %CPU   CPU  Command
Average:         3947    3.99    0.50    0.00    4.49     -  /usr/lib/firefox-11.0/firefox 
Average:         3998    0.33    0.83    0.00    1.16     -  /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja 

Instead of saying pidstat -p 3947,3998, we could have given the command as pidstat -C "firefox" to find processes with command name having the string "firefox". The string parameter can be a regular expression. But, while pidstat finds the processes with command name containing the string, it skips the processes with zero statistics values. Somehow, specifying the process ids seems to facilitate a more precise communication with the system.

The %usr column gives the percentage of CPU working in the user mode for the process. The %system column gives the percentage of CPU working in the kernel mode for the process. The %guest column gives percentage of the CPU time for the process running a virtual processor in a virtual machine. The %CPU column gives the total percentage of the CPU time. Had -I option been used, this would have been divided by the number of processors in the system. The CPU column tells which CPU has been assigned to the process. The last column gives the command and the parameters used.

The pidstat command is a part of the sysstat package for performance monitoring under Linux.

I/O Statistics

The -d option reports the I/O statistics for the concerned processes. For example,

$ pidstat -p 2093,2145 -l -I -d
Linux 3.0.0-14-generic (hostname) 	Wednesday 27 June 2012 	_i686_	(2 CPU)

09:27:47  IST       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
09:27:47  IST      2093     14.77     35.66      8.37  /usr/lib/firefox-11.0/firefox 
09:27:47  IST      2145      2.75      0.00      0.00  /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja  

The kB_rd/s column gives the kilobytes per second read by the process. Similarly, kB_wr/s column gives the kilobytes per second written by the process. The next column, kB_ccwr/s gives the kilobytes/sec writing which has been canceled by this process. Suppose some other process has written to the pagecache, which has been overwritten by this process. Thus the earlier I/O need not be done now and has been canceled, thanks to this process.

Page faults and memory utilization

The -r option gives the page fault and memory utilization statistics. For example,

$ pidstat -p 1985,2232 -l -r
Linux 3.0.0-14-generic (hostname) 	Thursday 28 June 2012 	_i686_	(2 CPU)

04:58:20  IST       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
04:58:20  IST      1985    159.51      0.61  643916 122204  12.73  /usr/lib/firefox-11.0/firefox 
04:58:20  IST      2232     12.83      0.17  170216  19200   2.00  /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja  

The minflt/s column gives the minor or soft page faults per second encountered by the process. Minor page faults occur when the concerned page is loaded in the memory but is not marked as loaded in the memory management unit as loaded. The minor page faults are not serious. The next column, majflt/s, gives the major page faults per second for the process. These require loading a memory page from the hard disk into the pagecache. A high value for major fault per second is definitely a cause for concern and the user should seriously think about increasing the main memory in the system. In the next column, VSZ, the virtual memory size for the process in kilobytes is given. And the next column, RSS, is the resident set size, the physical memory used by the process in kilobytes. Finally, the column, %MEM gives the physical memory used by the process as a percentage of the total available physical memory.

If while debugging a program, it is suspected to be having memory leaks, it can be checked using the pidstat command. For example,

$ pidstat -p 3245 -l -r 2
Linux 3.0.0-14-generic (hostname) 	Thursday 28 June 2012 	_i686_	(2 CPU)

06:36:50  IST       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
06:36:52  IST      3245      1.00      0.00    3804    244   0.03  ./buggy_program  
06:36:54  IST      3245      1.00      0.00    4068    244   0.03  ./buggy_program 
06:36:56  IST      3245      1.00      0.00    4332    244   0.03  ./buggy_program 
06:36:58  IST      3245      1.00      0.00    4596    244   0.03  ./buggy_program 
06:37:00  IST      3245      1.00      0.00    4860    244   0.03  ./buggy_program 
06:37:02  IST      3245      1.00      0.00    5124    244   0.03  ./buggy_program 
06:37:04  IST      3245      1.00      0.00    5388    244   0.03  ./buggy_program 
06:37:06  IST      3245      1.00      0.00    5652    244   0.03  ./buggy_program 
06:37:08  IST      3245      1.00      0.00    5916    244   0.03  ./buggy_program 
06:37:10  IST      3245      1.00      0.00    6180    244   0.03  ./buggy_program 
06:37:12  IST      3245      1.00      0.00    6444    244   0.03  ./buggy_program 
06:37:14  IST      3245      1.00      0.00    6708    504   0.05  ./buggy_program 
06:37:16  IST      3245      1.00      0.00    6972    504   0.05  ./buggy_program 
06:37:18  IST      3245      1.00      0.00    7236    504   0.05  ./buggy_program 
....

As the virtual memory size, VSZ, is increasing with time, the above process most likely has one or more memory leaks and the corresponding program should be checked for memory leaks.

Stack details

The -s option reports the stack utilization. The StkSize column gives the size of stack reserved for the process in kilobytes. All of it might not have been used. The StkRef column gives the amount of memory in kilobytes used as stack by the process. For example,

$ pidstat -p 2043,2194 -l -s
Linux 3.0.0-14-generic (hostname) 	Thursday 28 June 2012 	_i686_	(2 CPU)

12:16:25  IST       PID StkSize  StkRef  Command
12:16:25  IST      2043    224     140  /usr/lib/firefox-11.0/firefox 
12:16:25  IST      2194    136      40  /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja  

Thread display

The statistics for threads are obtained by using the -t option. For example

$ pidstat -p 2043 -l -t
Linux 3.0.0-14-generic (hostname) 	Thursday 28 June 2012 	_i686_	(2 CPU)

12:22:53  IST      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
12:22:53  IST      2043         -   16.18    1.11    0.00   17.29     1  /usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2043   14.96    0.82    0.00   15.79     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2049    0.02    0.02    0.00    0.03     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2051    0.61    0.02    0.00    0.63     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2052    0.19    0.22    0.00    0.41     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2055    0.03    0.03    0.00    0.06     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2057    0.00    0.00    0.00    0.00     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2059    0.00    0.00    0.00    0.00     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2061    0.02    0.00    0.00    0.02     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2062    0.00    0.00    0.00    0.01     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2077    0.01    0.00    0.00    0.02     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2100    0.01    0.01    0.00    0.01     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2101    0.02    0.00    0.00    0.02     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2102    0.00    0.00    0.00    0.00     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2109    0.00    0.00    0.00    0.00     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2111    0.01    0.01    0.00    0.01     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2117    0.00    0.00    0.00    0.00     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2118    0.22    0.03    0.00    0.25     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2127    0.00    0.00    0.00    0.01     0  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2193    0.00    0.00    0.00    0.00     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      2269    0.00    0.00    0.00    0.00     1  |__/usr/lib/firefox-11.0/firefox 
12:22:53  IST         -      3682    0.00    0.00    0.00    0.00     0  |__/usr/lib/firefox-11.0/firefox 

TGID and TID are the thread group id and thread id respectively.

Task switching

Task switching report is obtained by using the -w option. The cswch/s column gives the number of voluntary context switches per second. A voluntary context switch happens when the process or thread is not able to progress as it needs a resource which is not available and, hence, it relinquishes the processor. The nvcswch/s column gives the total number of involuntary context switches per second. An involuntary context switch happens when the process or thread is forced to relinquish the processor by the kernel as it has used the processor for its time slice. For example,

$ pidstat -p 1991 -w -l
Linux 3.0.0-14-generic (hostname) 	Thursday 28 June 2012 	_i686_	(2 CPU)  

03:39:57  IST       PID   cswch/s nvcswch/s  Command
03:39:57  IST      1991     96.15     17.14  /usr/lib/firefox-11.0/firefox