Linux CPU Performance Measurements with mpstat

The Linux tool mpstat provides information about the utilization of the individual CPUs of a system. For multi-core CPUs, each core is displayed as a CPU. Hyperthreaded CPUs display two CPUs per physical CPU or physical core. For most Linux distributions, mpstat is included in the sysstat package (see eg Debian http://packages.debian.org/sysstat ).

Basics of mpstat

mpstat is typically called with the option mpstat -P ALL. This will list all CPUs individually. A call to mpstat -P ALLshows the averages since the last boot:

wfischer @ ubuntu-1204: ~ $ sudo mpstat -P ALL
Linux 3.2.0-93-generic (ubuntu-1204) 11/11/2015 _x86_64_ (4 CPU)

09:38:16 AM CPU% usr% nice% sys% iowait% irq% soft% steal% guest% idle
09:38:16 AM all 1.94 0.09 0.57 6.95 0.00 0.04 0.00 0.00 90.42
09:38:16 AM 0 1.01 0.20 0.69 3.77 0.00 0.08 0.00 0.00 94.27
09:38:16 AM 1 3.70 0.08 0.67 10.18 0.00 0.07 0.00 0.00 85.30
09:38:16 AM 2 1.72 0.05 0.50 10.28 0.00 0.00 0.00 0.00 87.45
09:38:16 AM 3 1.32 0.04 0.41 3.57 0.00 0.00 0.00 0.00 94.66
wfischer @ ubuntu-1204: ~ $

For analysis over a period of time two further parameters can be given. At mpstat -P ALL 1 60every second, the current value is read out (and this 60 times – ie 1 minute). The last entry then shows the average values over the measurement period. You can mpstat -P ALL 1 60easily write the output to a file:

wfischer @ ubuntu-1204: ~ $ sudo mpstat -P ALL 1 60> mpstatanalyse20151111-1.txt
wfischer @ ubuntu-1204: ~ $

Here is an excerpt from this file:

Linux 3.2.0-93-generic (ubuntu-1204) 11/11/2015 _x86_64_ (4 CPU)

09:35:57 AM CPU% usr% nice% sys% iowait% irq% soft% steal% guest% idle
09:35:58 AM all 0.00 0.00 0.00 15.19 0.00 0.00 0.00 0.00 84.81
09:35:58 AM 0 0.00 0.00 0.00 18.37 0.00 0.00 0.00 0.00 81.63
09:35:58 AM 1 0.00 0.00 0.99 29.70 0.00 0.99 0.00 0.00 68.32
09:35:58 AM 2 0.00 0.00 1.00 11.00 0.00 0.00 0.00 0.00 88.00
09:35:58 AM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

[...]

09:36:56 AM CPU% usr% nice% sys% iowait% irq% soft% steal% guest% idle
09:36:57 AM all 3.26 0.00 1.50 0.75 0.00 0.00 0.00 0.00 94.49
09:36:57 AM 0 6.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 93.94
09:36:57 AM 1 0.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 98.00
09:36:57 AM 2 4.95 0.00 5.94 1.98 0.00 0.00 0.00 0.00 87.13
09:36:57 AM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

Average: CPU% usr% nice% sys% iowait% irq% soft% steal% guest% idle
Average: all 0.48 0.00 0.70 15.71 0.00 0.04 0.00 0.00 83.07
Average: 0 0.66 0.00 1.19 16.44 0.00 0.12 0.00 0.00 81.60
Average: 1 0.29 0.00 0.44 28.94 0.00 0.07 0.00 0.00 70.27
Average: 2 0.53 0.00 0.65 15.27 0.00 0.00 0.00 0.00 83.54
Average: 3 0.42 0.00 0.50 2.27 0.00 0.00 0.00 0.00 96.81

Meaning of the individual values

(Source man mpstat):

• CPU : Processor number. The keyword all statistics are calculated as averages among all processors.
• % usr : Show the percentage of CPU utilization that occurred while executing at the user level (application).
• % nice : Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
• % sys : Show the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time-consuming servicing hardware and software interrupts.
• % iowait : The CPU or CPUs were idle during which the system had an outstanding disk I / O request.
• % irq : Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.
• % soft : Show the percentage of time spent by the CPU or CPUs to service software interrupts.
• % steal : Show the percentage of time spent waiting for the CPU or CPUs while the hypervisor is servicing another virtual processor.
• % guest : Show the percentage of time used by the CPU or CPUs to run a virtual processor.
• % idle : The CPU or CPUs were idle and the system did not have an outstanding disk I / O request.

analysis

Utilization of the individual CPUs or cores

For a system with several CPUs or multiple cores, it makes sense to evenly load them as much as possible. If the average values of mpstat -P ALLone CPU show a very high% user share, but other CPUs with a very low% user share, optimizing the configuration of the applications may increase the overall performance of the system. A database that uses only a single thread or process can only use one core.

Analysis period

Since one mpstat -P ALLprovides the average values since the last boot process (and thus also times with low load – at night and the weekend – includes), for example, a ten-minute measurement can be mpstat -P ALL 1 600 > /root/mpstatanalyse.txtcarried out at peak times means . This also gives averages for those ten minutes at the end of the output.