Understanding 'Percentage Waiting for I/O'

Various operating systems have monitoring tools that report values for Percentage Waiting for I/O. Whilst this is not a new metric, there is often confusion surrounding the meaning and interpretation of this statistic. This article explains why Percentage Waiting for I/O is not a useful metric when considering the performance of the disk I/O subsystem.

Example
The following example shows the output of the sar -u command under Solaris 9:

# sar -u 60 5

SunOS sun-test2 5.9 Generic_117171-14 sun4u 06/03/2005

Percentage Waiting for I/O is given by the %wio column.

Meaning
A sample of what each CPU is doing is taken every clock tick. CPU usage is divided into four categories. Each one represents, after each clock tick, the average time the CPU has spent running processes in user, system (kernel), idle, and so-called "waiting for I/O" modes.

The following diagram explains the logic for determining the mode of operation for each CPU for each clock tick:

A CPU is idle when there is no thread running on it. As can be seen from the diagram above the CPU is also idle when described as waiting for I/O. In effect, Percentage Waiting for I/O is simply the component of idle time where the last thread that ran is waiting for an I/O to complete. However, large systems may have I/O subsystems consisting of many tens or hundreds of logical and physical volumes. Clearly a simple metric like Percentage Waiting for I/O is insufficient to provide any useful understanding of the source of any I/O bottlenecks.

Conclusions
At best, Percentage Waiting for I/O provides a vague indication of the amount of time that a system has outstanding I/Os, when the CPU is idle. There are many far superior methods of determining the I/O profile of a system (for example, the iostat tool available on most UNIX operating systems). In fact, Solaris 10 now always reports zero values for Percentage Waiting for I/O. This has removed the possibility of people misinterpreting Percentage Waiting for I/O, whilst ensuring that any scripts that are dependent on the output of commands such as sar -u do not break.