Beginner’s Guide to iostat

Ever wondered what’s going on “under the hood” of your Linux system when things feel a bit slow? While CPU and memory are often the first things we check, disk input/output (I/O) can be a major bottleneck. That’s where the iostat command comes in – a powerful yet often overlooked tool for understanding how your storage devices are performing.

Think of iostat as your system’s disk activity monitor. It provides valuable insights into how busy your disks are, how much data they’re reading and writing, and how efficiently they’re doing it. For those new to Linux, the output might look a bit intimidating at first, but with a few simple examples, you’ll be a disk detective in no time!

Why Should You Care About Disk I/O?

Imagine you’re trying to find a book in a massive library. If the librarian (your disk) is constantly busy fetching books for everyone else, or if the aisles are disorganized, it’s going to take you longer to get your book. Similarly, if your applications are constantly waiting for data from a slow or overworked disk, your entire system will feel sluggish.

Understanding disk I/O helps you:

Think of iostat as your system’s disk activity monitor. It provides valuable insights into how busy your disks are, how much data they’re reading and writing, and how efficiently they’re doing it. For those new to Linux, the output might look a bit intimidating at first, but with a few simple examples, you’ll be a disk detective in no time!

Why Should You Care About Disk I/O?

Imagine you’re trying to find a book in a massive library. If the librarian (your disk) is constantly busy fetching books for everyone else, or if the aisles are disorganized, it’s going to take you longer to get your book. Similarly, if your applications are constantly waiting for data from a slow or overworked disk, your entire system will feel sluggish.

Understanding disk I/O helps you:

• Identify performance bottlenecks: Is a slow disk causing your applications to crawl?
• Troubleshoot issues: Is a specific process hammering your disk?
• Capacity planning: Do you need a faster disk or more storage?

Getting Started with iostat

Before we dive into the examples, let’s make sure you have iostat installed. It’s usually part of the sysstat package. On most Debian/Ubuntu systems, you can install it with:

sudo apt-get install sysstat

On Fedora/RHEL/CentOS:

sudo yum install sysstat
# or for newer Fedora versions
sudo dnf install sysstat

Now, let’s run our first iostat command!

Basic iostat Output

Running iostat without any arguments gives you a snapshot of your system’s disk activity since the last reboot.

iostat

You’ll typically see two sections:

1. avg-cpu: This shows the average CPU utilization. While not directly disk-related, high iowait here can indicate that your CPU is waiting for disk operations to complete.
2. Device-specific statistics: This is where the magic happens! You’ll see a list of your storage devices (e.g., sda, sdb, nvme0n1) and various metrics for each.

Let’s break down some of the most important columns in the device-specific section:

• Device: The name of your storage device. sda typically refers to your first hard drive, sdb the second, and so on. dm-0, dm-1, etc., are often used for logical volumes or encrypted partitions.
• tps: Transfers per second. This tells you how many I/O requests are being sent to the device per second. A higher number means more activity.
• kB_read/s: Kilobytes read per second. The amount of data being read from the device.
• kB_wrtn/s: Kilobytes written per second. The amount of data being written to the device.
• kB_read: Total kilobytes read since boot.
• kB_wrtn: Total kilobytes written since boot.

Real-time Monitoring with iostat

The most useful way to use iostat for troubleshooting is to see live updates. You can specify an interval (in seconds) to continuously report statistics.

Let’s look at disk activity every 2 seconds:

iostat 2

This will print updated statistics every 2 seconds until you press Ctrl+C. This is incredibly helpful for seeing how your disk responds to various activities, like starting an application, copying files, or running a backup.

Focusing on Specific Disks

If you have multiple disks and only want to monitor a particular one, you can specify it:

iostat 2 sda

This will only show statistics for the sda device every 2 seconds.

Understanding iowait and %util

Two very important metrics to keep an eye on, especially when diagnosing performance issues, are %iowait (from avg-cpu) and %util (from device-specific stats).

• %iowait (in avg-cpu section): This percentage indicates the amount of time the CPU spends waiting for I/O operations to complete. A consistently high %iowait (e.g., above 20-30% for extended periods) suggests that your CPU is frequently idle because it’s waiting for your disk. This is a strong indicator of a disk bottleneck.
• %util (in device-specific section): This percentage tells you how busy the device is. A value close to 100% means the device is constantly busy and might be saturated. While high %util isn’t always bad (a fast SSD can handle 100% utilization without issue), if combined with high avgqu-sz (average queue size) or await (average wait time), it can indicate a bottleneck.

To see these metrics, we often use the -x option for extended statistics:

iostat -x 2

Now you’ll see additional columns like:

• %util: The percentage of time the device was busy.
• await: The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent in the queue and the time spent servicing them.
• svctm: The average service time (in milliseconds) for I/O requests issued to the device.
• avgqu-sz: The average queue length of requests that were issued to the device.

If you see a high %util along with high await and avgqu-sz, your disk is likely struggling to keep up with the demand.

Putting it All Together: A Scenario

Let’s say your Linux server is running a database, and users are complaining about slow queries. You log in and run iostat -x 2:

iostat -x 2

You observe the following:

• avg-cpu section: %iowait is consistently around 40-50%. This immediately tells you that the CPU is spending a lot of time waiting for disk operations.
• Device sda (your database disk):
  • %util is at 95-100%. The disk is constantly busy.
  • await is high, perhaps 50ms or more. Requests are taking a long time to be processed.
  • avgqu-sz is also high, indicating a backlog of requests.
  • kB_read/s and kB_wrtn/s show significant activity.

Conclusion: Your database queries are slow because the disk where the database is stored (sda) is overloaded. It’s struggling to read and write data fast enough, causing the CPU to wait and ultimately slowing down your application.

Possible Solutions:

• Optimize database queries: Reduce the amount of I/O required.
• Move database to a faster disk: Upgrade to an SSD or a RAID array.
• Add more RAM: Allow the database to cache more data in memory, reducing disk reads.

Conclusion

The iostat command is an indispensable tool in any Linux user’s or administrator’s arsenal. While it might seem complex at first, by focusing on a few key metrics like tps, kB_read/s, kB_wrtn/s, %iowait, and %util, you can quickly gain a clear picture of your disk’s health and performance.

So, the next time your Linux system feels sluggish, remember iostat. It might just help you solve your disk I/O mysteries!