Part I · How it works
4.4 IOPS, throughput and latency: how fast is a disk?
alsoiopsdisk speeddisk performancedisk latencythroughputiostatfioiotopqueue depthslow diskio waitiowait
TL;DR
- Three numbers describe a disk: latency (time for one request), IOPS (requests per second) and throughput (bytes per second). Throughput = IOPS × request size.
- Small random requests show IOPS and latency; big sequential ones show throughput. A disk can be great at one and poor at the other.
- Measure your own pattern:
fioreproduces it,iostat -xshows what a running system does.
iostat -x 1 # r_await/w_await: ms per request; aqu-sz: queue
sudo iotop -o # who is doing I/O now
fio --name=t --rw=randread --bs=4k --size=1G --direct=1 --iodepth=32
On your machine
$ atfq 4.4 nvme0n1 read 94.8 GiB in 2.8 million requests, 0.15 ms each; wrote 217.8 GiB in 5.1 million requests, 0.46 ms each; busy 1% of the time; 345,678 cache flushes sda read 396.7 MiB in 3,120 requests, 7.52 ms each; wrote 102.5 MiB in 812 requests, 121.63 ms each; busy under 1% of the time Since boot, 3 days, 4 h ago Pending 1.5 MiB not yet written, 0 B being written; data waits up to 30 s in RAM
An example: Alice's laptop, one of the test machines. In your terminal, this block shows your own machine, read live.
Pitfalls
%utilin iostat is the share of time with any request in flight. An NVMe drive at 100 % may still have plenty of capacity left.⚠ ddwith nooflag=directorconv=fsyncmeasures the page cache, not the disk.- A process stuck in state
Dwith highwaintopis waiting for the disk, not using the CPU (16.1). - The fio command above creates a 1 GiB test file in the current folder: run it on the disk you want to test, and delete the file after.
Deep dive 2 questions
What is queue depth?
The number of requests in flight at once. A hard drive gains little from more than a few: it has one head. An SSD has many flash chips working in parallel, so it only reaches its rated IOPS with dozens of requests queued. A program that reads one block, waits, then reads the next uses a depth of one, and sees a fraction of the box's figures.
How can disk time add up to more than the clock?
The counters in /proc/diskstats add the time of every request. With 32
requests in flight for 1 second, the total says 32 seconds. Dividing it
by the number of requests gives the average wait per request; the "busy"
time, counted once whatever the depth, is what %util uses.