Part I · How it works
4.3 From write() to the disk: the block layer
alsoblock layerblock deviceblock devicesio scheduleri/o schedulermq-deadlinebfqkyberblk-mqrequest queueo_directdirect io
TL;DR
- A
write()goes into the page cache and returns. Later, the filesystem turns dirty pages into block requests ("write these sectors"). - The block layer merges neighbouring requests, orders them through an I/O scheduler, and hands them to the driver. The device signals each completion with an interrupt.
- Each disk has its scheduler:
nonefor NVMe, which sorts nothing itself;mq-deadlineorbfqfor slower disks.
cat /sys/block/nvme0n1/queue/scheduler # the active one in [brackets]
lsblk -t # block sizes, scheduler
On your machine
$ atfq 4.3 nvme0n1 Samsung SSD 980 PRO 500GB: NVMe flash, 476.9 GiB · scheduler none · 8 hardware queues · volatile write cache · TRIM sda USB DISK 3.0: USB drive, 30.0 GiB · scheduler mq-deadline · no volatile write cache
An example: Alice's laptop, one of the test machines. In your terminal, this block shows your own machine, read live.
Pitfalls
bfqkeeps a desktop responsive while something copies in the background, at some CPU cost; on fast NVMe,noneis usually best.- Changing the scheduler with
echolasts until reboot; a udev rule makes it permanent (13.6). O_DIRECTskips the page cache, for databases that cache by themselves. It needs aligned buffers, and it isn't a durability promise: only fsync is (4.5).
Deep dive 2 questions
What's in between, layer by layer?
Your program calls write(); the VFS finds the file's filesystem; the
filesystem (ext4, btrfs, XFS) copies the data into the page cache and
decides where it goes on the device. At writeback, it builds bio
structures: a device, a sector, memory pages. Below, device-mapper layers
(LVM, LUKS encryption) may remap or encrypt them, then blk-mq queues them
for the driver: NVMe, SCSI for SATA and USB, or virtio in a VM.
Why do schedulers matter less than they did?
On a hard drive, sorting requests by position saved whole seeks: the
elevator algorithms. An SSD has no head to move and many internal queues,
and an NVMe drive has queues per CPU. Sorting costs more than it saves,
so the main remaining job is fairness between processes, which is what
bfq and cgroup I/O limits are for.