blktrace basics

Life of an I/O

Once a user issues an I/O request, this I/O enters block layer…then the magic begins…

1. IO request is remapped atop underlying logical/aggregated device device (MD, DM). Depending on alignment, size, …, requests are split into 2 separate I/Os
2. Requests added to the request queue
3. Merged with a previous entry on the queue -> All I/Os end up on a request queue at some point
4. The I/O is issued to a device driver, and submitted to a device
5. Later, the I/O is completed by the device, and posted by its driver

btt is a Linux utility that provides an analysis of the amount of time the I/O spent in the different areas of the I/O stack.

btt requires that you run blktrace first. Invoke blktrace specifying whatever devices and other parameters you want. You must save the traces to disk in this step,
In its current state, btt does not work in live mode.

After tracing completes, run blkrawverify, specifying all devices that were traced (or at least on all devices that you will use btt with.

If blkrawverify finds errors in the trace streams saved, it is best to recapture the data

Run blkparse with the -d option specifying a file to store the combined binary stream. (e.g.: blkparse -d bp.bin …).

blktrace produces a series of binary files containing parallel trace streams – one file per CPU per device. blkparse provides the ability to combine all the files into one time-ordered stream of traces for all devices.

Here’s some guidelines on the key indicators from the btt output:

Q — A block I/O is Queued
G — Get Request

A newly queued block I/O was not a candidate for merging with any existing request, so a new block layer request is allocated.

M — A block I/O is Merged with an existing request.
I — A request is Inserted into the device’s queue.
D — A request is issued to the D evice.
C — A request is Completed by the driver.
P — The block device queue is Plugged, to allow the aggregation of requests.
U — The device queue is Unplugged, allowing the aggregated requests to be issued to the device

Metrics of an I/O
Q2I – time it takes to process an I/O prior to it being inserted or merged onto a request queue

Includes split, and remap time
I2D – time the I/O is “idle” on the request queue

D2C – time the I/O is “active” in the driver and on the device

Q2I + I2D + D2C = Q2C
Q2C: Total processing time of the I/O

The latency data files which can be optionally produced by btt provide per-IO latency information, one for total IO time (Q2C) and one for latencies induced by lower layer drivers and devices (D2C).

In both cases, the first column (X values) represent runtime (seconds), while the second column (Y values) shows the actual latency for a command at that time (either Q2C or D2C).