BPF and the I/O Stack

This chapter is from the book

BPF Performance Tools

Learn More Buy

This chapter is from the book

This chapter is from the book 

BPF Performance Tools

Learn More Buy

9.4 BPF One-Liners

These sections show BCC and bpftrace one-liners. Where possible, the same one-liner is implemented using both BCC and bpftrace.

9.4.1 BCC

Count block I/O tracepoints:

funccount t:block:*

Summarize block I/O size as a histogram:

argdist -H 't:block:block_rq_issue():u32:args->bytes'

Count block I/O request user stack traces:

stackcount -U t:block:block_rq_issue

Count block I/O type flags:

argdist -C 't:block:block_rq_issue():char*:args->rwbs'

Trace block I/O errors with device and I/O type:

trace 't:block:block_rq_complete (args->error) "dev %d type %s error %d", args->dev,
args->rwbs, args->error'

Count SCSI opcodes:

argdist -C 't:scsi:scsi_dispatch_cmd_start():u32:args->opcode'

Count SCSI result codes:

argdist -C 't:scsi:scsi_dispatch_cmd_done():u32:args->result'

Count nvme driver functions:

funccount 'nvme*'

9.4.2 bpftrace

Count block I/O tracepoints:

bpftrace -e 'tracepoint:block:* { @[probe] = count(); }'

Summarize block I/O size as a histogram:

bpftrace -e 't:block:block_rq_issue { @bytes = hist(args->bytes); }'

Count block I/O request user stack traces:

bpftrace -e 't:block:block_rq_issue { @[ustack] = count(); }'

Count block I/O type flags:

bpftrace -e 't:block:block_rq_issue { @[args->rwbs] = count(); }'

Show total bytes by I/O type:

bpftrace -e 't:block:block_rq_issue { @[args->rwbs] = sum(args->bytes); }'

Trace block I/O errors with device and I/O type:

bpftrace -e 't:block:block_rq_complete /args->error/ {
    printf("dev %d type %s error %d\n", args->dev, args->rwbs, args->error); }'

Summarize block I/O plug time as a histogram:

bpftrace -e 'k:blk_start_plug { @ts[arg0] = nsecs; }
    k:blk_flush_plug_list /@ts[arg0]/ { @plug_ns = hist(nsecs - @ts[arg0]);
    delete(@ts[arg0]); }'

Count SCSI opcodes:

bpftrace -e 't:scsi:scsi_dispatch_cmd_start { @opcode[args->opcode] = count(); }'

Count SCSI result codes (all four bytes):

bpftrace -e 't:scsi:scsi_dispatch_cmd_done { @result[args->result] = count(); }'

Show CPU distribution of blk_mq requests:

bpftrace -e 'k:blk_mq_start_request { @swqueues = lhist(cpu, 0, 100, 1); }'

Count scsi driver functions:

bpftrace -e 'kprobe:scsi* { @[func] = count(); }'

Count nvme driver functions:

bpftrace -e 'kprobe:nvme* { @[func] = count(); }'

9.4.3 BPF One-Liners Examples

Including some sample output, as was done for each tool, is also useful for illustrating one-liners.

Counting Block I/O Type Flags

# bpftrace -e 't:block:block_rq_issue { @[args->rwbs] = count(); }'
Attaching 1 probe...
^C

@[N]: 2
@[WFS]: 9
@[FF]: 12
@[N]: 13
@[WSM]: 23
@[WM]: 64
@[WS]: 86
@[R]: 201
@[R]: 285
@[W]: 459
@[RM]: 1112
@[RA]: 2128
@[R]: 3635
@[W]: 4578

This frequency counts the rwbs field that encodes the I/O type. While tracing, where were 3635 reads (“R”) and 2128 read-ahead I/O (“RA”). The “rwbs” section at the start of this chapter describes this rwbs field.

This one-liner can answer workload characterization questions such as:

• What is the ratio of read versus read-ahead block I/O?

• What is the ratio of write versus synchronous write block I/O?

By changing count() to be sum(args->bytes), this one-liner will sum the bytes by I/O type.