There's also one interesting addition that people may not be aware of: Synchronous I/O that blocks (like pread64/pwrite64) will contribute to Linux system load

There's also one interesting addition that people may not be aware of: Synchronous I/O that blocks (like pread64/pwrite64) will contribute to Linux system load (threads in D state). Asynchronous I/O completion checks (libaio's io_getevents) that are willing to wait for I/O completion, will not contribute to Linux system load (threads in S state). Asynchronous I/O submissions (libaio's io_submit) either quickly submit their I/O (a small amount of time in R state) OR get stuck in io_submit() if the underlying block device I/O queue is full. When io_submit() gets stuck, then you're sleeping in D mode, thus contributing to system load again.