3 ms·

I don't understand why so much emphasis is put on optimizing for sequential reads and writes. Sequential IO is not something that happens in the real world exce

I don't understand why so much emphasis is put on optimizing for sequential reads and writes. Sequential IO is not something that happens in the real world except in rare circumstances. In most applications IO is random, even if it's a log file.

acscott 3y ago
When writing to a log file, isn't that sequential write?
dmitrygr 3y ago
> In most applications IO is random, even if it's a log file. I am all ears... how are log writes random?
- JdeBP 3y ago
  I think that I see what thelastparadise is getting at. At base, most logfile output is a read-modify-write of usually no more than the final block of the file, whereas the sequential writes that are generally optimized for by filesystem and disc drivers are multiple-block writes sent as single output requests. The twain are similar, but aren't exactly the same. Optimize for the cases where you are writing multiple blocks in a chunk, with contiguous free block allocation policies and write behind and whatnot, and you haven't in reality quite optimized for the case where what you're actually doing is overwriting the tail block in the file repeatedly, oftentimes changing as few as mere 10s of bytes in that block with each read-modify-write request. (And yes, logging mechanisms often do like to ensure that logs are synchronously flushed to disc line by line, effectively and intentionally defeating write caching and write behind optimizations.)
  - justsomehnguy 3y ago
    I would add what it is very rare to have 100MiB+ writes to a log file and anything less would be concurrent with the other operations, which would make 'sequential' here a moot point. > At base, most logfile output is a read-modify-write It's worse, actually, as you never can tell if this would be RMW to the same block or to the other free block. In the former you waste the writes for the whole blocks. In the latter you can hit a write amplification despite never have been writing any meaningful amount of data.
    - JdeBP 3y ago
      It's worse for an even simpler reason, which I thought about mentioning but it probably wasn't what thelastparadise was talking about. The line by line synchronization and flushing also causes i-node updates.

acscott 3y ago

When writing to a log file, isn't that sequential write?

dmitrygr 3y ago

> In most applications IO is random, even if it's a log file. I am all ears... how are log writes random?

JdeBP 3y ago
I think that I see what thelastparadise is getting at. At base, most logfile output is a read-modify-write of usually no more than the final block of the file, whereas the sequential writes that are generally optimized for by filesystem and disc drivers are multiple-block writes sent as single output requests. The twain are similar, but aren't exactly the same. Optimize for the cases where you are writing multiple blocks in a chunk, with contiguous free block allocation policies and write behind and whatnot, and you haven't in reality quite optimized for the case where what you're actually doing is overwriting the tail block in the file repeatedly, oftentimes changing as few as mere 10s of bytes in that block with each read-modify-write request. (And yes, logging mechanisms often do like to ensure that logs are synchronously flushed to disc line by line, effectively and intentionally defeating write caching and write behind optimizations.)
- justsomehnguy 3y ago
  I would add what it is very rare to have 100MiB+ writes to a log file and anything less would be concurrent with the other operations, which would make 'sequential' here a moot point. > At base, most logfile output is a read-modify-write It's worse, actually, as you never can tell if this would be RMW to the same block or to the other free block. In the former you waste the writes for the whole blocks. In the latter you can hit a write amplification despite never have been writing any meaningful amount of data.
  - JdeBP 3y ago
    It's worse for an even simpler reason, which I thought about mentioning but it probably wasn't what thelastparadise was talking about. The line by line synchronization and flushing also causes i-node updates.