5 ms·

> Latency? Throughput? I, don't think SSDs are 1,000 times slower than RAM in either category, assuming fairly typical consumer machines, but I could be wrong.

> Latency? Throughput? I, don't think SSDs are 1,000 times slower than RAM in either category, assuming fairly typical consumer machines, but I could be wrong. Everything in this business is like a great big onion of caching and buffering layers. Disks have them too. This is why you often see consumer SSDs have very fast writes only as long as you don't write continuously. Continuous writes causes their internal buffers to fill up and the real write performance to be revealed. There's also factors like write amplification and overprovisioning that enter into this set of equations. But if there's any takeaway from this, it's that I/O benchmarking is hell.

dmazin 3y ago
I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Like, let's say you are doing some writing using the C standard library... * the library itself will buffer your writes, to reduce the number of syscalls * I think your bio objects might get buffered, via plugging, to reduce the number of block requests * your block requests will get buffered, so that they can be canceled or merged (I'm actually not sure if this happens in the "software staging queue" part of blk-mq, or prior...) * as you mentioned, the disk controller may very well buffer the actual write operations
- hoten 3y ago
  John Carmack's lecture on systems engineering / reducing latency for VR mentions buffers repeatedly: https://youtu.be/lHLpKzUxjGk https://youtu.be/lHLpKzUxjGk
- formerly_proven 3y ago
  > I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Well there's a reason it takes a click or key press to register about as long as it takes sending a packet halfway across the world (or a few times around the world, depending on what software you're running).
  - ryandrake 3y ago
    It must have been a real blast back in the 8-bit days, when you'd have some peripheral's port mapped to a specific address in the memory map, and setting a byte to that address actually slammed the electrons into that peripheral's register. No wasteful 15 layers of OS abstractions waiting for buffers to fill and copying bytes from buffer to buffer all the way down.
jchw 3y ago
Oh yeah, that I can't deny. The peak throughput numbers used in marketing are definitely questionable at times. Kinda makes sense considering SSDs and HDDs are basically computers of their own that give you some abstraction over the actual raw storage media. You could compare memcpy to sequential write speed or something, but it'd definitely not tell the full story.
- marginalia_nu 3y ago
  It can be a pretty real problem. I encountered a problem where I needed to construct a file with out of order writes 64 bits long. This works fine for small files, but this file was dozens of gigabytes. Turns out if you do this the naive way on a consumer disk, it will take weeks and it will wear out the disk incredibly quickly, because all the buffers will fill up in the worst possible way and the disk ends up with worst case write amplification for every write. I ended up creating a bunch of smaller files containing essentially assembly instructions, of the form "put X at offset Y", and grouping adjacent write instructions in ~100Mb chunks. This is just sequential writes, which the OS will gladly help buffer and the disk likes. Then I go over the files one by one and evaluate the instructions. Now since the "random" writes are concentrated to a relatively small area on disk, the buffering works well. Despite writing 2.5x as much data (and reading the data an additional time), this makes the write operation take like an hour instead of several weeks.
  - gavinray 3y ago
    > ...but this file was dozens of gigabytes. Turns out if you do this the naive way on a consumer disk, it will take weeks and it will wear out the disk incredibly quickly This is what C's "fallocate(2)" with "FALLOC_FL_ZERO_RANGE" or "FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE" flags are for, along with "pwrite()"
    - marginalia_nu 3y ago
      It's mostly a SSD problem. They really do not like sustained loads of random small IO, since their write cycles have much larger granularity.

dmazin 3y ago

I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Like, let's say you are doing some writing using the C standard library... * the library itself will buffer your writes, to reduce the number of syscalls * I think your bio objects might get buffered, via plugging, to reduce the number of block requests * your block requests will get buffered, so that they can be canceled or merged (I'm actually not sure if this happens in the "software staging queue" part of blk-mq, or prior...) * as you mentioned, the disk controller may very well buffer the actual write operations

hoten 3y ago
John Carmack's lecture on systems engineering / reducing latency for VR mentions buffers repeatedly: https://youtu.be/lHLpKzUxjGk https://youtu.be/lHLpKzUxjGk
formerly_proven 3y ago
> I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Well there's a reason it takes a click or key press to register about as long as it takes sending a packet halfway across the world (or a few times around the world, depending on what software you're running).
- ryandrake 3y ago
  It must have been a real blast back in the 8-bit days, when you'd have some peripheral's port mapped to a specific address in the memory map, and setting a byte to that address actually slammed the electrons into that peripheral's register. No wasteful 15 layers of OS abstractions waiting for buffers to fill and copying bytes from buffer to buffer all the way down.

jchw 3y ago

Oh yeah, that I can't deny. The peak throughput numbers used in marketing are definitely questionable at times. Kinda makes sense considering SSDs and HDDs are basically computers of their own that give you some abstraction over the actual raw storage media. You could compare memcpy to sequential write speed or something, but it'd definitely not tell the full story.

marginalia_nu 3y ago
It can be a pretty real problem. I encountered a problem where I needed to construct a file with out of order writes 64 bits long. This works fine for small files, but this file was dozens of gigabytes. Turns out if you do this the naive way on a consumer disk, it will take weeks and it will wear out the disk incredibly quickly, because all the buffers will fill up in the worst possible way and the disk ends up with worst case write amplification for every write. I ended up creating a bunch of smaller files containing essentially assembly instructions, of the form "put X at offset Y", and grouping adjacent write instructions in ~100Mb chunks. This is just sequential writes, which the OS will gladly help buffer and the disk likes. Then I go over the files one by one and evaluate the instructions. Now since the "random" writes are concentrated to a relatively small area on disk, the buffering works well. Despite writing 2.5x as much data (and reading the data an additional time), this makes the write operation take like an hour instead of several weeks.
- gavinray 3y ago
  > ...but this file was dozens of gigabytes. Turns out if you do this the naive way on a consumer disk, it will take weeks and it will wear out the disk incredibly quickly This is what C's "fallocate(2)" with "FALLOC_FL_ZERO_RANGE" or "FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE" flags are for, along with "pwrite()"
  - marginalia_nu 3y ago
    It's mostly a SSD problem. They really do not like sustained loads of random small IO, since their write cycles have much larger granularity.