3 ms·

Command queuing is such a dangerous thing. This is the disk equivalent of 'buffer bloat'[1] but with disk data. The question you have to ask is what happens wh

Command queuing is such a dangerous thing. This is the disk equivalent of 'buffer bloat'[1] but with disk data. The question you have to ask is what happens when you lose power and there are 'n' writes queued to the storage device. How has the kernel marked that data? Historically once the disk accepted the write and returned kernels would say "ok, data on disk" buffer clean now. Of course if you lost power and that write never actually completed, well you now had file system corruption (possibly silent if it was just a data block). Another problem that cropped up is "ok we can re-use this data buffer" so it gets fill with different data, and then the storage device says "oh hey, send me that data you told me write ..." whoops. The third problem that crops up is that the computer on the storage device crashes (or watchdog resets) and it goes back and doesn't remember what commands it had in the queue. So some time later your kernel better ask "hey why haven't you acknowledged this write we sent 50mS ago?" and have the disk say "What write?" and replay it. One school of thought was "Command queuing in the storage device is always bad, the kernel knows more, has more memory, and is ultimately responsible for what is and what is not on disk." So the kernel would 'hold' writes to the disk until the write was sufficiently aged or until enough writes had accumulated to do a "streaming" write (multiple sectors on the same track). But for non-spinning media there is no seek time so there is no advantage. Except for flash, you have to erase a page which can be a lot of data which then has to be rewritten with the new stuff. I enjoyed the writeup but it really needs to start with a highly fragmented SD card (one with lots of random I/Os to it) so that it can measure the latency hit for page rewrites. [1] https://www.bufferbloat.net/projects/bloat/wiki/Introduction/ https://www.bufferbloat.net/projects/bloat/wiki/Introduction...

throwaway2048 7y ago
additionally SSDs use garbage collection algorithms so are fraught with the possibility of a write taking an unbounded amount of time, even if nothing is malfunctioning, which is pretty scary. To head off replies about fixed time garbage collection, sure it exists, but you cant guarantee it will reclaim enough space in a fixed timeslot (think of a massively pathological case where 99% of every flash chip is full, but super fragmented and data needs to be shuffled around perhaps hundreds or even thousands of times before enough space can be reclaimed)
- Dylan16807 7y ago
  > To head off replies about fixed time garbage collection, sure it exists, but you cant guarantee it will reclaim enough space in a fixed timeslot (think of a massively pathological case where 99% of every flash chip is full, but super fragmented and data needs to be shuffled around perhaps hundreds or even thousands of times before enough space can be reclaimed) That's why you have spare space. It's not overly hard to clamp write amplification to a number like 10x or 20x, even in the absolute worst case. Note that the SD standard here requires a minimum number of writes per second inside a 256MB area, and even 1% spare space on a 32GB card would give you a >256MB buffer.
the8472 7y ago
PC drives have had write caches and the necessary barrier/flush commands for many years. This is nothing new or dangerous. Additionally operating systems also play exactly the same "dangerous" game with writes, they keep dirty pages in memory and flush them out gradually. You are only guaranteed that things actually got to disk when you fsync them which sets off a cascade of writes and flushes if they didn't already happen in the background. Write performance is largely a very well-maintained illusion if you don't have something like Optane. If you open files in SYNC mode write performance craters, even on NVMe. What filesystems do to maintain sanity among this madness is to order writes so that if a crash occurs you lose a few seconds of data, but you don't lose consistency, you just get an older state of whatever made it to disk. SD cards are not special in this regard.
gruez 7y ago
>The question you have to ask is what happens when you lose power and there are 'n' writes queued to the storage device. How has the kernel marked that data? Historically once the disk accepted the write and returned kernels would say "ok, data on disk" buffer clean now. Is there a reason why storage devices can't accept the write, and then notify the kernel when the the write is complete? I'm fairly certain that's how it works right now. >The third problem that crops up is that the computer on the storage device crashes (or watchdog resets) and it goes back and doesn't remember what commands it had in the queue. So some time later your kernel better ask "hey why haven't you acknowledged this write we sent 50mS ago?" and have the disk say "What write?" and replay it. Don't you run into the same issue if you were doing writes and the user yanked out the power? What's written after the last fsync is pretty much undefined behavior and it's up to the application/filesystem's journaling mechanism to deal with it. >One school of thought was "Command queuing in the storage device is always bad, the kernel knows more, has more memory, and is ultimately responsible for what is and what is not on disk." So the kernel would 'hold' writes to the disk until the write was sufficiently aged or until enough writes had accumulated to do a "streaming" write (multiple sectors on the same track). But for non-spinning media there is no seek time so there is no advantage. Except for flash, you have to erase a page which can be a lot of data which then has to be rewritten with the new stuff. So command queuing is bad except on hard disks and flash. Then what else is left? Those two cover 99.99% of consumer storage.

throwaway2048 7y ago

additionally SSDs use garbage collection algorithms so are fraught with the possibility of a write taking an unbounded amount of time, even if nothing is malfunctioning, which is pretty scary. To head off replies about fixed time garbage collection, sure it exists, but you cant guarantee it will reclaim enough space in a fixed timeslot (think of a massively pathological case where 99% of every flash chip is full, but super fragmented and data needs to be shuffled around perhaps hundreds or even thousands of times before enough space can be reclaimed)

Dylan16807 7y ago
> To head off replies about fixed time garbage collection, sure it exists, but you cant guarantee it will reclaim enough space in a fixed timeslot (think of a massively pathological case where 99% of every flash chip is full, but super fragmented and data needs to be shuffled around perhaps hundreds or even thousands of times before enough space can be reclaimed) That's why you have spare space. It's not overly hard to clamp write amplification to a number like 10x or 20x, even in the absolute worst case. Note that the SD standard here requires a minimum number of writes per second inside a 256MB area, and even 1% spare space on a 32GB card would give you a >256MB buffer.

the8472 7y ago

PC drives have had write caches and the necessary barrier/flush commands for many years. This is nothing new or dangerous. Additionally operating systems also play exactly the same "dangerous" game with writes, they keep dirty pages in memory and flush them out gradually. You are only guaranteed that things actually got to disk when you fsync them which sets off a cascade of writes and flushes if they didn't already happen in the background. Write performance is largely a very well-maintained illusion if you don't have something like Optane. If you open files in SYNC mode write performance craters, even on NVMe. What filesystems do to maintain sanity among this madness is to order writes so that if a crash occurs you lose a few seconds of data, but you don't lose consistency, you just get an older state of whatever made it to disk. SD cards are not special in this regard.

gruez 7y ago

>The question you have to ask is what happens when you lose power and there are 'n' writes queued to the storage device. How has the kernel marked that data? Historically once the disk accepted the write and returned kernels would say "ok, data on disk" buffer clean now. Is there a reason why storage devices can't accept the write, and then notify the kernel when the the write is complete? I'm fairly certain that's how it works right now. >The third problem that crops up is that the computer on the storage device crashes (or watchdog resets) and it goes back and doesn't remember what commands it had in the queue. So some time later your kernel better ask "hey why haven't you acknowledged this write we sent 50mS ago?" and have the disk say "What write?" and replay it. Don't you run into the same issue if you were doing writes and the user yanked out the power? What's written after the last fsync is pretty much undefined behavior and it's up to the application/filesystem's journaling mechanism to deal with it. >One school of thought was "Command queuing in the storage device is always bad, the kernel knows more, has more memory, and is ultimately responsible for what is and what is not on disk." So the kernel would 'hold' writes to the disk until the write was sufficiently aged or until enough writes had accumulated to do a "streaming" write (multiple sectors on the same track). But for non-spinning media there is no seek time so there is no advantage. Except for flash, you have to erase a page which can be a lot of data which then has to be rewritten with the new stuff. So command queuing is bad except on hard disks and flash. Then what else is left? Those two cover 99.99% of consumer storage.