3 ms·

The disk controller may decide to write out blocks in a different order than the logical layout in the log file itself, and be interrupted before completing thi

by adambb 1y ago

The disk controller may decide to write out blocks in a different order than the logical layout in the log file itself, and be interrupted before completing this work.

hinkley 1y ago
We still have the elevator algorithm on NVMe?
- lxgr 1y ago
  SQLite runs on anything from servers to Internet-connected lightbulbs.
  - jrockway 1y ago
    Which lightbulbs include SQLite? I kind of want one.
    - natebc 1y ago
      these guys have a Cree logo on their homepage so maybe Cree? https://imaginovation.net/case-study/cree/ https://imaginovation.net/case-study/cree/ At least what I could turn up with a quick web search.
- bob1029 1y ago
  Anything that uses NAND storage technology is going to be optimized in some way like this. NVMe is just the messenger.
- jrockway 1y ago
  NVMe drives do their own manipulation of the datastream. Wear leveling, GC, trying to avoid rewriting an entire block for your 1 bit change, etc. NVMe drives have CPUs and RAM for this purpose; they are full computers with a little bit of flash memory attached. And no, of course they're not open source even though they have full access to your system.
  - djfivyvusn 1y ago
    Skynet gotta start somewhere.
johncolanduoni 1y ago
It’s worth noting this is also dependent on filesystem behavior; most that do copy-on-write will not suffer from this issue regardless of drive behavior, even if they don’t do their own checksumming.
grumbelbart2 1y ago
Just wondering how SQLite would ever work if it had zero control over this. Surely there must be some "flush" operation that guarantees that everthing so far is written to disk? Otherwise, any "old" block that contains data might have not been written. SQLite says: > Local devices also have a characteristic which is critical for enabling database management software to be designed to ensure ACID behavior: When all process writes to the device have completed, (when POSIX fsync() or Windows FlushFileBuffers() calls return), the filesystem then either has stored the "written" data or will do so before storing any subsequently written data.
- mschuster91 1y ago
  A "flush" command does indeed exist... but disk and controller vendors are like patients in Dr. House [1] - everybody lies. Especially if there are benchmarks to be "optimized". Other people here have written up that better than I ever could [2]. [1] https://house.fandom.com/wiki/Everybody_lies https://house.fandom.com/wiki/Everybody_lies [2] https://news.ycombinator.com/item?id=30371403 https://news.ycombinator.com/item?id=30371403

by adambb 1y ago

The disk controller may decide to write out blocks in a different order than the logical layout in the log file itself, and be interrupted before completing this work.

hinkley 1y ago

We still have the elevator algorithm on NVMe?

lxgr 1y ago
SQLite runs on anything from servers to Internet-connected lightbulbs.
- jrockway 1y ago
  Which lightbulbs include SQLite? I kind of want one.
  - natebc 1y ago
    these guys have a Cree logo on their homepage so maybe Cree? https://imaginovation.net/case-study/cree/ https://imaginovation.net/case-study/cree/ At least what I could turn up with a quick web search.
bob1029 1y ago
Anything that uses NAND storage technology is going to be optimized in some way like this. NVMe is just the messenger.
jrockway 1y ago
NVMe drives do their own manipulation of the datastream. Wear leveling, GC, trying to avoid rewriting an entire block for your 1 bit change, etc. NVMe drives have CPUs and RAM for this purpose; they are full computers with a little bit of flash memory attached. And no, of course they're not open source even though they have full access to your system.
- djfivyvusn 1y ago
  Skynet gotta start somewhere.

johncolanduoni 1y ago

It’s worth noting this is also dependent on filesystem behavior; most that do copy-on-write will not suffer from this issue regardless of drive behavior, even if they don’t do their own checksumming.

grumbelbart2 1y ago

Just wondering how SQLite would ever work if it had zero control over this. Surely there must be some "flush" operation that guarantees that everthing so far is written to disk? Otherwise, any "old" block that contains data might have not been written. SQLite says: > Local devices also have a characteristic which is critical for enabling database management software to be designed to ensure ACID behavior: When all process writes to the device have completed, (when POSIX fsync() or Windows FlushFileBuffers() calls return), the filesystem then either has stored the "written" data or will do so before storing any subsequently written data.

mschuster91 1y ago
A "flush" command does indeed exist... but disk and controller vendors are like patients in Dr. House [1] - everybody lies. Especially if there are benchmarks to be "optimized". Other people here have written up that better than I ever could [2]. [1] https://house.fandom.com/wiki/Everybody_lies https://house.fandom.com/wiki/Everybody_lies [2] https://news.ycombinator.com/item?id=30371403 https://news.ycombinator.com/item?id=30371403