5 ms·

There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead

There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead I/O? I wonder if this is related. https://forums.unraid.net/bug-reports/prereleases/sqlite-data-corruption-testing-r664/?do=findComment&comment=6882 https://forums.unraid.net/bug-reports/prereleases/sqlite-dat... From that report: ====== 8< ====== > In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD which tells I/O driver this is a read-ahead. In this case, if there are insufficient resources at the time this request is received, the driver is permitted to terminate the operation with BLK_STS_IOERR status. Here is an example in Linux md/raid5 driver. > In case of Unraid it can definitely happen under heavy load that a read-ahead comes along and there are no 'stripe buffers' immediately available. In this case, instead of making calling process wait, it terminated the I/O. This has worked this way for years. [...] > What I suspect is that this is a bug in SQLite - I think SQLite is using direct-I/O (bypassing page cache) and issuing it's own read-aheads and their logic to handle failing read-ahead is broken. But I did not follow that rabbit hole - too many other problems to work on :/ ====== 8< ====== Some related bugs they brought up: https://bugzilla.kernel.org/show_bug.cgi?id=201685 https://bugzilla.kernel.org/show_bug.cgi?id=201685 https://patchwork.kernel.org/patch/10712695/ https://patchwork.kernel.org/patch/10712695/ Edit: it appears that there's _something_ causing corruption when drivers fail read-ahead I/O. Whether it's SQLite or something else in Linux is another question.

geofft 7y ago
> I think SQLite is using direct-I/O (bypassing page cache) Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no? strace -e open -o >(grep O_DIRECT) ./myprogram
- lelf 7y ago
  -o >(grep O_DIRECT) This is cool. Never seen it (but it makes sense.)
  - mmastrac 7y ago
    Its cousin diff <(cmd a) <(cmd b) is also very handy.
    - lostmsu 7y ago
      Not a command line guru, but this looks interesting. Does it compare outputs of two commands? How does it do it?
      - arendtio 7y ago
        <(command) is being replaced by a path to a file which contains the output of the command: $ ls <(echo A) /dev/fd/63 $ cat <(echo A) A So diff <(command1) <(command2) becomes diff /dev/fd/62 /dev/fd/63
      - smartmic 7y ago
        It is called process substitution and specific to Bash (not POSIX shell compliant). The output of the commands in parenthesis will become runtime (file) inputs for the diff command. More: https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html https://www.gnu.org/software/bash/manual/html_node/Process-S...
        emmelaich 7y ago
        It's not specific to bash -- it's in ksh as well. And zsh. I'm also pretty sure it originated in ksh, not bash.
      - hans1729 7y ago
        diff compares files line by line, here we just redirect the output of cmd1 and cmd2 via <, so they serve as 'proxies' for the file-args diff expects you might want to read into redirection on the commandline, it's amazingly powerful
      - mmastrac 7y ago
        It might be easiest to view it like so: $ echo <(echo 1) <(echo 2) /dev/fd/63 /dev/fd/62 The shell spawns two commands connected to pipes, then replaces those with a file that represents the other (read) side of that pipe. The command above with >(grep something) does the same thing, just with the other end of a pipe.
pdw 7y ago
I don't think sqlite uses direct IO -- at least there's no mention of O_DIRECT in the source code
blattimwind 7y ago
SQLite doesn't use direct I/O.

geofft 7y ago

> I think SQLite is using direct-I/O (bypassing page cache) Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no? strace -e open -o >(grep O_DIRECT) ./myprogram

lelf 7y ago
-o >(grep O_DIRECT) This is cool. Never seen it (but it makes sense.)
- mmastrac 7y ago
  Its cousin diff <(cmd a) <(cmd b) is also very handy.
  - lostmsu 7y ago
    Not a command line guru, but this looks interesting. Does it compare outputs of two commands? How does it do it?
    - arendtio 7y ago
      <(command) is being replaced by a path to a file which contains the output of the command: $ ls <(echo A) /dev/fd/63 $ cat <(echo A) A So diff <(command1) <(command2) becomes diff /dev/fd/62 /dev/fd/63
    - smartmic 7y ago
      It is called process substitution and specific to Bash (not POSIX shell compliant). The output of the commands in parenthesis will become runtime (file) inputs for the diff command. More: https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html https://www.gnu.org/software/bash/manual/html_node/Process-S...
      - emmelaich 7y ago
        It's not specific to bash -- it's in ksh as well. And zsh. I'm also pretty sure it originated in ksh, not bash.
    - hans1729 7y ago
      diff compares files line by line, here we just redirect the output of cmd1 and cmd2 via <, so they serve as 'proxies' for the file-args diff expects you might want to read into redirection on the commandline, it's amazingly powerful
    - mmastrac 7y ago
      It might be easiest to view it like so: $ echo <(echo 1) <(echo 2) /dev/fd/63 /dev/fd/62 The shell spawns two commands connected to pipes, then replaces those with a file that represents the other (read) side of that pipe. The command above with >(grep something) does the same thing, just with the other end of a pipe.

pdw 7y ago

I don't think sqlite uses direct IO -- at least there's no mention of O_DIRECT in the source code

blattimwind 7y ago

SQLite doesn't use direct I/O.