4 ms·

I hope the claim to being fully async I/O is just a buzz term, as it's no longer supported in SQLite. https://www.sqlite.org/asyncvfs.html https://www.sqlite.o

I hope the claim to being fully async I/O is just a buzz term, as it's no longer supported in SQLite. https://www.sqlite.org/asyncvfs.html https://www.sqlite.org/asyncvfs.html

tyingq 7y ago
The very top of that page suggests WAL with PRAGMA synchronous = 0 accomplishes roughly the same objective.
- rockwotj 7y ago
  Based on the following header file, it would seem that they are using this unsupported module. https://github.com/canonical/dqlite/blob/master/src/vfs.h https://github.com/canonical/dqlite/blob/master/src/vfs.h
  - tyingq 7y ago
    They would have to, wouldn't they? The Sqlite VFS subsystem is the obvious place to intercept the usually local DB and WAL reads/writes and make them distributed functions that use Raft.
  - cyphar 7y ago
    Yes, you have to use a forked SQLite in order to make use of dqlite. I believe they (the LXD team) are working on upstreaming the WAL changes but due to SQLite's very strong compatibility guarantees they want to be very certain the API and protocol are correct before carving it in stone. Not to mention they are the only major users of the feature, so more widespread use would also be nice before merging it upstream.
    - free-ekanayaka 7y ago
      Everything you said is very accurate (dqlite author here).
free-ekanayaka 7y ago
It's not a buzz term. It's really fully async disk I/O. Dqlite does not use SQLite's stock vfs implementation for writing to disk, as it's an entirely different model (based on raft).
- liability 7y ago
  > Q: How does dqlite behave during conflict situations? Does Raft select a winning WAL write and any others in flight writes are aborted? > A: There can’t be a conflict situation. Raft’s model is that only the leader can append new log entries, which translated to dqlite means that only the leader can write new WAL frames. So this means that any attempt to perform a write transaction on a non-leader node will fail with a ErrNotLeader error (and in this case clients are supposed to retry against whoever is the new leader). Correct me if I'm wrong, but isn't that essentially the same limitation WAL mode in normal sqlite has? With WAL you can have as many reads going on as you like, in parallel to a single write. That seems directly comparable to what the dqlite FAQ says, unless I'm missing something.

I hope the claim to being fully async I/O is just a buzz term, as it's no longer supported in SQLite. https://www.sqlite.org/asyncvfs.html https://www.sqlite.org/asyncvfs.html

tyingq 7y ago

The very top of that page suggests WAL with PRAGMA synchronous = 0 accomplishes roughly the same objective.

rockwotj 7y ago
Based on the following header file, it would seem that they are using this unsupported module. https://github.com/canonical/dqlite/blob/master/src/vfs.h https://github.com/canonical/dqlite/blob/master/src/vfs.h
- tyingq 7y ago
  They would have to, wouldn't they? The Sqlite VFS subsystem is the obvious place to intercept the usually local DB and WAL reads/writes and make them distributed functions that use Raft.
- cyphar 7y ago
  Yes, you have to use a forked SQLite in order to make use of dqlite. I believe they (the LXD team) are working on upstreaming the WAL changes but due to SQLite's very strong compatibility guarantees they want to be very certain the API and protocol are correct before carving it in stone. Not to mention they are the only major users of the feature, so more widespread use would also be nice before merging it upstream.
  - free-ekanayaka 7y ago
    Everything you said is very accurate (dqlite author here).

free-ekanayaka 7y ago

It's not a buzz term. It's really fully async disk I/O. Dqlite does not use SQLite's stock vfs implementation for writing to disk, as it's an entirely different model (based on raft).

liability 7y ago
> Q: How does dqlite behave during conflict situations? Does Raft select a winning WAL write and any others in flight writes are aborted? > A: There can’t be a conflict situation. Raft’s model is that only the leader can append new log entries, which translated to dqlite means that only the leader can write new WAL frames. So this means that any attempt to perform a write transaction on a non-leader node will fail with a ErrNotLeader error (and in this case clients are supposed to retry against whoever is the new leader). Correct me if I'm wrong, but isn't that essentially the same limitation WAL mode in normal sqlite has? With WAL you can have as many reads going on as you like, in parallel to a single write. That seems directly comparable to what the dqlite FAQ says, unless I'm missing something.