3 ms·

Many smart people have pondered long and hard over this exact problem for decades. Why not climb on their shoulders and adopt the WAL approach that's working so

by moe 16y ago

Many smart people have pondered long and hard over this exact problem for decades. Why not climb on their shoulders and adopt the WAL approach that's working so well for most RDBMS?

jbellis 16y ago
bq. Why not climb on their shoulders and adopt the WAL approach The WAL approach solves a different problem: "how do we provide durable writes with minimal performance impact?" WAL is not queryable, so the question of "how do we efficiently provide random, indexed access to data" is a separate one. (And the traditional answer of B-trees is working increasingly poorly in the face of a growing capacity:seek time gap.)
- btilly 16y ago
  The WAL approach solves a different problem: "how do we provide durable writes with minimal performance impact?" WAL does more than that. Part of providing durable writes is to be able to fix up the potentially corrupted associated binary data structure on disk on startup. So, for instance, if you wrote a link before the associated node is actually saved to disk, then when you replay the WAL you're guaranteed to write the link and the node, and are not going to be corrupted.
- moe 16y ago
  You are right, I realize I misjudged/simplified the problem somewhat. My mind had spontaneously formed this idea of an in-memory overlay b-tree of committed (WAL) but not yet merged changes... But thinking it through further I have to admit that this would quickly lead to a complexity explosion (MVCC'ish). I also notice my tone in that comment was more snarky than I meant to be, sorry for that. However I still believe that this is a wheel that's been reinvented often enough (and that is complex enough) that it'd be a good idea to focus on researching how the others have eventually solved it - to minimize the risk of repeating history.
ot 16y ago
RDBMS have consistency guarantees that require the engine to fsync at each commit. This ensures that if the system crashes after a transaction no data is lost, but the transaction throughput is limited by the disk writes. Conversely Redis (with BGSAVE or diskstore) and other systems only ensure integrity, which means that if there is a crash you don't end up with a corrupted database, but you may lose the latest data that you wrote in the DB because they haven't yet been committed to disk. Relaxing these guarantees completely changes the rules of the game, so the tradeoffs that work well with RDBMS, such as WAL, may not be the best fit in this world.

by moe 16y ago

Many smart people have pondered long and hard over this exact problem for decades. Why not climb on their shoulders and adopt the WAL approach that's working so well for most RDBMS?

jbellis 16y ago

bq. Why not climb on their shoulders and adopt the WAL approach The WAL approach solves a different problem: "how do we provide durable writes with minimal performance impact?" WAL is not queryable, so the question of "how do we efficiently provide random, indexed access to data" is a separate one. (And the traditional answer of B-trees is working increasingly poorly in the face of a growing capacity:seek time gap.)

btilly 16y ago
The WAL approach solves a different problem: "how do we provide durable writes with minimal performance impact?" WAL does more than that. Part of providing durable writes is to be able to fix up the potentially corrupted associated binary data structure on disk on startup. So, for instance, if you wrote a link before the associated node is actually saved to disk, then when you replay the WAL you're guaranteed to write the link and the node, and are not going to be corrupted.
moe 16y ago
You are right, I realize I misjudged/simplified the problem somewhat. My mind had spontaneously formed this idea of an in-memory overlay b-tree of committed (WAL) but not yet merged changes... But thinking it through further I have to admit that this would quickly lead to a complexity explosion (MVCC'ish). I also notice my tone in that comment was more snarky than I meant to be, sorry for that. However I still believe that this is a wheel that's been reinvented often enough (and that is complex enough) that it'd be a good idea to focus on researching how the others have eventually solved it - to minimize the risk of repeating history.

ot 16y ago

RDBMS have consistency guarantees that require the engine to fsync at each commit. This ensures that if the system crashes after a transaction no data is lost, but the transaction throughput is limited by the disk writes. Conversely Redis (with BGSAVE or diskstore) and other systems only ensure integrity, which means that if there is a crash you don't end up with a corrupted database, but you may lose the latest data that you wrote in the DB because they haven't yet been committed to disk. Relaxing these guarantees completely changes the rules of the game, so the tradeoffs that work well with RDBMS, such as WAL, may not be the best fit in this world.