3 ms·

If you're willing to accept that durability is a lie, and also to compromise (possibly a lot) on performance and trust that the CPU and RAM aren't actively help

by a1369209993 9y ago

If you're willing to accept that durability is a lie, and also to compromise (possibly a lot) on performance and trust that the CPU and RAM aren't actively helping the disk to sabatoge you, you can make a fairly full-featured ACI database in a weekend using Merkle trees and a single-threaded rendevous serializer. The hard part is getting those guarantees while also having latency and throughput vaguely comparable to naive file writes. (I'd add something about a better API than assembly language, but database people decided to imitate COBOL, so I can't really criticise assembly with a straight face.)

mmalone 9y ago
Log structured merge is the last piece of your architecture. It solves the latency part. It can also solve your durability issues, if you fsync when necessary and you understand your hardware.
- a1369209993 9y ago
  Yeah, sorry, I assumed log/journal plus reapply-on-recovery went without saying. You can't have guaranteed durability (short of blockchain-style distributed stores) because it's always possible for your database server to suffer a unrecoverable hardware failure between transmitting "transaction success" and taping out its weekly backup. If all else success, your datacenter catching fire and burning down would suffice. So it's only ever a matter of reducing (not eliminating) the probability of having to roll back commited transactions. (Tldr: fsync can't help if your hard drive melted.)
  - mmalone 9y ago
    Oh, I assumed we were talking distributed. Fsync to more than one disk. My biases :).
    - a1369209993 9y ago
      Fsync to more than one disk is great way to reduce the probability of forced rollback, but simutaneous failure is a still possible. Then again, ACI does assume you aren't having CPU/memory/etc failures, so to the extent that you count anything as guaranteed, it is probably possible to 'guarantee' durablity. My original point was more that "minimally adequate" isn't that hard compared to going from there to a high performance, general purpose system (and also a offhand snipe at SQL).

mmalone 9y ago

Log structured merge is the last piece of your architecture. It solves the latency part. It can also solve your durability issues, if you fsync when necessary and you understand your hardware.

a1369209993 9y ago
Yeah, sorry, I assumed log/journal plus reapply-on-recovery went without saying. You can't have guaranteed durability (short of blockchain-style distributed stores) because it's always possible for your database server to suffer a unrecoverable hardware failure between transmitting "transaction success" and taping out its weekly backup. If all else success, your datacenter catching fire and burning down would suffice. So it's only ever a matter of reducing (not eliminating) the probability of having to roll back commited transactions. (Tldr: fsync can't help if your hard drive melted.)
- mmalone 9y ago
  Oh, I assumed we were talking distributed. Fsync to more than one disk. My biases :).
  - a1369209993 9y ago
    Fsync to more than one disk is great way to reduce the probability of forced rollback, but simutaneous failure is a still possible. Then again, ACI does assume you aren't having CPU/memory/etc failures, so to the extent that you count anything as guaranteed, it is probably possible to 'guarantee' durablity. My original point was more that "minimally adequate" isn't that hard compared to going from there to a high performance, general purpose system (and also a offhand snipe at SQL).