4 ms·
Ehhhh, doesn't align with my experience. I think FDB is actually really poorly tested. When I was evaluating it for replacement of the metadata key-value store
by jeffbee 5y ago
Ehhhh, doesn't align with my experience. I think FDB is actually really poorly tested. When I was evaluating it for replacement of the metadata key-value store at a major, public web services company we found that injecting faults into virtual NVMe devices on individual replicas would cause corrupt results returned to clients. We also found that it would just crash-loop on Linux systems with huge pages, because although someone from the project had written a huge-page-aware C++ allocator "for performance", evidently nobody had ever actually tried to use it, including the author.
It's also really, really weird that their non-scalable architecture hits a brick wall at 25 machines. Ignoring the correctness flaws, it only works if you can either design around that limit by sharding, and never off cross-shard transactions, or if you can assure yourself that your use case will never outgrow half a rack of equipment.
- bpicolo 5y agoWhat were the strong contenders?
- rbranson 5y agoWere there other distributed databases that did pass the fault injection testing?
- jeffbee 5y agoThere weren't any, which is why that particular shop elected to roll their own distributed system on top of rocks. In general I think people who think they want to do FoundationDB owe themselves a serious contemplation of the cost/benefit of using Cloud Spanner instead. Obviously you cannot do your own fault injection testing of Spanner, but it does have end-to-end checksums.
- sigstoat 5y ago> There weren't any, which is why that particular shop elected to roll their own distributed system on top of rocks. that's nuts. rocks could've been added as a storage engine to fdb far more easily.
- ryanworl 5y agoThis is currently in progress right now. https://github.com/apple/foundationdb/blob/e7d7b39f12afa8ea21aeb94972fc79cd7b5edbab/fdbserver/KeyValueStoreRocksDB.actor.cpp https://github.com/apple/foundationdb/blob/e7d7b39f12afa8ea2...
- jeffbee 5y agoFor the record, I said the same thing. But it's a management problem because on the one hand you have a known open project with demonstrable flaws, and on the other you have your own in-house developers and you will tend to discount the bugs they haven't written yet. But, also for the same record, thinking you can implement a reliable, globally-replicated key-value store on top of FoundationDB that is cheaper and better than Cloud Spanner may be evidence of the same cognitive bias.
- sigstoat 5y ago> But, also for the same record, thinking you can implement a reliable, globally-replicated key-value store on top of FoundationDB that is cheaper and better than Cloud Spanner may be evidence of the same cognitive bias. man, good thing nobody made any claim like that.
- fnordpiglet 5y agoCan you fix a point in time? Software evolves and I think a point I saw is that it wasn’t well tested then they changed once production workloads told them it needs to change.