4 ms·
His logic is beyond broken. "Why do I need ECC when I can just have more servers in my cluster in case something bad happens" Umm, I don't know -- perhaps data
by feld 11y ago
His logic is beyond broken. "Why do I need ECC when I can just have more servers in my cluster in case something bad happens"
Umm, I don't know -- perhaps data integrity? Having multiple servers doesn't solve the "I just wrote bad data to disk" problem.
- lmm 11y agoIf the client calls the cluster nodes individually then yes it does, provided there's enough overlap. Say the client considers a write committed when it's written to 5/7 of the cluster, and a read is only valid when you've heard from 5/7 of the cluster. Then whenever you read you're guaranteed to read at least three saved values, so if they disagree you can take the majority value (and write a correction). And that's the kind of system you already need to be safe against hard drive failures (just with the numbers at 4/7 and 4/7), so it's cheap to enable. In a modern distributed system you're already effectively doing ECC at a cross-machine level, so there's little value in doing it again within each individual machine.
- Retric 11y agoDepends on how much processing you’re doing. It's not hard to get to the point where 7/7 machines all give different answers.
- xorcist 11y agoDo you end clients really write to all nodes? I know pretty much no application for which this is the case.
- lmm 11y agoSometimes. Cassandra and similar datastores follow this model. I've been out of the web game a while but it seems like a sensible way to do a "single page application" - indeed it seems like the only way to have true high availability, if the client only makes a single call then whatever initially receives that call (e.g. the load balancer) becomes a single point of failure.
- feld 11y agoI upload a file to one server. Memory flips a bit before it is flushed to disk, and then the file is replicated to other servers. Now every server has a bad copy of the data. Or even more fun: I do an OS update and something is slightly corrupted -- enough to harm data being processed, but not enough to crash applications. Fun!
- lmm 11y ago> I upload a file to one server. Memory flips a bit before it is flushed to disk, and then the file is replicated to other servers. I explicitly excluded this case when I said "if the client calls the cluster nodes individually". > Or even more fun: I do an OS update and something is slightly corrupted -- enough to harm data being processed, but not enough to crash applications. Fun! Any such OS corruption (already an unlikely scenario) would almost certainly manifest as the equivalent of frequent memory errors and be handled by the same mechanism.
- feld 11y ago"almost certainly" means "I've never experienced the joy of this kind of problem" People see this stuff in the real world. It sucks.
- lmm 11y agoI've seen it. The most fun case I remember was where a copy of ed got somehow corrupted and worked fine as far as I could tell but somehow broke the TeX build process on that machine. No doubt it sucks, but what's going to manifest that's not going to be handled by that protocol? Either you get consistent corruption, in which case you notice quickly, or you get sporadic corruption, in which case it's handled by the redundancy with the rest of the cluster and eventually you notice that one node's failure rates are higher than the others. What other failure modes even are there?
- undersuit 11y agoWasn't his logic more like, "I need lots of single thread performance, but the only way to get that is consumer hardware and anyways my database servers will have ECC"?