18 ms·

I upload a file to one server. Memory flips a bit before it is flushed to disk, and then the file is replicated to other servers. Now every server has a bad co

• lmm 11y ago
  > I upload a file to one server. Memory flips a bit before it is flushed to disk, and then the file is replicated to other servers. I explicitly excluded this case when I said "if the client calls the cluster nodes individually". > Or even more fun: I do an OS update and something is slightly corrupted -- enough to harm data being processed, but not enough to crash applications. Fun! Any such OS corruption (already an unlikely scenario) would almost certainly manifest as the equivalent of frequent memory errors and be handled by the same mechanism.
  • feld 11y ago
    "almost certainly" means "I've never experienced the joy of this kind of problem" People see this stuff in the real world. It sucks.
    • lmm 11y ago
      I've seen it. The most fun case I remember was where a copy of ed got somehow corrupted and worked fine as far as I could tell but somehow broke the TeX build process on that machine. No doubt it sucks, but what's going to manifest that's not going to be handled by that protocol? Either you get consistent corruption, in which case you notice quickly, or you get sporadic corruption, in which case it's handled by the redundancy with the rest of the cluster and eventually you notice that one node's failure rates are higher than the others. What other failure modes even are there?