3 ms·

> And then you run across network failures. Missed events. Doubled events. Events that arrived while a database was down. Events that were missed due to a minor

> And then you run across network failures. Missed events. Doubled events. Events that arrived while a database was down. Events that were missed due to a minor software bug. And so on. You layer on fixes, and the system develops more complex edge cases, and more subtle bugs. Sure...if all the ilities weren't accounted for in the original design. Async or out-of-process event systems have increased points of failure. If those aren't accounted for, then yes -- problems occur. Doesn't make an anti-pattern, though.

btilly 10y ago
Sure...if all the ilities weren't accounted for in the original design. Virtually nobody is able to account for everything in the original design. Convincing yourself that you got it right is easy. Actually getting it right is HARD. It is possible. For example zookeeper seems to have. But your odds of success are very, very low. See https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen for many, many examples of competent people who thought they had it right, being proven wrong. Over and over again.
- jroseattle 10y ago
  You're right -- getting _everything_ right out of the gate is impossible. > You layer on fixes, and the system develops more complex edge cases, and more subtle bugs. "Accounting for" != implementing to address. Failure is expected in distributed systems. If your architecture allows no room to address future failure, that's the problem.
  - btilly 10y ago
    Failure is not only expected in distributed systems, it is mathematically impossible to avoid. The best that you can do is document your failure modes and what guarantees you will provide despite that. Unfortunately, as aphyr proves over and over again, is that the guarantees we are given for how distributed software is supposed to work don't hold. Over and over again, across virtually every piece of well-known distributed piece of software. And my experience is that in house software is at least an order of magnitude worse. However you think your software will work, it doesn't.

btilly 10y ago

Sure...if all the ilities weren't accounted for in the original design. Virtually nobody is able to account for everything in the original design. Convincing yourself that you got it right is easy. Actually getting it right is HARD. It is possible. For example zookeeper seems to have. But your odds of success are very, very low. See https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen for many, many examples of competent people who thought they had it right, being proven wrong. Over and over again.

jroseattle 10y ago
You're right -- getting _everything_ right out of the gate is impossible. > You layer on fixes, and the system develops more complex edge cases, and more subtle bugs. "Accounting for" != implementing to address. Failure is expected in distributed systems. If your architecture allows no room to address future failure, that's the problem.
- btilly 10y ago
  Failure is not only expected in distributed systems, it is mathematically impossible to avoid. The best that you can do is document your failure modes and what guarantees you will provide despite that. Unfortunately, as aphyr proves over and over again, is that the guarantees we are given for how distributed software is supposed to work don't hold. Over and over again, across virtually every piece of well-known distributed piece of software. And my experience is that in house software is at least an order of magnitude worse. However you think your software will work, it doesn't.