From an experience of working on a distributed enterprise messaging server (one of the top sellers with some super advanced stuff like automatic failover, a cli

From an experience of working on a distributed enterprise messaging server (one of the top sellers with some super advanced stuff like automatic failover, a client opaque to errors, TX and HA), the conclusion is that what could go wrong will go wrong in a distributed system. My favorite were distributed deadlocks; losing messages with certain cluster configurations under specific conditions was close second. If you support multiple OSes, they are all very different at the very bottom level and sometimes socket states and their transitions are just plainly weird (and undocumented). Fun is also caused by the "observer effect" - if you try to maximize throughput and something goes wrong, once you enable logging, the problem disappears (and of course throughput goes way down). Transaction processing is also yummy, 2PC obviously can't be recovered in all possible failure cases, 3PC is sensitive to network partitioning, checkpointing might run into local storage problem etc. We had our own distributed testing suite as well as standard conformance tests, yet these were badly insufficient and were more like firing off a shotgun and hoping it would help.