This was a core cluster db for about 150 web sites that had high profiles and user traffic. It was a long time ago and the client didn't want to pay for 20 mor

This was a core cluster db for about 150 web sites that had high profiles and user traffic. It was a long time ago and the client didn't want to pay for 20 more boxes to do something more durable. These things are always a balance between durability and speed. In this case the client wanted cheap speed. Sadly it worked very well except when people used a big hammer on it. We had replication slaves but in this incident the salves were not replicating and somehow the script checking them wasn't alerting. But even so the admin didn't check before bashing the keyboard and thus we were left with manual reconstruction from backups and other sundry sources of information.