FWIW, we're a reasonably successful social application that's _heavy_ on real time networking; so there's always a large number of users active and our services

FWIW, we're a reasonably successful social application that's _heavy_ on real time networking; so there's always a large number of users active and our services are generally humming along at high throughput. Lots of automated tests. A good community team. Code reviews. A _rigid_ feature-branch-and-test process. A _rigid_ closed beta->open beta->release process. And we still have bugs. But things don't catch fire often, and when they do, it's usually not our fault. ;)