This is a risk you run when you have continuous deployments. Luckily, incremental roll out, sufficient unit testing, and other safeguards mitigate the problem,

This is a risk you run when you have continuous deployments. Luckily, incremental roll out, sufficient unit testing, and other safeguards mitigate the problem, but sometimes things are still going to slip through. I'd bet that significantly man more problems have occurred, but this is simply the first big, obvious one worth talking about. This goes back to what I was preaching earlier about Microsoft and Google's varied approaches to testing. If this sort of bug got burned on to a disk and shipped to a million customers, it wouldn't have been a 40-minute problem... Sadly, Microsoft's testing culture bleeds into its ailing web culture and is becoming less effective in the modern auto-update world. See: Google grows impatient, so they patch Chrome with a workaround for Hotmail. Meanwhile, the Hotmail team is waiting for the next scheduled release. Lucky for Google, Chrome is mostly useless without an active internet connection useful for pushing updates.