The worst kinds of bugs are the ones that only happen in production, when you have some crazy set up with a load balancer, a CDN and/or Varnish, and a few app s

The worst kinds of bugs are the ones that only happen in production, when you have some crazy set up with a load balancer, a CDN and/or Varnish, and a few app servers, high traffic, middle of the day, and a bunch of executives hovering. I have seen a crazy bug that happened only on one of the identical app servers. Turns out SVN export didn't perform correctly (still don't know why), and one of the files was messed up. That was not easy to find. I have seen Apache ignore updated PHP files, and serving old code, which should never happen. This a real bitch to debug.