4 ms·
Ahaha. Reminds me of when I worked there. One day a large service tanked in some datacenter because BigTable replication in that location just stopped. Digging
by thu2111 6y ago
Ahaha. Reminds me of when I worked there. One day a large service tanked in some datacenter because BigTable replication in that location just stopped. Digging in, it turned out the BigTable should have been replicating from YQ but had started trying to use QQ instead, which didn't exist. Q being one bit away from Y. Or it was something like that, I don't remember exactly. There'd been a bit flip in the exact part of memory that contained the name of the database cluster to replicate from!
- jeffbee 6y agoBeautiful. As long as we're swapping war stories there was once a frontend server at that company which died of bit flip related causes but just before it did it managed to charge some product group for 2^63 bytes worth of network traffic in the internal quota system, which set off every budgetary alarm that capacity planners had.