4 ms·

Reads familiar? The outage pattern is identical to the one impacting GitHub January 8 [1] with my comment in the thread discussing the root cause [2]. Systems g

by jjoe 13y ago

Reads familiar? The outage pattern is identical to the one impacting GitHub January 8 [1] with my comment in the thread discussing the root cause [2]. Systems generating config files and then pushing them out to services within the infrastructure without proper checking and linting. In google's case, I just can't believe their systems are so delicately integrated and such critical component so botched up. [1] https://github.com/blog/1759-dns-outage-post-mortem https://github.com/blog/1759-dns-outage-post-mortem [2] https://news.ycombinator.com/item?id=7081913 https://news.ycombinator.com/item?id=7081913

brianr 13y ago
Also reminded me of this CloudFlare outage from last year, where a bad configuration rule caused their routers to crash: http://blog.cloudflare.com/todays-outage-post-mortem-82515 http://blog.cloudflare.com/todays-outage-post-mortem-82515 In that instance the rule was written by a human, but deployed automatically to their entire infrastructure before they could notice the problem it caused.
antimora 13y ago
Sounds like a problem what Dropbox had experienced recently too: https://tech.dropbox.com/2014/01/outage-post-mortem/ https://tech.dropbox.com/2014/01/outage-post-mortem/
rachelbythebay 13y ago
I wonder if it was gen_aggregator.py again, 6 years later.
bigiain 13y ago
Modern DevOps - the ability to deploy catastrophic mistakes globally and instantaneously! (what could possibly go wrong?)
magicalist 13y ago
Can you be more specific? They don't sound very similar at all, though to be fair, google didn't provide that many details here. They both involve generated configuration files? > Systems generating config files and then pushing them out to services within the infrastructure without proper checking and linting. It's not really that easy, as "proper checking and linting" might as well be phrased as "sufficiently smart checking and linting". You can have amazing checking and linting and still let a bug pass through.
rbanffy 13y ago
I once experienced a similar issue (on a much smaller scale, obviously). Our DNS zone files were generated by a programand, thanks to a subtle bug, we went offline. The catch was that the generated files were valid and no amount of linting would have saved us from syntactically perfect wrong data. The checks need to be quite sophisticated. We patched the generator, but never got to implement a test deploy mechanism to ensure the config made sense before deployment to a real production environment.

by jjoe 13y ago

brianr 13y ago

Also reminded me of this CloudFlare outage from last year, where a bad configuration rule caused their routers to crash: http://blog.cloudflare.com/todays-outage-post-mortem-82515 http://blog.cloudflare.com/todays-outage-post-mortem-82515 In that instance the rule was written by a human, but deployed automatically to their entire infrastructure before they could notice the problem it caused.

antimora 13y ago

Sounds like a problem what Dropbox had experienced recently too: https://tech.dropbox.com/2014/01/outage-post-mortem/ https://tech.dropbox.com/2014/01/outage-post-mortem/

rachelbythebay 13y ago

I wonder if it was gen_aggregator.py again, 6 years later.

bigiain 13y ago

Modern DevOps - the ability to deploy catastrophic mistakes globally and instantaneously! (what could possibly go wrong?)

magicalist 13y ago

Can you be more specific? They don't sound very similar at all, though to be fair, google didn't provide that many details here. They both involve generated configuration files? > Systems generating config files and then pushing them out to services within the infrastructure without proper checking and linting. It's not really that easy, as "proper checking and linting" might as well be phrased as "sufficiently smart checking and linting". You can have amazing checking and linting and still let a bug pass through.

rbanffy 13y ago

I once experienced a similar issue (on a much smaller scale, obviously). Our DNS zone files were generated by a programand, thanks to a subtle bug, we went offline. The catch was that the generated files were valid and no amount of linting would have saved us from syntactically perfect wrong data. The checks need to be quite sophisticated. We patched the generator, but never got to implement a test deploy mechanism to ensure the config made sense before deployment to a real production environment.