3 ms·
You missed the part where they described the root issue: > Rather the failure was caused by the growth in the number of servers in our datacenters. Their con
by toast0 5y ago
You missed the part where they described the root issue:
> Rather the failure was caused by the growth in the number of servers in our datacenters.
Their config server ran out of sockets and screwed up everything. Or something like that. Too many servers = nothing works has many ways to unfold and isn't obvious the first time.
- rawoke083600 5y ago>too many servers... My best advise about scaling, I once read (might have been right here on HN) is many devs know their primary bottleneck. Be it the "master-db", 3rd-party-api, login-service etc. But few devs knows the "2nd slowest/bottleneck" in their service stack. Not saying this was the case here, but always good to think (at least has a mental exercise) about what comes after your primary-bottleneck. If it's a long standing bottleneck, one day it gets fixed (sw update, better arch) and suddenly everything goes down since now you hitting a new bottleneck you never even thought about scaling.