3 ms·
1 Million hosts over 45 days = 15 min per host That's a very realistic/optimistic number (especially as you do want to wait for all services to be running and
by raverbashing 3y ago
1 Million hosts over 45 days = 15 min per host
That's a very realistic/optimistic number (especially as you do want to wait for all services to be running and marked as healthy)
"Oh but you can batch this" sure, but you don't want too much of a big batch that will make your service slow or want to risk shooting yourself in the foot - like rebooting your whole control plane then figuring out it doesn't work like that
(The 45 days is probably an estimate as well, I'm not sure they actually do that server by server)
- exitheone 3y agoThat's still ridiculously slow. I'd expect them to have hundreds of Microservices. Each one of those should be able to handle a random restart at any point in time so they should absolutely be able to restart 100s of servers concurrently without major disruptions. Hell on Facebook scale a whole-Datacenter going down should not cause service disruptions.
- Closi 3y agoThis does assume that nothing is getting broken along the way. Taking 45 days is probably more about caution and resolving issues systematically rather than pushing a big button and hoping you don’t cause issues. I’d expect them to have thousands of microservices - and you only have to find a way to break one to cause big issues.
- exitheone 3y agoRegular random crashes should be exercised regardless at Facebook scale. Not being resilient to that would be very unprofessional.
- BlackFingolfin 3y agoThat's not 15 minutes per host; that's 15 hosts per minute.