3 ms·
> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a f
by cortesoft 3y ago
> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server.
I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because some hardware is going to fail every day just from probability. You have to design everything for failure.
Given that, it shouldn't be hard to take servers out of production for patching and updates.
In other words, a hyperscaler is going to have less incentive to minimize down time than smaller shops.
- bradknowles 3y agoIf you've seen any of Werner Vogel's talks, you will notice that Amazon and AWS feel the same way. Any scalable service should be able to withstand the loss of some of their components and keep operating.
- vbezhenar 3y agoI don't follow. Reboot is downtime. Of course your architecture must allow for downtime if it happens, but it's lost money either way, your hardware is not doing any useful work while rebooting. So more computers you have, more money is lost. At small scale that's not significant, but at large scale that might become significant so there's more incentive to reduce downtime.
- jonhohle 3y agoA reboot, a software deployment (kernel upgrade), server replacement, etc. are all the same process. That simplifies things dramatically. You can micro-optimize the 30s it takes to reboot a server, or you can simplify a runbook to have one process for any “deployment”. Different scenarios require different things but for most “web scale” things that need to be overprovisioned anyway, I’d take the simpler process.
- nickstinemates 3y agoThese servers don't take 30s to reboot. Some servers take many minutes. It's a lot.
- bradknowles 3y agoSpin new servers up before you take the old ones down. Effectively zero loss of time for that service.
- latchkey 3y agoWorse, some just don't come back without manual intervention. Power supplies don't last forever and might run fine while the machine is on, but after a reboot... boom, gone.
- yjftsjthsd-h 3y agoI'd prefer kexec to kpatch, then
- jononor 3y agoSounds like something to fix rather than to paper over?
- tentacleuno 3y agoIsn't it more significant at smaller scale? That is, if you have less computers running to serve requests, the downtime of the singular system will be more pronounced (as opposed to rebooting one machine out of 20 in a rack).
- j16sdiz 3y agoYes, you need to overprovision the server.a little bit. But you got a much simpler process. Process ain't free either.
- cortesoft 3y agoIf it isn’t an emergency patch, we do all our maintenance at low traffic times (e.g. the middle of the night local time for the data center). Your capacity planning is based on peak traffic, so you can afford to have more machines out during low traffic times.
- MarkSweep 3y agoEdit: ignore the below numbers, I got hours and minutes confused. If you each sever spends 7.5 minutes each month rebooting, that is 1% of all your computers wasted. If you have 10,000 servers, that’s worth 100 severs. If you have 1 million servers, that’s worth 10,000 servers. If each server costs $10,000, that’s $100 million dollars of compute capacity. You can see how that amount of lost computer capacity can start to justify spending engineering time on driving down the amount of time servers are rebooting.
- sghiassy 3y agoThere’s 1,440 minutes per day. In a month with 30 days that’s 43,200 minutes per month. 7.5min/43,200min = 0.00017361 Where are you getting 1% of all your computers wasted per month???
- humanxperiments 3y ago[dead]
- MarkSweep 3y agoSorry I got hours and minutes confused and overstated the benefits. So the correct numbers would be 7.5 minutes of downtime divided by 43,200 minutes times 1,000,000 servers. That’s 173 servers wasted. That is probably still enough servers wasted to devote some engineering time to increasing utilization.
- cortesoft 3y agoCapacity is measured by peak usage, but most data centers experience a daily traffic cycle with low times at night. We just patch/upgrade our servers at the local low time, when traffic is much lower and a lot of your machines are idle.
- solatic 3y ago> You have to design everything for failure. Given that, it shouldn't be hard to take servers out of production for patching and updates. There's a big, big difference, especially at the scale of hundreds of thousands / millions of servers, between designing such that your architecture can suffer 1% of servers being offline and 10% of servers being offline. If you have 1 million servers, even if you could take 1% of them offline at once (i.e. 10,000 servers), if it takes 5 minutes to reboot, you then need to wait 5 minutes * 100 one-percent-buckets = 500 minutes, or 8.3 hours to do a full patch. When you have critical security updates (like Heartbleed) you simply cannot have unpatched servers exposed to the Internet for that much time. And that's not including the amount of time it takes to actually send reboot/patch commands to 10,000 servers. The larger the bucket, the more likely a bad patch is noticed by the public, and the more likely that an ordinary traffic spike (for which your extra capacity is ordinarily there for) will overwhelm your servers (since your extra capacity is being used to handle patch rollout and rebooting). Sure, you can plan and add even more capacity to compensate, which makes it take even longer to rollout a patch, and now Finance is knocking at your door wondering if you really need all these servers and maybe you could decommission some of them to save money. It's a fundamentally difficult problem at hyperscaler scale.
- bradknowles 3y agoI think AWS does pretty well with that philosophy at hyper scale. AWS has millions of servers in a single AZ.