6 ms·
I hardly ever felt so strongly about anything in software engineering, although I will concede this is an acquired taste, and it took me some time to see the ob
by m0zg 7y ago
I hardly ever felt so strongly about anything in software engineering, although I will concede this is an acquired taste, and it took me some time to see the obvious truth in this.
I will reiterate. If your programs are designed to be kill-safe, it's a waste of time to shut them down any other way. It is also harmful to your systems design, because you can't guarantee that your hardware won't give out, especially if you run on e.g. 50-100K cores (as many Google services do). You can basically be certain at that point that individual tasks (and hardware they run on) will die from time to time. Note that this only applies to shut down, not to e.g. draining traffic. That part can and should still be orderly in most systems so as not to disrupt the user experience too much, and Google does have that in their RPC stack (lame duck mode etc). For everything else you end up doing a lot of logging, checkpointing, and 2PC. For distributed systems you end up using consensus mechanisms and eliminating SPOFs.
If you rely on orderly shutdown for correctness in your distributed system, I'd like to know the name of your product so I can avoid it.
- ownagefool 7y agoI think you're talking past each other a bit here. Nothing the parent is discussing is regarding correctness, so I think your last sentence is a bit uncalled for.
- m0zg 7y agoI'm actually serious about that last bit. If your distributed system relies on guarantees it _does not have_ in order to operate correctly, one would be well advised to stay away from it.
- ownagefool 7y agoYou can be super serious about it all you want, but it's accusatory and doesn't reflect the post you responsed to, which referred to performance and user experience implications with shooting nodes in the head. Personally I think there was the formation of a really good debate. As you said, if you drain the nodes traffic before killing it, you're probably right, any costs associated with maintaining consistency is probably saved by the human aspect of just not waiting for a clean shutdown of processes at scale. But when you throw in the sass, people stop listening and we all get dumber for it.
- m0zg 7y agoBut there's no "debate" here to be had. All the high scale companies (Google, FB, Amazon, Microsoft, Netflix, others) do not rely on their distributed system nodes being able to wind down in an orderly fashion. Shit, Netflix and Google (and likely others as well) stage fault tolerance exercises, taking random nodes (or entire datacenters) out of rotation and checking if things still work. There's no way to get to five nines if you expect your program to always behave. Here's one from Netflix that will give you an ulcer: https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey Here's what Google does: https://www.usenix.org/conference/lisa15/conference-program/presentation/krishnan https://www.usenix.org/conference/lisa15/conference-program/... >> we all get dumber for it Not _all_. Only those who feel inclined to reject the obvious.
- ownagefool 7y agoAgain though, you're misrepresenting the position. But sure, lets all just be dicks to each other justified with appeals to authority. Not gonna bother though, you're obviously above being polite.
- joatmon-snoo 7y agoYou didn't read my comment at all. I explicitly said that yes, you need to design and plan for disorderly shutdown, but orderly shutdown is an order of magnitude more common and you therefore need to also account for that. We don't design products based on guarantees that we don't have, we design for the problem space. FWIW, I work at Google on distributed systems.