4 ms·

>Shutdown sequences include freeing up resources that hold on to file descriptors or other system resources that may leak if not cleared correctly. Startup wor

>Shutdown sequences include freeing up resources that hold on to file descriptors or other system resources that may leak if not cleared correctly. Startup workflow is important, and it is nice to have an abstraction that makes it easier. I am less convinced that shutdown workflow is important, and may be more trouble than it is worth. On modern operating systems, the OS will handle process shutdown and clean up system resources such as file descriptors etc. In addition, with things like the OOM killer, etc, the process may not have a chance to actually clean up. Trying to engineer graceful shutdowns is extra code with subtle edge cases (like circular dependencies), and generally, less well tested and exercised. Finally, the operating system needs to be robust against leaking resources due to processes crashing, anyway. So on the balance, I think it is worthwhile to do a startup workflow and engineer your service so that you can always crash safely without messing up your system or leaking resources.

liuliu 6y ago
Agree in principle (you shouldn't rely on shutdown sequence to do correct thing). It is still nice to have a grace shutdown option. For some cases, it is just a little bit "nicer". Imagining you have one service instance need to shutdown for either migration or maintaince, it is just a bit nicer to report to your service discovery manager that this instance will go offline. While service discovery manager will eventually figure it out by its own through heartbeat monitoring, proactively notifying the service discovery manager would reduce a little bit on the failed retries etc.
jfkebwjsbx 6y ago
There are resources that won't be cleanly cleared on crash. Others won't be removed at all. There is also the debugging analysis and the monitoring sides of it too. Thus avoiding shutdown logic is sloppy engineering to say the least. Same for ignoring the need to design the system to avoid crashes to begin with, including OOM ones. This has nothing to do with ensuring your system is OK on crash, which is another side of robustness.
- hamandcheese 6y ago
  If there are resources that won’t be cleared on crash, then how can you say a system is OK to crash?
  - pjmlp 6y ago
    Assuming the system was designed in such a way that relaunching the crash process will recover those resources for the freshly started process. Now if the error cause has led to an endless crash cycle, then not.
hamandcheese 6y ago
I came here to say more or less the same thing - my understanding was that system resources (like memory allocations, file descriptors) would be freed when the process exits. That said, I still see the value of doing graceful shutdowns.
- pjmlp 6y ago
  Not necessarily, for example IPC connection points depending on what kind they are.
  - twic 6y ago
    Ah yes, the various highly cursed SysV IPC constructs. And named pipes. Any others? A communication channel which survives a process exit is fine as long as the next process start can pick it up again. You can do that with the SysV things and named pipes. Of course, you can also not do that - i remember the first program i wrote to play with SysV message queues created a fresh private queue each time, and didn't delete it on exit, so i ended up with piles of anonymous queues i couldn't delete. I had to go and cry to a system administrator who knew how to get rid of them.
PaulRobinson 6y ago
If your code is crashing more often than it is gracefully shutting down, you have a problem. Yes, you need to engineer for the crash scenario, but you shouldn't be relying on that in the same way you shouldn't tell users the only way to stop your software is to `kill -9 [PID]`. In a normal deployment it might be useful to use shutdown parts of code to signal to orchestrators that the operation has been completed successfully so they can take further steps, for example. Everybody's circumstance is different, but I definitely recommend thinking more about shutdown workflows.
- twic 6y ago
  > If your code is crashing more often than it is gracefully shutting down, you have a problem. You have this exactly backwards. If your software crashes rarely, you have a problem - when it does crash. Because then it's doing something you aren't routinely and thoroughly testing by doing it all the time. This is an old, counter-intuitive, but powerful idea - if you're not familiar with it, start with this paper: https://dslab.epfl.ch/pubs/crashonly.pdf https://dslab.epfl.ch/pubs/crashonly.pdf
danpalmer 6y ago
Surely a webserver at least wants to finish serving in-flight requests?
jmull 6y ago
For some things fine, or at least you can sorta get away with it. But in scenarios where you want/need to avoid service interruptions that's not good. The basic scenario is when you have an instance of your service you want to shut down for whatever reason (maintenance, update): (1) start a new instance of your service; (2) route requests to the new instance; (3) shutdown the old instance. In (3) you'd prefer not to just kill any requests in-progress. It's also nice to flush write-buffers (can be used for high-write-frequency but not critical data -- perhaps stats collection or other informational data). I'd definitely keep shutdown simple, but you don't need to go without.

liuliu 6y ago

Agree in principle (you shouldn't rely on shutdown sequence to do correct thing). It is still nice to have a grace shutdown option. For some cases, it is just a little bit "nicer". Imagining you have one service instance need to shutdown for either migration or maintaince, it is just a bit nicer to report to your service discovery manager that this instance will go offline. While service discovery manager will eventually figure it out by its own through heartbeat monitoring, proactively notifying the service discovery manager would reduce a little bit on the failed retries etc.

jfkebwjsbx 6y ago

There are resources that won't be cleanly cleared on crash. Others won't be removed at all. There is also the debugging analysis and the monitoring sides of it too. Thus avoiding shutdown logic is sloppy engineering to say the least. Same for ignoring the need to design the system to avoid crashes to begin with, including OOM ones. This has nothing to do with ensuring your system is OK on crash, which is another side of robustness.

hamandcheese 6y ago
If there are resources that won’t be cleared on crash, then how can you say a system is OK to crash?
- pjmlp 6y ago
  Assuming the system was designed in such a way that relaunching the crash process will recover those resources for the freshly started process. Now if the error cause has led to an endless crash cycle, then not.

hamandcheese 6y ago

I came here to say more or less the same thing - my understanding was that system resources (like memory allocations, file descriptors) would be freed when the process exits. That said, I still see the value of doing graceful shutdowns.

pjmlp 6y ago
Not necessarily, for example IPC connection points depending on what kind they are.
- twic 6y ago
  Ah yes, the various highly cursed SysV IPC constructs. And named pipes. Any others? A communication channel which survives a process exit is fine as long as the next process start can pick it up again. You can do that with the SysV things and named pipes. Of course, you can also not do that - i remember the first program i wrote to play with SysV message queues created a fresh private queue each time, and didn't delete it on exit, so i ended up with piles of anonymous queues i couldn't delete. I had to go and cry to a system administrator who knew how to get rid of them.

PaulRobinson 6y ago

If your code is crashing more often than it is gracefully shutting down, you have a problem. Yes, you need to engineer for the crash scenario, but you shouldn't be relying on that in the same way you shouldn't tell users the only way to stop your software is to `kill -9 [PID]`. In a normal deployment it might be useful to use shutdown parts of code to signal to orchestrators that the operation has been completed successfully so they can take further steps, for example. Everybody's circumstance is different, but I definitely recommend thinking more about shutdown workflows.

twic 6y ago
> If your code is crashing more often than it is gracefully shutting down, you have a problem. You have this exactly backwards. If your software crashes rarely, you have a problem - when it does crash. Because then it's doing something you aren't routinely and thoroughly testing by doing it all the time. This is an old, counter-intuitive, but powerful idea - if you're not familiar with it, start with this paper: https://dslab.epfl.ch/pubs/crashonly.pdf https://dslab.epfl.ch/pubs/crashonly.pdf

danpalmer 6y ago

Surely a webserver at least wants to finish serving in-flight requests?

jmull 6y ago

For some things fine, or at least you can sorta get away with it. But in scenarios where you want/need to avoid service interruptions that's not good. The basic scenario is when you have an instance of your service you want to shut down for whatever reason (maintenance, update): (1) start a new instance of your service; (2) route requests to the new instance; (3) shutdown the old instance. In (3) you'd prefer not to just kill any requests in-progress. It's also nice to flush write-buffers (can be used for high-write-frequency but not critical data -- perhaps stats collection or other informational data). I'd definitely keep shutdown simple, but you don't need to go without.