3 ms·

But it can also make things more stable. I laughed when 30 years ago people recommended to restart Windows NT to make it more stable. I now have some Go website

But it can also make things more stable. I laughed when 30 years ago people recommended to restart Windows NT to make it more stable. I now have some Go websites running for a long time with daily Systemd restarts.

justinclift 3y ago
> I now have some Go websites running for a long time with daily Systemd restarts. Are the restarts needed because there's malicious over-size content being sent to it, trying to exhaust the server resources? If so, then "http.MaxBytesReader" might be helpful: https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f59b25fb78a576c0cac429323cad/webui/main.go#L5447 https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f...
- ilyt 3y ago
  I'm guessing he forgot null check
gbraad 3y ago
Edit: posted this as an independent comment
nonameiguess 3y ago
I believe this is one of the claims of chaos engineering. If you randomly restart services, you have to build them in a way that they're resilient to random failures. If you do this to entire servers, VM, or containers, whatever your unit of OS userspace is, you can also kill an attacker's foothold if they manage to get one. Sort of how if humans were capable of respawning in a healthy base state but with their memories intact, you could stop the spread of a disease by just killing everyone.
- hashhar 3y ago
  You'll find Altered Carbon very very interesting then. Similar concepts being explored there for human "resleeving". https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series) https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series)
regularfry 3y ago
Heroku notably does (did? It's been a while) daily restarts. I believe they started it to take away the support hassle of folks whose apps had memory leaks, but it's also a convenient mechanism to allow cycling the infrastructure without needing live migration.
KronisLV 3y ago
> I now have some Go websites running for a long time with daily Systemd restarts. I went a step further and all of my servers do staggered restarts at the start of every month (previously daily/weekly, but once I worked most stuff out, a month is enough now). No more update related issues catching me off guard when I suddenly NEED to do a restart in short order, no issues regarding manual processes or a service that doesn't start automatically after a restart/crash, or other things that you might not catch otherwise, load balancing etc. Even stuff like restarting the Swarm cluster leader/followers close to one another, yet expecting them to eventually communicate with one another properly and schedule all of the containers, pull them and launch them as needed, join them to the networks and actually communicate properly to them. Knowing that restarts are inevitable (sometimes after a crash), might as well make sure that they work properly. And when you discover a new failure mode and something indeed doesn't work, it's nice to test your monitoring and alerting, as well as learn how to deal with that particular failure.
toast0 3y ago
Or it can make things worse. I tried a load balancer once that had a reboot once a month. Not worth it when the underlying service hosts had failures more like once every three months. Round robin DNS was better than that.

justinclift 3y ago

> I now have some Go websites running for a long time with daily Systemd restarts. Are the restarts needed because there's malicious over-size content being sent to it, trying to exhaust the server resources? If so, then "http.MaxBytesReader" might be helpful: https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f59b25fb78a576c0cac429323cad/webui/main.go#L5447 https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f...

ilyt 3y ago
I'm guessing he forgot null check

gbraad 3y ago

Edit: posted this as an independent comment

nonameiguess 3y ago

I believe this is one of the claims of chaos engineering. If you randomly restart services, you have to build them in a way that they're resilient to random failures. If you do this to entire servers, VM, or containers, whatever your unit of OS userspace is, you can also kill an attacker's foothold if they manage to get one. Sort of how if humans were capable of respawning in a healthy base state but with their memories intact, you could stop the spread of a disease by just killing everyone.

hashhar 3y ago
You'll find Altered Carbon very very interesting then. Similar concepts being explored there for human "resleeving". https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series) https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series)

regularfry 3y ago

Heroku notably does (did? It's been a while) daily restarts. I believe they started it to take away the support hassle of folks whose apps had memory leaks, but it's also a convenient mechanism to allow cycling the infrastructure without needing live migration.

KronisLV 3y ago

> I now have some Go websites running for a long time with daily Systemd restarts. I went a step further and all of my servers do staggered restarts at the start of every month (previously daily/weekly, but once I worked most stuff out, a month is enough now). No more update related issues catching me off guard when I suddenly NEED to do a restart in short order, no issues regarding manual processes or a service that doesn't start automatically after a restart/crash, or other things that you might not catch otherwise, load balancing etc. Even stuff like restarting the Swarm cluster leader/followers close to one another, yet expecting them to eventually communicate with one another properly and schedule all of the containers, pull them and launch them as needed, join them to the networks and actually communicate properly to them. Knowing that restarts are inevitable (sometimes after a crash), might as well make sure that they work properly. And when you discover a new failure mode and something indeed doesn't work, it's nice to test your monitoring and alerting, as well as learn how to deal with that particular failure.

toast0 3y ago

Or it can make things worse. I tried a load balancer once that had a reboot once a month. Not worth it when the underlying service hosts had failures more like once every three months. Round robin DNS was better than that.