4 ms·

It'd be funny if they ended up with bugs in their systems that only show up if the systems aren't turned off often enough.

by acid__ 4y ago

swsieber 4y ago
This is very likely. I worked for a company where started deploying code less frequently and we discovered a few uptime related issues.
gkop 4y ago
Heroku used to cycle every dyno approximately once a day (for all I know, it still does…). Don’t ask me how I know my apps were free of memory leaks.
- ceejayoz 4y ago
  They still do that.
leros 4y ago
This reminds me of the time I "temporarily" fixed a memory leak in our servers by writing a cron job that rebooted them every 20 minutes. And then I forgot about it.
- philliphaydon 4y ago
  Ah memories. We have a web service in IIS which is used to generate some HTML and take a screen grab of it to generate images to display of some data. The library was created in .net framework 2.0 and no longer exists or is maintained. It’s been on the list for 8 years to replace… It has a memory leak which causes IIS to consume gigs of data over time, resulting in IIS completely restarting itself and causing all hosted sites to go offline for 30s. so I set the IIS pool to recycle every hour and it hasn’t caused any issues in ~ 5 years. Due to this I feel like it’s dropped even lower in priority. :(
  - leros 4y ago
    That's the dirty pragmatic side of engineering. Technically it is fixed.
- acid__ 4y ago
  Hah, yes, I've unironically used https://github.com/kzk/unicorn-worker-killer https://github.com/kzk/unicorn-worker-killer in production before. ~900 stars on Github...
bigiain 4y ago
Like, say, an airliner? https://www.theregister.com/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
- spicybright 4y ago
  It's imperfect code, but I actually wish full reboots were more common for critical tech like this. How many times have we rebooted our machines this week to fix some weird issue? (or maybe my OS install is buggy, but still :P)
  - bigiain 4y ago
    Lemme check: $ uptime 19:14 up 42 days, 6:49, 11 users, load averages: 7.85 8.24 8.07 Zero times this week (or month).
spicybright 4y ago
That would probably be a good thing, means you can't break procedure easily!
nickm12 4y ago
I've seen this on services that do CI/CD, but institute a freeze for peak traffic events. In theory, the freeze serves the purpose of increasing capacity, since all services are serving traffic rather than updating software, and increasing reliability, since software isn't changing. However, it can then expose latent memory leaks that weren't visible when the services were cycling all the time.

It'd be funny if they ended up with bugs in their systems that only show up if the systems aren't turned off often enough.

by acid__ 4y ago

It'd be funny if they ended up with bugs in their systems that only show up if the systems aren't turned off often enough.

swsieber 4y ago

This is very likely. I worked for a company where started deploying code less frequently and we discovered a few uptime related issues.

gkop 4y ago

Heroku used to cycle every dyno approximately once a day (for all I know, it still does…). Don’t ask me how I know my apps were free of memory leaks.

ceejayoz 4y ago
They still do that.

leros 4y ago

This reminds me of the time I "temporarily" fixed a memory leak in our servers by writing a cron job that rebooted them every 20 minutes. And then I forgot about it.

philliphaydon 4y ago
Ah memories. We have a web service in IIS which is used to generate some HTML and take a screen grab of it to generate images to display of some data. The library was created in .net framework 2.0 and no longer exists or is maintained. It’s been on the list for 8 years to replace… It has a memory leak which causes IIS to consume gigs of data over time, resulting in IIS completely restarting itself and causing all hosted sites to go offline for 30s. so I set the IIS pool to recycle every hour and it hasn’t caused any issues in ~ 5 years. Due to this I feel like it’s dropped even lower in priority. :(
- leros 4y ago
  That's the dirty pragmatic side of engineering. Technically it is fixed.
acid__ 4y ago
Hah, yes, I've unironically used https://github.com/kzk/unicorn-worker-killer https://github.com/kzk/unicorn-worker-killer in production before. ~900 stars on Github...

bigiain 4y ago

Like, say, an airliner? https://www.theregister.com/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ https://www.theregister.com/2020/04/02/boeing_787_power_cycl...

spicybright 4y ago
It's imperfect code, but I actually wish full reboots were more common for critical tech like this. How many times have we rebooted our machines this week to fix some weird issue? (or maybe my OS install is buggy, but still :P)
- bigiain 4y ago
  Lemme check: $ uptime 19:14 up 42 days, 6:49, 11 users, load averages: 7.85 8.24 8.07 Zero times this week (or month).

spicybright 4y ago

That would probably be a good thing, means you can't break procedure easily!

nickm12 4y ago

I've seen this on services that do CI/CD, but institute a freeze for peak traffic events. In theory, the freeze serves the purpose of increasing capacity, since all services are serving traffic rather than updating software, and increasing reliability, since software isn't changing. However, it can then expose latent memory leaks that weren't visible when the services were cycling all the time.