5 ms·

I forget the name, but I interviewed at a medical technology company a while back. They said they failed over a data center every single day and left it offline

by leros 5y ago

I forget the name, but I interviewed at a medical technology company a while back. They said they failed over a data center every single day and left it offline for a while. They needed to make sure their redundancy was super reliable so they failed over daily to ensure they engineered everything super robustly.

exdsq 5y ago
I know this can be a good idea because it helps you catch stuff, but man how could you convince patients that you're a reliable company if you did lose something by deliberately turning stuff off and on again. I wonder if you could argue it's negligence. Every day seems like you're just inviting a mistake at some point!
- SteveNuts 5y ago
  You could say the same thing about never testing your DR/backup capabilities. That's way more negligent.
- sverhagen 5y ago
  Doesn't routine make you better at things? (And since everybody is now going to share war stories... When I worked at Con-way, now XPO Logistics, they had the pop-tart incident, where a kitchen mishap triggered fire regulations that mandated the data center be taken down, and since they had absolutely *zero* routine in restarting their data center, folks got to debug the backup plans in real-time, until two weeks later all service had been restored. Kind of. Me suggesting we should test this more often was only perceived as a joke in bad taste. I thought it was a swell idea, since our Sundays were scheduled downtime anyway. Yep, the whole day. Every one or two weeks. Suffice to say, it wasn't a very long stint.)
  - edmundsauto 5y ago
    People improve with routine when the environment is static, or within certain ranges. Routine degrades performance when previous assumptions shift (or are changed by dynamic agents).
- makeitdouble 5y ago
  The gamble is to have small, contained issues you can deal with in a timely manner, vs full scale propagated failures you'd have to deal with at the worse time ever. It's like accidents during fire drills, they happen, yet it's worth doing all things considered.
- notjustanymike 5y ago
  You never want to use the parachute, but should always know it works.
- leros 5y ago
  That's the whole point. Every feature they build absolutely has to take failure into account so they do. Whereas if you only failed in emergencies, you would find stuff that didn't handle the failure well.
  - exdsq 5y ago
    Sure I get this in principle but if you keep rebooting something can you instigate errors that are very unlikely to normally occur, by increasing the probability? Feels like you're going to trip an edge case maybe even lower in the hardware stack than your software if you do this so often.
    - bobthebuilders 5y ago
      I mean restarting hardware once a day isn't going to hurt the hardware (indeed its the norm in some places to shut down many racks after business hours), and software will probably only see less bugs.
acid__ 5y ago
It'd be funny if they ended up with bugs in their systems that only show up if the systems aren't turned off often enough.
- swsieber 5y ago
  This is very likely. I worked for a company where started deploying code less frequently and we discovered a few uptime related issues.
- gkop 5y ago
  Heroku used to cycle every dyno approximately once a day (for all I know, it still does…). Don’t ask me how I know my apps were free of memory leaks.
  - ceejayoz 5y ago
    They still do that.
- leros 5y ago
  This reminds me of the time I "temporarily" fixed a memory leak in our servers by writing a cron job that rebooted them every 20 minutes. And then I forgot about it.
  - philliphaydon 5y ago
    Ah memories. We have a web service in IIS which is used to generate some HTML and take a screen grab of it to generate images to display of some data. The library was created in .net framework 2.0 and no longer exists or is maintained. It’s been on the list for 8 years to replace… It has a memory leak which causes IIS to consume gigs of data over time, resulting in IIS completely restarting itself and causing all hosted sites to go offline for 30s. so I set the IIS pool to recycle every hour and it hasn’t caused any issues in ~ 5 years. Due to this I feel like it’s dropped even lower in priority. :(
    - leros 5y ago
XorNot 5y ago
This is pretty much my infrastructure ideology now: everything, everything within my control has a well-defined, short lifetime (ideally less then a day).
andi999 5y ago
Does their redundency extend to two failures at the same time? If not then having one intentional failure per day is risky.
- bobbylarrybobby 5y ago
  Oh come on, what are the odds a second system fails right when the first one is taken offline? /s
  - andi999 5y ago
    100% :-)

by leros 5y ago

exdsq 5y ago

I know this can be a good idea because it helps you catch stuff, but man how could you convince patients that you're a reliable company if you did lose something by deliberately turning stuff off and on again. I wonder if you could argue it's negligence. Every day seems like you're just inviting a mistake at some point!

SteveNuts 5y ago
You could say the same thing about never testing your DR/backup capabilities. That's way more negligent.
sverhagen 5y ago
Doesn't routine make you better at things? (And since everybody is now going to share war stories... When I worked at Con-way, now XPO Logistics, they had the pop-tart incident, where a kitchen mishap triggered fire regulations that mandated the data center be taken down, and since they had absolutely *zero* routine in restarting their data center, folks got to debug the backup plans in real-time, until two weeks later all service had been restored. Kind of. Me suggesting we should test this more often was only perceived as a joke in bad taste. I thought it was a swell idea, since our Sundays were scheduled downtime anyway. Yep, the whole day. Every one or two weeks. Suffice to say, it wasn't a very long stint.)
- edmundsauto 5y ago
  People improve with routine when the environment is static, or within certain ranges. Routine degrades performance when previous assumptions shift (or are changed by dynamic agents).
makeitdouble 5y ago
The gamble is to have small, contained issues you can deal with in a timely manner, vs full scale propagated failures you'd have to deal with at the worse time ever. It's like accidents during fire drills, they happen, yet it's worth doing all things considered.
notjustanymike 5y ago
You never want to use the parachute, but should always know it works.
leros 5y ago
That's the whole point. Every feature they build absolutely has to take failure into account so they do. Whereas if you only failed in emergencies, you would find stuff that didn't handle the failure well.
- exdsq 5y ago
  Sure I get this in principle but if you keep rebooting something can you instigate errors that are very unlikely to normally occur, by increasing the probability? Feels like you're going to trip an edge case maybe even lower in the hardware stack than your software if you do this so often.
  - bobthebuilders 5y ago
    I mean restarting hardware once a day isn't going to hurt the hardware (indeed its the norm in some places to shut down many racks after business hours), and software will probably only see less bugs.

acid__ 5y ago

It'd be funny if they ended up with bugs in their systems that only show up if the systems aren't turned off often enough.

swsieber 5y ago
This is very likely. I worked for a company where started deploying code less frequently and we discovered a few uptime related issues.
gkop 5y ago
Heroku used to cycle every dyno approximately once a day (for all I know, it still does…). Don’t ask me how I know my apps were free of memory leaks.
- ceejayoz 5y ago
  They still do that.
leros 5y ago
This reminds me of the time I "temporarily" fixed a memory leak in our servers by writing a cron job that rebooted them every 20 minutes. And then I forgot about it.
- philliphaydon 5y ago
  Ah memories. We have a web service in IIS which is used to generate some HTML and take a screen grab of it to generate images to display of some data. The library was created in .net framework 2.0 and no longer exists or is maintained. It’s been on the list for 8 years to replace… It has a memory leak which causes IIS to consume gigs of data over time, resulting in IIS completely restarting itself and causing all hosted sites to go offline for 30s. so I set the IIS pool to recycle every hour and it hasn’t caused any issues in ~ 5 years. Due to this I feel like it’s dropped even lower in priority. :(
  - leros 5y ago

XorNot 5y ago

This is pretty much my infrastructure ideology now: everything, everything within my control has a well-defined, short lifetime (ideally less then a day).

andi999 5y ago

Does their redundency extend to two failures at the same time? If not then having one intentional failure per day is risky.

bobbylarrybobby 5y ago
Oh come on, what are the odds a second system fails right when the first one is taken offline? /s
- andi999 5y ago
  100% :-)