7 ms·

The dangers of resetting everything at once

137 points by luu 12y ago

IvyMike 12y ago
For those who aren't catching the "May 1 + 248 days" reference: http://arstechnica.com/information-technology/2015/05/01/boeing-787-dreamliners-contain-a-potentially-catastrophic-software-bug/ http://arstechnica.com/information-technology/2015/05/01/boe...
- zatkin 12y ago
  What is she referring to when she says 208 days in the parenthesized paragraph? (After the 2nd paragraph.)
  - robotmlg 12y ago
    The 208 day bug: https://www.novell.com/support/kb/doc.php?id=7009834 https://www.novell.com/support/kb/doc.php?id=7009834 The 49.7 day bug: https://support.microsoft.com/en-us/kb/216641 https://support.microsoft.com/en-us/kb/216641
    - _lce0 12y ago
      Interesting to read. Apparently it's caused by uptime being stored in a 32-bit integer which simply overflows at that point in time https://www.ibm.com/developerworks/community/blogs/anthonyv/entry/497_the_number_of_the_it_beast2?lang=en https://www.ibm.com/developerworks/community/blogs/anthonyv/...
  - ipozgaj 12y ago
    TSC overflow bug, see https://access.redhat.com/solutions/68466 https://access.redhat.com/solutions/68466
spectre256 12y ago
This is hard won advice. I've worked at places where we managed lots of systems, and we weren't quite organized enough to jot down when known issues would crop up far in the future and remember them. SSL certs specifically bit us once or twice.
- vacri 12y ago
  Stick an SSL expiry warning into your alerting system. We have one in our nagios system - checking once a day, it gives a 45-day warning for an impending expiry. As long as your alerting system allows custom alerts, you'll always be able to run such a check.
  - bigiain 12y ago
    This assumes "an alerting system". Not guaranteed a valid assumption. (In fact, anecdotal evidence suggests it's the exception rather than the rule, at lest for businesses below a certain size.)
    - mellavora 12y ago
      google calendar? My version of it does alerts.
      - mkesper 12y ago
        You'd have to update that manually, which, in case of emergency operation, will quite sure be forgotten. This has nothing to do with proper alerting.
    - jacquesm 12y ago
      That's a very big bad red flag then. If you run even a single service you need something that monitors that service and that something needs to be on a different bit of hardware and needs to be able to reach you even when it can no longer reach its own network uplink.
      - Silhouette 12y ago
        If you run even a single service you need something that monitors that service and that something needs to be on a different bit of hardware and needs to be able to reach you even when it can no longer reach its own network uplink. I think that's too much of a generalisation. If you're talking about an established public service, that you're charging real money for, where something that actually matters will be affected by even minor downtime, sure. But if you're talking about a small team or individual, running a new service that does something simple to help someone do something else, you probably have many higher prioritises than that level of monitoring and alerting, but you might still get messed around by something like all your certs expiring overnight.
hueving 12y ago
The heartbleed reference doesn't make much sense. Allowing active certs to expire has little to do with heartbleed. Either people have process in place to replace expiring certificates or they don't. It may have produced a global phenomenon where people observe multiple expired certificates around that date; however, there is absolutely no difference to an end-user of a particular site whether the cert is expired in April or October. They are both as equally bad. In response to heartbleed, a system admin would have gained no advantage by waiting to react and would have been exposing his/her users to MiTM attacks by waiting longer.
- bigiain 12y ago
  It makes sense to me - how many of the top stories here do you suppose are HeartBleed cert replacement issues (and look at the dates, and see who looks like they updated their certs days or weeks before the rest of us found out about it): https://hn.algolia.com/?query=ssl%20expired&sort=byDate&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=ssl%20expired&sort=byDate&pref... I suspect the real issue is that in "emergency situations" people get the important shit done (and replace certs immediately), but don't always do the non-emergency process type stuff that'd normally get done when doing important shit by the schedule (updating reminders about cert expiry dates). My guess is Instagram, senate.gov. gmail, and Docker (amongst others) are going to have ops people wandering around over the next few months saying "Hey, we just got the ssl cert update reminder, but someone's already renewed the cert months ago and didn't update 'the system(tm)'. What gives, people?"
- rogerbinns 12y ago
  If they replaced all certs in their organisation because they could have been compromised, then all the replacements are likely to expire at the same time. The article isn't saying that you should wait longer, but that if you do nothing then one day all of the replacements will expire and cause a massive workload to update. A fix would be to update those new certificates proactively, spreading out the workload and new expiry times. But also marking the calendar for mass expiry day because there are bound to be some that were forgotten.
flashman 12y ago
Considering Boeing said no airframes even came close to 248 days uptime, I would consider this a non-issue.
- kens 12y ago
  Also, the affected planes are now required by law to have power deactivated at an interval of no more than 120 days, so they will never get anywhere near 248 days. https://www.federalregister.gov/articles/2015/05/01/2015-10066/airworthiness-directives-the-boeing-company-airplanes https://www.federalregister.gov/articles/2015/05/01/2015-100...
  - brudgers 12y ago
    I suspect that the Federal Regulations also requires the planes not to fail at 248 days (or words to that effect). Specifications are not the thing specified.
- kwhitefoot 12y ago
  It is an issue because it suggests that both code and requirements were not adequately reviewed. Unless the requirements explicitly stated that the generators would never run for more that 2^31 x 100th of second allowing that counter to overflow without coping properly with it is a failure. I have some production code that uses ints as serial numbers with a factory method that just hands out the next one in sequence, however, I take care to ensure that there will be no negative consequences if they roll over (they exist only in debug builds).
- Silhouette 12y ago
  The problem isn't necessarily that specific issue, because now that it's known, practical measures can be taken to compensate for the risk. The problem is that the existence of that one issue demonstrates a serious failure in the tools and development processes and quality checks used to create the control software. There could therefore be other issues in the code that haven't been found yet that could cause similarly catastrophic failures. Failing to reliably identify this kind of arithmetic error is a huge deal when you're working with high-reliability software. It simply shouldn't happen, and there should have been several different stages in the development process that would have prevented it. So, the development team should have the book thrown at them until someone figures out exactly how it did happen, closes every related tool and process loophole so it can never happen again, and audits all potentially affected code to make sure the same combination of loopholes hasn't allowed any similar mistakes into production elsewhere in the system.
stevewilhelm 12y ago
> We'll also assume they have a bug which makes the machine lock up after 300 days of uptime. Nobody knows about this yet, but the bug exists. > So here's the trick: any time you see an announcement on date X of something bad that happens after item Y has been up for more than Z days, calculate what X + Z is and make a note in your calendar. That's the first possible date you should see a cluster of events beginning. What? Doesn't this assume the announcement date X was when the bug was introduced into the software (in the example the Linux OS) and installed on your servers?
- IvyMike 12y ago
  No. Let me try restating one sentence. "That's the first possible date you should see a news story or otherwise notice a cluster of events around the world as sysadmins everywhere inadvertently synchronized their systems by rebooting on day X"
- bigiain 12y ago
  No, She's assuming on date X a whole bunch of people are going to kneejerk react and reboot things (like Boeing 787s) without enough thought or putting any process in place to actually manage or mitigate the problem, then on date X+Z all those things are going to crash at the same time (possibly into many smoking holes in the ground).
  - deleted 12y ago
    [deleted]
  - stevewilhelm 12y ago
    Ah. my confusion stems from this line. > So here's the trick: any time you see an announcement on date X of something bad that happens after item Y has been up for more than Z days, calculate what X + Z is and make a note in your calendar. I thought she was making this suggestion to the 'kneejerk' admins, not us clever admins watching for failures externally.
GeorgeHahn 12y ago
Forekast event so we don't forget to look out for 787s falling from the sky: https://forekast.com/events/5547023b666b770ea12b0000 https://forekast.com/events/5547023b666b770ea12b0000
- iyn 12y ago
  Never heard of Forekast, interesting idea. There is also similar(?) site - PredictionBook (http://predictionbook.com/ http://predictionbook.com/).
Zardoz84 12y ago
Interesting. Were I work, We have scheduled a reboot every night some develop Tomcat servers. At same time, we apply automatic database scheme update and automatic updated from the latest version from our local repository. Also, users don't touch any of these servers, so not should be any problem. But I see that here is a mind-share that rebooting Tomcat server pass X time is a good idea as with time they begin to do funny things, but we don't do that.
weems 12y ago
Isn't it supposed to be 2020-02-03 00:00:00?
amelius 12y ago
Just reboot your systems on a regular basis, and time-interleaved.
perlgeek 12y ago
Instead of rebooting all the affected systems, create a monitoring check that alerts the ops team a few days or weeks before the error condition (system will hang/reboot, SSL cert expires) comes to pass. Then it's at their discretion what do (for example, decide when to reboot).
- techsupporter 12y ago
  > alerts the ops team "Everybody" knows that no good Web 3.0 company has an ops team. We're too expensive and too easily replaced with a handful of shell scripts or can just be fobbed off as another responsibility of the devs who wrote the code. (After all, if the code was perfect, there wouldn't need to be an ops team, right?) Silly rabbit, ops teams are for lumbering companies who "Just Don't Get It." Signed, Slightly miffed ops person who thinks he does a lot for his company but feels woefully underappreciated in the new IT.
  - kjs3 12y ago
    Meh...you want underappreciated? Try being in infosec. Those new-fangled companies don't think there's a thing wrong with "We'll worry about security after we get acquired...", regardless of how sensitive the data their holding is.
    - Kalium 12y ago
      Ah! Someone else understands my pain.
    - Silhouette 12y ago
      And sadly that will continue to be the case until data protection regulators have real teeth, at which point due diligence before any acquisition should obviously include a thorough audit of these areas. A potential acquisition target that hasn't looked after its data properly and is a large regulatory action waiting to happen should then, rightly, be unlikely to exit successfully until they get their house in order.
      - 12y ago
grecy 12y ago
I work at a telco with a lot of aging systems and servers. A couple of years ago we had a total power failure of our major data center and central office, including the backup gen so literally everything failed and an entire province of Canada (and most of two more) were without any telco, data, wireless. It was eye opening and extremely scary to see what happens when everything is turned off at once and you need to bring stuff back up. Some of the equipment had never been turned off, and certainly not all at once. During the next weeks we encountered many, many catch-22 scenarios like "Before we turn on System A, it needs system B to be online. B needs C, C needs A." Oh. For reference of how severe this is, paramedics were posting up at major intersections in towns and communities in case people needed help - and the radio stations in half of Canada were telling people what to do. When even 911 doesn't work you have problems.
- OldSchoolJohnny 12y ago
  As a Canadian I would love to see a citation to this because it would have been pretty big news here and I recall nothing about it.
  - grecy 12y ago
    http://www.ctvnews.ca/canada/power-outage-in-yukon-knocks-out-communication-systems-1.964656 http://www.ctvnews.ca/canada/power-outage-in-yukon-knocks-ou... It took out everything in the Yukon, and tons of stuff in NWT, Nunavut and Northern BC and Alberta.
- serve_yay 12y ago
  This is how "if it ain't broke, don't fix it" calcifies. People point to old systems like this (when they are working, that is) all the time as proof that developers are too obsessed with shiny new things. And I'm not saying that isn't true, but there is such a thing as being too conservative.
  - grecy 12y ago
    > there is such a thing as being too conservative In my experience, a telco is the definition of "too conservative".

137 points by luu 12y ago

For those who aren't catching the "May 1 + 248 days" reference: http://arstechnica.com/information-technology/2015/05/01/boeing-787-dreamliners-contain-a-potentially-catastrophic-software-bug/ http://arstechnica.com/information-technology/2015/05/01/boe...

zatkin 12y ago
What is she referring to when she says 208 days in the parenthesized paragraph? (After the 2nd paragraph.)
- robotmlg 12y ago
  The 208 day bug: https://www.novell.com/support/kb/doc.php?id=7009834 https://www.novell.com/support/kb/doc.php?id=7009834 The 49.7 day bug: https://support.microsoft.com/en-us/kb/216641 https://support.microsoft.com/en-us/kb/216641
  - _lce0 12y ago
    Interesting to read. Apparently it's caused by uptime being stored in a 32-bit integer which simply overflows at that point in time https://www.ibm.com/developerworks/community/blogs/anthonyv/entry/497_the_number_of_the_it_beast2?lang=en https://www.ibm.com/developerworks/community/blogs/anthonyv/...
- ipozgaj 12y ago
  TSC overflow bug, see https://access.redhat.com/solutions/68466 https://access.redhat.com/solutions/68466

spectre256 12y ago

This is hard won advice. I've worked at places where we managed lots of systems, and we weren't quite organized enough to jot down when known issues would crop up far in the future and remember them. SSL certs specifically bit us once or twice.

vacri 12y ago
Stick an SSL expiry warning into your alerting system. We have one in our nagios system - checking once a day, it gives a 45-day warning for an impending expiry. As long as your alerting system allows custom alerts, you'll always be able to run such a check.
- bigiain 12y ago
  This assumes "an alerting system". Not guaranteed a valid assumption. (In fact, anecdotal evidence suggests it's the exception rather than the rule, at lest for businesses below a certain size.)
  - mellavora 12y ago
    google calendar? My version of it does alerts.
    - mkesper 12y ago
      You'd have to update that manually, which, in case of emergency operation, will quite sure be forgotten. This has nothing to do with proper alerting.
  - jacquesm 12y ago
    That's a very big bad red flag then. If you run even a single service you need something that monitors that service and that something needs to be on a different bit of hardware and needs to be able to reach you even when it can no longer reach its own network uplink.
    - Silhouette 12y ago
      If you run even a single service you need something that monitors that service and that something needs to be on a different bit of hardware and needs to be able to reach you even when it can no longer reach its own network uplink. I think that's too much of a generalisation. If you're talking about an established public service, that you're charging real money for, where something that actually matters will be affected by even minor downtime, sure. But if you're talking about a small team or individual, running a new service that does something simple to help someone do something else, you probably have many higher prioritises than that level of monitoring and alerting, but you might still get messed around by something like all your certs expiring overnight.

hueving 12y ago

The heartbleed reference doesn't make much sense. Allowing active certs to expire has little to do with heartbleed. Either people have process in place to replace expiring certificates or they don't. It may have produced a global phenomenon where people observe multiple expired certificates around that date; however, there is absolutely no difference to an end-user of a particular site whether the cert is expired in April or October. They are both as equally bad. In response to heartbleed, a system admin would have gained no advantage by waiting to react and would have been exposing his/her users to MiTM attacks by waiting longer.

bigiain 12y ago
It makes sense to me - how many of the top stories here do you suppose are HeartBleed cert replacement issues (and look at the dates, and see who looks like they updated their certs days or weeks before the rest of us found out about it): https://hn.algolia.com/?query=ssl%20expired&sort=byDate&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=ssl%20expired&sort=byDate&pref... I suspect the real issue is that in "emergency situations" people get the important shit done (and replace certs immediately), but don't always do the non-emergency process type stuff that'd normally get done when doing important shit by the schedule (updating reminders about cert expiry dates). My guess is Instagram, senate.gov. gmail, and Docker (amongst others) are going to have ops people wandering around over the next few months saying "Hey, we just got the ssl cert update reminder, but someone's already renewed the cert months ago and didn't update 'the system(tm)'. What gives, people?"
rogerbinns 12y ago
If they replaced all certs in their organisation because they could have been compromised, then all the replacements are likely to expire at the same time. The article isn't saying that you should wait longer, but that if you do nothing then one day all of the replacements will expire and cause a massive workload to update. A fix would be to update those new certificates proactively, spreading out the workload and new expiry times. But also marking the calendar for mass expiry day because there are bound to be some that were forgotten.

flashman 12y ago

Considering Boeing said no airframes even came close to 248 days uptime, I would consider this a non-issue.

kens 12y ago
Also, the affected planes are now required by law to have power deactivated at an interval of no more than 120 days, so they will never get anywhere near 248 days. https://www.federalregister.gov/articles/2015/05/01/2015-10066/airworthiness-directives-the-boeing-company-airplanes https://www.federalregister.gov/articles/2015/05/01/2015-100...
- brudgers 12y ago
  I suspect that the Federal Regulations also requires the planes not to fail at 248 days (or words to that effect). Specifications are not the thing specified.
kwhitefoot 12y ago
It is an issue because it suggests that both code and requirements were not adequately reviewed. Unless the requirements explicitly stated that the generators would never run for more that 2^31 x 100th of second allowing that counter to overflow without coping properly with it is a failure. I have some production code that uses ints as serial numbers with a factory method that just hands out the next one in sequence, however, I take care to ensure that there will be no negative consequences if they roll over (they exist only in debug builds).
Silhouette 12y ago
The problem isn't necessarily that specific issue, because now that it's known, practical measures can be taken to compensate for the risk. The problem is that the existence of that one issue demonstrates a serious failure in the tools and development processes and quality checks used to create the control software. There could therefore be other issues in the code that haven't been found yet that could cause similarly catastrophic failures. Failing to reliably identify this kind of arithmetic error is a huge deal when you're working with high-reliability software. It simply shouldn't happen, and there should have been several different stages in the development process that would have prevented it. So, the development team should have the book thrown at them until someone figures out exactly how it did happen, closes every related tool and process loophole so it can never happen again, and audits all potentially affected code to make sure the same combination of loopholes hasn't allowed any similar mistakes into production elsewhere in the system.

stevewilhelm 12y ago

> We'll also assume they have a bug which makes the machine lock up after 300 days of uptime. Nobody knows about this yet, but the bug exists. > So here's the trick: any time you see an announcement on date X of something bad that happens after item Y has been up for more than Z days, calculate what X + Z is and make a note in your calendar. That's the first possible date you should see a cluster of events beginning. What? Doesn't this assume the announcement date X was when the bug was introduced into the software (in the example the Linux OS) and installed on your servers?

IvyMike 12y ago
No. Let me try restating one sentence. "That's the first possible date you should see a news story or otherwise notice a cluster of events around the world as sysadmins everywhere inadvertently synchronized their systems by rebooting on day X"
bigiain 12y ago
No, She's assuming on date X a whole bunch of people are going to kneejerk react and reboot things (like Boeing 787s) without enough thought or putting any process in place to actually manage or mitigate the problem, then on date X+Z all those things are going to crash at the same time (possibly into many smoking holes in the ground).
- deleted 12y ago
  [deleted]
- stevewilhelm 12y ago
  Ah. my confusion stems from this line. > So here's the trick: any time you see an announcement on date X of something bad that happens after item Y has been up for more than Z days, calculate what X + Z is and make a note in your calendar. I thought she was making this suggestion to the 'kneejerk' admins, not us clever admins watching for failures externally.

GeorgeHahn 12y ago

Forekast event so we don't forget to look out for 787s falling from the sky: https://forekast.com/events/5547023b666b770ea12b0000 https://forekast.com/events/5547023b666b770ea12b0000

iyn 12y ago
Never heard of Forekast, interesting idea. There is also similar(?) site - PredictionBook (http://predictionbook.com/ http://predictionbook.com/).

Zardoz84 12y ago

Interesting. Were I work, We have scheduled a reboot every night some develop Tomcat servers. At same time, we apply automatic database scheme update and automatic updated from the latest version from our local repository. Also, users don't touch any of these servers, so not should be any problem. But I see that here is a mind-share that rebooting Tomcat server pass X time is a good idea as with time they begin to do funny things, but we don't do that.

weems 12y ago

Isn't it supposed to be 2020-02-03 00:00:00?

amelius 12y ago

Just reboot your systems on a regular basis, and time-interleaved.

perlgeek 12y ago

Instead of rebooting all the affected systems, create a monitoring check that alerts the ops team a few days or weeks before the error condition (system will hang/reboot, SSL cert expires) comes to pass. Then it's at their discretion what do (for example, decide when to reboot).

techsupporter 12y ago
> alerts the ops team "Everybody" knows that no good Web 3.0 company has an ops team. We're too expensive and too easily replaced with a handful of shell scripts or can just be fobbed off as another responsibility of the devs who wrote the code. (After all, if the code was perfect, there wouldn't need to be an ops team, right?) Silly rabbit, ops teams are for lumbering companies who "Just Don't Get It." Signed, Slightly miffed ops person who thinks he does a lot for his company but feels woefully underappreciated in the new IT.
- kjs3 12y ago
  Meh...you want underappreciated? Try being in infosec. Those new-fangled companies don't think there's a thing wrong with "We'll worry about security after we get acquired...", regardless of how sensitive the data their holding is.
  - Kalium 12y ago
    Ah! Someone else understands my pain.
  - Silhouette 12y ago
    And sadly that will continue to be the case until data protection regulators have real teeth, at which point due diligence before any acquisition should obviously include a thorough audit of these areas. A potential acquisition target that hasn't looked after its data properly and is a large regulatory action waiting to happen should then, rightly, be unlikely to exit successfully until they get their house in order.
    - 12y ago

grecy 12y ago

I work at a telco with a lot of aging systems and servers. A couple of years ago we had a total power failure of our major data center and central office, including the backup gen so literally everything failed and an entire province of Canada (and most of two more) were without any telco, data, wireless. It was eye opening and extremely scary to see what happens when everything is turned off at once and you need to bring stuff back up. Some of the equipment had never been turned off, and certainly not all at once. During the next weeks we encountered many, many catch-22 scenarios like "Before we turn on System A, it needs system B to be online. B needs C, C needs A." Oh. For reference of how severe this is, paramedics were posting up at major intersections in towns and communities in case people needed help - and the radio stations in half of Canada were telling people what to do. When even 911 doesn't work you have problems.

OldSchoolJohnny 12y ago
As a Canadian I would love to see a citation to this because it would have been pretty big news here and I recall nothing about it.
- grecy 12y ago
  http://www.ctvnews.ca/canada/power-outage-in-yukon-knocks-out-communication-systems-1.964656 http://www.ctvnews.ca/canada/power-outage-in-yukon-knocks-ou... It took out everything in the Yukon, and tons of stuff in NWT, Nunavut and Northern BC and Alberta.
serve_yay 12y ago
This is how "if it ain't broke, don't fix it" calcifies. People point to old systems like this (when they are working, that is) all the time as proof that developers are too obsessed with shiny new things. And I'm not saying that isn't true, but there is such a thing as being too conservative.
- grecy 12y ago
  > there is such a thing as being too conservative In my experience, a telco is the definition of "too conservative".