5 ms·

>"The rule of thumb is when something is working you don't change it," Hertz said. "We'd like to change as few things as possible when we bring Hubble back into

>"The rule of thumb is when something is working you don't change it," Hertz said. "We'd like to change as few things as possible when we bring Hubble back into service."

AnIdiotOnTheNet 5y ago
A lesson not taught to any modern software developer. Instead they change things all the time for no real reason other than that they want to change things.
- 35fbe7d3d5b9 5y ago
  One of the best senior engineers I worked with taught me how to run an outage. The most important thing? Stop what you are doing, take charge, and get everyone else to stop what they are doing. The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.
  - londons_explore 5y ago
    In my experience, the best strategy depends a lot on the severity of the outage. If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment. If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.
    - ComputerGuru 5y ago
      > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
    - CGamesPlay 5y ago
      > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.
  - boardwaalk 5y ago
    Sounds like “locking the doors” (Space Shuttle disasters). Although, there was really not much to recover from there.
    - etskinner 5y ago
      I hadn't heard of this before, chilling but cool: https://www.theguardian.com/world/2003/feb/13/columbia.spaceexploration https://www.theguardian.com/world/2003/feb/13/columbia.space...
  - sunstone 5y ago
    I envy those innocent souls who have never heard of the concept of configuration management.
- grumple 5y ago
  This is not true. If something's working, you don't change it for no reason. Business requirements and requests change all the time. 90% of our work is done in response to that. The other 10% is fixing up technical problems due to increased scaling or bugs found, and then basically never do we upgrade a system to keep up with security updates or change to a more modern tech.
  - AnIdiotOnTheNet 5y ago
    > This is not true. If something's working, you don't change it for no reason Sure, there's often a reason like "we wanted to write it in a different language" or "we've overhauled the UI to be slower and more cumbersome".
    - Nextgrid 5y ago
      "we need to justify our front-end developers' and designers' jobs"
    - nonameiguess 5y ago
      This highly depends. I believe what Grumple is talking about is contracted work for a specific customer that is asking you for something different, even though what you've already delivered works. You're talking about consumer-facing product teams that make whatever they want and try to sell it after the fact, coming to their own decisions regarding what their "requirements" should be. When your work is funded by a specific customer, the requirements are whatever that customer says they are. And they're almost never going to let you change something just because you wanted to in order to learn a technology you can put on your resume.

>"The rule of thumb is when something is working you don't change it," Hertz said. "We'd like to change as few things as possible when we bring Hubble back into service."

AnIdiotOnTheNet 5y ago

A lesson not taught to any modern software developer. Instead they change things all the time for no real reason other than that they want to change things.

35fbe7d3d5b9 5y ago
One of the best senior engineers I worked with taught me how to run an outage. The most important thing? Stop what you are doing, take charge, and get everyone else to stop what they are doing. The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.
- londons_explore 5y ago
  In my experience, the best strategy depends a lot on the severity of the outage. If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment. If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.
  - ComputerGuru 5y ago
    > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
  - CGamesPlay 5y ago
    > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.
- boardwaalk 5y ago
  Sounds like “locking the doors” (Space Shuttle disasters). Although, there was really not much to recover from there.
  - etskinner 5y ago
    I hadn't heard of this before, chilling but cool: https://www.theguardian.com/world/2003/feb/13/columbia.spaceexploration https://www.theguardian.com/world/2003/feb/13/columbia.space...
- sunstone 5y ago
  I envy those innocent souls who have never heard of the concept of configuration management.
grumple 5y ago
This is not true. If something's working, you don't change it for no reason. Business requirements and requests change all the time. 90% of our work is done in response to that. The other 10% is fixing up technical problems due to increased scaling or bugs found, and then basically never do we upgrade a system to keep up with security updates or change to a more modern tech.
- AnIdiotOnTheNet 5y ago
  > This is not true. If something's working, you don't change it for no reason Sure, there's often a reason like "we wanted to write it in a different language" or "we've overhauled the UI to be slower and more cumbersome".
  - Nextgrid 5y ago
    "we need to justify our front-end developers' and designers' jobs"
  - nonameiguess 5y ago
    This highly depends. I believe what Grumple is talking about is contracted work for a specific customer that is asking you for something different, even though what you've already delivered works. You're talking about consumer-facing product teams that make whatever they want and try to sell it after the fact, coming to their own decisions regarding what their "requirements" should be. When your work is funded by a specific customer, the requirements are whatever that customer says they are. And they're almost never going to let you change something just because you wanted to in order to learn a technology you can put on your resume.