4 ms·

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariably push a hotfix that breaks something else and then you are in a world of hurt. \ Muphy's law is a cruel thing to sysadmins and developers.

JohnMakin 3y ago
Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”
- hammyhavoc 3y ago
  I read "varying degrees of severity" as "varying degrees of sanity" and was all "omg, he just like me fr" until I reread it. ༼ ༎ຶ ᆺ ༎ຶ༽
- tetha 3y ago
  > I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” There is coincidental function, and planned function. If you just throw numbers at a configuration parameter until it works and then call it "fixed",it may be fixed, but that's just coincidental. If the system load changes, you can't really tell how to modify the value afterwards. On the other hand, if you can explain why a certain setting will fix it, that's planned function, and if something changes, you most likely know how to proceed from there.
- oldandboring 3y ago
  > I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” Thanks, I'm going to steal this. Similarly we say: you must be sure you fixed THE problem, not just A problem.
- bombcar 3y ago
  My way of looking at it is if I can’t rebreak it, I haven’t found the cause yet.
- lamontcg 3y ago
  I hate it when button mashers are in charge of incident response. Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.
  - dmoy 3y ago
    To be fair, if your rollbacks are cheap (1-3 button pushes), safe, and automated, it doesn't really hurt to try a rollback while you look for other causes.
    - lamontcg 3y ago
      Most of my production experience precedes CI/CD as a concept by a number of years. You still don't know though if the rollback fixed it or if just bouncing the software alone fixed some kind of brownout, and you may not be able to replicate what happened and get stuck unsure of what the bug was and be in a state where you don't know if you can roll forward again or not. Then a developer throws a bit of spackle on the code to address what the problem "might" be -- then when you roll forward people assume that fixed it, which it might not, which is how cargo cults get formed.
    - JohnMakin 3y ago
      Code changes should be trivial to roll back. Infra changes can be very difficult to roll back, however.
    - rurp 3y ago
      Not to mention the times someone has said that an issue couldn't possibly have been caused by a recent change they made, only to find out that it was.
- throwaway_673 3y ago
  This is unfortunately very true. Example: At my work place we were made aware of some instability issues and the first place to look is of course the logs. It had apparently been happening for a while, but the most important parts of the system were not affected noticeably. This was early morning and we didn't have much traffic. We noticed a lot of network connection errors for SQS, so one of the team members started blaming Amazon. My response was that if SQS had these instabilities, we would have noticed posts on HN or downtime on other sites. The developer ignored this and jumped into the code to add toggles and debug log messages. I spent 2 more minutes and found other cases where we had connection issues to different services. I concluded that the reason why we saw so many SQS errors was because it is called much more frequently. The other developer pushed changes to prod which caused even more problems, because now the service could not even connect to the database. He wrongly assumed that the older versions of the service, which was still running, was holding on to the connections and killed them in order to make room for his new version. We went from a system with instabilities, to a system that was down. The reason was a network error that was fixed by someone in the ops team.
sa46 3y ago
Oh man, last week we had a small outage. Database queries took much longer than normal. I just so happened to be doing ad-hoc querying on the same table at the same time. “Luckily”, the problem was unrelated to my querying but two coincidences are proper scary.
- dylan604 3y ago
  nah, unless you were the intern, you should be safe it's like Star Trek where you're fine unless you're the one wearing the red shirt
  - ashton314 3y ago
    If I ever run a company, all the interns will get red shirts
    - ok_dad 3y ago
      Think about the optics of your funny joke. You’re saying you’ll advertise that the interns are replaceable fodder. Even if that’s true, it’s quite a dick move.
      - PNewling 3y ago
        It also could have just been a joke, not one they were actually going to put into practice...
        dylan604 3y ago
        Some people just come across as the type of person totally not any fun to hang out with.
        ok_dad 3y ago
        I’ve seen enough brainless execs and startup founders around here that you have to take stupid ideas like that seriously. I hope it was a joke, but it’s not very funny for an intern.
jnsaff2 3y ago
Yeah, the "coincidence" leads you to jump to conclusions about your change being the cause. This is very human, we do it all the time. However having been through these enough times I have picked up a habit of questioning more assumptions and not flagging something as verified data before it is. I'm still nowhere near perfect at removing these biases and early conclusions but it has helped. Having an open mind is hard work.
tnolet 3y ago
Oh yes, the amount of times I rolled back a change during an outage that had nothing to do with the outage... It's a critical skill for engineers: being able to critically reason, debug and "test in isolation" any changes that address outages. Much harder than it seems and typically a "senior" skill to have.
- lesam 3y ago
  Assuming you have a tested rollback strategy, I’m all for rolling back first and asking questions later. It at least can often eliminate the rolled back change as the cause, even if it doesn’t fix the problem.
  - eklavya 3y ago
    That's essentially on call 101, you try to mitigate as fast as possible and root cause later when you are not half asleep. Surprised to see some comments here.

JohnMakin 3y ago

Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”

hammyhavoc 3y ago
I read "varying degrees of severity" as "varying degrees of sanity" and was all "omg, he just like me fr" until I reread it. ༼ ༎ຶ ᆺ ༎ຶ༽
tetha 3y ago
> I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” There is coincidental function, and planned function. If you just throw numbers at a configuration parameter until it works and then call it "fixed",it may be fixed, but that's just coincidental. If the system load changes, you can't really tell how to modify the value afterwards. On the other hand, if you can explain why a certain setting will fix it, that's planned function, and if something changes, you most likely know how to proceed from there.
oldandboring 3y ago
> I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” Thanks, I'm going to steal this. Similarly we say: you must be sure you fixed THE problem, not just A problem.
bombcar 3y ago
My way of looking at it is if I can’t rebreak it, I haven’t found the cause yet.
lamontcg 3y ago
I hate it when button mashers are in charge of incident response. Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.
- dmoy 3y ago
  To be fair, if your rollbacks are cheap (1-3 button pushes), safe, and automated, it doesn't really hurt to try a rollback while you look for other causes.
  - lamontcg 3y ago
    Most of my production experience precedes CI/CD as a concept by a number of years. You still don't know though if the rollback fixed it or if just bouncing the software alone fixed some kind of brownout, and you may not be able to replicate what happened and get stuck unsure of what the bug was and be in a state where you don't know if you can roll forward again or not. Then a developer throws a bit of spackle on the code to address what the problem "might" be -- then when you roll forward people assume that fixed it, which it might not, which is how cargo cults get formed.
  - JohnMakin 3y ago
    Code changes should be trivial to roll back. Infra changes can be very difficult to roll back, however.
  - rurp 3y ago
    Not to mention the times someone has said that an issue couldn't possibly have been caused by a recent change they made, only to find out that it was.
throwaway_673 3y ago
This is unfortunately very true. Example: At my work place we were made aware of some instability issues and the first place to look is of course the logs. It had apparently been happening for a while, but the most important parts of the system were not affected noticeably. This was early morning and we didn't have much traffic. We noticed a lot of network connection errors for SQS, so one of the team members started blaming Amazon. My response was that if SQS had these instabilities, we would have noticed posts on HN or downtime on other sites. The developer ignored this and jumped into the code to add toggles and debug log messages. I spent 2 more minutes and found other cases where we had connection issues to different services. I concluded that the reason why we saw so many SQS errors was because it is called much more frequently. The other developer pushed changes to prod which caused even more problems, because now the service could not even connect to the database. He wrongly assumed that the older versions of the service, which was still running, was holding on to the connections and killed them in order to make room for his new version. We went from a system with instabilities, to a system that was down. The reason was a network error that was fixed by someone in the ops team.

sa46 3y ago

Oh man, last week we had a small outage. Database queries took much longer than normal. I just so happened to be doing ad-hoc querying on the same table at the same time. “Luckily”, the problem was unrelated to my querying but two coincidences are proper scary.

dylan604 3y ago
nah, unless you were the intern, you should be safe it's like Star Trek where you're fine unless you're the one wearing the red shirt
- ashton314 3y ago
  If I ever run a company, all the interns will get red shirts
  - ok_dad 3y ago
    Think about the optics of your funny joke. You’re saying you’ll advertise that the interns are replaceable fodder. Even if that’s true, it’s quite a dick move.
    - PNewling 3y ago
      It also could have just been a joke, not one they were actually going to put into practice...
      - dylan604 3y ago
        Some people just come across as the type of person totally not any fun to hang out with.
      - ok_dad 3y ago
        I’ve seen enough brainless execs and startup founders around here that you have to take stupid ideas like that seriously. I hope it was a joke, but it’s not very funny for an intern.

jnsaff2 3y ago

Yeah, the "coincidence" leads you to jump to conclusions about your change being the cause. This is very human, we do it all the time. However having been through these enough times I have picked up a habit of questioning more assumptions and not flagging something as verified data before it is. I'm still nowhere near perfect at removing these biases and early conclusions but it has helped. Having an open mind is hard work.

tnolet 3y ago

Oh yes, the amount of times I rolled back a change during an outage that had nothing to do with the outage... It's a critical skill for engineers: being able to critically reason, debug and "test in isolation" any changes that address outages. Much harder than it seems and typically a "senior" skill to have.

lesam 3y ago
Assuming you have a tested rollback strategy, I’m all for rolling back first and asking questions later. It at least can often eliminate the rolled back change as the cause, even if it doesn’t fix the problem.
- eklavya 3y ago
  That's essentially on call 101, you try to mitigate as fast as possible and root cause later when you are not half asleep. Surprised to see some comments here.