4 ms·

There wasn't a post-mortem in my case. I had thought of the case for <title/> but basically out of laziness talked myself out of writing a separate test case f

by KMag 4y ago

There wasn't a post-mortem in my case. I had thought of the case for <title/> but basically out of laziness talked myself out of writing a separate test case for it, presuming the test cases for a zero-length title and an un-closed title covered the corner cases. (The entire document was guaranteed to be converted to valid UTF-8, perhaps with invalid character substitution characters, by that late in the Content Converter pipeline.) So, as soon as someone asked me if I had changed the title parsing code, I was 50% sure of which corner case I had screwed up before looking at any code. It took me about 30 minutes to submit a code fix with updated test cases. I think less than 1 billion documents had been processed, resulting in less than 1,000 pages missing updates due to my bug.

danuker 4y ago
Parent comment is onto something. You sound traumatized.
- KMag 4y ago
  I was a couple years out of school, and felt a lot of responsibility. It wasn't really a big deal, but it didn't feel like it at the time.
  - quakeguy 4y ago
    What a great story nevertheless.
    - KMag 4y ago
      It helped me empathize with young engineers dealing with their first high responsibility bug. About 4 years ago, I was managing a guy right out of school who pushed a minor bug that broke real-time risk calculations for a major multinational financial institution in the middle of the European trading day, prior to the NYC open, and people were yelling over email that they were trading blind. Someone had committed an important change right after his, so a simple rollback was highly sub-optimal. Remembering how I felt years ago, I reassured the new guy that people were yelling over email because it was important, not because they were mad at him. I told him that I thought he was the most familiar with his change and the most capable person to fix it, and that he should do his best to calm down and focus, but he should let me know if he needed help, and I would do my best to calm folks down. I told him he would probably remember that mistake the rest of his life, but nobody else was going to remember it a week later. He had the bug fix in production in under an hour. He sent me an email from home that night worried that he had let the team down, and I reiterated that he was going to be the only one who remembered the mistake longer than a week. The post-mortem follow-up was just to reiterate to authors and reviewers the importance of corner-case tests, and nobody brought it up later. I really only remember it because my manager sent me an email that night praising how well I handled the new guy's first big production bug.
      - mattclarkdotnet 4y ago
        "I reassured the new guy that people were yelling over email because it was important, not because they were mad at him" This is outstanding advice, and very well put. I shall be borrowing it, thank you!
      - koide 4y ago
        For someone who sort of justified a humiliating response to a mistake a few comments above, this seems really well done! Congratulations.
      - jseban 4y ago
        It's your fault, you were the manager, the yelling should not even have leaked to the developer, it should have stopped with you. Edit: if you are managing someone who is new in a job, it's your job to make sure they don't push bugs to break important stuff in the first place.
        stavros 4y ago
        I don't think we can assign blame in a complicated situation based on a two-paragraph retelling, please have a bit more empathy.
        Forgeties79 4y ago
        Did you consider the fact that you probably know nothing about the dynamics of their workplace, the structure of their management/leadership, etc. before assigning blame?
        webmaven 4y ago
        > It's your fault, you were the manager, the yelling should not even have leaked to the developer, it should have stopped with you. That's impractical in organizations with flatter structures and general purpose communication channels. What would you suggest, kicking everyone off of internal IRC/Slack/mailing lists/etc.?
        KMag 4y ago
        The "yelling" was coming to the team email list, asking for ETAs and progress updates for when real-time risk would be back up. Roughly 4 people at the time knew the bug could be traced to the new guy's commit, and none of those people were doing the "yelling". And it was Goldman, so the "yelling" was kept very professional (no swearing, strictly enforced). But, there were literally tens of billions of dollars that needed to be dynamically hedged, but that wasn't possible without real-time risk, the European markets were open, and markets in the Americas were going to be open within a couple of hours. Trading and management were making sure that that everyone on the team email list understood that this was drop-everything important, perhaps using all caps. Yes, I and the person who reviewed the change bear more responsibility than the new developer. Also, I say "new guy", but the person who had interned with us the Summer after "the new guy" had already joined full time at that point, so "the new guy" had been working full time with the team for at least 9 months at that point. I also remember the room where it happened, which wasn't the first room we were in, so maybe he had been with us full time more like 18 months. In any case, it was the first time when he was trying keep the weight of billions of dollars out of his head and calmly but quickly fix a bug.
andrewljohnson 4y ago
I don’t think laziness (in this comment or the ones about the bash-like C macros) is the right word. I guess there is some pressure/tiredness/newness/taste/external factors driving the choices, test count, and style guide aberrations.
- andi999 4y ago
  Probably typing on a non english keyboard. When I learned C a long time ago, I read somewhere 'isnt it nice that one doesnt need a lot of keystrokes like for begin and end', and I thought pls give me begin and end instead of this unpleasent slow hand movement.
  - atq2119 4y ago
    Spot on. It's the reason many non-native speaker developers I know nevertheless use an English keyboard layout. I personally made my own hybrid layout that is basically an English US layout with the letters rearranged according to my native layout.
  - KMag 4y ago
    The author was born and raised in Brooklyn and went to Cornell, if I remember correctly. As far as I know, English was his only language, and he was almost certainly using either US QWERTY or Dvorak keyboard layout.
    - andi999 4y ago
      In that case it is bizarr.
  - KMag 4y ago
    > I thought pls give me begin and end instead of this unpleasent slow hand movement Have you tried trigraphs? (/me ducks.)
- KMag 4y ago
  Yes, "laziness" is unfair and imprecise, a laziness on my part. :( I think it was an issue of familiarity and comfort, not newness. The author joined Google before I did. If you're basically working with one other person, and you're rarely getting code reviews from outside your coding pair, and few other people interact with your code, it's easy to develop some bad habits and forget that your code choices have externalities. To be fair, the externalities were usually rather small.

by KMag 4y ago

danuker 4y ago

Parent comment is onto something. You sound traumatized.

KMag 4y ago
I was a couple years out of school, and felt a lot of responsibility. It wasn't really a big deal, but it didn't feel like it at the time.
- quakeguy 4y ago
  What a great story nevertheless.
  - KMag 4y ago
    It helped me empathize with young engineers dealing with their first high responsibility bug. About 4 years ago, I was managing a guy right out of school who pushed a minor bug that broke real-time risk calculations for a major multinational financial institution in the middle of the European trading day, prior to the NYC open, and people were yelling over email that they were trading blind. Someone had committed an important change right after his, so a simple rollback was highly sub-optimal. Remembering how I felt years ago, I reassured the new guy that people were yelling over email because it was important, not because they were mad at him. I told him that I thought he was the most familiar with his change and the most capable person to fix it, and that he should do his best to calm down and focus, but he should let me know if he needed help, and I would do my best to calm folks down. I told him he would probably remember that mistake the rest of his life, but nobody else was going to remember it a week later. He had the bug fix in production in under an hour. He sent me an email from home that night worried that he had let the team down, and I reiterated that he was going to be the only one who remembered the mistake longer than a week. The post-mortem follow-up was just to reiterate to authors and reviewers the importance of corner-case tests, and nobody brought it up later. I really only remember it because my manager sent me an email that night praising how well I handled the new guy's first big production bug.
    - mattclarkdotnet 4y ago
      "I reassured the new guy that people were yelling over email because it was important, not because they were mad at him" This is outstanding advice, and very well put. I shall be borrowing it, thank you!
    - koide 4y ago
      For someone who sort of justified a humiliating response to a mistake a few comments above, this seems really well done! Congratulations.
    - jseban 4y ago
      It's your fault, you were the manager, the yelling should not even have leaked to the developer, it should have stopped with you. Edit: if you are managing someone who is new in a job, it's your job to make sure they don't push bugs to break important stuff in the first place.
      - stavros 4y ago
        I don't think we can assign blame in a complicated situation based on a two-paragraph retelling, please have a bit more empathy.
      - Forgeties79 4y ago
        Did you consider the fact that you probably know nothing about the dynamics of their workplace, the structure of their management/leadership, etc. before assigning blame?
      - webmaven 4y ago
        > It's your fault, you were the manager, the yelling should not even have leaked to the developer, it should have stopped with you. That's impractical in organizations with flatter structures and general purpose communication channels. What would you suggest, kicking everyone off of internal IRC/Slack/mailing lists/etc.?
      - KMag 4y ago
        The "yelling" was coming to the team email list, asking for ETAs and progress updates for when real-time risk would be back up. Roughly 4 people at the time knew the bug could be traced to the new guy's commit, and none of those people were doing the "yelling". And it was Goldman, so the "yelling" was kept very professional (no swearing, strictly enforced). But, there were literally tens of billions of dollars that needed to be dynamically hedged, but that wasn't possible without real-time risk, the European markets were open, and markets in the Americas were going to be open within a couple of hours. Trading and management were making sure that that everyone on the team email list understood that this was drop-everything important, perhaps using all caps. Yes, I and the person who reviewed the change bear more responsibility than the new developer. Also, I say "new guy", but the person who had interned with us the Summer after "the new guy" had already joined full time at that point, so "the new guy" had been working full time with the team for at least 9 months at that point. I also remember the room where it happened, which wasn't the first room we were in, so maybe he had been with us full time more like 18 months. In any case, it was the first time when he was trying keep the weight of billions of dollars out of his head and calmly but quickly fix a bug.

andrewljohnson 4y ago

I don’t think laziness (in this comment or the ones about the bash-like C macros) is the right word. I guess there is some pressure/tiredness/newness/taste/external factors driving the choices, test count, and style guide aberrations.

andi999 4y ago
Probably typing on a non english keyboard. When I learned C a long time ago, I read somewhere 'isnt it nice that one doesnt need a lot of keystrokes like for begin and end', and I thought pls give me begin and end instead of this unpleasent slow hand movement.
- atq2119 4y ago
  Spot on. It's the reason many non-native speaker developers I know nevertheless use an English keyboard layout. I personally made my own hybrid layout that is basically an English US layout with the letters rearranged according to my native layout.
- KMag 4y ago
  The author was born and raised in Brooklyn and went to Cornell, if I remember correctly. As far as I know, English was his only language, and he was almost certainly using either US QWERTY or Dvorak keyboard layout.
  - andi999 4y ago
    In that case it is bizarr.
- KMag 4y ago
  > I thought pls give me begin and end instead of this unpleasent slow hand movement Have you tried trigraphs? (/me ducks.)
KMag 4y ago
Yes, "laziness" is unfair and imprecise, a laziness on my part. :( I think it was an issue of familiarity and comfort, not newness. The author joined Google before I did. If you're basically working with one other person, and you're rarely getting code reviews from outside your coding pair, and few other people interact with your code, it's easy to develop some bad habits and forget that your code choices have externalities. To be fair, the externalities were usually rather small.