5 ms·

What does it mean that you were right to feel embarrassed? You were wrong to make the mistake, sure, but that happens. That’s what a mistake is. It’s not clear

by jolux 4y ago

What does it mean that you were right to feel embarrassed? You were wrong to make the mistake, sure, but that happens. That’s what a mistake is. It’s not clear to me what shame helps in this instance. If somebody has a pattern of not meeting a quality bar with their code that’s one thing, but otherwise cutting people up for mistakes really seems like bad culture. Aren’t you supposed to practice blameless postmortems and all that at Google?

wutbrodo 4y ago
My recollection from my time there, as well as one of the biggest cultural differences noticeable between my and my workplaces since, is that big chunks of the company really do believe in the blameless postmortem ideal. But culture isn't a magic tool that completely neutralizes assholes, and there are assholes in _every_ organization of sufficient size, like the "[name redacted]" character in the previous post
- KMag 4y ago
  I probably should have left [redacted]'s name out of it. He's a nice guy, but a bit cynical and rough on the edges, at least he was 15 years ago. He fit in well in New York, brutally honest, but actually generally nice. Edit: I should point out that we were in Google NYC, as were the researchers. We had lunch with them some times. I remember the first name and face of the guy who submitted the slow code, but forget his family name and intentionally left his name out. New Yorker to New Yorker adds a lot of context. Google's corporate culture is generally very Californian, but this happened all within the New York office between people who generally got along pretty well and knew each other decently. In context, there was a heavy note of respect for someone's abilities and disappointment that they weren't performing at a top level. He wouldn't have been so harsh with someone who was new or was a weaker engineer, or someone who wasn't used to New York culture.
  - the-rc 4y ago
    Yeah, that might have made it sound as if he's always like that. To his credit, he was one of those that had to be involved when things got desperate, e.g. the insane and massive data recovery to prevent the index from growing stale during a PCR that wasn't properly planned for (if you were around, you know what I'm talking about).
    - KMag 4y ago
      Yea, nice guy and solid engineer. He was actually my TA for a systems engineering class back in school. He was actually very helpful in solving other people's problems, but a bit blunt in encouraging people to reach their full potential.
  - wutbrodo 4y ago
    > New Yorker to New Yorker adds a lot of context. Google's corporate culture is generally very Californian As a lifelong Californian who moved to NY, this makes sense. Though I don't know if it's a blanket excuse... There's a reason that my friends at Goldman would agree that Google's culture is better, and a lot of it has to do with the difference between CA and NY culture. I completely dismiss the claims that the difference is just about surface-level abrasiveness, instead of noticeable differences in how unkindly people treat each other. I regularly see strangers here treat each other in ways that I didn't see in 30+ years in California. Ie, "he's from New York" isn't quite a rebuttal to "he's being an asshole at work". The work culture is more tolerant of assholes, but that barely makes it better.
    - KMag 4y ago
      I grew up in the upper Midwest, went to school outside Boston, and spent most of my adult life in New York and Hong Kong. I haven't spent much time in California, so maybe I'm placing too many of the cultural problems I saw at Google at the foot of Californian culture. I saw lots of pressure to "be Googly", which seemed to result in a lot of passive-aggression. That matches the stereotypes I've heard of LA and SF cultures. New York and Hong Kong are both very impatient places that don't suffer fools lightly. They're both brutally honest places, but with strong shared identities. Both places have lots of people from elsewhere who are sympathetic to being new to the city. The quintessential New York story is the woman pushing a stroller (buggy, for my overseas friends) out of a subway car. A man in a hurry is passing her, looks over his shoulder, they make eye contact and silently agree he should help. He grabs the front of the stroller, they carry it up the stairs, and he sets the stroller down at the top of the stairs. They give each other a nod, and he runs off. Never a word is spoken. On the other hand, if you stop in the middle of the sidewalk to read a map or talk on the phone in New York, don't be surprised if people bump into you, perhaps on purpose. I just left Goldman after more than a decade. I actually really preferred the corporate culture to the Google culture, but most of my time was in Hong Kong. I even knew a couple of people who worked at Goldman, went to Google, and came back to Goldman. I don't doubt that there are some assholes there, particularly in trading, but my experience in both Strats and Technology was overwhelmingly positive. In my experience, abusive MDs didn't last long, but maybe I was very lucky. If you're having a tough time, I don't want to give out full names, but some pretty unique names: Laurent, Jia, Dunstan, and C.K. all really seem to care for their folks. The heads of SRE, One Delta Strats, and Flow Strats in Hong Kong also seem to be really great managers, but Grahaeme, Nick, and Alain are pretty common names. At Google, a friend of mine managed a couple of small teams (on a shoestring budget, using mostly interns) and completed a couple of projects ahead of time, only to get them cancelled by middle management right before launch. One of the projects even had heavy buy-in from upper management, but he got essentially zero credit for either one because they were cancelled before launching. It was incredibly frustrating for him and he left. In my experience, Goldman runs projects much better, particularly using "Projito" internal contracts that list the value proposition, deliverables, timetable, and whose budget is paying for it. Also, despite Goldman's reputation, I saw a lot less arrogance at Goldman than at Google. Goldman has a much more healthy respect for its competitors, though with good reason. I've also seen a few people transition from Google to Goldman and take a bit of time to adjust to the idea that Google's way of doing things is best in Goldman's operating environment. At Google, I got the sense that lots of people thought that Google was better than its competitors because they were smarter. At Goldman, I got the sense that lots of people felt they were better than their competitors because they had higher expectations of themselves.
  - jseban 4y ago
    "What happens in New York stays in New York"? What kind of bullshit is this
- jseban 4y ago
  It's also ironic that the original engineering culture at google was explicitly breaking away from this type of pennalistic shaming culture, and was seen as a big factor in the initial success.
- dang 4y ago
  (I've redacted the person's name from the parent comment and other comments in this thread. I realize you didn't introduce it originally. Probably best not to leave up though, and it isn't necessary for any of the substantive points here.)
  - deleted 4y ago
    [deleted]
KMag 4y ago
There wasn't a post-mortem in my case. I had thought of the case for <title/> but basically out of laziness talked myself out of writing a separate test case for it, presuming the test cases for a zero-length title and an un-closed title covered the corner cases. (The entire document was guaranteed to be converted to valid UTF-8, perhaps with invalid character substitution characters, by that late in the Content Converter pipeline.) So, as soon as someone asked me if I had changed the title parsing code, I was 50% sure of which corner case I had screwed up before looking at any code. It took me about 30 minutes to submit a code fix with updated test cases. I think less than 1 billion documents had been processed, resulting in less than 1,000 pages missing updates due to my bug.
- danuker 4y ago
  Parent comment is onto something. You sound traumatized.
  - KMag 4y ago
    I was a couple years out of school, and felt a lot of responsibility. It wasn't really a big deal, but it didn't feel like it at the time.
    - quakeguy 4y ago
      What a great story nevertheless.
      - KMag 4y ago
        It helped me empathize with young engineers dealing with their first high responsibility bug. About 4 years ago, I was managing a guy right out of school who pushed a minor bug that broke real-time risk calculations for a major multinational financial institution in the middle of the European trading day, prior to the NYC open, and people were yelling over email that they were trading blind. Someone had committed an important change right after his, so a simple rollback was highly sub-optimal. Remembering how I felt years ago, I reassured the new guy that people were yelling over email because it was important, not because they were mad at him. I told him that I thought he was the most familiar with his change and the most capable person to fix it, and that he should do his best to calm down and focus, but he should let me know if he needed help, and I would do my best to calm folks down. I told him he would probably remember that mistake the rest of his life, but nobody else was going to remember it a week later. He had the bug fix in production in under an hour. He sent me an email from home that night worried that he had let the team down, and I reiterated that he was going to be the only one who remembered the mistake longer than a week. The post-mortem follow-up was just to reiterate to authors and reviewers the importance of corner-case tests, and nobody brought it up later. I really only remember it because my manager sent me an email that night praising how well I handled the new guy's first big production bug.
mannykannot 4y ago
This may all be a matter of different nuances on words like "embarrassment" and "shame", but personally, I would put higher trust in someone who firstly recognizes that they could have done better and secondly regrets that they did not, and I would prefer to be that way myself, without being rendered ineffective by it. One can never, however, be sure that someone really feels that way ("The secret to success is sincerity - once you can fake that, you've got it made!") Nevertheless, faking it involves never saying certain things, and that turns out to be hard, so the fallible approach that I aspire to is to assume sincerity unless given evidence to the contrary. In this view, blameless postmortems are the right way to go, at least up to a point: neither embarrassment nor regret are things that can truly be imposed from outside, and definitely not sincerity.

by jolux 4y ago

wutbrodo 4y ago

My recollection from my time there, as well as one of the biggest cultural differences noticeable between my and my workplaces since, is that big chunks of the company really do believe in the blameless postmortem ideal. But culture isn't a magic tool that completely neutralizes assholes, and there are assholes in _every_ organization of sufficient size, like the "[name redacted]" character in the previous post

KMag 4y ago
I probably should have left [redacted]'s name out of it. He's a nice guy, but a bit cynical and rough on the edges, at least he was 15 years ago. He fit in well in New York, brutally honest, but actually generally nice. Edit: I should point out that we were in Google NYC, as were the researchers. We had lunch with them some times. I remember the first name and face of the guy who submitted the slow code, but forget his family name and intentionally left his name out. New Yorker to New Yorker adds a lot of context. Google's corporate culture is generally very Californian, but this happened all within the New York office between people who generally got along pretty well and knew each other decently. In context, there was a heavy note of respect for someone's abilities and disappointment that they weren't performing at a top level. He wouldn't have been so harsh with someone who was new or was a weaker engineer, or someone who wasn't used to New York culture.
- the-rc 4y ago
  Yeah, that might have made it sound as if he's always like that. To his credit, he was one of those that had to be involved when things got desperate, e.g. the insane and massive data recovery to prevent the index from growing stale during a PCR that wasn't properly planned for (if you were around, you know what I'm talking about).
  - KMag 4y ago
    Yea, nice guy and solid engineer. He was actually my TA for a systems engineering class back in school. He was actually very helpful in solving other people's problems, but a bit blunt in encouraging people to reach their full potential.
- wutbrodo 4y ago
  > New Yorker to New Yorker adds a lot of context. Google's corporate culture is generally very Californian As a lifelong Californian who moved to NY, this makes sense. Though I don't know if it's a blanket excuse... There's a reason that my friends at Goldman would agree that Google's culture is better, and a lot of it has to do with the difference between CA and NY culture. I completely dismiss the claims that the difference is just about surface-level abrasiveness, instead of noticeable differences in how unkindly people treat each other. I regularly see strangers here treat each other in ways that I didn't see in 30+ years in California. Ie, "he's from New York" isn't quite a rebuttal to "he's being an asshole at work". The work culture is more tolerant of assholes, but that barely makes it better.
  - KMag 4y ago
    I grew up in the upper Midwest, went to school outside Boston, and spent most of my adult life in New York and Hong Kong. I haven't spent much time in California, so maybe I'm placing too many of the cultural problems I saw at Google at the foot of Californian culture. I saw lots of pressure to "be Googly", which seemed to result in a lot of passive-aggression. That matches the stereotypes I've heard of LA and SF cultures. New York and Hong Kong are both very impatient places that don't suffer fools lightly. They're both brutally honest places, but with strong shared identities. Both places have lots of people from elsewhere who are sympathetic to being new to the city. The quintessential New York story is the woman pushing a stroller (buggy, for my overseas friends) out of a subway car. A man in a hurry is passing her, looks over his shoulder, they make eye contact and silently agree he should help. He grabs the front of the stroller, they carry it up the stairs, and he sets the stroller down at the top of the stairs. They give each other a nod, and he runs off. Never a word is spoken. On the other hand, if you stop in the middle of the sidewalk to read a map or talk on the phone in New York, don't be surprised if people bump into you, perhaps on purpose. I just left Goldman after more than a decade. I actually really preferred the corporate culture to the Google culture, but most of my time was in Hong Kong. I even knew a couple of people who worked at Goldman, went to Google, and came back to Goldman. I don't doubt that there are some assholes there, particularly in trading, but my experience in both Strats and Technology was overwhelmingly positive. In my experience, abusive MDs didn't last long, but maybe I was very lucky. If you're having a tough time, I don't want to give out full names, but some pretty unique names: Laurent, Jia, Dunstan, and C.K. all really seem to care for their folks. The heads of SRE, One Delta Strats, and Flow Strats in Hong Kong also seem to be really great managers, but Grahaeme, Nick, and Alain are pretty common names. At Google, a friend of mine managed a couple of small teams (on a shoestring budget, using mostly interns) and completed a couple of projects ahead of time, only to get them cancelled by middle management right before launch. One of the projects even had heavy buy-in from upper management, but he got essentially zero credit for either one because they were cancelled before launching. It was incredibly frustrating for him and he left. In my experience, Goldman runs projects much better, particularly using "Projito" internal contracts that list the value proposition, deliverables, timetable, and whose budget is paying for it. Also, despite Goldman's reputation, I saw a lot less arrogance at Goldman than at Google. Goldman has a much more healthy respect for its competitors, though with good reason. I've also seen a few people transition from Google to Goldman and take a bit of time to adjust to the idea that Google's way of doing things is best in Goldman's operating environment. At Google, I got the sense that lots of people thought that Google was better than its competitors because they were smarter. At Goldman, I got the sense that lots of people felt they were better than their competitors because they had higher expectations of themselves.
- jseban 4y ago
  "What happens in New York stays in New York"? What kind of bullshit is this
jseban 4y ago
It's also ironic that the original engineering culture at google was explicitly breaking away from this type of pennalistic shaming culture, and was seen as a big factor in the initial success.
dang 4y ago
(I've redacted the person's name from the parent comment and other comments in this thread. I realize you didn't introduce it originally. Probably best not to leave up though, and it isn't necessary for any of the substantive points here.)
- deleted 4y ago
  [deleted]

KMag 4y ago

There wasn't a post-mortem in my case. I had thought of the case for <title/> but basically out of laziness talked myself out of writing a separate test case for it, presuming the test cases for a zero-length title and an un-closed title covered the corner cases. (The entire document was guaranteed to be converted to valid UTF-8, perhaps with invalid character substitution characters, by that late in the Content Converter pipeline.) So, as soon as someone asked me if I had changed the title parsing code, I was 50% sure of which corner case I had screwed up before looking at any code. It took me about 30 minutes to submit a code fix with updated test cases. I think less than 1 billion documents had been processed, resulting in less than 1,000 pages missing updates due to my bug.

danuker 4y ago
Parent comment is onto something. You sound traumatized.
- KMag 4y ago
  I was a couple years out of school, and felt a lot of responsibility. It wasn't really a big deal, but it didn't feel like it at the time.
  - quakeguy 4y ago
    What a great story nevertheless.
    - KMag 4y ago
      It helped me empathize with young engineers dealing with their first high responsibility bug. About 4 years ago, I was managing a guy right out of school who pushed a minor bug that broke real-time risk calculations for a major multinational financial institution in the middle of the European trading day, prior to the NYC open, and people were yelling over email that they were trading blind. Someone had committed an important change right after his, so a simple rollback was highly sub-optimal. Remembering how I felt years ago, I reassured the new guy that people were yelling over email because it was important, not because they were mad at him. I told him that I thought he was the most familiar with his change and the most capable person to fix it, and that he should do his best to calm down and focus, but he should let me know if he needed help, and I would do my best to calm folks down. I told him he would probably remember that mistake the rest of his life, but nobody else was going to remember it a week later. He had the bug fix in production in under an hour. He sent me an email from home that night worried that he had let the team down, and I reiterated that he was going to be the only one who remembered the mistake longer than a week. The post-mortem follow-up was just to reiterate to authors and reviewers the importance of corner-case tests, and nobody brought it up later. I really only remember it because my manager sent me an email that night praising how well I handled the new guy's first big production bug.

mannykannot 4y ago

This may all be a matter of different nuances on words like "embarrassment" and "shame", but personally, I would put higher trust in someone who firstly recognizes that they could have done better and secondly regrets that they did not, and I would prefer to be that way myself, without being rendered ineffective by it. One can never, however, be sure that someone really feels that way ("The secret to success is sincerity - once you can fake that, you've got it made!") Nevertheless, faking it involves never saying certain things, and that turns out to be hard, so the fallible approach that I aspire to is to assume sincerity unless given evidence to the contrary. In this view, blameless postmortems are the right way to go, at least up to a point: neither embarrassment nor regret are things that can truly be imposed from outside, and definitely not sincerity.