3 ms·

It's funny that the big tech companies have such high hiring bars and yet when you get inside there are large teams of people actively working to protect the co

It's funny that the big tech companies have such high hiring bars and yet when you get inside there are large teams of people actively working to protect the company from the stupidity of their colleagues.

hinkley 6y ago
I’m mostly trying to protect myself from my own stupidity. That other people fall prey to the same issues in many scenarios just makes the pill go down easier for them.
- EdwardDiego 6y ago
  That's the story of my development life! I know I'm human, and as such, I will err, but I really don't want to be erring as humans do in production. I feel like my "capacity to err over time" graph is U shaped with a bit of an M tendency towards the end - as a newbie, lots of ways to make mistakes without realising, then as my skills increased, realised the "known unknowns", and coded and released very cautiously, but when I blurred into a senior, ran the risk of thinking "I know this" without checking that all my assumptions and prior knowledge held true, and learned the caution of the intermediate developer again.
  - hinkley 6y ago
    I had a boss who insisted on calling the root account stupiduser, because when a server gets properly fucked up it's invariably because some stupid user did the wrong thing...
- aahortwwy 6y ago
  When I say stupidity I don't mean mistakes. From the article: > At that point, it was decided that something easier had to be done. The barrier to entry for safe and sane rollouts had to be lowered even more to bring more teams on board. Why does something have to be easy for a team of professional software engineers to do it correctly? Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.
  - joshuamorton 6y ago
    When you serve hundreds or thousands of teams, as this tool did, one of them is bound to be lazy. Defaults should be safe.
  - ratww 6y ago
    > Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners. In practice it ends up being more like a request. :)
  - hinkley 6y ago
    If you meet an asshole in the morning, you met an asshole. If you meet assholes all day long, you're the asshole. I can think of a few current or former coworkers who talk about how everybody is an idiot and they are the biggest idiots I know. Insisting people use a tool that is capable of doing dangerous things correctly, every time, is white-knuckling. Yelling at them about screwing it up is running your organization by shame. When you do that, eventually only the people with very high pain tolerance ever get promoted, or even heard. Do you really want to work on a team run by masochists/sadists? Similar to make the change easy, make the easy change, if you want to have a dialog with people about doing better, the easiest way to do that is to give them a reliable tool (literal or figurative), and then start insisting they use it when it becomes obvious they haven't. The people who insist on doing things the hardest way possible can still do it as long as nobody else feels the consequences. But typically what they're going to learn is that where self-reflection is concerned, they're bad at math and they have way more frequent problems and far more collateral damage than they like to tell themselves.
    - aahortwwy 6y ago
      There are a few strawmen in this post. 1. I never said everybody is stupid. The vast majority of people in Big Tech are great, but there ARE stupid people there too. I resent the implication of your first paragraph. 2. Who said anything about yelling or other toxic behaviors? There are options other than "engineer a foolproof system" and "institute a sado-masochistic management strategy". 3. I never advocated for doing things the hardest way possible, I advocated for doing things correctly (to the best of your ability) even when it is difficult to do so. The example given in the article is a team choosing to do a global rollout when the option existed to do a staged rollout. There's myriad technical things you can do to make it easier and faster to do a staged rollout, harder to do a global rollout, reduce the risk of a global rollout, etc. etc. ... but what about instilling a healthy respect for production in your colleagues? That's been conspicuously absent from most postmortems I've sat in on.
csours 6y ago
I think you've hit the nail on the head. When the company is small (like a traditional startup), the main risk is not delivering features. When the company is large, the main risk is loss of {data, privacy, uptime, reputation, lawsuits, unexpected licensing fees}. So large companies put in place systems to reduce those risks. Unfortunately the default seems to be to make it hard to deliver features.
gitgud 6y ago
Smarter people work on harder problems, breaking things in more obscure ways, so you need other smart people to protect the service from breaking for customers...
- aahortwwy 6y ago
  Deploying to 100% of production when the option exists to do a staged rollout[0] isn't "breaking things in more obscure ways," it's ignorance or laziness. [0] As mentioned in the article.
  - gitgud 6y ago
    > "it's ignorance or laziness" Well this kind of bugs me, that explanation can be applied to literally anyone who's ever broken anything by accident. With the benefit of hindsight you can always call them out as ignorant or lazy. The circumstances in the article were not explicitly lazy or ignorant in my opinion. It was an unusually large update (2 gigabytes), which due to being able to deploy all servers simultaneously, created an outage. Now, sure they could have done a staged roll-out, but why were they even allowed to deploy to all servers like that? How much due-diligence to they need to do in order to do their job? Reminds me of [1] that guy who accidentally deleted the entire production database on his first day. Was that "ignorance or laziness"? I would say it's a failure of the company's internal processes and systems. [1] https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/ https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/a...
davedx 6y ago
Wasn't that one of the golang design philosophies? :)
acdha 6y ago
Your mistake is thinking it’s stupidity: what you’re seeing is due to conflicting priorities. Smart, motivated people like to hit their goals and will bend whatever rules they need to so – and if you’re busy it’s easy to think you don’t have time to do it the proper way if that takes a while. That’s why investing in tools makes sense: when those people think it’s harder to do things the wrong way then your incentives are aligned. SRE culture around error budgets can help, too: having issues reschedule your project team is a much better schedule than chucking it over the wall to be ops’ problem because people learn that they’re not saving time by cutting corners.
draw_down 6y ago
Is it stupid to use a tool in the way that seems more expedient, or is it stupid to design and release a tool that can cause an outage when used in the more expedient fashion?

hinkley 6y ago

I’m mostly trying to protect myself from my own stupidity. That other people fall prey to the same issues in many scenarios just makes the pill go down easier for them.

EdwardDiego 6y ago
That's the story of my development life! I know I'm human, and as such, I will err, but I really don't want to be erring as humans do in production. I feel like my "capacity to err over time" graph is U shaped with a bit of an M tendency towards the end - as a newbie, lots of ways to make mistakes without realising, then as my skills increased, realised the "known unknowns", and coded and released very cautiously, but when I blurred into a senior, ran the risk of thinking "I know this" without checking that all my assumptions and prior knowledge held true, and learned the caution of the intermediate developer again.
- hinkley 6y ago
  I had a boss who insisted on calling the root account stupiduser, because when a server gets properly fucked up it's invariably because some stupid user did the wrong thing...
aahortwwy 6y ago
When I say stupidity I don't mean mistakes. From the article: > At that point, it was decided that something easier had to be done. The barrier to entry for safe and sane rollouts had to be lowered even more to bring more teams on board. Why does something have to be easy for a team of professional software engineers to do it correctly? Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.
- joshuamorton 6y ago
  When you serve hundreds or thousands of teams, as this tool did, one of them is bound to be lazy. Defaults should be safe.
- ratww 6y ago
  > Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners. In practice it ends up being more like a request. :)
- hinkley 6y ago
  If you meet an asshole in the morning, you met an asshole. If you meet assholes all day long, you're the asshole. I can think of a few current or former coworkers who talk about how everybody is an idiot and they are the biggest idiots I know. Insisting people use a tool that is capable of doing dangerous things correctly, every time, is white-knuckling. Yelling at them about screwing it up is running your organization by shame. When you do that, eventually only the people with very high pain tolerance ever get promoted, or even heard. Do you really want to work on a team run by masochists/sadists? Similar to make the change easy, make the easy change, if you want to have a dialog with people about doing better, the easiest way to do that is to give them a reliable tool (literal or figurative), and then start insisting they use it when it becomes obvious they haven't. The people who insist on doing things the hardest way possible can still do it as long as nobody else feels the consequences. But typically what they're going to learn is that where self-reflection is concerned, they're bad at math and they have way more frequent problems and far more collateral damage than they like to tell themselves.
  - aahortwwy 6y ago
    There are a few strawmen in this post. 1. I never said everybody is stupid. The vast majority of people in Big Tech are great, but there ARE stupid people there too. I resent the implication of your first paragraph. 2. Who said anything about yelling or other toxic behaviors? There are options other than "engineer a foolproof system" and "institute a sado-masochistic management strategy". 3. I never advocated for doing things the hardest way possible, I advocated for doing things correctly (to the best of your ability) even when it is difficult to do so. The example given in the article is a team choosing to do a global rollout when the option existed to do a staged rollout. There's myriad technical things you can do to make it easier and faster to do a staged rollout, harder to do a global rollout, reduce the risk of a global rollout, etc. etc. ... but what about instilling a healthy respect for production in your colleagues? That's been conspicuously absent from most postmortems I've sat in on.

csours 6y ago

I think you've hit the nail on the head. When the company is small (like a traditional startup), the main risk is not delivering features. When the company is large, the main risk is loss of {data, privacy, uptime, reputation, lawsuits, unexpected licensing fees}. So large companies put in place systems to reduce those risks. Unfortunately the default seems to be to make it hard to deliver features.

gitgud 6y ago

Smarter people work on harder problems, breaking things in more obscure ways, so you need other smart people to protect the service from breaking for customers...

aahortwwy 6y ago
Deploying to 100% of production when the option exists to do a staged rollout[0] isn't "breaking things in more obscure ways," it's ignorance or laziness. [0] As mentioned in the article.
- gitgud 6y ago
  > "it's ignorance or laziness" Well this kind of bugs me, that explanation can be applied to literally anyone who's ever broken anything by accident. With the benefit of hindsight you can always call them out as ignorant or lazy. The circumstances in the article were not explicitly lazy or ignorant in my opinion. It was an unusually large update (2 gigabytes), which due to being able to deploy all servers simultaneously, created an outage. Now, sure they could have done a staged roll-out, but why were they even allowed to deploy to all servers like that? How much due-diligence to they need to do in order to do their job? Reminds me of [1] that guy who accidentally deleted the entire production database on his first day. Was that "ignorance or laziness"? I would say it's a failure of the company's internal processes and systems. [1] https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/ https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/a...

davedx 6y ago

Wasn't that one of the golang design philosophies? :)

acdha 6y ago

Your mistake is thinking it’s stupidity: what you’re seeing is due to conflicting priorities. Smart, motivated people like to hit their goals and will bend whatever rules they need to so – and if you’re busy it’s easy to think you don’t have time to do it the proper way if that takes a while. That’s why investing in tools makes sense: when those people think it’s harder to do things the wrong way then your incentives are aligned. SRE culture around error budgets can help, too: having issues reschedule your project team is a much better schedule than chucking it over the wall to be ops’ problem because people learn that they’re not saving time by cutting corners.

draw_down 6y ago

Is it stupid to use a tool in the way that seems more expedient, or is it stupid to design and release a tool that can cause an outage when used in the more expedient fashion?