4 ms·

> I want to serve 5TB of data. Ok, spin up an instance in AWS and put it there... it was amusing when we hired people from Google who were confused by the lack

> I want to serve 5TB of data. Ok, spin up an instance in AWS and put it there... it was amusing when we hired people from Google who were confused by the lack of process and approvals. Quoting from Velocity in Software Engineering https://queue.acm.org/detail.cfm?id=3352692 https://queue.acm.org/detail.cfm?id=3352692: In 2003, at a time in Amazon's history when we were particularly frustrated by our speed of software engineering, we turned to Matt Round, an engineering leader who was a most interesting squeaky wheel in that his team appeared to get more done than any other, yet he remained deeply impatient and complained loudly and with great clarity about how hard it was to get anything done. He wrote a six-pager that had a great hook in the first paragraph: "To many of us Amazon feels more like a tectonic plate than an F-16." Matt's paper had many recommendations... including the maximization of autonomy for teams and for the services operated by those teams by the adoption of REST-style interfaces, platform standardization, removal of roadblocks or gatekeepers (high-friction bureaucracy), and continuous deployment of isolated components. He also called... for an enduring performance indicator based on the percentage of their time that software engineers spent building rather than doing other tasks. Builders want to build, and Matt's timely recommendations influenced the forging of Amazon's technology brand as "the best place where builders can build." ...leading up to the creation of AWS.

bcoughlan 5y ago
What does "platform standardization" mean in this context?
ryandrake 5y ago
The "approval paralysis" thing happens at a lot of companies, large and small, not just GiantTech. It creeps up on you slowly: 1. A big problem happens that gains the attention of leadership. 2. The problem is root-caused to some risky thing an employee did trying to accomplish XYZ. 3. To correct this, a process is put in place that must be followed when one wants to do XYZ, and (critically) gatekeepers are anointed who must approve the activity. 4. These gatekeepers are inevitably senior already-busy people who become bottlenecks. Now we can't do this critical thing without hounding approvers. 5. Some other big problem happens and the above cycle starts all over again. Before you know it, every even slightly risky task you need to do through the course of your job requires the blessing of approvers who are well-intentioned, but all so overloaded they don't even answer their E-mail or chats. They sometimes need to be physically grabbed in the hallway in order to unblock your project. Progress grinds to a halt and it still has not stopped production problems--just those particular classes of problems that the approval processes caught. EDIT: Not sure what the right solution is, but it must be one that doesn't rely on a particular overloaded human doing something. Maybe an automated approval system that produces a paper trail (to help with postmortem and corrective action later) and ensuring all changes can be rolled back effortlessly. Easier said than done, obviously.
- Zababa 5y ago
  I've read on HN that "processes are organizational scar tissue", I think it applies here.
  - riknos314 5y ago
    Yep. A wise engineer once told me "Runbooks [written SOPs] are just solving bugs with people instead of code"
  - rShergold 5y ago
    That's an excellent phrase. It reminds me of the navy saying "regulations are written in blood"
    - MauranKilom 5y ago
      It's actually super related, given that (at least in the medical software sector) you won't get anything approved by the FDA before spelling out the entire software development operation in processes.
      - NaturalPhallacy 5y ago
        Makes sense in the medical domain where people are way more likely to be injured or die. But most software isn't that critical.
- david422 5y ago
  What is the solution? I've worked at big companies that are mired in process because they would rather spend more time crossing i's and dotting t's than risk breaking something. I can see why. And I've worked at smaller companies where the clients are small and it's easy to fix things that break. Move fast and break things at a small scale maybe. But how do you grow to be a big company and still operate like a small company? I can't seem to see an answer.
  - 5y ago
jll29 5y ago
> we turned to Matt Round, an engineering leader who was a most interesting squeaky wheel in that his team appeared to get more done than any other Matt went on to study theology, and he's started a church community in Scotland: https://www.linkedin.com/in/mattround/ https://www.linkedin.com/in/mattround/ "Leader Company Name: Hope City Church Edinburgh Dates Employed: Sep 2017 – Present Driving a new church start-up."

bcoughlan 5y ago

What does "platform standardization" mean in this context?

ryandrake 5y ago

The "approval paralysis" thing happens at a lot of companies, large and small, not just GiantTech. It creeps up on you slowly: 1. A big problem happens that gains the attention of leadership. 2. The problem is root-caused to some risky thing an employee did trying to accomplish XYZ. 3. To correct this, a process is put in place that must be followed when one wants to do XYZ, and (critically) gatekeepers are anointed who must approve the activity. 4. These gatekeepers are inevitably senior already-busy people who become bottlenecks. Now we can't do this critical thing without hounding approvers. 5. Some other big problem happens and the above cycle starts all over again. Before you know it, every even slightly risky task you need to do through the course of your job requires the blessing of approvers who are well-intentioned, but all so overloaded they don't even answer their E-mail or chats. They sometimes need to be physically grabbed in the hallway in order to unblock your project. Progress grinds to a halt and it still has not stopped production problems--just those particular classes of problems that the approval processes caught. EDIT: Not sure what the right solution is, but it must be one that doesn't rely on a particular overloaded human doing something. Maybe an automated approval system that produces a paper trail (to help with postmortem and corrective action later) and ensuring all changes can be rolled back effortlessly. Easier said than done, obviously.

Zababa 5y ago
I've read on HN that "processes are organizational scar tissue", I think it applies here.
- riknos314 5y ago
  Yep. A wise engineer once told me "Runbooks [written SOPs] are just solving bugs with people instead of code"
- rShergold 5y ago
  That's an excellent phrase. It reminds me of the navy saying "regulations are written in blood"
  - MauranKilom 5y ago
    It's actually super related, given that (at least in the medical software sector) you won't get anything approved by the FDA before spelling out the entire software development operation in processes.
    - NaturalPhallacy 5y ago
      Makes sense in the medical domain where people are way more likely to be injured or die. But most software isn't that critical.
david422 5y ago
What is the solution? I've worked at big companies that are mired in process because they would rather spend more time crossing i's and dotting t's than risk breaking something. I can see why. And I've worked at smaller companies where the clients are small and it's easy to fix things that break. Move fast and break things at a small scale maybe. But how do you grow to be a big company and still operate like a small company? I can't seem to see an answer.
- 5y ago

jll29 5y ago

> we turned to Matt Round, an engineering leader who was a most interesting squeaky wheel in that his team appeared to get more done than any other Matt went on to study theology, and he's started a church community in Scotland: https://www.linkedin.com/in/mattround/ https://www.linkedin.com/in/mattround/ "Leader Company Name: Hope City Church Edinburgh Dates Employed: Sep 2017 – Present Driving a new church start-up."