5 ms·

I appreciate that they have a programming philosophy that they want people at the company to adopt. A common problem I see at companies that don't have onboardi

by Sevii 2y ago

I appreciate that they have a programming philosophy that they want people at the company to adopt. A common problem I see at companies that don't have onboarding is that people join the team with assumptions from previous jobs but you never level set them with the company. So 12 months down the line the new guy wants to change the process and you have to repeat the same discussions about what agile means for the nth time. Amazon does a good job of training new hire on the 'Amazon way'. Amazon does 6 pagers, they do design docs. Amazon does SOA. Amazon does not use relational databases. Everything has an API. Because of the 'Amazon way' and the training they do new team members understand at least some of the context and expectations. Is it the best way? Probably not but no one knows what the best way is anyway. At least they have a way. Saves a lot of effort compared to every new hire relitigating the process and architecture.

rrr_oh_man 2y ago
> Amazon does not use relational databases Huh?
- deskglass 2y ago
  Friends say they typically use Dynamo and that using a relational database requires approval from a vp (because of scaling concerns).
  - hyperliner 2y ago
    Related: Amazon kicked out Oracle from the company. Somewhere along the same timeline, the operational recommendation for teams was to not use relational databases. https://www.theregister.com/2019/10/16/amazon_ditches_oracle/ https://www.theregister.com/2019/10/16/amazon_ditches_oracle...
    - rrr_oh_man 2y ago
      Kicking out Oracle might have to do with factors outside the usage of relational databases...
      - hyperliner 2y ago
        [dead]
- vamega 2y ago
  Relational databases are not the preferred storage mechanism at Amazon. If a team wants to use an OLTP relational database it’s possible that it will be a decision they will need to defend at any kind of design review. Of course there are relational databases running OLTP workloads, but it’s far away from the norm. There was a program a while ago to shift many RDBMS systems onto something else.
  - emmelaich 2y ago
    So they do joins in code rather than SQL? Wouldn't that risk hiding scaling problems?
    - grogenaut 2y ago
      It can but it's usually more obvious what's happening with code and how to fix it. Amazon wants you to think about the scaling issues while building as they don't want to lose the area under the customer curve on the far right. The theory is that with rdbms you have a magical box that scales vertically until it doesn't. And when it doesn't all you can do is scale back the customers until you fix it with sharding or a re-architecture. Basically you tend to hang yourself with indexes and transactions. Also generally when an RDBMS fails it fails down to like 30% throughput.
      - mickael-kerjean 2y ago
        I recently finished a contract at a company who has gone full on dynamo with the idea that if we have slow queries and dynamo is good for Amazon, then it's good for us too. I've ran some explain on the queries causing issues and of course those queries didn't leverage indexes like they thought ....
        redditor98654 2y ago
        How did you run explain queries on DynamoDB? Or may be you mean something different and I misunderstood you?
saghm 2y ago
> Amazon does a good job of training new hire on the 'Amazon way'. Amazon does 6 pagers, they do design docs. Amazon does SOA. Amazon does not use relational databases. Everything has an API. Because of the 'Amazon way' and the training they do new team members understand at least some of the context and expectations. As a counterpoint, a huge part of Amazon's culture (or at least, AWS's) in my experience was the emphasis on operations and the fact that they didn't have any separation between SREs/on-call engineers from the people who implement their services, and at least for me as someone who had never been on-call before in any meaningful capacity (due to my previous job working on a libraries rather than services), the training for it was basically non-existent on the two teams I spent time on. The "training" I did receive essentially consisted of being put on the rotation once to shadow, where I was able to sort of see what the actual on-call person did but didn't really have any explanation for how to know how to do them other than being told to read the runbooks, which were not really written in a way that was easy to understand for me as someone who was so new to learning all of the internal AWS tooling and ops in general. The next time I came up on the rotation, I was expected to be able to manage on my own, which essentially meant that literally no matter what occurred, I ended up having to escalate because I wasn't knowledgeable enough to fix literally anything within a timeframe that would have been reasonable.
- xwolfi 2y ago
  Which is the only way to learn tbh, you can receive as much positive reinforcement imaginable, nothing prepares you for a large scale incident like living through one, building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure.
  - pyrale 2y ago
    Nothing teaches you to swim like being thrown in the middle of the atlantic.
  - saghm 2y ago
    I'm not really sure what you mean by "positive reinforcement", but I don't think it's possible to disagree more with this sentiment. "building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure" isn't a strategy for teaching for something; it's a coping mechanism for someone trying to brute force their way through something that they weren't adequately trained for. Most people seem to think it's fine for companies to offload the entirety of the burden of learning to individual employees, and maybe I'm an outlier in this regard, but to me, this seems more like a cop out to avoid trying to actually solve the problem at the cost of the employee's emotional health. I'm not surprised that companies default to this, but it's also not surprising that burnout is so common in our industry when this is considered the "best" or "only" way to do things.
infomaniac 2y ago
> Amazon does not use relational databases This is false, at least in my very thin exposure to the company: I interviewed for a team last year which was maintaining EC2 SSH keys using MySQL.
- mnahkies 2y ago
  I watched an interesting talk from the RDS team about how they dogfood RDS the other day https://www.pgevents.ca/events/pgconfdev2024/schedule/session/178-scaling-past-rds-postgresqls-vertical-scaling-limits-lessons-and-guidance-on-the-largest-postgresql-workloads/ https://www.pgevents.ca/events/pgconfdev2024/schedule/sessio...
- chupasaurus 2y ago
  If it was using MyISAM relations are in question.
- 9rx 2y ago
  SQL, and therefore MySQL by extension, isn't relational.
  - snapcaster 2y ago
    He typed smugly, confident that everyone reading this will appreciate his pedantry that totally contributed to the conversation
    - 9rx 2y ago
      A lot to unpack in this comment. 1. Wherein do you find the smugness? It does not speak to any person, let alone the first person. 2. What would give the impression that comments on the internet are written for others? Carly Simon once recorded a popular song about this type of falsehood. 3. It remains that SQL isn't relational. That is why it "won", after all. Relations are too complex for the layman to understand. Tables are much more familiar to the people in charge and arguably a better model for most business problems.

by Sevii 2y ago

rrr_oh_man 2y ago

> Amazon does not use relational databases Huh?

deskglass 2y ago
Friends say they typically use Dynamo and that using a relational database requires approval from a vp (because of scaling concerns).
- hyperliner 2y ago
  Related: Amazon kicked out Oracle from the company. Somewhere along the same timeline, the operational recommendation for teams was to not use relational databases. https://www.theregister.com/2019/10/16/amazon_ditches_oracle/ https://www.theregister.com/2019/10/16/amazon_ditches_oracle...
  - rrr_oh_man 2y ago
    Kicking out Oracle might have to do with factors outside the usage of relational databases...
    - hyperliner 2y ago
      [dead]
vamega 2y ago
Relational databases are not the preferred storage mechanism at Amazon. If a team wants to use an OLTP relational database it’s possible that it will be a decision they will need to defend at any kind of design review. Of course there are relational databases running OLTP workloads, but it’s far away from the norm. There was a program a while ago to shift many RDBMS systems onto something else.
- emmelaich 2y ago
  So they do joins in code rather than SQL? Wouldn't that risk hiding scaling problems?
  - grogenaut 2y ago
    It can but it's usually more obvious what's happening with code and how to fix it. Amazon wants you to think about the scaling issues while building as they don't want to lose the area under the customer curve on the far right. The theory is that with rdbms you have a magical box that scales vertically until it doesn't. And when it doesn't all you can do is scale back the customers until you fix it with sharding or a re-architecture. Basically you tend to hang yourself with indexes and transactions. Also generally when an RDBMS fails it fails down to like 30% throughput.
    - mickael-kerjean 2y ago
      I recently finished a contract at a company who has gone full on dynamo with the idea that if we have slow queries and dynamo is good for Amazon, then it's good for us too. I've ran some explain on the queries causing issues and of course those queries didn't leverage indexes like they thought ....
      - redditor98654 2y ago
        How did you run explain queries on DynamoDB? Or may be you mean something different and I misunderstood you?

saghm 2y ago

> Amazon does a good job of training new hire on the 'Amazon way'. Amazon does 6 pagers, they do design docs. Amazon does SOA. Amazon does not use relational databases. Everything has an API. Because of the 'Amazon way' and the training they do new team members understand at least some of the context and expectations. As a counterpoint, a huge part of Amazon's culture (or at least, AWS's) in my experience was the emphasis on operations and the fact that they didn't have any separation between SREs/on-call engineers from the people who implement their services, and at least for me as someone who had never been on-call before in any meaningful capacity (due to my previous job working on a libraries rather than services), the training for it was basically non-existent on the two teams I spent time on. The "training" I did receive essentially consisted of being put on the rotation once to shadow, where I was able to sort of see what the actual on-call person did but didn't really have any explanation for how to know how to do them other than being told to read the runbooks, which were not really written in a way that was easy to understand for me as someone who was so new to learning all of the internal AWS tooling and ops in general. The next time I came up on the rotation, I was expected to be able to manage on my own, which essentially meant that literally no matter what occurred, I ended up having to escalate because I wasn't knowledgeable enough to fix literally anything within a timeframe that would have been reasonable.

xwolfi 2y ago
Which is the only way to learn tbh, you can receive as much positive reinforcement imaginable, nothing prepares you for a large scale incident like living through one, building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure.
- pyrale 2y ago
  Nothing teaches you to swim like being thrown in the middle of the atlantic.
- saghm 2y ago
  I'm not really sure what you mean by "positive reinforcement", but I don't think it's possible to disagree more with this sentiment. "building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure" isn't a strategy for teaching for something; it's a coping mechanism for someone trying to brute force their way through something that they weren't adequately trained for. Most people seem to think it's fine for companies to offload the entirety of the burden of learning to individual employees, and maybe I'm an outlier in this regard, but to me, this seems more like a cop out to avoid trying to actually solve the problem at the cost of the employee's emotional health. I'm not surprised that companies default to this, but it's also not surprising that burnout is so common in our industry when this is considered the "best" or "only" way to do things.

infomaniac 2y ago

> Amazon does not use relational databases This is false, at least in my very thin exposure to the company: I interviewed for a team last year which was maintaining EC2 SSH keys using MySQL.

mnahkies 2y ago
I watched an interesting talk from the RDS team about how they dogfood RDS the other day https://www.pgevents.ca/events/pgconfdev2024/schedule/session/178-scaling-past-rds-postgresqls-vertical-scaling-limits-lessons-and-guidance-on-the-largest-postgresql-workloads/ https://www.pgevents.ca/events/pgconfdev2024/schedule/sessio...
chupasaurus 2y ago
If it was using MyISAM relations are in question.
9rx 2y ago
SQL, and therefore MySQL by extension, isn't relational.
- snapcaster 2y ago
  He typed smugly, confident that everyone reading this will appreciate his pedantry that totally contributed to the conversation
  - 9rx 2y ago
    A lot to unpack in this comment. 1. Wherein do you find the smugness? It does not speak to any person, let alone the first person. 2. What would give the impression that comments on the internet are written for others? Carly Simon once recorded a popular song about this type of falsehood. 3. It remains that SQL isn't relational. That is why it "won", after all. Relations are too complex for the layman to understand. Tables are much more familiar to the people in charge and arguably a better model for most business problems.