28 ms·

I’ve worked at Amazon. I made it a few years, but was almost pipped in the first few months. Management dropped a huge project on me, a brand new tier 1 service

I’ve worked at Amazon. I made it a few years, but was almost pipped in the first few months. Management dropped a huge project on me, a brand new tier 1 service, with full dns resolution, api, database, distributed system, along with micro services. The deadline was three months. Brand new tech stack. When I was struggling to meet the deadline, we started having “performance conversations”. I ended up switching teams and had a decent time at the company, but the nightmare situations are very real. You are replaceable and there’s no mercy

Aeolun 5y ago
> You are replaceable and there’s no mercy I don’t understand this though? Eventually someone’s head needs to roll. If you fail your project and get fired, how is your manager immune from consequence? They failed too (probably more).
BossingAround 5y ago
Maybe a silly question on my part; did you manage to learn a lot? In the past, I thought of putting myself in positions like these to increase my understanding in short time, at the short term expense of personal life. I wonder if that's just a stupid thought that leads to nothing but burnout, or actually a viable short-term strategy to get up to speed with some of the tech stack you know but haven't really used in prod at the scale of Amazon. Nowadays, I have health issues (go figure) so it's off the table and all I can do is wonder :).
- ngc248 5y ago
  You may/may not learn, but living in situations like the OP was put in is horrible. Every day feels like hell, and if you burn out it may take years to recover.
  - BossingAround 5y ago
    This is my personal experience, which is why I'd advise people not to go through with the above thought experiment. Then again, I keep interviewing people with 5-10 years of experience, who will go something like this: > How would you manage a server? (leaving out a ton of context about what it means to "manage a server") > Very easy, SSH connection is very fast, reliable, ... > Awesome, love me some SSH, how about 5 servers? > Hmm, probably write shell scripts. > How about 50 servers? > Shell scripts? > Even with 500 servers? > Hire more people..? And I keep feeling that these people don't have 10 YoE, they have 1-2 YoE times some coefficient. I feel like if you're in a "successful rut", that's what it will do to you. You feel like you're successful, but in fact, you have fallen horribly behind, without even knowing it. Your quality of life is probably fine, until you're laid off one day.
    - AyyWS 5y ago
      I updated a custom package on 500 ESX servers using ssh. Just kicked it off asynchronously, used ssh to execute a single command and not get an interactive terminal, and logged the results to check for failures. I was in the support org, so I was pretty constrained on tool choice.
    - ngc248 5y ago
      Agree. There are people with 10 YOE and ppl who have repeated 1 YOE 10 times. Some people get complacent once they get a job and stop learning. The trick is to keep on learning. I have worked for a company like Amazon with aggressive schedules where we used to work for 12 hrs/day. One time I didnt go home for 3 days. I can say I learnt a lot, but there are better companies with mature planning where you would still learn the same amount. How different ppl react to toxic environments is also different. Some people may thrive, while some may wither. I say it is a random walk searching for the best place (for you) to work. You keep iterating until you find what you want.
    - pid-1 5y ago
      In many companies software got so specialized devs won't ever get close to infrastructure stuff. They ask for a VM or a DB to the DevOps / SRE team, wait a bit days and get a URL. If a team uses CI/CD and K8s, they will write a yaml file from a template, wait a few minutes and app will be deployed.
    - DarylZero 5y ago
      You're confusing experience-time with working on big expensive things.
    - Aeolun 5y ago
      > Even with 500 servers? Not sure what answer you expect here? Just because you have 10 years of experience doesn’t mean you’ve ever worked with more than 25 servers at a time (at least I haven’t). I consider it a source of pride, I think. You absolutely don’t need mega scale for most things (or maybe mega scale, but still not many servers)
      - deleted 5y ago
- tpxl 5y ago
  You will learn more if you're working under a competent lead in a similar situation.
- klodolph 5y ago
  My go-to aphorism in for questions like these is "you learn more from success than from failure". People like to talk about how great a teacher failure is, and how it teaches you valuable lessons... and it's true! Failure does teach you lots of valuable things, including things that you don't learn by succeeding. However, succeeding at something challenging generally teaches you more. Or to cast it in ML terms... you need examples of both success and failure in your data set, but the "success" data points are more valuable.
  - xmprt 5y ago
    Isn't that simply because you experience failure many more times than you experience success. In fact on your path to success you are almost guaranteed to experience multiple failures. So by that metrics one success teaches you the lessons of many failures.
    - DarylZero 5y ago
      No.
      - xmprt 5y ago
        Could you elaborate? What do you learn from a success where everything you did went right? How replicable is a success like that?
        DarylZero 5y ago
        For example if you apply for jobs and get an interview, you'll learn from it. If you get ghosted and never get an interview, you won't.
- greatpostman 5y ago
  I’ll be honest, I learned an absurd amount. Like more in those six months than in four years if previous work. It was real software development, with actual distributed systems problems. But was also scarring, I have huge trust issues with management now
  - mytailorisrich 5y ago
    > But was also scarring, I have huge trust issues with management now This is the normal approach. You are experienced and wise now.
    - pm90 5y ago
      Is it really? Seems an unnecessarily stressful way to get experience. I kinda agree with the OP, I would have a lot of trust issues with management if they pulled this shit on me.
      - southerntofu 5y ago
        > I would have a lot of trust issues with management if they pulled this shit on me. You should have trust issues with "management" in any corporation because these people have the interest of the shareholders at heart, not of the workers. It's just sad some of us have to go through a traumatic burnout to learn this lesson. Of course, what i said doesn't apply in a self-organized workers cooperatives where management is the workers.
        mytailorisrich 5y ago
        Yes, exactly my point. I was not commenting that it is normal to have a rather traumatic experience but that it ought to be normal (and frankly, healthy) to have "trust issues with management".
        DarylZero 5y ago
        They don't have the interest of the shareholders at heart, they have their own career interests at heart. The shareholders try to build a social control structure that aligns these, but it never fully succeeds.
- atoav 5y ago
  In my experience you can learn from such stress situations, but the goal of a company is usually not to form a bootcamp for new employees, but to have it's devs create great, lasting, secure and stable software. And like any great, secure, lasting and stable engineering giving you the knowledge and time is key here. When a company overloads the safety officer with tasks, so they and up not being able to do their work propperly and an incident happens, the company is at fault because it failed to provide the officer with the resources needed for the job. If they don't give a dev the resources to do their job it is not different in my eyes. The fact that there might be some magical dev who would be able to get something working out of the same circumstances doesn't matter — maybe it has a company crippling flaw in it because of all the haste.
  - urthor 5y ago
    Amazon's mentality is pretty clever I think. The entire company is designed to repeatedly test all its employees till failure by piling on the workload until they say uncle, then backing off 20% if they push-back. It's a crazy approach that probably falls apart if you're unsuccessful. But they're very successful so they can just keep burning money to attract new hires. Also, I imagine it's a huge career builder if you work in FMCG or development to have "Amazon retail" or "Amazon Web Services" on your resume. Since everyone uses those platforms. I imagine what is left at the end is a residue of "born again" workaholics, who're probably pretty great at their jobs. Who then ascend to and form middle management. And the global express trundles on, powered by these workaholics.
- skinkestek 5y ago
  I worked for three years in a extremely customer driven company. The last 1.5 of those I worked with what I consider two amazingly brilliant a####les. I ended up leaving the most amazing place I had ever worked at until then (worldwide traveling, learning new stuff, actual high performance systems). It has taken years to recover and I still cover myself more than I should. My boss (who I liked) tried to get me back twice after he realized I had not been the problem and the two others had left but by then my salary had increased so much I was out of reach for him (edit:,) and telling my family I'd go down a double digit percentage in salary to work for someone who had not stood up for me back the was out of the question.
cabirum 5y ago
No idea how things work at Amazon, but why did you accept such a tight deadline? Have you provided your own estimates for dev time?
- ngc248 5y ago
  In big companies, there is program management which comes up with the overall schedule. There will be buffer time, but you still have to work within the given time whatever challenges there may be.
- slt2021 5y ago
  amazon has Leadership Principles (corporate koolaid) that makes deadlines top-down and non-negotiable, otherwise you are deemed as not meeting a bar
  - deltaonefour 5y ago
    What if the deadline is fundamentally impossible? No room for leeway here?
    - MattGaiser 5y ago
      At one company I worked for, they fired the product manager, then fired all of them.
    - altacc 5y ago
      Then either the people given the deadline are being "managed out of the business" or they should be managing their own way out And yes, I've been a manager in a company where the phrase "managed out of the business" was used. I no longer work there.
      - bregma 5y ago
        The term that is usually used in law is "constructive dismissal".
Cthulhu_ 5y ago
I mean, you even attempted it. I would've looked at it and told them their deadline or scope was unrealistic. I mean I don't want to do any project with a deadline, ever again, if possible, and I haven't even been exposed to the level of stress and pressure that people get at certain companies. I thought deadlines had gotten rid of a decade ago in software engineering with the rise of agile software development and the realization that software is difficult to neatly scope, and development time is almost impossible to predict.
- mjrbrennan 5y ago
  I’m in the same boat as you, haven’t worked on anything with a deadline for the past 2 and a half years and I never want to go back to it, totally pointless arbitrary dates that are never met.
hughrr 5y ago
A side effect of working for asshole companies like that for years is I’ve slowly developed a zero fucks attitude and an immunity to threats and carrots dangled in front of me. I can sit there as a project is falling over the edge of a cliff with total inner peace. My interest stops at the pay cheque and by the clock.
- square_usual 5y ago
  > I can sit there as a project is falling over the edge of a cliff with total inner peace. This is what I'm (slowly) learning as well, even though I'm not entirely sure I want to. Working on projects where you care but almost nobody else does does that to you, but I'm afraid I won't be able to make it back to a position where I can care for projects.
  - hughrr 5y ago
    Don’t sweat it; it might be healthy. I have made it a feature of my life that I don’t actually care about work and will not be changing it. It turned out that I’d spent at least 20 years being tricked into caring for things which ultimately someone else leveraged most of the benefit for with measly returns by proportion on that investment of time back to me. Never a thank you, always difficult to extract any entitlements from it. Another thing to note is my father who had completely invested his entire life into this way of thinking and is currently spending his retirement with bad mental and physical health. That’s the end game these working practices tend to lead into. It’s true that you should work to live, not live to work.
    - DarylZero 5y ago
      But your life is spent at work, so if you don't care, your life is spent doing something you don't care about.
      - np- 5y ago
        You can care about your career and your self improvement while still maintaining healthy boundaries around the mental effort expended for your job. We’re in it for a long time but jobs come and go
giantg2 5y ago
"You are replaceable and there’s no mercy" Sounds generally like most jobs.
abledon 5y ago
Having all that responsibility were you SDE-2 ? SDE-3 ?
wing-_-nuts 5y ago
I'm surprised you were allowed to switch teams. Most companies explicitly disallow it for people under any sort of performance review (even informal).

Aeolun 5y ago

> You are replaceable and there’s no mercy I don’t understand this though? Eventually someone’s head needs to roll. If you fail your project and get fired, how is your manager immune from consequence? They failed too (probably more).

BossingAround 5y ago

Maybe a silly question on my part; did you manage to learn a lot? In the past, I thought of putting myself in positions like these to increase my understanding in short time, at the short term expense of personal life. I wonder if that's just a stupid thought that leads to nothing but burnout, or actually a viable short-term strategy to get up to speed with some of the tech stack you know but haven't really used in prod at the scale of Amazon. Nowadays, I have health issues (go figure) so it's off the table and all I can do is wonder :).

ngc248 5y ago
You may/may not learn, but living in situations like the OP was put in is horrible. Every day feels like hell, and if you burn out it may take years to recover.
- BossingAround 5y ago
  This is my personal experience, which is why I'd advise people not to go through with the above thought experiment. Then again, I keep interviewing people with 5-10 years of experience, who will go something like this: > How would you manage a server? (leaving out a ton of context about what it means to "manage a server") > Very easy, SSH connection is very fast, reliable, ... > Awesome, love me some SSH, how about 5 servers? > Hmm, probably write shell scripts. > How about 50 servers? > Shell scripts? > Even with 500 servers? > Hire more people..? And I keep feeling that these people don't have 10 YoE, they have 1-2 YoE times some coefficient. I feel like if you're in a "successful rut", that's what it will do to you. You feel like you're successful, but in fact, you have fallen horribly behind, without even knowing it. Your quality of life is probably fine, until you're laid off one day.
  - AyyWS 5y ago
    I updated a custom package on 500 ESX servers using ssh. Just kicked it off asynchronously, used ssh to execute a single command and not get an interactive terminal, and logged the results to check for failures. I was in the support org, so I was pretty constrained on tool choice.
  - ngc248 5y ago
    Agree. There are people with 10 YOE and ppl who have repeated 1 YOE 10 times. Some people get complacent once they get a job and stop learning. The trick is to keep on learning. I have worked for a company like Amazon with aggressive schedules where we used to work for 12 hrs/day. One time I didnt go home for 3 days. I can say I learnt a lot, but there are better companies with mature planning where you would still learn the same amount. How different ppl react to toxic environments is also different. Some people may thrive, while some may wither. I say it is a random walk searching for the best place (for you) to work. You keep iterating until you find what you want.
  - pid-1 5y ago
    In many companies software got so specialized devs won't ever get close to infrastructure stuff. They ask for a VM or a DB to the DevOps / SRE team, wait a bit days and get a URL. If a team uses CI/CD and K8s, they will write a yaml file from a template, wait a few minutes and app will be deployed.
  - DarylZero 5y ago
    You're confusing experience-time with working on big expensive things.
  - Aeolun 5y ago
    > Even with 500 servers? Not sure what answer you expect here? Just because you have 10 years of experience doesn’t mean you’ve ever worked with more than 25 servers at a time (at least I haven’t). I consider it a source of pride, I think. You absolutely don’t need mega scale for most things (or maybe mega scale, but still not many servers)
    - deleted 5y ago
tpxl 5y ago
You will learn more if you're working under a competent lead in a similar situation.
klodolph 5y ago
My go-to aphorism in for questions like these is "you learn more from success than from failure". People like to talk about how great a teacher failure is, and how it teaches you valuable lessons... and it's true! Failure does teach you lots of valuable things, including things that you don't learn by succeeding. However, succeeding at something challenging generally teaches you more. Or to cast it in ML terms... you need examples of both success and failure in your data set, but the "success" data points are more valuable.
- xmprt 5y ago
  Isn't that simply because you experience failure many more times than you experience success. In fact on your path to success you are almost guaranteed to experience multiple failures. So by that metrics one success teaches you the lessons of many failures.
  - DarylZero 5y ago
    No.
    - xmprt 5y ago
      Could you elaborate? What do you learn from a success where everything you did went right? How replicable is a success like that?
      - DarylZero 5y ago
        For example if you apply for jobs and get an interview, you'll learn from it. If you get ghosted and never get an interview, you won't.
greatpostman 5y ago
I’ll be honest, I learned an absurd amount. Like more in those six months than in four years if previous work. It was real software development, with actual distributed systems problems. But was also scarring, I have huge trust issues with management now
- mytailorisrich 5y ago
  > But was also scarring, I have huge trust issues with management now This is the normal approach. You are experienced and wise now.
  - pm90 5y ago
    Is it really? Seems an unnecessarily stressful way to get experience. I kinda agree with the OP, I would have a lot of trust issues with management if they pulled this shit on me.
    - southerntofu 5y ago
      > I would have a lot of trust issues with management if they pulled this shit on me. You should have trust issues with "management" in any corporation because these people have the interest of the shareholders at heart, not of the workers. It's just sad some of us have to go through a traumatic burnout to learn this lesson. Of course, what i said doesn't apply in a self-organized workers cooperatives where management is the workers.
      - mytailorisrich 5y ago
        Yes, exactly my point. I was not commenting that it is normal to have a rather traumatic experience but that it ought to be normal (and frankly, healthy) to have "trust issues with management".
      - DarylZero 5y ago
        They don't have the interest of the shareholders at heart, they have their own career interests at heart. The shareholders try to build a social control structure that aligns these, but it never fully succeeds.
atoav 5y ago
In my experience you can learn from such stress situations, but the goal of a company is usually not to form a bootcamp for new employees, but to have it's devs create great, lasting, secure and stable software. And like any great, secure, lasting and stable engineering giving you the knowledge and time is key here. When a company overloads the safety officer with tasks, so they and up not being able to do their work propperly and an incident happens, the company is at fault because it failed to provide the officer with the resources needed for the job. If they don't give a dev the resources to do their job it is not different in my eyes. The fact that there might be some magical dev who would be able to get something working out of the same circumstances doesn't matter — maybe it has a company crippling flaw in it because of all the haste.
- urthor 5y ago
  Amazon's mentality is pretty clever I think. The entire company is designed to repeatedly test all its employees till failure by piling on the workload until they say uncle, then backing off 20% if they push-back. It's a crazy approach that probably falls apart if you're unsuccessful. But they're very successful so they can just keep burning money to attract new hires. Also, I imagine it's a huge career builder if you work in FMCG or development to have "Amazon retail" or "Amazon Web Services" on your resume. Since everyone uses those platforms. I imagine what is left at the end is a residue of "born again" workaholics, who're probably pretty great at their jobs. Who then ascend to and form middle management. And the global express trundles on, powered by these workaholics.
skinkestek 5y ago
I worked for three years in a extremely customer driven company. The last 1.5 of those I worked with what I consider two amazingly brilliant a####les. I ended up leaving the most amazing place I had ever worked at until then (worldwide traveling, learning new stuff, actual high performance systems). It has taken years to recover and I still cover myself more than I should. My boss (who I liked) tried to get me back twice after he realized I had not been the problem and the two others had left but by then my salary had increased so much I was out of reach for him (edit:,) and telling my family I'd go down a double digit percentage in salary to work for someone who had not stood up for me back the was out of the question.

cabirum 5y ago

No idea how things work at Amazon, but why did you accept such a tight deadline? Have you provided your own estimates for dev time?

ngc248 5y ago
In big companies, there is program management which comes up with the overall schedule. There will be buffer time, but you still have to work within the given time whatever challenges there may be.
slt2021 5y ago
amazon has Leadership Principles (corporate koolaid) that makes deadlines top-down and non-negotiable, otherwise you are deemed as not meeting a bar
- deltaonefour 5y ago
  What if the deadline is fundamentally impossible? No room for leeway here?
  - MattGaiser 5y ago
    At one company I worked for, they fired the product manager, then fired all of them.
  - altacc 5y ago
    Then either the people given the deadline are being "managed out of the business" or they should be managing their own way out And yes, I've been a manager in a company where the phrase "managed out of the business" was used. I no longer work there.
    - bregma 5y ago
      The term that is usually used in law is "constructive dismissal".

Cthulhu_ 5y ago

I mean, you even attempted it. I would've looked at it and told them their deadline or scope was unrealistic. I mean I don't want to do any project with a deadline, ever again, if possible, and I haven't even been exposed to the level of stress and pressure that people get at certain companies. I thought deadlines had gotten rid of a decade ago in software engineering with the rise of agile software development and the realization that software is difficult to neatly scope, and development time is almost impossible to predict.

mjrbrennan 5y ago
I’m in the same boat as you, haven’t worked on anything with a deadline for the past 2 and a half years and I never want to go back to it, totally pointless arbitrary dates that are never met.

hughrr 5y ago

A side effect of working for asshole companies like that for years is I’ve slowly developed a zero fucks attitude and an immunity to threats and carrots dangled in front of me. I can sit there as a project is falling over the edge of a cliff with total inner peace. My interest stops at the pay cheque and by the clock.

square_usual 5y ago
> I can sit there as a project is falling over the edge of a cliff with total inner peace. This is what I'm (slowly) learning as well, even though I'm not entirely sure I want to. Working on projects where you care but almost nobody else does does that to you, but I'm afraid I won't be able to make it back to a position where I can care for projects.
- hughrr 5y ago
  Don’t sweat it; it might be healthy. I have made it a feature of my life that I don’t actually care about work and will not be changing it. It turned out that I’d spent at least 20 years being tricked into caring for things which ultimately someone else leveraged most of the benefit for with measly returns by proportion on that investment of time back to me. Never a thank you, always difficult to extract any entitlements from it. Another thing to note is my father who had completely invested his entire life into this way of thinking and is currently spending his retirement with bad mental and physical health. That’s the end game these working practices tend to lead into. It’s true that you should work to live, not live to work.
  - DarylZero 5y ago
    But your life is spent at work, so if you don't care, your life is spent doing something you don't care about.
    - np- 5y ago
      You can care about your career and your self improvement while still maintaining healthy boundaries around the mental effort expended for your job. We’re in it for a long time but jobs come and go

giantg2 5y ago

"You are replaceable and there’s no mercy" Sounds generally like most jobs.

abledon 5y ago

Having all that responsibility were you SDE-2 ? SDE-3 ?

wing-_-nuts 5y ago

I'm surprised you were allowed to switch teams. Most companies explicitly disallow it for people under any sort of performance review (even informal).