8 ms·

Maybe a silly question on my part; did you manage to learn a lot? In the past, I thought of putting myself in positions like these to increase my understandin

Maybe a silly question on my part; did you manage to learn a lot? In the past, I thought of putting myself in positions like these to increase my understanding in short time, at the short term expense of personal life. I wonder if that's just a stupid thought that leads to nothing but burnout, or actually a viable short-term strategy to get up to speed with some of the tech stack you know but haven't really used in prod at the scale of Amazon. Nowadays, I have health issues (go figure) so it's off the table and all I can do is wonder :).

ngc248 5y ago
You may/may not learn, but living in situations like the OP was put in is horrible. Every day feels like hell, and if you burn out it may take years to recover.
- BossingAround 5y ago
  This is my personal experience, which is why I'd advise people not to go through with the above thought experiment. Then again, I keep interviewing people with 5-10 years of experience, who will go something like this: > How would you manage a server? (leaving out a ton of context about what it means to "manage a server") > Very easy, SSH connection is very fast, reliable, ... > Awesome, love me some SSH, how about 5 servers? > Hmm, probably write shell scripts. > How about 50 servers? > Shell scripts? > Even with 500 servers? > Hire more people..? And I keep feeling that these people don't have 10 YoE, they have 1-2 YoE times some coefficient. I feel like if you're in a "successful rut", that's what it will do to you. You feel like you're successful, but in fact, you have fallen horribly behind, without even knowing it. Your quality of life is probably fine, until you're laid off one day.
  - AyyWS 5y ago
    I updated a custom package on 500 ESX servers using ssh. Just kicked it off asynchronously, used ssh to execute a single command and not get an interactive terminal, and logged the results to check for failures. I was in the support org, so I was pretty constrained on tool choice.
  - ngc248 5y ago
    Agree. There are people with 10 YOE and ppl who have repeated 1 YOE 10 times. Some people get complacent once they get a job and stop learning. The trick is to keep on learning. I have worked for a company like Amazon with aggressive schedules where we used to work for 12 hrs/day. One time I didnt go home for 3 days. I can say I learnt a lot, but there are better companies with mature planning where you would still learn the same amount. How different ppl react to toxic environments is also different. Some people may thrive, while some may wither. I say it is a random walk searching for the best place (for you) to work. You keep iterating until you find what you want.
  - pid-1 5y ago
    In many companies software got so specialized devs won't ever get close to infrastructure stuff. They ask for a VM or a DB to the DevOps / SRE team, wait a bit days and get a URL. If a team uses CI/CD and K8s, they will write a yaml file from a template, wait a few minutes and app will be deployed.
  - DarylZero 5y ago
    You're confusing experience-time with working on big expensive things.
  - Aeolun 5y ago
    > Even with 500 servers? Not sure what answer you expect here? Just because you have 10 years of experience doesn’t mean you’ve ever worked with more than 25 servers at a time (at least I haven’t). I consider it a source of pride, I think. You absolutely don’t need mega scale for most things (or maybe mega scale, but still not many servers)
    - deleted 5y ago
      [deleted]
  - drunkpotato 5y ago
    Whether someone is in a rut or not is a separate issue than whether they've worked with terraform. The key here isn't what technology they've used, it's a mindset towards automation. I recently came off a project that was technically using "GitOps" but where every deployment involved copying and pasting dozens of files across 3 or 4 repos, and the whole thing was very error prone, and if you didn't change exactly the right things your service would just silently fail to deploy, or be unreachable, with zero feedback. I've worked at a company with a bespoke CI/CD system hacked together with PHP scripts and Jenkins servers, but where every single process was ruthlessly automated, reversible, with multiple layers of fallback, failure recovery, garrulous feedback, and built-in approvals that you could automate, or if you didn't remember, you'd get a helpful reminder in email and a link to a page to fill in the required information. People could, and did, know how to do a deployment on it their first or second day. Guess which one was more pleasant to deploy with? It's the automation mindset that's important. You can find people with the right mindset who haven't had a chance yet to play with ansible or terraform, and you can find people who manage to create nightmarish complexities of manual processes out of ansible and terraform. Unfortunately, I haven't yet found this to correlate much with years of experience.
tpxl 5y ago
You will learn more if you're working under a competent lead in a similar situation.
klodolph 5y ago
My go-to aphorism in for questions like these is "you learn more from success than from failure". People like to talk about how great a teacher failure is, and how it teaches you valuable lessons... and it's true! Failure does teach you lots of valuable things, including things that you don't learn by succeeding. However, succeeding at something challenging generally teaches you more. Or to cast it in ML terms... you need examples of both success and failure in your data set, but the "success" data points are more valuable.
- xmprt 5y ago
  Isn't that simply because you experience failure many more times than you experience success. In fact on your path to success you are almost guaranteed to experience multiple failures. So by that metrics one success teaches you the lessons of many failures.
  - DarylZero 5y ago
    No.
    - xmprt 5y ago
      Could you elaborate? What do you learn from a success where everything you did went right? How replicable is a success like that?
      - DarylZero 5y ago
        For example if you apply for jobs and get an interview, you'll learn from it. If you get ghosted and never get an interview, you won't.
greatpostman 5y ago
I’ll be honest, I learned an absurd amount. Like more in those six months than in four years if previous work. It was real software development, with actual distributed systems problems. But was also scarring, I have huge trust issues with management now
- mytailorisrich 5y ago
  > But was also scarring, I have huge trust issues with management now This is the normal approach. You are experienced and wise now.
  - pm90 5y ago
    Is it really? Seems an unnecessarily stressful way to get experience. I kinda agree with the OP, I would have a lot of trust issues with management if they pulled this shit on me.
    - southerntofu 5y ago
      > I would have a lot of trust issues with management if they pulled this shit on me. You should have trust issues with "management" in any corporation because these people have the interest of the shareholders at heart, not of the workers. It's just sad some of us have to go through a traumatic burnout to learn this lesson. Of course, what i said doesn't apply in a self-organized workers cooperatives where management is the workers.
      - mytailorisrich 5y ago
        Yes, exactly my point. I was not commenting that it is normal to have a rather traumatic experience but that it ought to be normal (and frankly, healthy) to have "trust issues with management".
      - DarylZero 5y ago
        They don't have the interest of the shareholders at heart, they have their own career interests at heart. The shareholders try to build a social control structure that aligns these, but it never fully succeeds.
atoav 5y ago
In my experience you can learn from such stress situations, but the goal of a company is usually not to form a bootcamp for new employees, but to have it's devs create great, lasting, secure and stable software. And like any great, secure, lasting and stable engineering giving you the knowledge and time is key here. When a company overloads the safety officer with tasks, so they and up not being able to do their work propperly and an incident happens, the company is at fault because it failed to provide the officer with the resources needed for the job. If they don't give a dev the resources to do their job it is not different in my eyes. The fact that there might be some magical dev who would be able to get something working out of the same circumstances doesn't matter — maybe it has a company crippling flaw in it because of all the haste.
- urthor 5y ago
  Amazon's mentality is pretty clever I think. The entire company is designed to repeatedly test all its employees till failure by piling on the workload until they say uncle, then backing off 20% if they push-back. It's a crazy approach that probably falls apart if you're unsuccessful. But they're very successful so they can just keep burning money to attract new hires. Also, I imagine it's a huge career builder if you work in FMCG or development to have "Amazon retail" or "Amazon Web Services" on your resume. Since everyone uses those platforms. I imagine what is left at the end is a residue of "born again" workaholics, who're probably pretty great at their jobs. Who then ascend to and form middle management. And the global express trundles on, powered by these workaholics.
skinkestek 5y ago
I worked for three years in a extremely customer driven company. The last 1.5 of those I worked with what I consider two amazingly brilliant a####les. I ended up leaving the most amazing place I had ever worked at until then (worldwide traveling, learning new stuff, actual high performance systems). It has taken years to recover and I still cover myself more than I should. My boss (who I liked) tried to get me back twice after he realized I had not been the problem and the two others had left but by then my salary had increased so much I was out of reach for him (edit:,) and telling my family I'd go down a double digit percentage in salary to work for someone who had not stood up for me back the was out of the question.

ngc248 5y ago

You may/may not learn, but living in situations like the OP was put in is horrible. Every day feels like hell, and if you burn out it may take years to recover.

BossingAround 5y ago
This is my personal experience, which is why I'd advise people not to go through with the above thought experiment. Then again, I keep interviewing people with 5-10 years of experience, who will go something like this: > How would you manage a server? (leaving out a ton of context about what it means to "manage a server") > Very easy, SSH connection is very fast, reliable, ... > Awesome, love me some SSH, how about 5 servers? > Hmm, probably write shell scripts. > How about 50 servers? > Shell scripts? > Even with 500 servers? > Hire more people..? And I keep feeling that these people don't have 10 YoE, they have 1-2 YoE times some coefficient. I feel like if you're in a "successful rut", that's what it will do to you. You feel like you're successful, but in fact, you have fallen horribly behind, without even knowing it. Your quality of life is probably fine, until you're laid off one day.
- AyyWS 5y ago
  I updated a custom package on 500 ESX servers using ssh. Just kicked it off asynchronously, used ssh to execute a single command and not get an interactive terminal, and logged the results to check for failures. I was in the support org, so I was pretty constrained on tool choice.
- ngc248 5y ago
  Agree. There are people with 10 YOE and ppl who have repeated 1 YOE 10 times. Some people get complacent once they get a job and stop learning. The trick is to keep on learning. I have worked for a company like Amazon with aggressive schedules where we used to work for 12 hrs/day. One time I didnt go home for 3 days. I can say I learnt a lot, but there are better companies with mature planning where you would still learn the same amount. How different ppl react to toxic environments is also different. Some people may thrive, while some may wither. I say it is a random walk searching for the best place (for you) to work. You keep iterating until you find what you want.
- pid-1 5y ago
  In many companies software got so specialized devs won't ever get close to infrastructure stuff. They ask for a VM or a DB to the DevOps / SRE team, wait a bit days and get a URL. If a team uses CI/CD and K8s, they will write a yaml file from a template, wait a few minutes and app will be deployed.
- DarylZero 5y ago
  You're confusing experience-time with working on big expensive things.
- Aeolun 5y ago
  > Even with 500 servers? Not sure what answer you expect here? Just because you have 10 years of experience doesn’t mean you’ve ever worked with more than 25 servers at a time (at least I haven’t). I consider it a source of pride, I think. You absolutely don’t need mega scale for most things (or maybe mega scale, but still not many servers)
  - deleted 5y ago
    [deleted]
- drunkpotato 5y ago
  Whether someone is in a rut or not is a separate issue than whether they've worked with terraform. The key here isn't what technology they've used, it's a mindset towards automation. I recently came off a project that was technically using "GitOps" but where every deployment involved copying and pasting dozens of files across 3 or 4 repos, and the whole thing was very error prone, and if you didn't change exactly the right things your service would just silently fail to deploy, or be unreachable, with zero feedback. I've worked at a company with a bespoke CI/CD system hacked together with PHP scripts and Jenkins servers, but where every single process was ruthlessly automated, reversible, with multiple layers of fallback, failure recovery, garrulous feedback, and built-in approvals that you could automate, or if you didn't remember, you'd get a helpful reminder in email and a link to a page to fill in the required information. People could, and did, know how to do a deployment on it their first or second day. Guess which one was more pleasant to deploy with? It's the automation mindset that's important. You can find people with the right mindset who haven't had a chance yet to play with ansible or terraform, and you can find people who manage to create nightmarish complexities of manual processes out of ansible and terraform. Unfortunately, I haven't yet found this to correlate much with years of experience.

tpxl 5y ago

You will learn more if you're working under a competent lead in a similar situation.

klodolph 5y ago

My go-to aphorism in for questions like these is "you learn more from success than from failure". People like to talk about how great a teacher failure is, and how it teaches you valuable lessons... and it's true! Failure does teach you lots of valuable things, including things that you don't learn by succeeding. However, succeeding at something challenging generally teaches you more. Or to cast it in ML terms... you need examples of both success and failure in your data set, but the "success" data points are more valuable.

xmprt 5y ago
Isn't that simply because you experience failure many more times than you experience success. In fact on your path to success you are almost guaranteed to experience multiple failures. So by that metrics one success teaches you the lessons of many failures.
- DarylZero 5y ago
  No.
  - xmprt 5y ago
    Could you elaborate? What do you learn from a success where everything you did went right? How replicable is a success like that?
    - DarylZero 5y ago
      For example if you apply for jobs and get an interview, you'll learn from it. If you get ghosted and never get an interview, you won't.

greatpostman 5y ago

I’ll be honest, I learned an absurd amount. Like more in those six months than in four years if previous work. It was real software development, with actual distributed systems problems. But was also scarring, I have huge trust issues with management now

mytailorisrich 5y ago
> But was also scarring, I have huge trust issues with management now This is the normal approach. You are experienced and wise now.
- pm90 5y ago
  Is it really? Seems an unnecessarily stressful way to get experience. I kinda agree with the OP, I would have a lot of trust issues with management if they pulled this shit on me.
  - southerntofu 5y ago
    > I would have a lot of trust issues with management if they pulled this shit on me. You should have trust issues with "management" in any corporation because these people have the interest of the shareholders at heart, not of the workers. It's just sad some of us have to go through a traumatic burnout to learn this lesson. Of course, what i said doesn't apply in a self-organized workers cooperatives where management is the workers.
    - mytailorisrich 5y ago
      Yes, exactly my point. I was not commenting that it is normal to have a rather traumatic experience but that it ought to be normal (and frankly, healthy) to have "trust issues with management".
    - DarylZero 5y ago
      They don't have the interest of the shareholders at heart, they have their own career interests at heart. The shareholders try to build a social control structure that aligns these, but it never fully succeeds.

atoav 5y ago

In my experience you can learn from such stress situations, but the goal of a company is usually not to form a bootcamp for new employees, but to have it's devs create great, lasting, secure and stable software. And like any great, secure, lasting and stable engineering giving you the knowledge and time is key here. When a company overloads the safety officer with tasks, so they and up not being able to do their work propperly and an incident happens, the company is at fault because it failed to provide the officer with the resources needed for the job. If they don't give a dev the resources to do their job it is not different in my eyes. The fact that there might be some magical dev who would be able to get something working out of the same circumstances doesn't matter — maybe it has a company crippling flaw in it because of all the haste.

urthor 5y ago
Amazon's mentality is pretty clever I think. The entire company is designed to repeatedly test all its employees till failure by piling on the workload until they say uncle, then backing off 20% if they push-back. It's a crazy approach that probably falls apart if you're unsuccessful. But they're very successful so they can just keep burning money to attract new hires. Also, I imagine it's a huge career builder if you work in FMCG or development to have "Amazon retail" or "Amazon Web Services" on your resume. Since everyone uses those platforms. I imagine what is left at the end is a residue of "born again" workaholics, who're probably pretty great at their jobs. Who then ascend to and form middle management. And the global express trundles on, powered by these workaholics.

skinkestek 5y ago

I worked for three years in a extremely customer driven company. The last 1.5 of those I worked with what I consider two amazingly brilliant a####les. I ended up leaving the most amazing place I had ever worked at until then (worldwide traveling, learning new stuff, actual high performance systems). It has taken years to recover and I still cover myself more than I should. My boss (who I liked) tried to get me back twice after he realized I had not been the problem and the two others had left but by then my salary had increased so much I was out of reach for him (edit:,) and telling my family I'd go down a double digit percentage in salary to work for someone who had not stood up for me back the was out of the question.