53 ms·

Laying myself off from Amazon

• yolovoe 4y ago
  I want to offer a different perspective. I work on EC2 and my experience is completely different. Lots of good internal tooling. When something critical to us breaks, we usually cut high severity ticket to get other team's attention. We rarely have actual customer affecting issues in production, there's no PIP threat, promotions feel achievable within a couple years if you really want them and the extra responsibility, automation is very helpful and we're constantly looking for ways to reduce our operational load. Yes, testing could be better. Yes, we could have more engineers. Yes, oncall could be less stressful. Yes, I spend 10+ hours on meetings every week. Yes, I write a lot of documents and do a lot if ops to debug issues. I have probably merged just 10-20 lines if code in the last couple months. But EC2 dataplane is very complicated. Every small change needs to be given a lot of thought. Success and scale bring responsibility. Plus, I've learned so much here. I'm never bored. And my work life balance is good overall. Slightly less than 40 serious pomodoro hours a week except when oncall. Ec2 is a very stable system and we keep making it more stable for our customers. Do use it. It's not brittle at all. Expect occasional instance failures and design around that with cross-AZ redundancy and you should be golden.
• ArcMex 4y ago
  Daniel. I can only wish you the best. 6 months ago, I was in your shoes. Today, I have my own startup building applications I am passionate about. I have never been happier. I’m grateful for everything that I’ve learned and endured so far. I just wanted to tell you that I hope you stay positive even during the dark days and that you maintain your integrity. Truly, I wish you the very best. Regards, Daniel :)
• ezoe 4y ago
  Let's face it. Most programing jobs are a bullshit job. I've been chasing obscure bugs for weeks and ended up fixing a few lines of code some guy wrote years ago and it improve the software so little. It's really difficult to keep the motivation besides the money.
• lowbloodsugar 4y ago
  So twice as much time writing unit tests for your code as writing your code (20% of total vs 10% of total), and that's a complaint? Got it. But, sure, it's everyone elses code that is a shitshow. Ok.
• r0rshrk 4y ago
  I would like to say, I am going through a very similar experience at Meta. Is there any Big Tech company where internal tooling isn't broken for front-end development? Does such a company exist?
• retrocryptid 4y ago
  I'm reminded of the quip: "Amazon is a place where smart people go to feel bad about themselves."
• CelticBard 4y ago
  This was a good read. I guess we do have to rip off the bandaid sometimes.
  • dimmke 4y ago
    Thanks. An upvote would help, I'd love to have more people read it.
• nyarlathotep_ 4y ago
  Worked for AWS for just under a year and a half; this mirrors my experience exactly. Poorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter. Automated tooling and tests that were ostensibly created to make life easier were the number one pain point and constantly failed in obscure ways that required cutting tickets to teams responsible for the automation. More often than not these tickets would be ignored and issues would persist for weeks beyond what's reasonable. I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use. That, paired with the awful corporate culture and constant fear of PIP/job loss forced me to finally leave. Generally a very unpleasant experience all around, sans the compensation and name on the resume.
  • dimmke 4y ago
    >I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use. I did not mention this in my blog post, but I have actually done similar. That's incredible. Thanks for sharing this comment. Our tenures were almost the exact same length too.
    • scruple 4y ago
      When I worked for a BigTechCo, I was doing exercism every single morning for ~30 minutes for the same reason. It really is incredible. I know a few of my former colleagues were doing the same. Today I'm on a team with 4 other engineers, so I keep busy coding.
      • arcturus17 4y ago
        I cannot code in prod anymore and have definitely been looking to develop a habit like this. Any tips from you and others on: a) How to make it stick b) How to maximize its efficacy/efficiency Would be awesome.
        • 4y ago
• bryanlarsen 4y ago
  No voluntary layoff plan you could take advantage of to get some severance pay? Might have been worth holding off a week to see if they rolled one of those out. Water under the bridge now, though.
  • yumbrand 4y ago
    Literally today -- about three hours ago, Amazon extended voluntary buyout offers to some employees.
    • FunnyLookinHat 4y ago
      Yup - I heard about this from a friend of mine. The offer seemed pretty good: lump sum 4 - 9 months base salary depending on tenure, extra for COBRA for 12 weeks, benefits through end of the year, etc.
• adamwk 4y ago
  Amazon's internal tooling really is the ninth circle of hell. It seems unsalvageable too; ostensibly the teams are supposed to be customer obsessed and the developers are the customer, but there's no accountability for anything to work well at all.
  • yathaid 4y ago
    Can you elaborate on this a bit? I am really interested in which area of the tools space was annoying to this extent.
  • philsnow 4y ago
    Well what are they (the developer-customers) going to do, buy from another vendor?
  • feyr 4y ago
    Buildertool's guiding leadership principle is "Contempt for their Customers"
• stuff4ben 4y ago
  How was the work/life balance though? Did you find you were able to have downtime and rest or were you constantly worrying about unfinished work and looming deadlines?
  • dangerwill 4y ago
    I don't think I've ever heard a testimony of Amazon having good work life balance. The best you hear is that some people get lucky and things are "ok" on that front.
    • belval 4y ago
      My work life balance at Amazon as an SDE is great but I don't have 24/7 oncall, make of that what you will. That being said, I agree that the internal tooling at Amazon is not that great. It's actually a place where newer employees tend to clash with older Amazonian because it compares unfavourably with GitHub actions or any modern CI/CD while being significantly better than anything that existed in the early 2010s. I've heard a lot of testimonies of SDEs maintaining pipelines that are making them miserable.
      • scubbo 4y ago
        This is really baffling to me. In fact, since leaving Amazon in June, I've been really frustrated by how much extra work setting up a CI/CD pipeline is in (say) Drone and Argo/Flux than with Amazon's internal tooling. You can set up a standard Amazon pipeline with a single command and answering a few prompts about naming, whereas with the open-source systems you have to hand-craft everything yourself, hack in a way to update the infra repo every time the source repo changes (I blogged more about that here: https://blog.scubbo.org/posts/ci-cd-cd-oh-my/ https://blog.scubbo.org/posts/ci-cd-cd-oh-my/), and it seems like Drone literally doesn't have metrics that indicate broken builds. To say nothing of how well-integrated Pipelines is with alerting and ticketing - yet _another_ integration you would have to build for yourself in OSS-land. Literally the only advantage I'm aware of for OSS systems is that they allow you to use whatever language, dependencies, build system, etc. that you want - which, yes, fair, if that's something that you need, then that's a deal-breaker, but for the 99.99...% of internal cases that Pipelines works for, it (seems to me to) work flawlessly and smoothly. What are some unfavourable comparisons that I'm missing?
• swyx 4y ago
  same reasons here that contributed towards deciding leaving Amazon despite it being a dream job entering the industry (i did have a significant "pull factor" to another job that accelerated the timing). when you realize you wouldn't be happy even with the best case, it's time to go. However, i've heard many reports of talented, smart people doing fulfilling, challenging work at Amazon and its probably the luck of the draw on what team you get staffed on. So the case could be made that you couldve simply tried for a transfer. i suspect that, if you work on a "slow and kludgy product", the responsible thing to do would have been to write a memo laying out why it sucks and what needs to be done. I've seen a rare few folks actually get heard and get the mandate to do what needed to be done. but I know it can feel overwhelming for most and ultimately its the responsibility of the GM/PM to know whether this is true or just the whining of an underling with no context. also kudos for having the guts/integrity to do this before thanksgiving. make it count!
• d_burfoot 4y ago
  > 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc… This gives the lie to Jeff Bezos' whole "Day 1" philosophy. If you're at a Day 1 company you don't spend 40% of your time fighting with the crappy internal tools.
  • dilyevsky 4y ago
    Not sure what the Amzn situation is but something I've noticed at few previous gigs that there're some folks who claim they spent significant time wrestling with tooling but when you actually go in and make them formulate the issue it's either something really obvious or a 10-min google/github search like 9/10 times. These also weren't coming from people who I considered lazy or unintelligent (well, for the most part) so I can't really explain it logically. I suspect it may have something to do with motivation or cultural issues ("not my job" sort of folks).
    • zug_zug 4y ago
      I've seen people complain about it who were wrong, but I've also seen people fail to complain about it when they really should have been. I just worked at a fang company, and here's an example - to search CI logs to see if a certain log happened frequently you would : WRITE A SCRIPT to download 100MB CI logs and search them for you.
      • dilyevsky 4y ago
        Your example does not sound as outrageous as you seem to think - I can imagine a number of scenarios where this would be totally acceptable solution assuming we’re talking about software engineers here. Consider the fact that most startups don’t even have dedicated teams running CI unlike at fangs and it’s up to devs to set everything up. Yet I don’t really hear about folks complaining about internal tooling at early startups
        • zug_zug 4y ago
          The fact that you don't think that's ridiculous is a great example of terrible tooling be accepted as "normal." Any startup can get this by piping their jenkins (etc) logs into ELK/Splunk/Sumo. And indeed this is exactly what I built at the last startup I worked for. But again I was talking about a FAANG company not having a way to search logs (again, each individual log file is 100mb and there are 100k of them generated a day). If it's not clear why that's entirely unacceptable, imagine your team is running the CI for over 10,000 engineers and you're landing changes to this CI system daily. Engineers are seeing all kinds of logs and bugs daily and you need to ascertain if these errors are new, unique to some subset of jobs, lead to failed jobs, etc.
• unnouinceput 4y ago
  This mirrors my experience with my prestigious job I had at SiemensVDO in 2005/2006. Same stuff of wasting time on anything else than coding. He said 10% of his time was coding. I suspect mine was like 20% and I felt miserable after a year. The honeymoon of getting to write embedded code for real cars, cars that would be driven by millions of people only lasted like 3 to 4 months. After that the job had nothing else to teach me and over a little year later I quit too, with a total of one month shy of 1.5 years there.
• amzn-throw 4y ago
  Amazon has a lot of bad internal tools, but this person's experience doesn't match mine (being here for 8 years) at all > 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc… The tools for code submission, pull requests, pipelines, metrics, and logging are fantastic. Google is better. Most companies aren't. I have never spent 40% of my time battling internal tools.... > 20% of my time in meetings Developers complain when they're not invited to meetings, and they complain when they're invited. On my team we brutally introspect the value of every meeting, and if it looks like it's not delivering value, we find a new process. > 20% of my time writing unit tests to hit the 100% coverage requirement of the codebase I worked on. This makes no sense. This isn't a company mandate, every team is free to determine what code coverage percentage makes sense for them. Give this feedback to your tech lead, nearest Sr. SDE or PE -> 100% test coverage should never be "required" > 10% of my time tracking down bugs in other team’s codebases for either internal tools or frameworks and trying to get them to acknowledge the problem by filing tickets. So, software engineering?
  • zeroonetwothree 4y ago
    I’ve never complained about not being invited to a meeting. I’d be happy never going to a single one.
  • fdgsdfogijq 4y ago
    Yeah I actually think this guy was just underperforming
  • marcinzm 4y ago
    >I have never spent 40% of my time battling internal tools.... In my experience, not at Amazon, long tenure employees get used to the quirky tools but the impact on new employees can be massive. Same with bad code bases, bad documentation and so on.
    • dimmke 4y ago
      I really think this is true. Like compare the CR process to Github's PR process. To me, it's horrendous. But to others who are embedded, maybe they love it.
• dilyevsky 4y ago
  Is this just me or aside from 40% spent on tooling (which I don't know what the deal is in this case but tbh I've seen some folks really spent their time inefficiently trying to make everything perfect) the breakdown looks completely normal for an established project at medium to big corp?
  • TheRealPomax 4y ago
    While it's not just you, and there are lots of folks who think that's just fine: if someone hires a dev and then make them spend only 10% doing the job they applied for, whoever hired them lied about the job, and the product they're hiring for has fundamental problems that you can either commit to solve, or burn bodies over in the hopes that you get transferred out to a different position and it's not your problem anymore. One of those may be good for business, but it's not the one that makes for a good company.
    • dilyevsky 4y ago
      So 20% spent on tests and another 20% on meetings where presumably you decide what to build is not part of the job they applied for? Nonsense! If we’re doing hot takes here there’s no way everyone else on his team is spending 40% of their time on tooling without management doing anything about it
      • TheRealPomax 4y ago
        What a lovely thought, I do hope that's true, but everything I've seen about this story and people replying to it points to "no, actually, everyone else apparently spends 40% on fighting their internal tooling, too, and not a single manager seems to give a fuck". So there's that.
  • leros 4y ago
    It's really difficult getting hundreds or thousands of people working together. Tooling, communication, etc become really critical. Otherwise, it's just chaos. As a result, you spend a lot less time "actually working" and a lot more time "collaborating" and "project managing". Spending less than 50% of your time on "actually working" is very normal in large companies
• Yhippa 4y ago
  To be completely honest, what he describes seems to be what I've run into at nearly every stop I've been at as a consultant. I honestly never thought of it as hell, but I've also never been in a much better, tech tooling-friendly environment like at one of the FAAMNGs. I suspect the people who end up leaving those environments for a more normal F500 company are going to be in for a shock in terms of work ergonomics and pay. This guy was lucky that he was able to squirrel away so much tech-worker money to be able to quit without a backup plan.
• ElijahLynn 4y ago
  The key word here is "savings". And good on you for doing it and great you don't have obligations to provide for others. What a freeing decision you made for your self!
• robofanatic 4y ago
  You should have waited few days for the package.
  • keyle 4y ago
    You can't wait for what you don't know.
• mabbo 4y ago
  > I also know that every team at Amazon is different, so please don’t take my description for what it’s like working there as a whole. No, no, this sounds pretty much like every team I was on for the nearly-decade I was there.
  • nemo44x 4y ago
    Why did you waste 10 years of your life there if it was that bad to you? Legit curious here. I can’t imagine spending more than a couple months in an awful situation.
• newaccount2021 4y ago
• bogomipz 4y ago
  Does anyone know what role or title the the author might have had there? They mentioned DevOps but then also in another paragraph they mentioned front-end development.
  • bolt7469 4y ago
    https://www.linkedin.com/in/danielimmke/ https://www.linkedin.com/in/danielimmke/
  • p0rkbelly 4y ago
    If you write code at Amazon you also are responsible for the availability of your service. So, you rotate in as an on-call. There is no central SRE teams, SysAdmins, etc. There is no throwing it over the fence. There is nobody better to fix a production issue than the person who wrote the code which broke. Plus, you are more motivated to write better code.
    • bogomipz 4y ago
      Yes I am well-aware of how SDEs work at Amazon. However there is a role at AWS called SDE - System Development Engineer which is very much DevOps which is very different than SRE and the reason I was asking.
• atlgator 4y ago
  How does this compare to your experience at CDC?
• olladecarne 4y ago
  My experience has been similar at other large companies (MSFT, Google). My advice is to look for founder-led teams (not companies). If there are plenty of founders still on the team, they will likely care a lot about the product and code and hold everyone to high standards which make the work easier in the long run. On the other hand when I've been on teams where all the original founders left, it's been a constant struggle to make anyone care about anything. At that point there is no ownership and most people are there for a paycheck. Lots of sloppy code gets shipped, everyone approves anything that looks like code, and eventually adding a single line anywhere feels like playing Russian roulette. Extreme short term thinking takes hold and no one cares that in 2-3 years the code will be unmaintainable because they don't plan to be on the team by then.
  • gruffle 4y ago
    Yup, and the other flipside is devs being extremely nitpicky and regurgitating irrelevant Amazon principles BS in code reviews because they're trying to get promoted.
    • neongodzilla 4y ago
      That shit never stopped sounding culty to me. When they broke out "Best employer on Earth", it was a new level of weird.
    • innocentoldguy 4y ago
      Agreed! If I never hear "You should employ a bias towards <some Amazon principle of dipshittery>" again it will be too soon.
      • gruffle 4y ago
        Everytime I bypass some stupid process or break glass, I now just put 'bias for action' as the reason. Ahahaha I don't give a shit anymore.
        • innocentoldguy 4y ago
• muh_gradle 4y ago
  I completely understand where the author is coming from. I did literally the same thing. I am fortunate to be in a similar circumstance where I have the savings to weather the storm. The only difference is I was working for a highly dysfunctional manager and team. Towards the last days before I resigned, I was nearly on the brink of panic attacks from every meeting I had with my manager. I could see that my skills as a developer in doing actual, meaningful work had severely atrophied in a short period of time. To anyone that had to make difficult decisions like this, do what is best for you and your loved ones because a job shouldn't define you.
• iLoveOncall 4y ago
  If you're thinking of joining big tech, try to join a team that develops an internal tool. You have as much impact and are exposed to as much scale (maybe not in terms of TPS but in terms of data, etc.), but you usually have much lighter development processes and a much more frequent release cycle. Your ops will also be a lot lighter.
  • craftyguy98 4y ago
    > as much scale I'm an early career software engineer and am curious - why do people say working "at scale" like it's a good thing? Why is that desirable? What are you doing differently from someone who doesn't work "at scale"?
    • fragmede 4y ago
      it just means they deal with some obvious and non-obvious problems having to do with being A Big Deal. (there's only a little bragging involved.) the most usual example is scaling up. say you have a website which you run on a server, and it's humming along serving users. everything's great. but then you get popular, and your one lil server can no longer keep up. So you swap that in for a bigger server and everything's great again. but then you get even more popular. and suddenly there don't exist bigger computers. so then you split up the programs on that one computer so that you can use two big ones and serve more people than you could before with one. and then three servers. and so on. Someone who's dealt with websites at scale before is going to be able to skip the first 20 of that process, so if you're a startup dreaming to get big, you find someone that's worked "at scale" eg Amazon or Google to design and build your systems and avoid some pain points like the website going down because the system can't keep up.
    • int_19h 4y ago
      If you look at tools in general as something that's intended to help other people solve some problem - and derive some enjoyment from that aspect of your job as a tool writer - tooling that operates "at scale" usually also helps more people.
    • rswail 4y ago
• techsupporter 4y ago
  > This is not counting the weeks where I was “on call” and forced to drop all of this to work on a backlog of DevOps related issues. I want to say sorry at the beginning if I misread what the author (you?) intended when you wrote this. And I'm coming at this from someone who is not a developer and who, compared to the lofty Software Engineer salaries in this industry, feels underpaid and underappreciated for doing the operations/administration work. With that said: I truly wish those who are leaving their roles where this kind of requirement exists--feels "forced"--would speak more loudly about the need to have actual people doing system administration work. This is especially true, I feel, in large companies who operate "the Cloud" and who ought to know better; someone needs to be doing that work and it can't always, or usually, be the people who are writing the features and implementing the updates to the core product you are selling. But what seems like has happened is everyone in the tech industry has forgotten it, or named it "Site Reliability Engineer" with a job of 55% failure analysis, 35% coding, and 10% "are the infrastructure and products actually online and functional." And then this role gets looked down on and paid less because the people in the role--people like me--are not seen as "delivering" "value". Which culminates in the proper software developers seeing it as a thing they are forced to do, resulting in disdain, and furthering the cycle.
  • titanomachy 4y ago
    The SRE role comes from Google, where they are neither looked down on nor underpaid. Perhaps other companies have corrupted that title to mean something else.
  • deathanatos 4y ago
    How, pray tell, is someone utterly unfamiliar with a code base supposed to be able to deal with unforeseen issues in production for that service? (Worse, the incentives for improvements to production quality, and thus on-call quality, are utterly mis-aligned.) Ironically, I say this as someone presently on-call for a bunch of stuff for which I am utterly unfamiliar with the code base of, and have no time to become familiar with. It's going predictably badly.
    • greedo 4y ago
• perpil 4y ago
  You must have not tried god mode, the internal tool that flips the table on all of that SDE-0 bullshit. If you work at AMZN, search broadcast for "god mode hackathon" and start using it. If you are external, watch the hype videos here: https://photos.app.goo.gl/8enVGZLYFvpcgj2V9 https://photos.app.goo.gl/8enVGZLYFvpcgj2V9
• latchkey 4y ago
  Writing tests is dysfunctional? Funny, I thought that was part of the job.
  • Smaug123 4y ago
    You appear to have forgotten, for example, the "40% of my time trying to tame the bad internal tooling I was forced to use…".
    • latchkey 4y ago
      Did you read any of his other posts on his blog? He's a digital nomad who took off to Mexico (and Belize) and seems a bit lost in where he is going (he says it himself). I've been there myself (except I moved to Vietnam), so I understand it. This is a lot deeper than just writing tests or 40% internal tooling issues. I also suspect that being remote, he feels like his hands are tied in a company that isn't used to working remotely. Or at least, if I was struggling with tooling, I'd work to find a way to make it better. I also never complain about writing tests. I can't tell you how many times tests have saved my bacon or resulted in writing cleaner code.
  • ender341341 4y ago
    it is when the focus is on having 100% coverage and not on test quality. When 100% becomes the metric it tends to get gamed pretty heavily with tests that have such a huge amount of mocks as to make the tests useless, or doing something to make it hit a branch, but ignoring actually verifying it cause that branch is just a log statement.
    • latchkey 4y ago
      I seriously doubt it is actually 100%... let's verify that number.
      • ender341341 4y ago
        on our team it was more "Don't lower it, get accolades for increasing it", and someone gamed it up to 100% using the methods I described.
• outside1234 4y ago
  Why does anyone work there?
  • mkl95 4y ago
    Resume would be my first guess. Even if the actual job sucks it's a solid career investment.
  • danrocks 4y ago
    Former Amazonian here. The learning is unmatched. I've been in different FAANGs but the way that Amazon pushes one to learn and challenge the status quo is unthinkable. I never saw "no, you can't do that, that system/business/area is sacred". Everything is up for grabs all the time. While there are pathological side effects to this (AWS promotion-per-launch, constantly battle for scope), the amount of knowledge one can gain in different areas from world-class people is pretty incredible. One of my systems at Amazon had more traffic in one day than my system at another FAANG had in 5 years. Working on a system that needs to support 5, 6, 7 MILLION TPs to perform complex operations was a great lesson for me.
    • stuff4ben 4y ago
      Yet the internal tooling has been cited by several here as being one of the major time wasters and reasons they don't want to work there. It seems sacred to me...
      • danrocks 4y ago
        I would say that 1) the experience varies by team 2) there's a lot of truth to that. But it's not a matter of being sacred, but the (sad) fact that Amazon being a ruthlessly data-driven business, investment on the tools and on reducing technical debt (real or perceived) is always prioritized down in OP1s.
      • NathanKP 4y ago
        I've been at AWS for almost 6 years now. It depends on what you mean by "internal tooling". But considering all the tooling I've used over the years at least 75% of the internal tooling that was in use when I joined is no longer in use as it has been replaced by newer, better versions of what came before. Things definitely move at a slower pace than at new startups, but systems within AWS still change at a pretty fast pace for such a large organization. A fair number of employees don't stay past two years because of how compensation works, so they don't stay long enough to observe tooling changes, and are left with the assumption that tools are sacred. The reality is that it takes about two years between major rewritten versions of most tools. Then there is a leap forward, stasis for a while while feedback is gathered and the limits of the existing tooling are found, then in roughly two years there is another leap forward, etc. This churn requires work to keep up with, so some teams also fall behind if their product team prioritizes features over keeping up, so they may also be stuck on old versions of tooling for even longer. So in summary I'd say yes in the short term some of the tools appear sacred. In the long term pretty much every internal tool or framework is discarded and replaced by newer versions on a fairly regular cadence.
• lightbendover 4y ago
  I hated the internal tooling so much that I switched to a manager role almost immediately and only went back to IC when I was sufficiently situated to not have to code anything that wasn't a flashy proof-of-concept.
• scubbo 4y ago
  I left Amazon in June of this year, in a similar-ish position to the author. I didn't have anywhere near that level of problems with the internal tooling[0], my frustrations were more related to seeing poor product prioritization decisions, neglect of tech debt, and repeated inefficiencies and mistakes in org-wide projects. Nonetheless, the feeling choosing to prioritize one's own mental health and recovery really resonates. Good for you, author! Take a break, get back in touch with yourself and what you care about, prioritize your joy, and I wish you all the best of luck in your next project! [0] which I've found to be pretty-to-extremely-good compared with what I've discovered since leaving, and would actively welcome new perspectives on what flaws I'm missing.
  • sakopov 4y ago
    Curious, did you go somewhere that at least matched your salary? I know Amazon compensates very well, so much so that it's probably difficult to find something else unless it's another FAANG company. I have a friend working there who basically deals with all the not-so-good things mostly because the compensation is so good.
    • scubbo 4y ago
      Haven't moved on yet (intentionally took some time off to relax and refresh), but I expect I'll have to take a bit of a pay cut. > I know Amazon compensates very well That's not been my understanding - I thought it was noticeably below other FAANG companies (at least until the pay hike earlier this year)? I have only just started reaching out to possible employers so I don't really have any points of comparison.
      • innocentoldguy 4y ago
        Amazon's compensation wasn't anything special. The first two years were awesome because I got a $100K+ sign-on bonus prorated and tacked onto my paycheck every month. Once my first two years ended, my regular salary was pretty lackluster and I was easily able to find a job making a lot more elsewhere.
• matai_kolila 4y ago
  Sigh... another loss at the hands of, "Code is Art!" It's not art. Art requires no function, it exists as a representation. (Professional) code, and software engineering generally, must produce valuable things, and sometimes the process of making those valuable things isn't "fun", but that's never been part of the equation. Aligning "fun" with "valuable" is nearly impossible, and when you try you're dooming yourself to almost certain failure. This is really burnout, and the mental health effects that cause burnout. I'm sorry the author has gotten to this place, but wrapping it up in "passion" is a disguise.
  • prepend 4y ago
    Nothing requires a function, you can have code without a function. I think good art (to me) requires some function. And code is art. Like architecture , or furniture design, or gardening, or dance, or many art where the design and precision and purpose is all tied into what the artist is doing. I don’t think coding is pointless and think that coding for the purpose of being beautiful can be dangerous and painful to support or refactor. But I’ve read a lot of code and appreciated it on some aesthetic level and remember thinking, “that’s neat.”
  • altruios 4y ago
    besides your point - I tangentially would like to mention: art HAS a function. It sometimes appears invisible because it is we who are affected by it. Art is anything that functions to manipulate a {conscious being}'s internal state (traditionally emotional state). That said - I agree mostly with your point besides that one pedantic complaint :)
  • at_compile_time 4y ago
    Code may not be art, but there is beauty in design.
    • matai_kolila 4y ago
      Beauty doesn't put food on the table, is my point, and I'm beyond sick of working with people who put "code beauty" before "code usefulness". Ugly code that works pays bills. Beautiful code that doesn't work is, in a very literal sense, worthless. I'm not against writing clean code (there's utility in that), but I feel like some folks lose the plot and quit their jobs when they can't make "beautiful" things anymore, and I think that's more often a sign of burnout than it is a desire to "return" to creating art.
• thrway2312312 4y ago
  I only apply to roles that permit Rust, Crystal, Go, Erlang/Elixir, Clojure, or Racket. I have zero tolerance for anything else at this point.
• ChrisMarshallNY 4y ago
  > but also a job that made me feel more unhappy than any other job I’ve ever had. I can relate. I worked for many years for one "those" companies that is a household name. The pay wasn't outstanding, and the job was not one that gave me much fulfillment, but it was at a marquee brand. My business card opened a lot of doors. When that job finally ended (after almost 27 years), I started looking at working for other companies. I would have been a particularly good fit for startups, as I was financially secure (thus, didn't need to be paid too much), and have a huge arsenal of skills and experience, that can be quite useful to any small outfit that has "many hats" employees. What I found, is that us older folks are quite unpopular, and got tired of slamming doors and passive-aggressive insults. So I just decided that I was in early retirement. Best decision I ever made. I would have liked having the extra money, but the soul-sucking aspect would have probably killed me, by now. I am in a good position. I have enough set aside to be OK, but not at an exorbitant level, and I get to write awesome code every single day. It's basically, my "dream job." I feel terrible for over-leveraged young folks, having to deal with this kind of thing, early on. I feel even worse, for the ones that have "crossed over" into "old," and are still over-leveraged. They won't feel old, but they will be treated as such. It's really humbling and infuriating.
  • verelo 4y ago
    I really want to know what you do and how i can do it with you.
    • ChrisMarshallNY 4y ago
      Feel free to reach out (my handle has links). I generally prefer keeping it out of the limelight. May or may not be able to work with me, but always happy to expand my network.
  • throwawaylinux 4y ago
    > What I found, is that us older folks are quite unpopular, and got tired of slamming doors and passive-aggressive insults. Some of the older engineers I've worked with are among the most brilliant and productive people in the groups. In those cases I have seen bar none, they have been revered and respected among the other members. Age discrimination is definitely a thing, but not everywhere. Maybe that rotted mindset of prejudice has taken hold in the "I'm saving the world broken by boomers by working at a corporation that sells advertisements on the internet" crowd.
• conqrr 4y ago
  Been there done that too.The thing that hit me really hard was Software Engineering vs Writing code/Computer Science are two very different things. Yes they overlap in theory and behavior of systems.There's no deny Amazon chooses to amplify the worst parts of Software Engineering today (poor tooling etc), but most places aren't much better and even if it is, its not forever.
• deleted 4y ago
  [deleted]
• einpoklum 4y ago
  IMHO you shouldn't have worked for Amazon in the first place. Most of what they do is more negative than positive for society overall. Good that you're out of there - please consider doing something more redeeming with your skills.
  • rswail 4y ago
    I wouldn't say that AWS is negative. That is seperate to the business of Amazon, which exploits warehouse and delivery employees, but is not unusual for the US.
• delecti 4y ago
  Has the tooling gotten dramatically worse in the past 5 years? I willingly left almost exactly 5 years ago because my team's responsibilities had shifted too much, but I remember the tooling being pretty great. Also, everyone being more or less on the same stack also meant that there was a ton of resources to solve problems, just search the wiki for your error message. I still miss how well Odin worked for service credential management, nothing I've used since comes remotely close to how seamless it worked. Also, I found the PIP process to be overly dramatized relative to my experience. I had a PIP imposed on me a couple years before I willingly left, and I "graduated" out of it with support and cooperation of my manager. Everyone involved seemed to want me to succeed. Also, when I left, I opted into a second PIP, because HR and my manager wanted me to leave on good terms. Doing so let me get severance while quitting (it was ~3 months worth based on my 5+ year tenure). I'm not sure if this option still works though.
  • jerglingu 4y ago
    I left about a year ago, but the universality of tools and stack was not a thing in the ~3 years I was there. Retail and AWS are split and use different stacks. Additionally, the company was in a drawn out multi-year process of deprecating old tooling (including Odin), so you had teams or orgs that were in-between these old internal services and new internal services (or off internal services altogether and on "native AWS"). If cross-team interoperability demanded that certain tooling was to be used, many times you could inadvertently hold back or screw over another team by deciding to stay or move to the new tooling. Not to say it was all bad - some of it was certainly necessary. Especially for security - IMO the systems of Amazon are great with security, but not necessarily the spirit of the company. Without all that bureaucratic annoyance, there 100% would be existential leaks or breaches coming out of Amazon with how fast and loose everybody treated security in the name of "Deliver Results" .
• epolanski 4y ago
  You could replace Amazon with any other company and it would make no difference (bar on calls).
• anyfoo 4y ago
  I did not feel super sympathetic when reading the breakdown of percentages (with only 10% "actually writing code"). Yeah, those specific percentages were a bit egregious, but working on a big project it's not uncommon at all to spend more time thinking and talking things through than to actually write code. There's a lot of complexity to be tamed, and in my experience that can be a very enjoyable part of actually engineering. This however: > And the worst part is, I feel like the product we were shipping was really bad. It was slow and kludgy. It was the opposite of the qualities I value in software products. I didn’t feel like I was adding any value, and I was miserable. Yeah, there is absolutely no excuse for that. And if that is the case about the product you're working on, what I said just before about "enjoyable engineering" will never apply. Good for getting out of there.
  • Zvez 4y ago
    >There's a lot of complexity to be tamed, and in my experience that can be a very enjoyable part of actually engineering In my experience there is almost no correlation between complexity of the project and time needed to do 'non-technical' work. Quite contrary, the most complex things I did as a dev required me to do a lot of coding - to prototype, to experiment, to load testing and so on. And at the same time the easiest project in the larger corporations required me to write down design docs for trivial stuff and spent months in meetings to 'align' with people, who don't have time to even read your design doc, but want you to have bi-weekly meeting about it. So no, I don't buy this excuse anymore. Unless you are building space ship or do very unique work, there shouldn't be any need to spend 80% of your time in non-strictly-technical work.
    • anyfoo 4y ago
      I fundamentally disagree with the notion that design and discussion, of often very detailed technical aspects, is "non-technical work" of any kind, or that sitting down and writing code is the only "technical work".
      • hinkley 4y ago
        I’m not sure who to reply to in this thread about whether they mean to imply that debugging and simplifying aren’t technical work too. Finding the right spot to make a 2 line change instead of the wrong spot to make 8 five line changes is seriously technical work
• innocentoldguy 4y ago
  I just laid myself off from Amazon too after 2+ years. I just couldn't take the janky-as-shit internal tooling, the myriad pointless/redundant meetings, and pointlessly stressful and toxic work environment anymore. Also, I enjoyed my first two managers at Amazon, but my third manager was a tool, which is a systemic problem at Amazon. I'm glad to have Amazon in the rearview mirror of my career.
  • yathaid 4y ago
    I am seeing a lot of hate on the internal tooling. Can you elaborate on that a bit more?
• sys_64738 4y ago
  20% meetings is two story points per 2 week sprint. I hope you have a recurring task for this. I do this and does it go down like a lead balloon with project managers.
  • ramphastidae 4y ago
    Why not skip the recurring ticket and reduce your sprint capacity by 2?
    • sys_64738 4y ago
      It's all about perception. If I had 2 less story points per sprint then I would get more tasks to fill up those 2 story points. PMs don't like it when the whole day isn't accounted for.
• rgmerk 4y ago
  One key lesson from these kinds of stories - money gives you options. Pile that money up high (in something like index funds) while you're earning the big bucks, and there comes a time where you can choose whether you want to keep doing it, or move to the beach and surf and still be young enough to enjoy it.
• wmichelin 4y ago
  This author sounds insufferable. Tooling sucks? Fix it. It's important to lead with a culture of quality and impact. Wasting 40 hours a month on something broken? It'd literally be faster to stop your feature work, fix that thing, and then go back to your feature work. Be the change you want to see.
  • nicksf 4y ago
    Easier said than done. Amazon is a top-down org, where each pod's (aka pizza team) work is planned 12-18 months in advance through their OLR process.
    • p0rkbelly 4y ago
      OLR is employee performance reviews.
    • mguterl 4y ago
      OP1
  • forgot_old_user 4y ago
    But the problem is, fixing existing broken stuff might not be rewarded as much as working on the "new feature". This means that if I focus on just fixing the broken stuff, I'd be managed out. Even say I put in the extra effort to fix the broken stuff, and deliver my new features, other thousands of engineers won't do that same- meaning I can't make a meaningful dent. (i dont work for amazon, but I can imagine it's similar everywhere)
  • iLoveOncall 4y ago
    I'm sorry but you honestly have no idea of what you are talking about. The tooling in Amazon is developed by hundreds of engineers and it doesn't suck because of small bugs here and there but rather because of systemic issues related to how it's been architected. It's not something that is fixable at all, let alone by a lone engineer. That said, I also want to be more tempered than the author, I don't find the tooling to be that bad. Most teams are on full Native AWS now and the few non-native tools actually do make your life easier because of how well they're integrated in the whole Amazon ecosystem. Some tooling does suck, but it's more like 2% of my time spent fighting bad tools, not 40%.
• icedchai 4y ago
  This reminds me of a place I used to work. I used to get my real work done between useless meetings. A couple months after I left, they laid off like 20%.
• chromejs10 4y ago
  Only 20% time in meetings? That's nothing if he worked at Amazon unless he was a junior.
  • I_AM_A_SMURF 4y ago
    The breakdown definitely reads SDE1-2
• Waterluvian 4y ago
  Where did this image of “prestige” come from if everyone has the same stories: the place was dysfunctional and the tooling and processes were mediocre ?
• cat_plus_plus 4y ago
  If it works for you fine, but most of us just find another job and then resign. Few situations are intolerable for a few more months, although Twitter seems to qualify right now. As you said, you have a simpler financial situation that enables more flexibility, good for you!
• ksubedi 4y ago
  I quit Amazon a couple weeks ago to start a startup, and I felt like I was reading my own story at some points. The tooling was the biggest thing dragging me back. It's hard to get excited about what you are building when you have to wait 2-4 minutes to preview your changes when most industry standard tools / stacks can do it instantly. I left this feedback a bunch of times with different people in the org and I really hope they scrap their janky "frameworks" and dev tooling. They should just move to more standard open source tools options that evolve and get better quicker than barely maintained internal tooling. Open Source tools also have bigger communities and resources online to debug and solve issues, compared to mediocre documentation from internal wiki. If absolutely needed, make thin wrappers above open source tools. Some other teams within Amazon had the luxury of using better tools, but I bet a lot of people are in the shoes I was in.
  • shepherdjerred 4y ago
    2-4 minutes???? Minimum 20 minutes to build my team's packages. Integration tests didn't run locally and you'd have to push. Same thing for LPT/CFN changes. Tooling at Amazon is _incredibly advanced_, but no teams know how to use it properly.
• fortylove 4y ago
  This mirrored my (much longer) experience at Amazon a bit. The internal tooling is just absolutely awful, _especially_ the newer stuff NAWS stuff. The log4j incident was an absolute nightmare for many teams in Amazon, and the sad thing is that I don't think any lessons were learned by tech leadership.
• wojciii 4y ago
  "40% of my time trying to tame the bad internal tooling I was forced to use to submit my code" .. I experienced something similar while working for a big Korean multinational. This combined with development practices from the 80ies was (when looking back) the cause of severe physiological issues that took months resolve.
• faangiq 4y ago
  >20% of my time writing unit tests to hit the 100% coverage requirement of the codebase I worked on. heh. try one of the teams with no coverage reqs…
• siliconc0w 4y ago
  People focus on coverage but I think that has diminishing returns. There is a lot of 'art' to building clean reusable code which you can't easily standardize but what you can do is pay attention to things like: * Build times * Testing times * Test Flakiness * High quality fast unit and integration tests * High quality fakes * Fast, high fidelity non-prod environments * Time from commit to production, number of cherry picks/rollbacks * Good Coverage, but focus where it matters. If any of these go out of whack it needs to be a P0-P1 to fix because it really is debt that grows to become unmanageable (not to mention repelling to the best people). In most companies, even big tech, this is difficult because non-coders set priorities and allocate the resources. Even once-upon-a-time engineers who make it up the org chart find themselves preoccupied with other things and invariably sacrifice these things for features, compliance, conformity, toolchain complexity/sophistication, security, outage/incident scar tissue, etc. (And maybe rightly so as there are few successful businesses that manage this consistently)
• dbg31415 4y ago
  A few things here... 1) Work is work. The percentage breakdowns here don't seem too far off from any other company. It would be interesting to see what the author feels the percentages should be. My gut feeling for anyone willing to walk away from a job is that they probably do need a break for a few months. 2) If you're looking for reasons to be unhappy, or trying to justify unhappiness... you may be depressed. Switching jobs may help, but if you walk around smelling shit all day remember to check your own shoes. Everything is going to smell like shit until you get the help you need to get back to being healthy. Getting help is hard, but it really makes life (and work) a lot easier. 3) Amazon... I've written about this in the past, but I really have a love / hate around my time there. Looking back, lots of talent, lots of cool projects. But I just felt like Amazon was perhaps the worst place I ever worked for people giving praise or sharing appreciation. Amazon... I can't put my finger on it exactly, but it did often feel like their mission was to suck all the fun out of the room.
• th3h4mm3r 4y ago
  Same situation here, and I don't work for a big tech but for a little company. I'm unable to realize how things could change in a similar environment: the sensation is that there's a lot of time wasted in calls and similar activities. And also into the process of "how to do things". This is very stressful, in particular when what there's to do is clear and glaring.
• retrocryptid 4y ago
  I spent just over a year up here at the Seattle mothership back in the mid-20-teens. I met a few really smart / kind people that I'm still friends with (I even liked my boss, who I'm going to have coffee with after the holiday craziness.) I last a year. Most everyone I know there I know left. My experience was similar. On the first day of orientation we were trying to build an environment to run some tests in. But some aspect of it was borked. I got to file a Sev 2 ticket my first day of work. The component that completed the dependency closure was borked (what was that, brazil? can't remember the name.) But the whole company spent two hours not building software. I love that all teams advertise their capabilities through online services, but man.. it would have been cool if maybe they tested that update to the build system since we were all forced to use it. The other big hilarious problem was when we ran out of IP addresses for new sites. Long story short someone made a mistake three or four teams up the chain and it only became obvious when we tried to allocate a new routable (public) IP address. The problem was in the DNS team, but we didn't use their service interface directly, so we had to wait while the team upstream of us responded. And then they had to file a ticket with the team upstream of them. And so on. It always seemed to me that while they were trying to prevent their interfaces from being brittle, they missed out on some opportunities to understand what was going on by looking at data flowing from one team to another. I hope there's SOMEONE in the company doing that, but we never met them. Not all my work relationships were toxic, but I saw quite a bit of toxicity while there. I quit the day after my then new boss wore a MAGA hat to work and made comments about killing trans-folk (I'm f2m.) Our HR rep was out on vacation and the main office indicated they weren't interested in doing anything about it. I probably should have sued, but it wasn't clear I had a strong case (he said/xe said). At the very least I could have forced everyone to take another sensitivity training class. Which is to say... There are some amazing people there. Also some complete shits. If you have a high tolerance for death threats, you'll probably do fine there. Some of the technology is amazing. Some of it is completely bonkers. When things are good, they're amazing and very good. When they're bad, they're hellish.
• dxuh 4y ago
  Sadly where I work I split my time very similarly and I thought it was a sign of a dysfunctional organization and a sign that I should leave. It's been my only professional software job, so I thought it's likely going to be better somewhere else, but now I am concerned this might be more normal than I thought. Is that so?
• tibanne 4y ago
• ranting-moth 4y ago
  > ... 100% coverage requirement of the codebase ... Is 100% really practical? Probably in life or finance critical code. What about those "should never happen" cases? Incredibly hard to trigger because you only know from gut feeling it can happen in some really weird way. I know it would feel very nice to have 100% coverage, but my gut feeling is that it will also affect the code base in a negative way. Developers might skip catching errors that would be very hard to reach.
• simonebrunozzi 4y ago
  I suggest, in this discussion, to not consider a single company a real, monolithic entity, where everything works the same way everywhere on the planet. I was at AWS 2008-2014, two years each in EMEA, APAC, Americas. I didn't contribute to code directly, but I built my own software demos (I was the tech evangelist for these regions). My experience has been vastly different between these 3 "regions", and also when interacting with HQ in Seattle (which I visited very frequently). There are certain things that unfortunately heavily affect the company - the internal tools were already considered quite bad back then, and I'm not surprised it hasn't changed much. There are others, though, that in my view are even more important. I had a total of 4 managers, and three of them were amazing, average, and quite bad. But in all honesty, I was probably not perfect all the time (real modesty, uh? :D), and it might be that the "bad" manager and me were just a poor match, or that I failed to understand things that made me perceive him as bad. I will never work for someone else ever again in my life (I can afford to, both economically and career-wise; in fact, I just launched a VC fund in Europe), and I love that. But if I had to go back to a corporate job, my single best piece of advice would be: pick your manager wisely, especially if you're early in your career. Everything else is important, yes, but your manager is going to have a huge influence on how you develop, how happy you are, etc. If you're curious, this is how I got hired by Amazon back in 2008 [0]. I find that post a bit naive in retrospect, but I also wrote it ~15 years ago. [0]: https://simon.medium.com/2008-how-i-got-hired-by-amazon-com-314ee8634da9 https://simon.medium.com/2008-how-i-got-hired-by-amazon-com-...
• _jppe 4y ago
  I quit Amazon over a year ago with my managers encouraging me to stay. I was on a cross road of decisions between being stuck in an engineering work that intensify my depression vs taking the high road and hope for the best in uncharted startup territory. Engineering at Amazon is not for a fainted heart, and internal tooling is definitely the source of most people's miseries. On multiple occasions, I had to fix external tools at the expense of making progress on my own work, in the name of exhibiting a "deep dive" ethos. On another hand, however, I think Amazon is a good place to work for people who will eventually get into startups.