7 ms·
Working at Amazon
- top256 9y agoit's interesting how Amazon is now positioning themselves as a tier 1 employer. They used to be a tier2. AFAIK they also have a brutal pager culture where you're awoken in the middle of the night.
- babaganoosh89 9y agoIsn't that the point of a pager system?
- jorblumesea 9y agoMany companies have overseas teams that take the night shift, eg: every on call dev has 12 on during the day, 12 off at night. Amazon does not based on cultural reasons (allegedly) because it promotes ownership and accountability for those teams. Idea being if you're woken up at 4am because of a flaky test or some issue you'll be more inclined to fix it. And if it's a recurring issue, you'll really want to get it sorted or else your life will be hell. It also pushes emphasis on serious testing methodology and the idea that Amazon is a 24/7 business. It actually makes a fair amount of sense, but it's also brutal and contributes to burnout and attrition. I also think this is starting to change for some teams.
- openasocket 9y agoAmazon encourages developers to be on call, but there are also rolling on-call systems so no one gets paged in the middle of the night. A team I work closely with does this. It isn't very stressful for me, I'm just on call for small, well-defined intervals, something like 1 week out of every 6, and only high severity issues will cause a page. In the last year or so I've been on call I was only paged after normal business hours once. Of course, Amazon is an absolutely massive company, so different teams may have wildly different experiences.
- jorblumesea 9y agoI think it's different for each team, especially the AWS groups. I know some support engineers that do 12on/12off rolling calls like you said, but others where it's one person on call, 24/7 for a week and they catch everything, not just SEV 1. It really just depends on your org/boss but the AWS org is known to be particularly pager-slaved.
- wink 9y agoMeh, I really don't buy that reasoning - but maybe it's my current job where 9/10 "got woken up at night" is stuff that's not really our fault (e.g. something in the datacenter is broken, or missing network connectivity, etc.pp. - and yes, I've heard of multihoming, but it's not the team's budget and decision to not keep stuff redundant...)
- twayamznacct 9y agoDisclaimer: Ridiculous pager duty was one of the reasons I quit Amazon (combined with massive failures in management, which I'll describe shortly). The team I was on (retail-related, not AWS), shifted away from the rolling schedule with your counterparts in India taking over for the other 12 hours in order to push the "promote ownership" BS. The only problem? Management was constantly pushing for new things to get done on extremely tight deadlines (including "emergency features" that needed to be done and in prod in days when they likely required a week of design efforts to get right, never mind actual dev time) so you have two options: (1) Develop something stable, with good test coverage and the like, and work to fix it if it breaks... and work 20 hours to get it done. (2) Shit something out as fast as you can and hope it breaks when someone else is on call (who likely will be too busy triaging SEV3-5's during business hours to even think about spending time fixing the root cause of the SEV1/2, rather than mitigating it and moving on...or, even worse (!) (/s) try to get the actual feature owners to fix it) in order to maintain some semblance of work-life balance. I'll leave it as an exercise to the reader to guess at which route was generally taken.
- snorkel 9y agoI've experienced that when the coders are getting paged they'll focus on quality and writing better proactive tests. But when you throw the ops responsilities over to another team then that other team gets to decide how and when new code gets shipped which slows down the whole operation. I've experienced that Dev vs. Ops arrangement too where the ops team was incentivized to maintain stability over allowing new code to be shipped which resulted in no code being shipped! When devs own the ops responsibility too then things move faster.
- noahl 9y agoDo you think the tier 1 thing started after the NYTimes article on them, or before?
- top256 9y agoI think it started with AWS success
- falcolas 9y ago> pager culture where you're awoken in the middle of the night For multinational corporations, this has never made sense to me. Why do we insist on relying groggy, "at the bottom of their performance curve" engineers to keep your system up, instead of someone in the middle of their work day? IMO, if you're waking someone up in the middle of the night, something is horribly wrong, and it's not the issue you woke that person up for.
- dsfyu404ed 9y agoMost don't page people at 1am for the initial page. People only get paged at 1am when the problem has been narrowed down to some specific software and all the people with expertise in that thing are in the same time zone.
- mabbo 9y agoHaving done Amazon on-call for five years, I can give some background on that. First, the front-line pager duty is usually hit by an ops eng team first. Each of those teams has an India group and an North America group. If they can't resolve the issue, they page in the developers. Having good Ops Eng guys supporting you is a blessing. There's also a very strong incentive for teams to write good software and test before deploying when you know damn well that if you're deploying something that doesn't work, you're the guy who will have to deal with it at 3am. Having strong integration and system tests suddenly becomes incredibly important. Edit: and as dsfyu404ed said, the Ops Eng guys will be smart enough to narrow down where the problem is usually. If you're being paged, it's usually your team's fault.
- colmmacc 9y agoAmazon on-call engineer here! I'm on-call for three rotations. Two are "Call leader" rotations where a call leader like me is engaged if there's any kind of large event (e.g. Amazon.com orders aren't working, or an AWS service is having issues). I volunteer for those because it's super rewarding to be able to help materially make a tangible difference now and then, wouldn't trade them away for anything. For those rotations, I am paged in the middle of the night, and there's nothing I can really do about that. The other rotation is for one of my dev-team's software, and I'm hardly ever paged - like once or twice a year max. That's how it's supposed to be: by having skin in the game and being directly on the hook for the reliability of what we produce, I'm sure that we automate to a much greater degree than we might otherwise. We have a culture of strong ownership: it's not uncommon to see senior engineers elect to be engaged on all of their team's live issues, even if they aren't on-call themselves. Some other teams make different trade-offs, and have follow-the-sun rotations or engagements that are more reliant on humans following run-books than automations. There can be a place for that, but it gets old quickly and is not scalable. Good teams will prioritize the work to get out of that above almost anything.
- hsod 9y ago> The other rotation is for one of my dev-team's software, and I'm hardly ever paged - like once or twice a year max. I'm in an on-call rotation (not at AMZ) where I hardly ever get paged, and it's cold comfort. I still have to carry my laptop with me everywhere I go, make sure I have access to the Internet, make sure I don't have one too many at the bar on Saturday night, etc.
- deleted 9y ago[deleted]
- synicalx 9y ago> AFAIK they also have a brutal pager culture where you're awoken in the middle of the night. That's a fairly normal part of Operations work and has been for decades. Most places (I assume Amazon is one of them) will have a pool of lower-tier admins on deck 24/7 who handle and investigate 99% of the issues that crop up. But for that 1% they can't deal with, then they need to be able to reach someone who CAN deal with it. It's much cheaper to occasionally call an SME every once in a while when they're needed, than it is to hire another 3 or 4 of them to cover a 24 hour roster. I've been called out in the middle of the night to fix someone else's mess more times than I can count, so I'm 100% behind Amazon's cultural reasons for sticking to this sort of on-call system as well.
- gtsteve 9y agoThe site is now offline, probably due to load. Google cache link for the lazy: https://webcache.googleusercontent.com/search?q=cache:70VxjsUrScIJ:https://www.tbray.org/ongoing/When/201x/2017/10/26/Working-at-Amazon+&cd=1&hl=en&ct=clnk&gl=uk https://webcache.googleusercontent.com/search?q=cache:70Vxjs...
- asdfasdfasd333 9y agoThanks for writing this, and with all due respect: This exact article could have been copy/pasted from any of the other big-tech companies. What makes Amazon different?
- sumitgt 9y agoThe way Amazon does one-pagers is sort of unique.
- xchaotic 9y agoYou mean on six pages? ;)
- sumitgt 9y agoThe 20 minute quiet reading in the beginning is kinda different. At the other companies I've worked for, you were expected to come to the meeting having read the docs sent over email.
- sushisource 9y agoThe Amazon way appeals to me because in my experience pretty much no one ever does the pre-reading.
- jdbernard 9y agoAnd even if they do chances are they've context switched enough since reading that most of the details are lost.
- RandomOpinion 9y ago> At the other companies I've worked for, you were expected to come to the meeting having read the docs sent over email. It may be expected but large numbers of people don't do it. The 20 min. thing is a bit odd but I can see how it could be a net positive.
- xchaotic 9y agoThe biggest flaw in working for all those mega corps, is that despite their size, they still haven't fully embraced remote work. Yes there are island, but Amazon are still looking for that "2nd city" to locate in. Some people may dig going to an office, I don't. If you're lucky to be early in a given neighborhood, you can probably retire on renting out your house. If you're late, like applying to Amazon now would be, you're just a blue collar worker, with higher turnover - you earn more and spend on rent. Especially in places like Seattle if I was to relocate with the family, I'd be an indentured servant, with obligations to pay high rent/mortgage, I'd probably have nowhere to change jobs to, if Amazon didn't work out. One can dwell on small things like which version control does a company use, but ultimately, they are failing to deliver on the "quality of life" thing.
- sumitgt 9y ago> Especially in places like Seattle if I was to relocate with the family, I'd be an indentured servant, with obligations to pay high rent/mortgage, I'd probably have nowhere to change jobs to, if Amazon didn't work out. That's quite the opposite. Wouldn't living in a big city mean more choices? Seattle for instance has high paying job options from Microsoft, Google, FB if you decide to leave Amazon. However, yes having more acceptance of remote work would be great.
- alistairSH 9y agoWouldn't living in a big city mean more choices? I guess the OP is only willing to work for one of the big-4? Here in DC, between private sector software (where I work), public sector and defense, and consulting/contracting, there are tons of options all across the spectrum of job roles and salaries. And these are located all around the area, not just in a single part of the metro.
- methodover 9y agoAren't the Google and FB offices in Seattle relatively small, satellite offices?
- 9y ago
- amznthrow 9y agoAmazon has one thing going for it that I think is a huge deal for a standard BigCo job: if you don't like your initial team, you can switch ANY TIME. And if you're an engineer, you can probably join any team you want. The reason this is such a big deal is that when applying to any company, you'll get placed on a team and it's really hard to get enough information on the culture, mission, management, etc. Once you're inside? Tons of information, and a lot of coworkers you can ask. This really reduces the downside risk of a new job. I know this because I work at Amazon, and switched teams very early. I'm much happier on my second team, and probably would have quit on the first team.
- Danihan 9y agoWhat was wrong with the first team? How large are teams, generally? Are we talking like, six people, or sixty?
- 52-6F-62 9y agoI'd also like to hear more on this, if possible
- patch_cable 9y agoTeams are generally 4 to 8 people. At Amazon you frequently hear the term "two-pizza team", meaning the team can be fed with two large pizzas. That's the size they usually target.
- amznthrow 9y agoI didn't like the project I was assigned, didn't like the underlying design of the org's charter, didn't like the hours. I also got transferred between managers. I didn't really like it from the director down. It sounded way cooler from the pitch the hiring manager gave me. Team size is not a well-defined concept, TBH -- hierarchies can be structured in very different ways, and a team can be comprised of people with different managers. Also, if you love your director and adjacent teams, it would be unlikely you'd hate your own team. Once I was inside, I found a few teams that sounded cool. At that point, I could easily meet people on those teams, read survey results on the managers, ask around about the new teams, visit the offices at 6-7 PM and see who was still around. This is somewhere between 10-100x the information I could get as an internal employee than as an external applicant. This led me to joining a team where I'm very happy. And if things went really badly -- I could have switched again! (Might look bad if you switch a ton though)
- gamedna 9y agoWhile informative, this post is extremely non-contentious, and i suspect on purpose. Much of what is said here can be said about most companies that are at scale. Probably by design b/c if his employer or manager found out about it, i am sure there would be backlash.
- 0xbear 9y agoI heard they will actually cut your base pay if stock goes up “too much”. Is this true?
- ifhd 9y agoNo, but you won't get a raise. What does happen is this. You get a 2 year bonus to start, about half on your first cheque and the rest spread over each cheque in year 2. This is fine, but in year 3 you see a pretty big drop (you to really get your stock in years 3/4 though). you know that going in, but it still kind of sucks to check your pay stub when you start your third year and it's down over a thousand a month.
- skookum 9y agoThis is not true. However there is a target total compensation range for your [level X review ratings] and if the stock price growth results in your future total compensation projecting above your target compensation range you won't get refresh RSU grants.
- 0xbear 9y agoWhat’s the justification though?
- maxxxxx 9y agoI think AWS is probably the place to be Amazon. The whole idea of the cloud is that everything gets automated so good and clean engineering will be rewarded. I think working on the Amazon e-commerce is much less fun because it's much messier to deal with issues like customer support, taxes and other stuff.
- ifhd 9y ago> kid engineers If he described pretty much any other group with type of label he would be lambasted for it. Just saying. Anyway, I'd be wary of the experience of a well known senior-principal level AWS employee being anything like the day to day for someone coming at SDE-2 with 5-10 years experience. He's well past the point where some shitty low level dev manager can make his life hell.
- news_to_me 9y ago> I could go on about diversity ... and so on, but at the end of the day it’s just another high-tech company, nothing surprising. Dealbreaker. "On balance" in this industry isn't good enough, and if any company can take a leadership role in promoting diversity, it's one of the Big Four.