12 ms·

Amazon built a new unit to fix its engineering culture

112 points by firstSpeaker 4y ago

firstSpeaker 4y ago
https://archive.ph/aRauw https://archive.ph/aRauw
deleted 4y ago
[deleted]
snarkerson 4y ago
Maybe the adversarial evaluation process has a poisonous effect?
- jejeyyy77 4y ago
  They're trying to fix this with dev tooling, which is the least of their problems...
- jsnell 4y ago
  Stripped of fluff, the main point of the article seems to be that their engineers are doing too much manual repetitive work, due to bad tooling and processes. Why would that be caused by their performance review system?
  - anderber 4y ago
    Why spend time building tools that will help others if you're competing against them?
  - mathattack 4y ago
    I’ve found tooling lags behind in places where incentives are too short term, or most of the funding comes from projects (rather than durable product teams). I’m not saying this is Amazon.
  - innagadadavida 4y ago
    > Why would that be caused by their performance review system? May be because their performance review prioritizes employee contributions that impact bottom line immediately instead of building infra that might improve developer productivity?
  - sidlls 4y ago
    Easy: tooling and process are often seen as being of little importance because they’re not easily associated with metrics that are important in toxic review process cultures. Engineers who like working on these things are treated as second class citizens and performance reviews will reflect that. (I don’t know that this is the case with Amazon: I’m just providing an anecdote based on my observations at places I’ve worked.)
    - nerdponx 4y ago
      Not just engineers, but also engineering managers and product managers. Shipping bugfixes and tooling improvements is often harder than shipping new features, and is generally less visible to the people who make hiring/firing/promotion decisions.
  - closeparen 4y ago
    You can't manage what you don't measure. Developer tooling quality impact on revenue is very difficult to measure. Therefore, a rationally managed organization cannot authorize any work on developer tooling.
  - deleted 4y ago
    [deleted]
- choudharism 4y ago
  TBF, that was scrapped in 2017. Source: worked there. There was a clear culture shock for some of the more seasoned Amazonians (and you could see a fall out effect with some "mails to your manager" every now and then if you crossed a Grand Poobah the wrong way), but to all of us "newer" hires, the times of adversarial review were just a story.
  - stevewatson301 4y ago
    I’m not sure how you can say that given news reports like these in 2021, see https://www.theverge.com/2021/7/9/22570579/amazon-performance-focus-plans-hidden-employees https://www.theverge.com/2021/7/9/22570579/amazon-performanc...
    - choudharism 4y ago
      I'm not saying there aren't other issues - there certainly are. What you highlighted is one of them. All I'm saying is that The Grand "Me-vs-You" peer review process is gone. Anecdotally, I can say with confidence that some teams have a lot of autonomy in deciding to follow (or not follow) company-wide practices / guidance like what you've posted (obviously, only a few do it, since non-compliance isn't officially blessed). I was lucky enough to work in a team which was much better than baseline. But I try not to let that colour my memory - I heard and saw plenty of horror stories, and I saw plenty of people (including in my leadership) trying to do the right thing. Like most things, there are shades of grey here.
- paganel 4y ago
  Have never worked at Amazon nor do I know anyone who did, what exactly did that entail? "Snitching" on your co-workers? I'm asking because a quick web search didn't help me too much.
stevievee 4y ago
"...the retail giant has pledged to become the Earth's Best Employer, not just for frontline warehouse workers but for corporate and technical employees, too..."* * As long as it doesn't interfere with the share price and exec bonuses
- vertis 4y ago
  Barely a day goes by without a story about how badly Amazon treats it's non-tech workers. Combine that with the stack ranking and, like you, I'm not convinced. I would argue you can't fix the tech employment without fixing the rest. One bleeds into the other. I would never consider working for a company that can't treat it's more replaceable employees well.
  - gumby 4y ago
    Are they still doing stack ranking and hire-to-fire?
maybedev 4y ago
Brazil should be gone
- nickm12 4y ago
  I was confounded by Brazil for my first 3-4 years of working with it. The lightbulb clicked on just as I was ready to move on and at my next job, I missed Brazil terribly (maybe it was just Stockholm syndrome). I eventually boomeranged back to Amazon and have found the middle ground. When used right, Brazil solves a lot of problems and day-to-day I enjoy working with it much more than not. The trouble is that almost all teams don't know how to use it and tie themselves up in knots of their own making. Third party software is also a problem. I hope Peru comes through, but so far I haven't worked with it, for good or for bad.
- jonhohle 4y ago
  Really? I miss Brazil and version sets regularly and often think about reimplementing the parts I find valuable on top of an existing package system. Having self contained workspaces with all dependencies in any language, fine grained ability to replace any package with a local build, and the simplicity of package config (for most languages) was awesome. Having to use _n_ language dependent build systems without interface pinning or reproducible builds since leaving sucked, imho. Checking out a package and running `bb` to get an artifact was just so satisfying.
  - maybedev 4y ago
    3p packages...Python...
  - vamega 4y ago
    Nix will get you most of this. It's got a learning curve, but I think it pays off and is not bad if you're familiar with the Brazil concepts. Having to set up a million dependencies just to begin to start working on a codebase is a nightmare, and Brazil makes that easy.
    - jonhohle 4y ago
twic 4y ago
Is "software builders" an Amazonism, or did this guy come up with it, and either way, are they aware how weird and insular that makes Amazon sound?
- sokoloff 4y ago
  Jassy has used the "builders" term for years while leading AWS. It was a central theme in many of his keynote speeches. I think it was chosen specifically because it was not an industry-standardized (and watered-down) term. https://aws.amazon.com/builders-library/ https://aws.amazon.com/builders-library/ among many other AWS docs that use the builders language.
  - socialismisok 4y ago
    That language predates Jassy, it was there back around 2012 at least.
    - rawrenstein 4y ago
      Predates Jassy? He has literally worked at Amazon for decades and most of that on Amazon Web Services.
      - socialismisok 4y ago
        Predates Jassy running Amazon.
    - Cipater 4y ago
      Andy Jassy's been at Amazon since 1997 and headed AWS from its inception.
- discodave 4y ago
  Amazon has had an org called 'Builder Tools' or some variation on that for over a decade (maybe much longer?). That team has historically owned the systems for build, deployment, source control etc. AWS also uses the 'builder' terminology a lot in their marketing. E.g. Werner saying 'Now go build!' at the end of his keynotes, and the 'Build On AWS' campaign.
mmmeff 4y ago
I quit a few months back and hope it continues to crumble. You won’t be missed, Amazon. After 5 years and countless long nights, I was thanked with a pay cut and ignored when petitioned, even after rearchitecting the entirety of one of their floundering subsidiaries and leading a team of 10 thru it’s execution. The reason? One of my junior engineers missed a deadline. When petitioned, my cries were ignored and I was recommended to boomerang if I wanted a pay raise. At the same time, a friend was interviewing for the exact same senior engineering role and received an offer. The total compensation he was offered in his first year was nearly double my pay. He didn’t even take it - another company offered him more.
- bogomipz 4y ago
  >"When petitioned, my cries were ignored and I was recommended to boomerang if I wanted a pay raise." Is petitioning a process in AWS? What does boomerang mean here?
  - sokoloff 4y ago
    Boomerang is “leave and come back” (quit, get a job elsewhere, later re-apply).
  - 55555 4y ago
    I don't know but I imagine it means to leave and then come back.
  - deleted 4y ago
    [deleted]
  - jvolkman 4y ago
    Leave and get rehired.
  - ertemplin 4y ago
    (Disclaimer: I left Amazon about 6 months ago) Petitioning isn't a process in AWS, but you can ask your manager for a raise if your projected total compensation isn't to your satisfaction. Boomerang means you leave the company, get another job, and then come back to the company with much higher compensation. It's a way for otherwise happy employees to fight salary compression.
phillipcarter 4y ago
It'd be great to read about all the things this internal DevX org has been able to improve after some time. Internal developer experience is extremely important. One codebase I worked on used to have a 24-hour, nondeterministic CI run. A teammate brought that time down to ~45 minutes (it's a big codebase with a ton of tests) and made it deterministic. Everything got better: * Faster turnaround time for external contributions (it's OSS) * Ability to be relied on to get a critical fix in and tested in under an hour * Much faster local build times * Removed reliance on some old busted internal infrastructure so we could use much better, different infrastructure, and make the build/test runs the same across any OS rather than rely on windows so much * Everyone was happier in general If we could have gotten a full CI run down from 45 minutes to 15 minutes, that'd be great. And there was still quite a lot of technical debt in the tests themselves, since they should be able to run a lot faster than they did. But even with that in mind, the difference was night and day.
- johnbellone 4y ago
  This kind of stuff is right my ally and I'd love to connect with someone at Amazon about how they're tackling these problems. Does anyone know someone that would like to grab coffee to chat about it?
  - gumby 4y ago
    Your note is unlikely to yield a result. It does end with a call to action but doesn’t explain why someone should respond. It’s nice and short but needs one more sentence in the middle. This is up your alley — great! But do you want to compare notes? Do you want to get hired? What’s in it for someone from Amazon to respond (or harder, since you’re asking for an intro: why would someone send this to a friend?) I get a lot of messages like this and usually ignore them because they lack any reason to respond.
- zach_garwood 4y ago
  I would love to find a full-time position of just doing this kind of work. Nearly every place I've worked had some hacked together system that mostly worked, but wasn't enough of a pain for management to dedicate time to. I'd have to steal time away from the feature mill to make some little improvement, and everyone would agree that, yes, it did make things a easier for the devs, but no, we can't just have you out there working on things willy-nilly when there are features to churn out!
_fat_santa 4y ago
This type of bureaucracy is not only harmful to dev culture, it's also outrageously expensive. I've been on an engagement with a large healthcare provider and the dev culture there is very similar to Amazon, loads and loads of procedure and bureaucracy. Now i get a certain level of that at any company, but you often find yourself going: "does it really need to be this hard". Tasks that would take you an hour outside of this companies ecosystem would take you 3-4 inside the ecosystem. It's annoying to devs but should be alarming to management. You're literally spending 2x to 3x the required amount on your labor because of your inefficiencies. Personally I found it easy to convince management of changes like this when you pitch it like: "How would you like to reduce your labor costs by 2x". Now granted many people at these places are salaried which makes this a bit of a moot point but most software shops contract out alot of work, and that's where you're really paying for your inefficiencies.
- foobarian 4y ago
  Heh, watch out for empire-building middle managers. "Hey how would you like to reduce your headcount by 2x" will not go over well.
  - dicriseg 4y ago
    It’s more like, “How would you like to get twice as much done?” I don’t work at Amazon, but I’m sure there’s plenty for folks to do.
  - ryandvm 4y ago
    Yup. This is an especially dangerous suggestion when you work for a government contractor. Since a lot of them have their profit margin capped at a specific percentage, the only way for them to increase total profit is by increasing the size of the project. Suggesting efficiencies that would reduce headcount is a good way to get on executive leadership's threat radar. Man, I do not miss working for a fed contractor.
- bumby 4y ago
  While it's true that arbitrary bureaucratic inefficiencies can get out of control, sometimes "good" bureaucratic requirements can get inflated with the arbitrary ones, simply because people don't understand their reason in the first place. This leads to a Chesteron's fence scenario [1]. Anytime somebody say's something like "does it really need to be this hard" or "all you've got to do..." should make sure they fully understand the rationale behind the requirements. A lot of times, those requirements can be met in a more efficient manner, but sometimes the seemingly "efficient" path will cause more second-order problems down the road because of those blind spots. What you often find in both camps (those bluntly instituting the requirements and those saying they should go away) is a lack of understanding of the entire system dynamics. [1] https://fs.blog/chestertons-fence/ https://fs.blog/chestertons-fence/
jvolkman 4y ago
Sounds like a Day 2 move.
- phillipcarter 4y ago
  The corporate speak can be hard to stomach sometimes :)
ctvo 4y ago
Internal tooling is a mess at most companies I've worked at. At a certain scale it's worth the savings to standardize, improve, and reduce the shared friction costs. Stripe, as it has scaled, has ran into challenges and invests in the same efforts. FB internal tooling was (is?) a mess. It goes on. The surprise here is it's such a late investment from Amazon, but it's worked for them thus far. I'm guessing the returns on +1 engineer has plateaued or declined, leading to this effort.
- mv4 4y ago
  Re: FB - still is.
jollyllama 4y ago
>"Being an engineer at Amazon is more button-clicking than development," the slide said, quoting an unnamed Amazon developer. You guys get to click buttons??
- buscoquadnary 4y ago
  Lucky ducks. Development at my org largely consists of going around to every different possible "partner" to get buy in to have a meeting, to discuss approval of permissions to do something. I think the new metric used to asses an organization is PPS (Powerpoint Slides) to LOC ratio.
champagnepapi 4y ago
What about the hire to fire, and crazy amount of pips they do? How are they going to fix that part of the culture? These two are the biggest reason why I turn down amazon recruiters.
- endemic 4y ago
  Churn and burn, baby!
afroisalreadyin 4y ago
At the moment, the general impression I get from people working at Amazon is "Join for the experience, get it on your CV, move on after a year". This is even across departments and countries. The wife of a very good friend who lives in China is working there since nine months in some kind of operations role, and made up her mind months ago that she wouldn't take it any longer than a year. Here in Berlin, I just recently talked to a dev who left exactly a year after he started. He said it was interesting seeing the whole AWS machinery from the inside, and there was quite a lot to learn in terms of process and management, but shoving around legacy Java was just not worth it.
- breitling 4y ago
  > legacy Java I'm starting to feel very old Is Java considered legacy now?
  - azmarks 4y ago
    Java came out 3 (or is it 4) recessions ago. There are entry level developers who were born after I started using it. It's a legacy technology.
    - 1024core 4y ago
      If you remember Duke, you're old. I still remember when Sun was going around holding Java events, touting the advantages of Java, with Duke shirts, mugs, etc.
    - PathOfEclipse 4y ago
      For a technology to be legacy, I think it needs to be stale and falling out of use for new projects. I don't see that with Java. New releases with new language features are being released at a faster pace than ever, and companies are still building new products on it. Java hits a sweet spot of performance, tooling, and productivity that few other ecosystems can. It's biggest competitors are probably C# and golang, and, for various reasons, it is holding its own just fine against them.
  - 4y ago
bearjaws 4y ago
Amazon was a good gig to join (on the AWS side) 4 years ago, not so much now. Reason being everyone I know who got in around 2018 has options that cost a fraction of the current stock price. The job is still boring/miserable but when you are clearing 600k+ annual income... Now if you join, you can bet its a far worse deal and you will make far less than those who just happened to join earlier, even if they're checked out... There in lies one of their problems, the incentives are for checked out people to stay and 'click buttons' while performing as much CYA as possible.
- ogn3rd 4y ago
  For reference, entering in July of 2016 got you $700 RSU's. Today that's the equivalent of a $35 share.
- retinaros 4y ago
  not sure what you are saying the stock lasts 4 years any new stock you getting in between will be at market price so you not gonna clear 600k after 4years but only base +new stock.. if amazon stock goes up the next 4years newcomer will get 600k too
- Cthulhu_ 4y ago
  I'm hearing tales left and right like that; one part of the tales is that you get stocks or stock options, BUT you have to eat a shit sandwich for years before you can 'cash out'. This is the companies' golden handcuffs; they know the job is shit, but you'll get a big payout if you manage to keep it up for at least X time. The other one is that compensation and competition is sobering up. I have one former colleague who claims that they earn 3x as much working for a US company than they did our previous job, but I don't see much exorbitant wages being thrown around. Second, that company he works for now has set a hiring stop; the explosive growth and demand from US based tech companies is slowing down or coming to a halt.
fjabre 4y ago
AWS console is one of the most poorly developed builder tools Ive ever encountered for a company like Amazon, from a UX perspective at least. I may like some of the services like s3 but will never rely on AWS for anything more that that. Logging into the aws console makes my blood pressure go up.
pyb 4y ago
What do people think of DevX groups in general? I'm not sold, but in my (only n=1) experience, it'd be best to try to address the cultural problems that led to the perceived need for DevX. For instance : if build times are really bad, maybe it's because developers don't feel empowered to fix this as part of their day-to-day work.
- helen___keller 4y ago
  At first I was going to disagree but now I'm rethinking. The need for these DevX groups comes from the fact that at Amazon's scale, there's lots of powerful tools, such as a massive internal build system and such. So, one answer is that you need DevX groups, because it makes no sense to have each of the thousands of developers learn how to tinker/fix the huge custom amazon build system (IE empowering individual devs). A second answer might be that there shouldn't be massive internal systems in the first place. A lot of the brilliance of AWS was reducing the differentiation between internal and external teams as consumers of services[0]. The buildup of massive internal systems probably shows that Amazon hasn't gone far enough, and DevX failures are just a symptom. [0] related https://www.alexanderjarvis.com/steve-yegges-famous-rant-on-google-vs-amazon/ https://www.alexanderjarvis.com/steve-yegges-famous-rant-on-...
  - pyb 4y ago
    I can't speak for AWS ; my experience is at a much smaller company, where the DevX team was mostly busy with enforcing coding standards and other uniform "best practices" upon other teams. OTOH if your build system needs a lot of tinkering/deep knowledge to be used effectively, maybe it needs a redesign. This would be a job for the tools team rather than DevX.
deleted 4y ago
[deleted]
socialismisok 4y ago
Amazon built a business on hiring starry eyed engineers and then burning them to ash. I worked for them for 6 years and holy shit I'd never go back. Their reputation is catching up with them and their lack of stock growth is highlighting their comp deficiencies.
- ogn3rd 4y ago
  Amen. Worked for AWS for 6 years. 6 years I'll never get back, 6 years of abusive practices, corruption, lies, and bullshit. This is the bed they've made, let the sleep in it.
  - firstSpeaker 4y ago
    Is this just AWS that has such a broken culture or it is all other teams? Although AWS is big, but Amazon has other organisations and teams. (https://www.amazon.jobs/en/search-teamcategory https://www.amazon.jobs/en/search-teamcategory).
    - socialismisok 4y ago
      The rest of Amazon is worse.
- voidfunc 4y ago
  Always amused on Blind about people posting leaving Microsoft for Amazon because TC is the only thing folks seem to focus on there. I've known dozens of Amazon engineers and I've never met one that was happy.
  - mynameisash 4y ago
    At the risk of sounding the "+1" bell... I left Amazon due to it being a total shit show (for me and my mental well-being but also for the team I was on). Went to Microsoft and found it to be absolutely wonderful. I was working at 6pm one day and my manager, as he was leaving, chided me and said, "Go home to your family. Work will be here in the morning." I have had friends/colleagues leave to go to Amazon for the TC. One even tried to get me to go over there for a PE role that likely would have meant a huge pay increase. But I value my sanity and have no plans to go back there ever.
helen___keller 4y ago
Current AWS employee of 1 year, opinions are my own I largely agree that for me, working here feels "day 2", which just means it feels the same as working for any other mid to large sized corporation. Lots of meetings, building consensus, tasks that amount to overhead, and adjusting slowly moving pieces to advance whatever the corporate agenda is. Many of these things are directly contradictory to amazon's stated leadership principles (core values) but work happens at the scope of teams not at the scope of leadership principles. Some of it is team-dependent (not unrelated, my team has really good work life balance. I slog through annoying corporate BS 9-5, but don't work much beyond that) and some of it is related to the builder tools as mentioned in the article.
- ghostpepper 4y ago
  Is it just the luck of the draw who ends up on the teams with good managers or can you dig your heels in if you end up on one of the bad teams and just refuse to stay late? Or can you easily switch teams until you find a good one?
  - helen___keller 4y ago
    > can you dig your heels in if you end up on one of the bad teams and just refuse to stay late I mean, literally anybody anywhere can do this. Whether you get fired depends on your performance evaluations. Like any organization, the biggest reasons to stay late is (a) too high expectations were set, (b) failures to re-adjust to new information (e.g. we had to pull you off your project for 2 weeks but still expect you to make your deadline), (c) paging / oncall. At AWS, I'd say (c) is the biggest risk, but (a) and (b) can also be present at any team of course. > Is it just the luck of the draw who ends up on the teams with good managers Yeah, that's my impression. When you're already in Amazon though, you're empowered because you have a much greater ability to gather information about a team before deciding if you want to transfer. For example, message/meet with devs on the team, look at their ticket queue to see how often the oncall gets paged, and so on. > Or can you easily switch teams until you find a good one? I don't know about "until you find a good one", but transferring is pretty normalized. If you've been here a year or two, it's normal to want to transfer, and even before that it's not impossible. You can't transfer if you're on a performance improvement plan, but otherwise it's pretty straightforward
dev_tty01 4y ago
So their solution to management and process problems is to "manage harder." Yeah, this is going to go well.
user3939382 4y ago
Amazon's retail business is driven by the consumption of mass-produced garbage quality products from China that we destroy the environment producing, often abusing factory workers in the process, only to soon thereafter put them in a landfill. They've repeatedly stolen the IP of top selling product companies, then you've got this stuff https://www.forbes.com/sites/davidphelan/2019/04/12/amazon-confirms-staff-listen-to-alexa-conversations-heres-all-you-need-to-know/?sh=364d70fd5d9d https://www.forbes.com/sites/davidphelan/2019/04/12/amazon-c... Not to mention all the heinous shit (unconstitutional wars, dragnet spying) they help our military and intelligence services do https://aws.amazon.com/government-education/defense/ https://aws.amazon.com/government-education/defense/ Their abuse of their workers from warehouse to engineering is just one piece. As I tell their recruiters I'd rather eat a cactus than work for them.
- des429 4y ago
  > As I tell their recruiters I'd rather eat a cactus than work for them. wow, hardcore.
- d-trieu 4y ago
  > As I tell their recruiters I'd rather eat a cactus than work for them. That's not much of a statement https://en.wikipedia.org/wiki/Nopal https://en.wikipedia.org/wiki/Nopal
  - Justsignedup 4y ago
    Cactus tacos are very tasty!
- ugh123 4y ago
  >As I tell their recruiters I'd rather eat a cactus than work for them. lol. whats their response?
- d23 4y ago
  Unless they turn around soon, I don't see how their goose isn't cooked in the long run. The site has become too filled with garbage products, hijacked listings and fake reviews. Once the people who think they've never received a counterfeit realize there's no way they can really know and start to question why the products they get from Amazon break earlier than usual or have questionable packaging, it will be hard to undo the loss of trust.
syntaxing 4y ago
Even their interview is kind of soul crushing. Felt like I was a story teller instead of an engineer trying to get a job. I can only imagine how it is working there.
PaulDavisThe1st 4y ago
From TFA: > In an email to Insider, Amazon's spokesperson confirmed the existence of the team, saying it's part of an effort to maintain its Day 1 culture. This terminology is bullshit. As someone who was actually there at Day 1 of amzn, it's completely clear that the term "Day 1 culture" is truly meaningless. It refers not to a point in time, but to some management-imposed aspiration in which things never actually settle down to a routine, but instead everyone feels "empowered" by running around constantly reinventing everything. Of course, this is sold as "you can still make a big difference here", when in reality, you almost certainly cannot. Amazon's actual Day 1 culture meant bringing up a couple of SPARCstations and getting the T1 line functioning.
steveBK123 4y ago
I know a few senior engineers / manager types who left financial services tech to work at Amazon. Most of them left within two years, and the ones that stayed quickly got out of AWS org because it was too crushing. In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff.
- ogn3rd 4y ago
  "In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff." This isn't always the case. Ask a TAM. With the right blend of customers I've known some to work about 10 hours a week.
  - coredog64 4y ago
    JFC, that explains a lot.
d23 4y ago
Anybody got a non-paywalled link?
- firstSpeaker 4y ago
  Oldest comment is the non-paywalled link; https://archive.ph/aRauw https://archive.ph/aRauw
ogn3rd 4y ago
This is purely a PR stunt for the shareholders confidence in Andy's leadership. We've been screaming from the rooftops for years, then they hire someone even worse as HR head, Beth Galetti, who clearly shares distain for the companies talent. Andy, you've known about the culture problems for years, the fact that you're just now acknowledging them is only because the business is hurting, and burning out all the talent. You're scared shareholders will get involved if this isn't fixed. Sadly, this won't fix anything, this is the culture YOU built.
angryasian 4y ago
Management at Amazon was the biggest problem. The don't index for soft skills. They want procedural robots that can memorize 100 stories to cover every LP. Culture fit matters and you can't fix the culture when thats what they are hiring for.

112 points by firstSpeaker 4y ago

firstSpeaker 4y ago

https://archive.ph/aRauw https://archive.ph/aRauw

deleted 4y ago

[deleted]

snarkerson 4y ago

Maybe the adversarial evaluation process has a poisonous effect?

jejeyyy77 4y ago
They're trying to fix this with dev tooling, which is the least of their problems...
jsnell 4y ago
Stripped of fluff, the main point of the article seems to be that their engineers are doing too much manual repetitive work, due to bad tooling and processes. Why would that be caused by their performance review system?
- anderber 4y ago
  Why spend time building tools that will help others if you're competing against them?
- mathattack 4y ago
  I’ve found tooling lags behind in places where incentives are too short term, or most of the funding comes from projects (rather than durable product teams). I’m not saying this is Amazon.
- innagadadavida 4y ago
  > Why would that be caused by their performance review system? May be because their performance review prioritizes employee contributions that impact bottom line immediately instead of building infra that might improve developer productivity?
- sidlls 4y ago
  Easy: tooling and process are often seen as being of little importance because they’re not easily associated with metrics that are important in toxic review process cultures. Engineers who like working on these things are treated as second class citizens and performance reviews will reflect that. (I don’t know that this is the case with Amazon: I’m just providing an anecdote based on my observations at places I’ve worked.)
  - nerdponx 4y ago
    Not just engineers, but also engineering managers and product managers. Shipping bugfixes and tooling improvements is often harder than shipping new features, and is generally less visible to the people who make hiring/firing/promotion decisions.
- closeparen 4y ago
  You can't manage what you don't measure. Developer tooling quality impact on revenue is very difficult to measure. Therefore, a rationally managed organization cannot authorize any work on developer tooling.
- deleted 4y ago
  [deleted]
choudharism 4y ago
TBF, that was scrapped in 2017. Source: worked there. There was a clear culture shock for some of the more seasoned Amazonians (and you could see a fall out effect with some "mails to your manager" every now and then if you crossed a Grand Poobah the wrong way), but to all of us "newer" hires, the times of adversarial review were just a story.
- stevewatson301 4y ago
  I’m not sure how you can say that given news reports like these in 2021, see https://www.theverge.com/2021/7/9/22570579/amazon-performance-focus-plans-hidden-employees https://www.theverge.com/2021/7/9/22570579/amazon-performanc...
  - choudharism 4y ago
    I'm not saying there aren't other issues - there certainly are. What you highlighted is one of them. All I'm saying is that The Grand "Me-vs-You" peer review process is gone. Anecdotally, I can say with confidence that some teams have a lot of autonomy in deciding to follow (or not follow) company-wide practices / guidance like what you've posted (obviously, only a few do it, since non-compliance isn't officially blessed). I was lucky enough to work in a team which was much better than baseline. But I try not to let that colour my memory - I heard and saw plenty of horror stories, and I saw plenty of people (including in my leadership) trying to do the right thing. Like most things, there are shades of grey here.
paganel 4y ago
Have never worked at Amazon nor do I know anyone who did, what exactly did that entail? "Snitching" on your co-workers? I'm asking because a quick web search didn't help me too much.

stevievee 4y ago

"...the retail giant has pledged to become the Earth's Best Employer, not just for frontline warehouse workers but for corporate and technical employees, too..."* * As long as it doesn't interfere with the share price and exec bonuses

vertis 4y ago
Barely a day goes by without a story about how badly Amazon treats it's non-tech workers. Combine that with the stack ranking and, like you, I'm not convinced. I would argue you can't fix the tech employment without fixing the rest. One bleeds into the other. I would never consider working for a company that can't treat it's more replaceable employees well.
- gumby 4y ago
  Are they still doing stack ranking and hire-to-fire?

maybedev 4y ago

Brazil should be gone

nickm12 4y ago
I was confounded by Brazil for my first 3-4 years of working with it. The lightbulb clicked on just as I was ready to move on and at my next job, I missed Brazil terribly (maybe it was just Stockholm syndrome). I eventually boomeranged back to Amazon and have found the middle ground. When used right, Brazil solves a lot of problems and day-to-day I enjoy working with it much more than not. The trouble is that almost all teams don't know how to use it and tie themselves up in knots of their own making. Third party software is also a problem. I hope Peru comes through, but so far I haven't worked with it, for good or for bad.
jonhohle 4y ago
Really? I miss Brazil and version sets regularly and often think about reimplementing the parts I find valuable on top of an existing package system. Having self contained workspaces with all dependencies in any language, fine grained ability to replace any package with a local build, and the simplicity of package config (for most languages) was awesome. Having to use _n_ language dependent build systems without interface pinning or reproducible builds since leaving sucked, imho. Checking out a package and running `bb` to get an artifact was just so satisfying.
- maybedev 4y ago
  3p packages...Python...
- vamega 4y ago
  Nix will get you most of this. It's got a learning curve, but I think it pays off and is not bad if you're familiar with the Brazil concepts. Having to set up a million dependencies just to begin to start working on a codebase is a nightmare, and Brazil makes that easy.
  - jonhohle 4y ago

twic 4y ago

Is "software builders" an Amazonism, or did this guy come up with it, and either way, are they aware how weird and insular that makes Amazon sound?

sokoloff 4y ago
Jassy has used the "builders" term for years while leading AWS. It was a central theme in many of his keynote speeches. I think it was chosen specifically because it was not an industry-standardized (and watered-down) term. https://aws.amazon.com/builders-library/ https://aws.amazon.com/builders-library/ among many other AWS docs that use the builders language.
- socialismisok 4y ago
  That language predates Jassy, it was there back around 2012 at least.
  - rawrenstein 4y ago
    Predates Jassy? He has literally worked at Amazon for decades and most of that on Amazon Web Services.
    - socialismisok 4y ago
      Predates Jassy running Amazon.
  - Cipater 4y ago
    Andy Jassy's been at Amazon since 1997 and headed AWS from its inception.
discodave 4y ago
Amazon has had an org called 'Builder Tools' or some variation on that for over a decade (maybe much longer?). That team has historically owned the systems for build, deployment, source control etc. AWS also uses the 'builder' terminology a lot in their marketing. E.g. Werner saying 'Now go build!' at the end of his keynotes, and the 'Build On AWS' campaign.

mmmeff 4y ago

I quit a few months back and hope it continues to crumble. You won’t be missed, Amazon. After 5 years and countless long nights, I was thanked with a pay cut and ignored when petitioned, even after rearchitecting the entirety of one of their floundering subsidiaries and leading a team of 10 thru it’s execution. The reason? One of my junior engineers missed a deadline. When petitioned, my cries were ignored and I was recommended to boomerang if I wanted a pay raise. At the same time, a friend was interviewing for the exact same senior engineering role and received an offer. The total compensation he was offered in his first year was nearly double my pay. He didn’t even take it - another company offered him more.

bogomipz 4y ago
>"When petitioned, my cries were ignored and I was recommended to boomerang if I wanted a pay raise." Is petitioning a process in AWS? What does boomerang mean here?
- sokoloff 4y ago
  Boomerang is “leave and come back” (quit, get a job elsewhere, later re-apply).
- 55555 4y ago
  I don't know but I imagine it means to leave and then come back.
- deleted 4y ago
  [deleted]
- jvolkman 4y ago
  Leave and get rehired.
- ertemplin 4y ago
  (Disclaimer: I left Amazon about 6 months ago) Petitioning isn't a process in AWS, but you can ask your manager for a raise if your projected total compensation isn't to your satisfaction. Boomerang means you leave the company, get another job, and then come back to the company with much higher compensation. It's a way for otherwise happy employees to fight salary compression.

phillipcarter 4y ago

It'd be great to read about all the things this internal DevX org has been able to improve after some time. Internal developer experience is extremely important. One codebase I worked on used to have a 24-hour, nondeterministic CI run. A teammate brought that time down to ~45 minutes (it's a big codebase with a ton of tests) and made it deterministic. Everything got better: * Faster turnaround time for external contributions (it's OSS) * Ability to be relied on to get a critical fix in and tested in under an hour * Much faster local build times * Removed reliance on some old busted internal infrastructure so we could use much better, different infrastructure, and make the build/test runs the same across any OS rather than rely on windows so much * Everyone was happier in general If we could have gotten a full CI run down from 45 minutes to 15 minutes, that'd be great. And there was still quite a lot of technical debt in the tests themselves, since they should be able to run a lot faster than they did. But even with that in mind, the difference was night and day.

johnbellone 4y ago
This kind of stuff is right my ally and I'd love to connect with someone at Amazon about how they're tackling these problems. Does anyone know someone that would like to grab coffee to chat about it?
- gumby 4y ago
  Your note is unlikely to yield a result. It does end with a call to action but doesn’t explain why someone should respond. It’s nice and short but needs one more sentence in the middle. This is up your alley — great! But do you want to compare notes? Do you want to get hired? What’s in it for someone from Amazon to respond (or harder, since you’re asking for an intro: why would someone send this to a friend?) I get a lot of messages like this and usually ignore them because they lack any reason to respond.
zach_garwood 4y ago
I would love to find a full-time position of just doing this kind of work. Nearly every place I've worked had some hacked together system that mostly worked, but wasn't enough of a pain for management to dedicate time to. I'd have to steal time away from the feature mill to make some little improvement, and everyone would agree that, yes, it did make things a easier for the devs, but no, we can't just have you out there working on things willy-nilly when there are features to churn out!

_fat_santa 4y ago

This type of bureaucracy is not only harmful to dev culture, it's also outrageously expensive. I've been on an engagement with a large healthcare provider and the dev culture there is very similar to Amazon, loads and loads of procedure and bureaucracy. Now i get a certain level of that at any company, but you often find yourself going: "does it really need to be this hard". Tasks that would take you an hour outside of this companies ecosystem would take you 3-4 inside the ecosystem. It's annoying to devs but should be alarming to management. You're literally spending 2x to 3x the required amount on your labor because of your inefficiencies. Personally I found it easy to convince management of changes like this when you pitch it like: "How would you like to reduce your labor costs by 2x". Now granted many people at these places are salaried which makes this a bit of a moot point but most software shops contract out alot of work, and that's where you're really paying for your inefficiencies.

foobarian 4y ago
Heh, watch out for empire-building middle managers. "Hey how would you like to reduce your headcount by 2x" will not go over well.
- dicriseg 4y ago
  It’s more like, “How would you like to get twice as much done?” I don’t work at Amazon, but I’m sure there’s plenty for folks to do.
- ryandvm 4y ago
  Yup. This is an especially dangerous suggestion when you work for a government contractor. Since a lot of them have their profit margin capped at a specific percentage, the only way for them to increase total profit is by increasing the size of the project. Suggesting efficiencies that would reduce headcount is a good way to get on executive leadership's threat radar. Man, I do not miss working for a fed contractor.
bumby 4y ago
While it's true that arbitrary bureaucratic inefficiencies can get out of control, sometimes "good" bureaucratic requirements can get inflated with the arbitrary ones, simply because people don't understand their reason in the first place. This leads to a Chesteron's fence scenario [1]. Anytime somebody say's something like "does it really need to be this hard" or "all you've got to do..." should make sure they fully understand the rationale behind the requirements. A lot of times, those requirements can be met in a more efficient manner, but sometimes the seemingly "efficient" path will cause more second-order problems down the road because of those blind spots. What you often find in both camps (those bluntly instituting the requirements and those saying they should go away) is a lack of understanding of the entire system dynamics. [1] https://fs.blog/chestertons-fence/ https://fs.blog/chestertons-fence/

jvolkman 4y ago

Sounds like a Day 2 move.

phillipcarter 4y ago
The corporate speak can be hard to stomach sometimes :)

ctvo 4y ago

Internal tooling is a mess at most companies I've worked at. At a certain scale it's worth the savings to standardize, improve, and reduce the shared friction costs. Stripe, as it has scaled, has ran into challenges and invests in the same efforts. FB internal tooling was (is?) a mess. It goes on. The surprise here is it's such a late investment from Amazon, but it's worked for them thus far. I'm guessing the returns on +1 engineer has plateaued or declined, leading to this effort.

mv4 4y ago
Re: FB - still is.

jollyllama 4y ago

>"Being an engineer at Amazon is more button-clicking than development," the slide said, quoting an unnamed Amazon developer. You guys get to click buttons??

buscoquadnary 4y ago
Lucky ducks. Development at my org largely consists of going around to every different possible "partner" to get buy in to have a meeting, to discuss approval of permissions to do something. I think the new metric used to asses an organization is PPS (Powerpoint Slides) to LOC ratio.

champagnepapi 4y ago

What about the hire to fire, and crazy amount of pips they do? How are they going to fix that part of the culture? These two are the biggest reason why I turn down amazon recruiters.

endemic 4y ago
Churn and burn, baby!

afroisalreadyin 4y ago

At the moment, the general impression I get from people working at Amazon is "Join for the experience, get it on your CV, move on after a year". This is even across departments and countries. The wife of a very good friend who lives in China is working there since nine months in some kind of operations role, and made up her mind months ago that she wouldn't take it any longer than a year. Here in Berlin, I just recently talked to a dev who left exactly a year after he started. He said it was interesting seeing the whole AWS machinery from the inside, and there was quite a lot to learn in terms of process and management, but shoving around legacy Java was just not worth it.

breitling 4y ago
> legacy Java I'm starting to feel very old Is Java considered legacy now?
- azmarks 4y ago
  Java came out 3 (or is it 4) recessions ago. There are entry level developers who were born after I started using it. It's a legacy technology.
  - 1024core 4y ago
    If you remember Duke, you're old. I still remember when Sun was going around holding Java events, touting the advantages of Java, with Duke shirts, mugs, etc.
  - PathOfEclipse 4y ago
    For a technology to be legacy, I think it needs to be stale and falling out of use for new projects. I don't see that with Java. New releases with new language features are being released at a faster pace than ever, and companies are still building new products on it. Java hits a sweet spot of performance, tooling, and productivity that few other ecosystems can. It's biggest competitors are probably C# and golang, and, for various reasons, it is holding its own just fine against them.
- 4y ago

bearjaws 4y ago

Amazon was a good gig to join (on the AWS side) 4 years ago, not so much now. Reason being everyone I know who got in around 2018 has options that cost a fraction of the current stock price. The job is still boring/miserable but when you are clearing 600k+ annual income... Now if you join, you can bet its a far worse deal and you will make far less than those who just happened to join earlier, even if they're checked out... There in lies one of their problems, the incentives are for checked out people to stay and 'click buttons' while performing as much CYA as possible.

ogn3rd 4y ago
For reference, entering in July of 2016 got you $700 RSU's. Today that's the equivalent of a $35 share.
retinaros 4y ago
not sure what you are saying the stock lasts 4 years any new stock you getting in between will be at market price so you not gonna clear 600k after 4years but only base +new stock.. if amazon stock goes up the next 4years newcomer will get 600k too
Cthulhu_ 4y ago
I'm hearing tales left and right like that; one part of the tales is that you get stocks or stock options, BUT you have to eat a shit sandwich for years before you can 'cash out'. This is the companies' golden handcuffs; they know the job is shit, but you'll get a big payout if you manage to keep it up for at least X time. The other one is that compensation and competition is sobering up. I have one former colleague who claims that they earn 3x as much working for a US company than they did our previous job, but I don't see much exorbitant wages being thrown around. Second, that company he works for now has set a hiring stop; the explosive growth and demand from US based tech companies is slowing down or coming to a halt.

fjabre 4y ago

AWS console is one of the most poorly developed builder tools Ive ever encountered for a company like Amazon, from a UX perspective at least. I may like some of the services like s3 but will never rely on AWS for anything more that that. Logging into the aws console makes my blood pressure go up.

pyb 4y ago

What do people think of DevX groups in general? I'm not sold, but in my (only n=1) experience, it'd be best to try to address the cultural problems that led to the perceived need for DevX. For instance : if build times are really bad, maybe it's because developers don't feel empowered to fix this as part of their day-to-day work.

helen___keller 4y ago
At first I was going to disagree but now I'm rethinking. The need for these DevX groups comes from the fact that at Amazon's scale, there's lots of powerful tools, such as a massive internal build system and such. So, one answer is that you need DevX groups, because it makes no sense to have each of the thousands of developers learn how to tinker/fix the huge custom amazon build system (IE empowering individual devs). A second answer might be that there shouldn't be massive internal systems in the first place. A lot of the brilliance of AWS was reducing the differentiation between internal and external teams as consumers of services[0]. The buildup of massive internal systems probably shows that Amazon hasn't gone far enough, and DevX failures are just a symptom. [0] related https://www.alexanderjarvis.com/steve-yegges-famous-rant-on-google-vs-amazon/ https://www.alexanderjarvis.com/steve-yegges-famous-rant-on-...
- pyb 4y ago
  I can't speak for AWS ; my experience is at a much smaller company, where the DevX team was mostly busy with enforcing coding standards and other uniform "best practices" upon other teams. OTOH if your build system needs a lot of tinkering/deep knowledge to be used effectively, maybe it needs a redesign. This would be a job for the tools team rather than DevX.

deleted 4y ago

[deleted]

socialismisok 4y ago

Amazon built a business on hiring starry eyed engineers and then burning them to ash. I worked for them for 6 years and holy shit I'd never go back. Their reputation is catching up with them and their lack of stock growth is highlighting their comp deficiencies.

ogn3rd 4y ago
Amen. Worked for AWS for 6 years. 6 years I'll never get back, 6 years of abusive practices, corruption, lies, and bullshit. This is the bed they've made, let the sleep in it.
- firstSpeaker 4y ago
  Is this just AWS that has such a broken culture or it is all other teams? Although AWS is big, but Amazon has other organisations and teams. (https://www.amazon.jobs/en/search-teamcategory https://www.amazon.jobs/en/search-teamcategory).
  - socialismisok 4y ago
    The rest of Amazon is worse.
voidfunc 4y ago
Always amused on Blind about people posting leaving Microsoft for Amazon because TC is the only thing folks seem to focus on there. I've known dozens of Amazon engineers and I've never met one that was happy.
- mynameisash 4y ago
  At the risk of sounding the "+1" bell... I left Amazon due to it being a total shit show (for me and my mental well-being but also for the team I was on). Went to Microsoft and found it to be absolutely wonderful. I was working at 6pm one day and my manager, as he was leaving, chided me and said, "Go home to your family. Work will be here in the morning." I have had friends/colleagues leave to go to Amazon for the TC. One even tried to get me to go over there for a PE role that likely would have meant a huge pay increase. But I value my sanity and have no plans to go back there ever.

helen___keller 4y ago

Current AWS employee of 1 year, opinions are my own I largely agree that for me, working here feels "day 2", which just means it feels the same as working for any other mid to large sized corporation. Lots of meetings, building consensus, tasks that amount to overhead, and adjusting slowly moving pieces to advance whatever the corporate agenda is. Many of these things are directly contradictory to amazon's stated leadership principles (core values) but work happens at the scope of teams not at the scope of leadership principles. Some of it is team-dependent (not unrelated, my team has really good work life balance. I slog through annoying corporate BS 9-5, but don't work much beyond that) and some of it is related to the builder tools as mentioned in the article.

ghostpepper 4y ago
Is it just the luck of the draw who ends up on the teams with good managers or can you dig your heels in if you end up on one of the bad teams and just refuse to stay late? Or can you easily switch teams until you find a good one?
- helen___keller 4y ago
  > can you dig your heels in if you end up on one of the bad teams and just refuse to stay late I mean, literally anybody anywhere can do this. Whether you get fired depends on your performance evaluations. Like any organization, the biggest reasons to stay late is (a) too high expectations were set, (b) failures to re-adjust to new information (e.g. we had to pull you off your project for 2 weeks but still expect you to make your deadline), (c) paging / oncall. At AWS, I'd say (c) is the biggest risk, but (a) and (b) can also be present at any team of course. > Is it just the luck of the draw who ends up on the teams with good managers Yeah, that's my impression. When you're already in Amazon though, you're empowered because you have a much greater ability to gather information about a team before deciding if you want to transfer. For example, message/meet with devs on the team, look at their ticket queue to see how often the oncall gets paged, and so on. > Or can you easily switch teams until you find a good one? I don't know about "until you find a good one", but transferring is pretty normalized. If you've been here a year or two, it's normal to want to transfer, and even before that it's not impossible. You can't transfer if you're on a performance improvement plan, but otherwise it's pretty straightforward

dev_tty01 4y ago

So their solution to management and process problems is to "manage harder." Yeah, this is going to go well.

user3939382 4y ago

Amazon's retail business is driven by the consumption of mass-produced garbage quality products from China that we destroy the environment producing, often abusing factory workers in the process, only to soon thereafter put them in a landfill. They've repeatedly stolen the IP of top selling product companies, then you've got this stuff https://www.forbes.com/sites/davidphelan/2019/04/12/amazon-confirms-staff-listen-to-alexa-conversations-heres-all-you-need-to-know/?sh=364d70fd5d9d https://www.forbes.com/sites/davidphelan/2019/04/12/amazon-c... Not to mention all the heinous shit (unconstitutional wars, dragnet spying) they help our military and intelligence services do https://aws.amazon.com/government-education/defense/ https://aws.amazon.com/government-education/defense/ Their abuse of their workers from warehouse to engineering is just one piece. As I tell their recruiters I'd rather eat a cactus than work for them.

des429 4y ago
> As I tell their recruiters I'd rather eat a cactus than work for them. wow, hardcore.
d-trieu 4y ago
> As I tell their recruiters I'd rather eat a cactus than work for them. That's not much of a statement https://en.wikipedia.org/wiki/Nopal https://en.wikipedia.org/wiki/Nopal
- Justsignedup 4y ago
  Cactus tacos are very tasty!
ugh123 4y ago
>As I tell their recruiters I'd rather eat a cactus than work for them. lol. whats their response?
d23 4y ago
Unless they turn around soon, I don't see how their goose isn't cooked in the long run. The site has become too filled with garbage products, hijacked listings and fake reviews. Once the people who think they've never received a counterfeit realize there's no way they can really know and start to question why the products they get from Amazon break earlier than usual or have questionable packaging, it will be hard to undo the loss of trust.

syntaxing 4y ago

Even their interview is kind of soul crushing. Felt like I was a story teller instead of an engineer trying to get a job. I can only imagine how it is working there.

PaulDavisThe1st 4y ago

From TFA: > In an email to Insider, Amazon's spokesperson confirmed the existence of the team, saying it's part of an effort to maintain its Day 1 culture. This terminology is bullshit. As someone who was actually there at Day 1 of amzn, it's completely clear that the term "Day 1 culture" is truly meaningless. It refers not to a point in time, but to some management-imposed aspiration in which things never actually settle down to a routine, but instead everyone feels "empowered" by running around constantly reinventing everything. Of course, this is sold as "you can still make a big difference here", when in reality, you almost certainly cannot. Amazon's actual Day 1 culture meant bringing up a couple of SPARCstations and getting the T1 line functioning.

steveBK123 4y ago

I know a few senior engineers / manager types who left financial services tech to work at Amazon. Most of them left within two years, and the ones that stayed quickly got out of AWS org because it was too crushing. In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff.

ogn3rd 4y ago
"In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff." This isn't always the case. Ask a TAM. With the right blend of customers I've known some to work about 10 hours a week.
- coredog64 4y ago
  JFC, that explains a lot.

d23 4y ago

Anybody got a non-paywalled link?

firstSpeaker 4y ago
Oldest comment is the non-paywalled link; https://archive.ph/aRauw https://archive.ph/aRauw

ogn3rd 4y ago

This is purely a PR stunt for the shareholders confidence in Andy's leadership. We've been screaming from the rooftops for years, then they hire someone even worse as HR head, Beth Galetti, who clearly shares distain for the companies talent. Andy, you've known about the culture problems for years, the fact that you're just now acknowledging them is only because the business is hurting, and burning out all the talent. You're scared shareholders will get involved if this isn't fixed. Sadly, this won't fix anything, this is the culture YOU built.

angryasian 4y ago

Management at Amazon was the biggest problem. The don't index for soft skills. They want procedural robots that can memorize 100 stories to cover every LP. Culture fit matters and you can't fix the culture when thats what they are hiring for.