7 ms·

The away team model at Amazon (2022)

82 points by softwaredoug 3y ago

starkparker 3y ago
Very curious how this works in practice, because having worked as a non-software eng role (support, SRE, tech writer) for "tiger teams" and other out-of-band task forces with limited scope, any of those roles associated with away/host teams become the metaphorical red shirts[1], buried alive under the actual handoff/maintenance work that the engs are too valuable to the org to touch. 1: https://en.wikipedia.org/wiki/Redshirt_(stock_character) https://en.wikipedia.org/wiki/Redshirt_(stock_character)
- cjhowarddev 3y ago
  I'm at Amazon now and my team is often the target of away team work as our system is seen as "critical" to other business unit growth. it's a farce. Interesting idea in theory, terrible in practice. The away team incentive is "launch as fast as possible at all costs" and that goes about as well as you'd expect. We constantly deal with shit code and loose rules. Given that many away teams work in different time zones AND Amazon gives them carte blanche to get their work done, the host team can't manage them effectively. So yeah. On one hand management is like "push back! Make them follow your standards!" while also saying "Don't dedicate time to away teams, you have your own deliverables." It's a major drain on the home team and can cause a lot of unnecessary stress. These opinions are my own, obviously, lol.
  - deleted 3y ago
    [deleted]
  - billllll 3y ago
    I think it really depends on the leadership. The worse case scenario is when the scenario you given: the "pet project," where a team goes nuts on a codebase they aren't maintaining, in which case it's effectively out-sourcing maintenance and tech debt. Ideally, the home team leadership has a lot of leverage to push back on work by the away team, which means away team work is slow but at least it gets done.
  - angarg12 3y ago
    Fellow amazonian here. I have been on both sides of away teams and I agree that it sucks for everyone involved. You explains how it looks as the host teams. It also sucks for the guest team. * You have to work with an, at best, passive team, at worst, actively hostile team. * You need to work against very tight deadlines and a lot of pressure in a codebase that you know nothing about, and need to ramp up in quickly. * You spent weeks (months?) developing expertise in a system you will probably never work again. It is a terrible idea, but to be honest I can't think of another way to deal with the problem that it is trying to solve (mismatch between agendas of different teams).
    - SoftTalker 3y ago
      Bubble the issue up the hierarchy until someone with authority over both teams can tell them what is more important?
      - angarg12 3y ago
        We resort to away teaming when what you describe doesn't work, usually by telling both teams that both goals are important, and they need to figure out how to deliver. Since away teaming is an accepted technique, it's usually considered as a way to solve these issues. I guess we would need to completely give away the concept, and just accept that some thing won't get done unless they are important enough to keep two distant teams aligned (in other words, they are important enough for someone high in the hierarchy).
    - hoodedmongoose 3y ago
      The open source model? If the away team can't get the home team to fulfill their requirements, they fork home teams code and take full ownership of it for their purposes. If they want to take updates from home team, they do so but have to handle the merging. If they want to, they can try to get home team to take their changes, but that's at home teams full discretion.
      - Others 3y ago
        How does that work when the thing you want to change is a complex micro-service with a public API? Do you want to spin up your own version of that service and own it forever? Even if you’re okay with that, you’ll not have the clients the older version has Open-source model works for libraries, or small changes in service code (where the maintenance burden is trivial). But it doesn’t work for complex services with high maintenance burdens
  - bluedevilzn 3y ago
    Having worked at both amazon and Google, IMO Google does this significantly better by having OWNERS, away team member plus readability approver. This comes at the cost of much slower progress.
  - fluoride1002 3y ago
    Before Away Teams became a thing there we had people "loaned" to teams to update our code for team projects. I remember during one of the code reviews I did for someone "loaned" to us involved him commenting out validation checks in critical systems that had legal and financial ramifications. When I asked him why he commented it out he said "Oh, my code didn't work with those checks in there so I removed them." I get that the code is complex and he doesn't have the full domain knowledge, but he didn't even ask why they were there and why certain use cases were not allowed for that part of the code.
    - ramraj07 3y ago
      This model only works if the away teams are the most competent people you have. Generally the “loaning” business is a cowardly way for teams to trade the worst engineers you have (bonus point if that engineer is someone who everyone praises but is secretly shit and only the manager knows that reality).
      - 8note 3y ago
        I've only gotten completely new hires as loans, where you also have to train them up to being useful
    - hiyer 3y ago
      I don't see how having Away Teams fixes this issue. The Away Team member is still free to write shitty code (or un-write good code, as in this case), and it is still the responsibility of the home team to catch that in code reviews.
- grensley 3y ago
  This has been my experience as well (as a dev). Curious if there's others in the same sort of situation, and I'd love to hear more thoughts. My general tips are: - Use as much as you can of what they already have. - Talk to them, but don't waste their time. - Follow their patterns. - Understand at the end of the day its their domain. - Choose your advice carefully. - Try to throw a little extra gratitude their way. Organizational visibility, code reviews, documentation, mentorship, whatever feels appropriate and not over-stepping. - Put extra effort into handoff documentation.
- x86x87 3y ago
  [dead]
- jrd259 3y ago
  I'm at Amazon and am the senior developer on my team. For the last few months an "away" team has been adding a feature to a service I own. It's only n of 1, but the process worked well. I did the high level design and ensured alignment on requirements. The away team did an excellent job of learning our code base. The new feature required a tricky refactoring of some of our oldest and most rotten code, which I did myself as it seemed too hard for an away team. I saw no sign at all of the away team compromising on quality to meet deadlines.
  - gregw2 3y ago
    I am using a similar concept to scale my team's impact at a F100 company; making processes and ways for other teams to contribute code that is otherwise bottlenecked to/by us but ends up getting owned by us. It's worked for the last year or two semi-effectively, but I am loving this discussion for reminding me of the downsides/risks.
ilc 3y ago
The weight of having to support what you write is an important one for a developer IMHO. It is easy to slap together crap and toss it over the wall as good enough, and then, the "hone team" is stuck with supporting it. For example: Imagine building the feature on a O(n^2) data structure, it'll work fine to start, but long term, we'll need a O(log(n)) structure. I wonder who has to do the work on the "slow performance" bug, and data migration. These concerns aren't abstract. They are very concrete. There will be landmines left behind. Even when we try not to leave them behind we do as developers. Without responsibility... it would be even worse. This is also why I don't like having one team do all the Greenfield work, and one do all the fixing.
- softwaredoug 3y ago
  I much prefer an “operating system” model. The away teams should be writing “userland” code within the home teams well defined guardrails. The home teams act as OS devs for their service mostly write “kernel” code for their service that empower the away teams. Usually the “home team” are functional specialists. They are working on some piece of infrastructure or maintaining a service. The “away” team OTOH is building a feature and needs to touch a few dozen codebases It should be obvious which code is kernel and maintained by the service team, and which by the away “app” team. The away teams should be maintaining, on-call, and responsible for their code running within the context of the home teams service. Which hosts many such “apps” https://softwaredoug.com/blog/2023/07/08/why-search-orgs-fail.html https://softwaredoug.com/blog/2023/07/08/why-search-orgs-fai...
  - flaminHotSpeedo 3y ago
    It's painfully obvious you've never been on either end of this problem. In big orgs, if you have time to set up baby fences (or had them from the start) you have enough free time to build the feature yourself. No one who hosts an away team is prepared to do so
    - pfannkuchen 3y ago
      I have successfully done this repeatedly in a very big org, though it wasn’t easy and it also wasn’t at Amazon.
drewcoo 3y ago
How does this relate to Brooks' Law? https://en.wikipedia.org/wiki/Brooks%27s_law https://en.wikipedia.org/wiki/Brooks%27s_law Also, out of curiosity, does AMZN have red shirts on the away teams?
- x86x87 3y ago
  All people are red shirts at amazon
- gregw2 3y ago
  Don't know about Amazon-specific details but since I have setup a similar mechanism with my team and our "partner teams" within a big company to solve similar scaling challenges I will make a few observations. Per that wikipedia article, Brook's law is more about trying to use more people to make a late projects faster, than projects in general starting from scratch. Knowledge transfer (KT) mid-project, and knowledge transfer as the project inevitably evolves in design/implementation is the challenge due to the need for N people to communicate which can lead to n-squared communication paths which fails to scale. Brook's law also doesn't really apply if you have some segmentation so the "Away" team and "Home" team have semi-clear boundaries of responsibility allowing them to work independently. The architecture of whatever you are working on has to have some internal boundaries where the "Away" team works on something that generally doesn't require change (or awareness of changes) from the "Home" team and vice-versa. It is presumed in this Home/Away scenario that the "Home" platform/system is fairly stable for the most part and the "Away" team is just plugging in features/new-capabilities and there is some degree of organization and pattern in the core system already. Ability to scale with multiple Away teams is based on 1) the degree of that segmentation, and also 2) any reduction you can make in amount of upfront KT needed and judgement-based decisions that otherwise require interaction along the way and during the later review stages. The KT and communications along the way and review process will still cause some Amdahl's-law like situations in overall scalability but you can avoid the "n-squared" Brooks phenomena. In my experience, the problem with the "Home"/"Away" approach I've seen so far (from a manager/architect perspective more than IC) is that Away teams tend to be project based, so you spend a lot of time KT'ing between Home and Away, and when the project end, that KT goes to waste and is lost. If you have a steady stream of incremental work done by away teams, management should arrange to create a partner permanent team divided along some architectural boundary ideally (make the org match the architecture) or if needed, along some different mission statement. This is analogous to what happens when you have employee turnover on your team.
jawns 3y ago
I'm in a home team (not at Amazon) that wants to make our domain easier for away teams to work in. Aside from writing good documentation, what else can we do to make it a hospitable environment without it becoming a drain on our resources?
- skellera 3y ago
  Create IDE configs and other guardrails to limit churn and discussions on already determined style for your team. Clear goals and acceptance criteria.
- Others 3y ago
  1. Exercise good judgment about what projects are “away team-able”. Be okay saying “this needs to be done by our team” even when that’s hard. Otherwise you’ll end up with tricky features implemented by people who don’t understand the full context (this sucks very badly) 2. Make sure the amount of process matches the amount of complexity. If the change is very simple or only a few lines, make it trivial to submit code changes that will pass review. (So have really good automated style checks/test suites.) If the change is complex, make sure you’re doing design reviews before implementation, to ensure the away team doesn’t waste time with bad implementation strategies
billllll 3y ago
This passage: > ...it should not be the first option for developing necessary changes to a system dependency. Away team work is complex and inefficient. Is very telling. I've worked in a similar model, and I'd say this is basically the important part for everyone wondering how well it works. The main thing to note is that the arrangement should be of last resort, when the crushing bureaucracy means it's either away teams, or nothing at all.
- chrisandchris 3y ago
  I think your quote is contradicted by the heading > Away teams accelerate software delivery [...] So while it is compley and inefficient, it also accelerates delivery? Also from the comments here I get that away teams are mostly stuck in meetings. So I find it interesting to use away teams as a solution where crushing bureaucracy seems the main issue.
jtchang 3y ago
What problem specifically is the away team solving? It sounds like DLL dependency hell except at a higher scope.
WalterBright 3y ago
Boeing has an AOG team, which means "Airplane On Ground". As it was explained to me, airliners are a money generating machine for airlines. But only when they're flying. When they're sitting on the ground, they're a black hole sucking up buckets of cash. The airline business model is therefore maximizing the percentage of time an airplane is in the air generating cash. Boeing responds to this by expending a lot of design effort towards minimizing required maintenance, and making maintenance quick and easy to do. But, sometimes, an airplane develops a fault that the airline mechanics cannot fix. Enter the AOG team. The AOG team is composed of the best engineers and mechanics, and they are on a moment's notice to fly anywhere and get that AOG back in the air ASAP. I heard Microsoft has a similar team to go out and fix a customer's software. It's one reason Microsoft is able to sell expensive enterprise software in the face of free software. It's worth the money corporations pay to Microsoft to "come out and fix it!"
- SoftTalker 3y ago
  "X" only makes money when "X" is working is the case for most high-value assets whether it's an aircraft, ship, tugboat, other specialized heavy equipment, mainframe computer, or countless other things. Most manufacturers of this stuff have "go teams" that will fly out or express-ship parts, etc. it's part of why those things tend to be expensive.
- BrentOzar 3y ago
  > I heard Microsoft has a similar team to go out and fix a customer's software. About 20 years ago, they were called Rangers, modeled after the Army Rangers, soldiers who could venture out into enemy territory with no support and accomplish missions out there. About 15 years ago, the program evolved into Microsoft Certified Masters, and it included both internal staff and specialized outside consultants. (I was one of about 200 MCMs worldwide for SQL Server.)
  - runlaszlorun 3y ago
    This old soldier is cringing at the fact that they called themselves Rangers. I’m guessing no actual Rangers were involved. If they called themselves SEALs though, I’d be laughing my ass off.
fluoride1002 3y ago
It makes me nervous when I read articles like this and then do some research and find out that the author didn't work as an engineer in engineering teams. Someone will pick this up and say "Amazon does this - we should too!" Not really understanding the ramifications of this model for all the parties involved should be front and center when there are write-ups like this. It is so incredibly easy for this to go sideways and create so many problems once the work is done.
- kartoshechka 3y ago
  agree, the ambiguous context enables people in charge practicing quite elaborate mental gymnastics so they can justify copying big corp practices however giving enough details might be prohibited by NDAs and whatnot, or even render companies as toxic/disfunctional so nobody will ever join them :)
- dbish 3y ago
  Yeah, this seems to be another person trading on the name of amazon for clicks without actual relevant experience. I’ve seen this with folks talking about the amazon writing culture and PR/FAQs who I know for a fact had no products or features themselves that came out of any such writing.
dr_kretyn 3y ago
I used to be in the Away Team at Amazon. Terrible experience. Within 0.5y I changed 3 lines of code. Most of the time was spent in unproductive meetings with individual teams "gossiping" - he said it works that way but she thinks that's actually an edge case because of this so what do you think. Some code was secret, all documentation was outdated. Asking for a specific code change, or waiting for a promised deliverable usually was just idle time. All that while teams have their own goals and priorities, and you're treated like an outsider. Honestly I feel like I wasted a year of life while at that team.
- mr_o47 3y ago
  That’s very true. I had a similar experience while working there
yellow_lead 3y ago
Here's the problem: > It doesn’t affect ownership. After the away team finishes its changes, the service ownership remains with the host teams. The away team model also reflects how Amazon weakly assigns code ownership to service teams. The article quotes Martin Fowler to back this up: > Weak code ownership: Developers are allowed to change modules owned by other people. Module owners are expected to take responsibility for the modules they own and keep an eye on changes made by other people. However, this quote seems taken out of context. It is about individual developers making changes. When you have an entire team doing this for (months?) at a time, can maintainers really keep up with the changes and support them in perpetuity in addition to their other responsibilities?
tracerbulletx 3y ago
In theory this sounds like having everything be an internal open source project, where the home team are the core maintainers, and the away teams are contributors. Seems good in theory, though the other comments here make it sound like it didn't work well in practice. It would take quite a bit of discipline to make it work that may not be available within the confines of corporate politics and incentives. I can imagine an away team half assing something then complaining to a VP that the home team is the reason the product isn't released yet who then yells at the home team, who then says f it and stops caring (or other variants of this).
valbaca 3y ago
Working in away team models were my worst time at Amazon and this my worst times in my professional life. You’re not supported but being expected to deliver by your “home” team. Meanwhile the away team has no obligation to help you. Can spin their thumbs and give you three day turn arounds on code reviews only to move the goal posts and/or hold you to some unattainable standard. For example “all integration tests need to pass” while the integration tests haven’t even STARTED correctly in over a year. Not to mention the increased visibility and communication that comes with now effectively having 2+ managers you have to report to and keep in the loop. FUCK. THE. AWAY. TEAM. MODEL. I’ve lived through it and it seriously burned me out harder than anything else in my life.
- hyperliner 3y ago
  “Meanwhile the away team has no obligation to help you.” I think you meant the host team.
- deleted 3y ago
  [deleted]
beastcoast 3y ago
Away teaming is definitely a last resort at Amazon, and has all the pitfalls mentioned here - much better to do a budgetary transfer so the host team is properly resourced to make the changes, assuming the feature doesn’t make it into the host teams operational plan.
menzoic 3y ago
This is very common at Uber and Airbnb
fortylove 3y ago
As someone who did away team work at Amazon, and also hosted away teams: the model sucks in most ways and was unpleasant for everyone. It got the job done in extreme situations, but it was mostly a result of subpar managers taking the term "fungible" too seriously. Source: was sr engineer at Amazon
flaminHotSpeedo 3y ago
I can't think of a way to concisely convey my response without being crass, so here it is: The author of this piece is an idiot who has drunk the Kool aid and has very clearly never performed away team work. I have personally wasted a year of my time on away team work, have supported several away teams, and most importantly: operated systems/built on top of stuff built by away teams. All of this at Amazon, specifically. Having personally experienced every engineering facet of away teams, I feel as qualified as anyone to say this: away teams are a stupid concept that hurts everyone except management. You spend way more time to deliver a product you don't really understand and will have a hard time spinning on promos, which will almost certainly be a nightmare to maintain and operate, all because your employer is too cheap to allocate headcount for surprises and your team took a dependency on another team years ago so they have you by the balls.
- dbish 3y ago
  The author likely didn’t participate in the away team model as according to LI he was a solutions architect during his time at amazon, so kind of odd to write about it now.
aledalgrande 3y ago
For all the ex Amazonians saying the model suck, maybe it was the culture that sucked? Where I work we have loosely the same model, doesn't even have a name, but the component owners are always very helpful.
mkl95 3y ago
Take my comment with a grain of salt because I have never worked for Amazon and I don't have any relationship with the company. Since these are Amazon employees, I would expect them to experience the downsides of being a consultant, such as alienation, and the downsides of belonging to a product organization, such as extreme pressure. > As organizations grow, engineers’ code intertwines with other engineers’ work, making it difficult for teams to complete their features independently. The current consensus in the industry is that it should be the opposite. To grow in a healthy way, teams should be as decoupled as possible. > The away team model is a powerful and flexible mechanism for removing dependencies between teams. However, it should not be the first option for developing necessary changes to a system dependency. Did I miss something? You introduced this team model because there was a dependency, but you never removed it. This post is full of contradictions.
- jq-r 3y ago
  > Did I miss something? You introduced this team model because there was a dependency, but you never removed it. This post is full of contradictions. It's just another article by a manager expounding the virtues of a managerial fad to score points in his org. Sounds great to his peers, sounds like a nightmare to ICs.
  - dbish 3y ago
    Not even a manager, his LI says solutions architect which is neither an engineer or manager so he would not have first hand experience doing it for actual software dev.
    - jq-r 3y ago
      Right, those "solutions architects" like "Techical Account Managers" which act like moats against receiving any kind of good support. Those guys are just terrible IMO unfortunately.
kthejoker2 3y ago
I see a lot of comments saying they "wasted" months or years on away team work, it was hard to quantify for promos, the KT itself from the work went to waste etc. Totally makes sense for career- and purpose-driven employees. So one management takeaway is this type of work seems ideal for highly competent outsourcing. Offer a bonus for early delivery, and hire a small army of cheap out of school eager beaver BAs to just go around frantically documenting all the tech debt being incurred.
- dbish 3y ago
  Out of school BAs seem like the the best way to waste significant dev time explaining to people who don’t understand the tech or technical documentation what to do.
8note 3y ago
Something not captured here is that the host team tends to have so much turnover that there's little difference between the host team's engineers working on a project and an away team's.
NBJack 3y ago
Have worked for the A to Z company. Away teams as a concept exists because teams that are overworked and understaffed can't do everything asked of them (including things you can't say no to, like S-Team goals). The Search team was especially famous/infamous for using this, due to the sheer volume of asks (i.e. can we do a twister on fabric types for clothing and see if it improves dress sales?). Your team makes the ask, search says "sure we'll support you as you make the changes to it," and the asking team provides the brave souls diving into a strange codebase. Does it work? Technically, yes. Is it sustainable for the engineers on both sides? Not in my eexperience. >> An away team is a self-sufficient engineering team that works on code owned by another team (the host team) to deliver features they are dependent on. This is fundamentally wrong. The point of the away team is to get support from the host team as they work. It is not self sufficient by nature. Otherwise, this is just a team randomly contributing to a codebase you own.
- canttestthis 3y ago
  > brave souls diving into a strange codebase As someone who made a career out of being loaned out to other teams as an 'away resource' at Amazon, I've struggled to find a similar level of mental stimulation after leaving for other Big 4 companies. Being a loaner engineer meant something new to work on every few months, no maintenance or oncall burden, making lots of new connections, and just generally there being always something interesting/intense to do.
  - NBJack 3y ago
    I won't dispute your level of enjoyment from it, but I'd have to ask: did your fellow engineers feel the same way about it? What was your time commitment like to deliver? Also, if you're looking for a thrill, I would highly recommend going after cloud-based infra providers. There's never a shortage of unique problems there.
    - canttestthis 3y ago
      Time commitments were generally aggressive but reasonable with a 10-20% buffer to account for ramp up (realistically closer to 10 than 20). The reasonable-ness of most things eng-culture related at Amazon is probably a function of the competence of your management chain. As for my fellow engineers- I don’t know anyone else who did this regularly so I’m not sure

82 points by softwaredoug 3y ago

Very curious how this works in practice, because having worked as a non-software eng role (support, SRE, tech writer) for "tiger teams" and other out-of-band task forces with limited scope, any of those roles associated with away/host teams become the metaphorical red shirts[1], buried alive under the actual handoff/maintenance work that the engs are too valuable to the org to touch. 1: https://en.wikipedia.org/wiki/Redshirt_(stock_character) https://en.wikipedia.org/wiki/Redshirt_(stock_character)

cjhowarddev 3y ago
I'm at Amazon now and my team is often the target of away team work as our system is seen as "critical" to other business unit growth. it's a farce. Interesting idea in theory, terrible in practice. The away team incentive is "launch as fast as possible at all costs" and that goes about as well as you'd expect. We constantly deal with shit code and loose rules. Given that many away teams work in different time zones AND Amazon gives them carte blanche to get their work done, the host team can't manage them effectively. So yeah. On one hand management is like "push back! Make them follow your standards!" while also saying "Don't dedicate time to away teams, you have your own deliverables." It's a major drain on the home team and can cause a lot of unnecessary stress. These opinions are my own, obviously, lol.
- deleted 3y ago
  [deleted]
- billllll 3y ago
  I think it really depends on the leadership. The worse case scenario is when the scenario you given: the "pet project," where a team goes nuts on a codebase they aren't maintaining, in which case it's effectively out-sourcing maintenance and tech debt. Ideally, the home team leadership has a lot of leverage to push back on work by the away team, which means away team work is slow but at least it gets done.
- angarg12 3y ago
  Fellow amazonian here. I have been on both sides of away teams and I agree that it sucks for everyone involved. You explains how it looks as the host teams. It also sucks for the guest team. * You have to work with an, at best, passive team, at worst, actively hostile team. * You need to work against very tight deadlines and a lot of pressure in a codebase that you know nothing about, and need to ramp up in quickly. * You spent weeks (months?) developing expertise in a system you will probably never work again. It is a terrible idea, but to be honest I can't think of another way to deal with the problem that it is trying to solve (mismatch between agendas of different teams).
  - SoftTalker 3y ago
    Bubble the issue up the hierarchy until someone with authority over both teams can tell them what is more important?
    - angarg12 3y ago
      We resort to away teaming when what you describe doesn't work, usually by telling both teams that both goals are important, and they need to figure out how to deliver. Since away teaming is an accepted technique, it's usually considered as a way to solve these issues. I guess we would need to completely give away the concept, and just accept that some thing won't get done unless they are important enough to keep two distant teams aligned (in other words, they are important enough for someone high in the hierarchy).
  - hoodedmongoose 3y ago
    The open source model? If the away team can't get the home team to fulfill their requirements, they fork home teams code and take full ownership of it for their purposes. If they want to take updates from home team, they do so but have to handle the merging. If they want to, they can try to get home team to take their changes, but that's at home teams full discretion.
    - Others 3y ago
      How does that work when the thing you want to change is a complex micro-service with a public API? Do you want to spin up your own version of that service and own it forever? Even if you’re okay with that, you’ll not have the clients the older version has Open-source model works for libraries, or small changes in service code (where the maintenance burden is trivial). But it doesn’t work for complex services with high maintenance burdens
- bluedevilzn 3y ago
  Having worked at both amazon and Google, IMO Google does this significantly better by having OWNERS, away team member plus readability approver. This comes at the cost of much slower progress.
- fluoride1002 3y ago
  Before Away Teams became a thing there we had people "loaned" to teams to update our code for team projects. I remember during one of the code reviews I did for someone "loaned" to us involved him commenting out validation checks in critical systems that had legal and financial ramifications. When I asked him why he commented it out he said "Oh, my code didn't work with those checks in there so I removed them." I get that the code is complex and he doesn't have the full domain knowledge, but he didn't even ask why they were there and why certain use cases were not allowed for that part of the code.
  - ramraj07 3y ago
    This model only works if the away teams are the most competent people you have. Generally the “loaning” business is a cowardly way for teams to trade the worst engineers you have (bonus point if that engineer is someone who everyone praises but is secretly shit and only the manager knows that reality).
    - 8note 3y ago
      I've only gotten completely new hires as loans, where you also have to train them up to being useful
  - hiyer 3y ago
    I don't see how having Away Teams fixes this issue. The Away Team member is still free to write shitty code (or un-write good code, as in this case), and it is still the responsibility of the home team to catch that in code reviews.
grensley 3y ago
This has been my experience as well (as a dev). Curious if there's others in the same sort of situation, and I'd love to hear more thoughts. My general tips are: - Use as much as you can of what they already have. - Talk to them, but don't waste their time. - Follow their patterns. - Understand at the end of the day its their domain. - Choose your advice carefully. - Try to throw a little extra gratitude their way. Organizational visibility, code reviews, documentation, mentorship, whatever feels appropriate and not over-stepping. - Put extra effort into handoff documentation.
x86x87 3y ago
[dead]
jrd259 3y ago
I'm at Amazon and am the senior developer on my team. For the last few months an "away" team has been adding a feature to a service I own. It's only n of 1, but the process worked well. I did the high level design and ensured alignment on requirements. The away team did an excellent job of learning our code base. The new feature required a tricky refactoring of some of our oldest and most rotten code, which I did myself as it seemed too hard for an away team. I saw no sign at all of the away team compromising on quality to meet deadlines.
- gregw2 3y ago
  I am using a similar concept to scale my team's impact at a F100 company; making processes and ways for other teams to contribute code that is otherwise bottlenecked to/by us but ends up getting owned by us. It's worked for the last year or two semi-effectively, but I am loving this discussion for reminding me of the downsides/risks.

ilc 3y ago

The weight of having to support what you write is an important one for a developer IMHO. It is easy to slap together crap and toss it over the wall as good enough, and then, the "hone team" is stuck with supporting it. For example: Imagine building the feature on a O(n^2) data structure, it'll work fine to start, but long term, we'll need a O(log(n)) structure. I wonder who has to do the work on the "slow performance" bug, and data migration. These concerns aren't abstract. They are very concrete. There will be landmines left behind. Even when we try not to leave them behind we do as developers. Without responsibility... it would be even worse. This is also why I don't like having one team do all the Greenfield work, and one do all the fixing.

softwaredoug 3y ago
I much prefer an “operating system” model. The away teams should be writing “userland” code within the home teams well defined guardrails. The home teams act as OS devs for their service mostly write “kernel” code for their service that empower the away teams. Usually the “home team” are functional specialists. They are working on some piece of infrastructure or maintaining a service. The “away” team OTOH is building a feature and needs to touch a few dozen codebases It should be obvious which code is kernel and maintained by the service team, and which by the away “app” team. The away teams should be maintaining, on-call, and responsible for their code running within the context of the home teams service. Which hosts many such “apps” https://softwaredoug.com/blog/2023/07/08/why-search-orgs-fail.html https://softwaredoug.com/blog/2023/07/08/why-search-orgs-fai...
- flaminHotSpeedo 3y ago
  It's painfully obvious you've never been on either end of this problem. In big orgs, if you have time to set up baby fences (or had them from the start) you have enough free time to build the feature yourself. No one who hosts an away team is prepared to do so
  - pfannkuchen 3y ago
    I have successfully done this repeatedly in a very big org, though it wasn’t easy and it also wasn’t at Amazon.

drewcoo 3y ago

How does this relate to Brooks' Law? https://en.wikipedia.org/wiki/Brooks%27s_law https://en.wikipedia.org/wiki/Brooks%27s_law Also, out of curiosity, does AMZN have red shirts on the away teams?

x86x87 3y ago
All people are red shirts at amazon
gregw2 3y ago
Don't know about Amazon-specific details but since I have setup a similar mechanism with my team and our "partner teams" within a big company to solve similar scaling challenges I will make a few observations. Per that wikipedia article, Brook's law is more about trying to use more people to make a late projects faster, than projects in general starting from scratch. Knowledge transfer (KT) mid-project, and knowledge transfer as the project inevitably evolves in design/implementation is the challenge due to the need for N people to communicate which can lead to n-squared communication paths which fails to scale. Brook's law also doesn't really apply if you have some segmentation so the "Away" team and "Home" team have semi-clear boundaries of responsibility allowing them to work independently. The architecture of whatever you are working on has to have some internal boundaries where the "Away" team works on something that generally doesn't require change (or awareness of changes) from the "Home" team and vice-versa. It is presumed in this Home/Away scenario that the "Home" platform/system is fairly stable for the most part and the "Away" team is just plugging in features/new-capabilities and there is some degree of organization and pattern in the core system already. Ability to scale with multiple Away teams is based on 1) the degree of that segmentation, and also 2) any reduction you can make in amount of upfront KT needed and judgement-based decisions that otherwise require interaction along the way and during the later review stages. The KT and communications along the way and review process will still cause some Amdahl's-law like situations in overall scalability but you can avoid the "n-squared" Brooks phenomena. In my experience, the problem with the "Home"/"Away" approach I've seen so far (from a manager/architect perspective more than IC) is that Away teams tend to be project based, so you spend a lot of time KT'ing between Home and Away, and when the project end, that KT goes to waste and is lost. If you have a steady stream of incremental work done by away teams, management should arrange to create a partner permanent team divided along some architectural boundary ideally (make the org match the architecture) or if needed, along some different mission statement. This is analogous to what happens when you have employee turnover on your team.

jawns 3y ago

I'm in a home team (not at Amazon) that wants to make our domain easier for away teams to work in. Aside from writing good documentation, what else can we do to make it a hospitable environment without it becoming a drain on our resources?

skellera 3y ago
Create IDE configs and other guardrails to limit churn and discussions on already determined style for your team. Clear goals and acceptance criteria.
Others 3y ago
1. Exercise good judgment about what projects are “away team-able”. Be okay saying “this needs to be done by our team” even when that’s hard. Otherwise you’ll end up with tricky features implemented by people who don’t understand the full context (this sucks very badly) 2. Make sure the amount of process matches the amount of complexity. If the change is very simple or only a few lines, make it trivial to submit code changes that will pass review. (So have really good automated style checks/test suites.) If the change is complex, make sure you’re doing design reviews before implementation, to ensure the away team doesn’t waste time with bad implementation strategies

billllll 3y ago

This passage: > ...it should not be the first option for developing necessary changes to a system dependency. Away team work is complex and inefficient. Is very telling. I've worked in a similar model, and I'd say this is basically the important part for everyone wondering how well it works. The main thing to note is that the arrangement should be of last resort, when the crushing bureaucracy means it's either away teams, or nothing at all.

chrisandchris 3y ago
I think your quote is contradicted by the heading > Away teams accelerate software delivery [...] So while it is compley and inefficient, it also accelerates delivery? Also from the comments here I get that away teams are mostly stuck in meetings. So I find it interesting to use away teams as a solution where crushing bureaucracy seems the main issue.

jtchang 3y ago

What problem specifically is the away team solving? It sounds like DLL dependency hell except at a higher scope.

WalterBright 3y ago

Boeing has an AOG team, which means "Airplane On Ground". As it was explained to me, airliners are a money generating machine for airlines. But only when they're flying. When they're sitting on the ground, they're a black hole sucking up buckets of cash. The airline business model is therefore maximizing the percentage of time an airplane is in the air generating cash. Boeing responds to this by expending a lot of design effort towards minimizing required maintenance, and making maintenance quick and easy to do. But, sometimes, an airplane develops a fault that the airline mechanics cannot fix. Enter the AOG team. The AOG team is composed of the best engineers and mechanics, and they are on a moment's notice to fly anywhere and get that AOG back in the air ASAP. I heard Microsoft has a similar team to go out and fix a customer's software. It's one reason Microsoft is able to sell expensive enterprise software in the face of free software. It's worth the money corporations pay to Microsoft to "come out and fix it!"

SoftTalker 3y ago
"X" only makes money when "X" is working is the case for most high-value assets whether it's an aircraft, ship, tugboat, other specialized heavy equipment, mainframe computer, or countless other things. Most manufacturers of this stuff have "go teams" that will fly out or express-ship parts, etc. it's part of why those things tend to be expensive.
BrentOzar 3y ago
> I heard Microsoft has a similar team to go out and fix a customer's software. About 20 years ago, they were called Rangers, modeled after the Army Rangers, soldiers who could venture out into enemy territory with no support and accomplish missions out there. About 15 years ago, the program evolved into Microsoft Certified Masters, and it included both internal staff and specialized outside consultants. (I was one of about 200 MCMs worldwide for SQL Server.)
- runlaszlorun 3y ago
  This old soldier is cringing at the fact that they called themselves Rangers. I’m guessing no actual Rangers were involved. If they called themselves SEALs though, I’d be laughing my ass off.

fluoride1002 3y ago

It makes me nervous when I read articles like this and then do some research and find out that the author didn't work as an engineer in engineering teams. Someone will pick this up and say "Amazon does this - we should too!" Not really understanding the ramifications of this model for all the parties involved should be front and center when there are write-ups like this. It is so incredibly easy for this to go sideways and create so many problems once the work is done.

kartoshechka 3y ago
agree, the ambiguous context enables people in charge practicing quite elaborate mental gymnastics so they can justify copying big corp practices however giving enough details might be prohibited by NDAs and whatnot, or even render companies as toxic/disfunctional so nobody will ever join them :)
dbish 3y ago
Yeah, this seems to be another person trading on the name of amazon for clicks without actual relevant experience. I’ve seen this with folks talking about the amazon writing culture and PR/FAQs who I know for a fact had no products or features themselves that came out of any such writing.

dr_kretyn 3y ago

I used to be in the Away Team at Amazon. Terrible experience. Within 0.5y I changed 3 lines of code. Most of the time was spent in unproductive meetings with individual teams "gossiping" - he said it works that way but she thinks that's actually an edge case because of this so what do you think. Some code was secret, all documentation was outdated. Asking for a specific code change, or waiting for a promised deliverable usually was just idle time. All that while teams have their own goals and priorities, and you're treated like an outsider. Honestly I feel like I wasted a year of life while at that team.

mr_o47 3y ago
That’s very true. I had a similar experience while working there

yellow_lead 3y ago

Here's the problem: > It doesn’t affect ownership. After the away team finishes its changes, the service ownership remains with the host teams. The away team model also reflects how Amazon weakly assigns code ownership to service teams. The article quotes Martin Fowler to back this up: > Weak code ownership: Developers are allowed to change modules owned by other people. Module owners are expected to take responsibility for the modules they own and keep an eye on changes made by other people. However, this quote seems taken out of context. It is about individual developers making changes. When you have an entire team doing this for (months?) at a time, can maintainers really keep up with the changes and support them in perpetuity in addition to their other responsibilities?

tracerbulletx 3y ago

In theory this sounds like having everything be an internal open source project, where the home team are the core maintainers, and the away teams are contributors. Seems good in theory, though the other comments here make it sound like it didn't work well in practice. It would take quite a bit of discipline to make it work that may not be available within the confines of corporate politics and incentives. I can imagine an away team half assing something then complaining to a VP that the home team is the reason the product isn't released yet who then yells at the home team, who then says f it and stops caring (or other variants of this).

valbaca 3y ago

Working in away team models were my worst time at Amazon and this my worst times in my professional life. You’re not supported but being expected to deliver by your “home” team. Meanwhile the away team has no obligation to help you. Can spin their thumbs and give you three day turn arounds on code reviews only to move the goal posts and/or hold you to some unattainable standard. For example “all integration tests need to pass” while the integration tests haven’t even STARTED correctly in over a year. Not to mention the increased visibility and communication that comes with now effectively having 2+ managers you have to report to and keep in the loop. FUCK. THE. AWAY. TEAM. MODEL. I’ve lived through it and it seriously burned me out harder than anything else in my life.

hyperliner 3y ago
“Meanwhile the away team has no obligation to help you.” I think you meant the host team.
deleted 3y ago
[deleted]

beastcoast 3y ago

Away teaming is definitely a last resort at Amazon, and has all the pitfalls mentioned here - much better to do a budgetary transfer so the host team is properly resourced to make the changes, assuming the feature doesn’t make it into the host teams operational plan.

menzoic 3y ago

This is very common at Uber and Airbnb

fortylove 3y ago

As someone who did away team work at Amazon, and also hosted away teams: the model sucks in most ways and was unpleasant for everyone. It got the job done in extreme situations, but it was mostly a result of subpar managers taking the term "fungible" too seriously. Source: was sr engineer at Amazon

flaminHotSpeedo 3y ago

I can't think of a way to concisely convey my response without being crass, so here it is: The author of this piece is an idiot who has drunk the Kool aid and has very clearly never performed away team work. I have personally wasted a year of my time on away team work, have supported several away teams, and most importantly: operated systems/built on top of stuff built by away teams. All of this at Amazon, specifically. Having personally experienced every engineering facet of away teams, I feel as qualified as anyone to say this: away teams are a stupid concept that hurts everyone except management. You spend way more time to deliver a product you don't really understand and will have a hard time spinning on promos, which will almost certainly be a nightmare to maintain and operate, all because your employer is too cheap to allocate headcount for surprises and your team took a dependency on another team years ago so they have you by the balls.

dbish 3y ago
The author likely didn’t participate in the away team model as according to LI he was a solutions architect during his time at amazon, so kind of odd to write about it now.

aledalgrande 3y ago

For all the ex Amazonians saying the model suck, maybe it was the culture that sucked? Where I work we have loosely the same model, doesn't even have a name, but the component owners are always very helpful.

mkl95 3y ago

Take my comment with a grain of salt because I have never worked for Amazon and I don't have any relationship with the company. Since these are Amazon employees, I would expect them to experience the downsides of being a consultant, such as alienation, and the downsides of belonging to a product organization, such as extreme pressure. > As organizations grow, engineers’ code intertwines with other engineers’ work, making it difficult for teams to complete their features independently. The current consensus in the industry is that it should be the opposite. To grow in a healthy way, teams should be as decoupled as possible. > The away team model is a powerful and flexible mechanism for removing dependencies between teams. However, it should not be the first option for developing necessary changes to a system dependency. Did I miss something? You introduced this team model because there was a dependency, but you never removed it. This post is full of contradictions.

jq-r 3y ago
> Did I miss something? You introduced this team model because there was a dependency, but you never removed it. This post is full of contradictions. It's just another article by a manager expounding the virtues of a managerial fad to score points in his org. Sounds great to his peers, sounds like a nightmare to ICs.
- dbish 3y ago
  Not even a manager, his LI says solutions architect which is neither an engineer or manager so he would not have first hand experience doing it for actual software dev.
  - jq-r 3y ago
    Right, those "solutions architects" like "Techical Account Managers" which act like moats against receiving any kind of good support. Those guys are just terrible IMO unfortunately.

kthejoker2 3y ago

I see a lot of comments saying they "wasted" months or years on away team work, it was hard to quantify for promos, the KT itself from the work went to waste etc. Totally makes sense for career- and purpose-driven employees. So one management takeaway is this type of work seems ideal for highly competent outsourcing. Offer a bonus for early delivery, and hire a small army of cheap out of school eager beaver BAs to just go around frantically documenting all the tech debt being incurred.

dbish 3y ago
Out of school BAs seem like the the best way to waste significant dev time explaining to people who don’t understand the tech or technical documentation what to do.

8note 3y ago

Something not captured here is that the host team tends to have so much turnover that there's little difference between the host team's engineers working on a project and an away team's.

NBJack 3y ago

Have worked for the A to Z company. Away teams as a concept exists because teams that are overworked and understaffed can't do everything asked of them (including things you can't say no to, like S-Team goals). The Search team was especially famous/infamous for using this, due to the sheer volume of asks (i.e. can we do a twister on fabric types for clothing and see if it improves dress sales?). Your team makes the ask, search says "sure we'll support you as you make the changes to it," and the asking team provides the brave souls diving into a strange codebase. Does it work? Technically, yes. Is it sustainable for the engineers on both sides? Not in my eexperience. >> An away team is a self-sufficient engineering team that works on code owned by another team (the host team) to deliver features they are dependent on. This is fundamentally wrong. The point of the away team is to get support from the host team as they work. It is not self sufficient by nature. Otherwise, this is just a team randomly contributing to a codebase you own.

canttestthis 3y ago
> brave souls diving into a strange codebase As someone who made a career out of being loaned out to other teams as an 'away resource' at Amazon, I've struggled to find a similar level of mental stimulation after leaving for other Big 4 companies. Being a loaner engineer meant something new to work on every few months, no maintenance or oncall burden, making lots of new connections, and just generally there being always something interesting/intense to do.
- NBJack 3y ago
  I won't dispute your level of enjoyment from it, but I'd have to ask: did your fellow engineers feel the same way about it? What was your time commitment like to deliver? Also, if you're looking for a thrill, I would highly recommend going after cloud-based infra providers. There's never a shortage of unique problems there.
  - canttestthis 3y ago
    Time commitments were generally aggressive but reasonable with a 10-20% buffer to account for ramp up (realistically closer to 10 than 20). The reasonable-ness of most things eng-culture related at Amazon is probably a function of the competence of your management chain. As for my fellow engineers- I don’t know anyone else who did this regularly so I’m not sure