5 ms·

Very curious how this works in practice, because having worked as a non-software eng role (support, SRE, tech writer) for "tiger teams" and other out-of-band ta

Very curious how this works in practice, because having worked as a non-software eng role (support, SRE, tech writer) for "tiger teams" and other out-of-band task forces with limited scope, any of those roles associated with away/host teams become the metaphorical red shirts[1], buried alive under the actual handoff/maintenance work that the engs are too valuable to the org to touch. 1: https://en.wikipedia.org/wiki/Redshirt_(stock_character) https://en.wikipedia.org/wiki/Redshirt_(stock_character)

cjhowarddev 3y ago
I'm at Amazon now and my team is often the target of away team work as our system is seen as "critical" to other business unit growth. it's a farce. Interesting idea in theory, terrible in practice. The away team incentive is "launch as fast as possible at all costs" and that goes about as well as you'd expect. We constantly deal with shit code and loose rules. Given that many away teams work in different time zones AND Amazon gives them carte blanche to get their work done, the host team can't manage them effectively. So yeah. On one hand management is like "push back! Make them follow your standards!" while also saying "Don't dedicate time to away teams, you have your own deliverables." It's a major drain on the home team and can cause a lot of unnecessary stress. These opinions are my own, obviously, lol.
- deleted 3y ago
  [deleted]
- billllll 3y ago
  I think it really depends on the leadership. The worse case scenario is when the scenario you given: the "pet project," where a team goes nuts on a codebase they aren't maintaining, in which case it's effectively out-sourcing maintenance and tech debt. Ideally, the home team leadership has a lot of leverage to push back on work by the away team, which means away team work is slow but at least it gets done.
- angarg12 3y ago
  Fellow amazonian here. I have been on both sides of away teams and I agree that it sucks for everyone involved. You explains how it looks as the host teams. It also sucks for the guest team. * You have to work with an, at best, passive team, at worst, actively hostile team. * You need to work against very tight deadlines and a lot of pressure in a codebase that you know nothing about, and need to ramp up in quickly. * You spent weeks (months?) developing expertise in a system you will probably never work again. It is a terrible idea, but to be honest I can't think of another way to deal with the problem that it is trying to solve (mismatch between agendas of different teams).
  - SoftTalker 3y ago
    Bubble the issue up the hierarchy until someone with authority over both teams can tell them what is more important?
    - angarg12 3y ago
      We resort to away teaming when what you describe doesn't work, usually by telling both teams that both goals are important, and they need to figure out how to deliver. Since away teaming is an accepted technique, it's usually considered as a way to solve these issues. I guess we would need to completely give away the concept, and just accept that some thing won't get done unless they are important enough to keep two distant teams aligned (in other words, they are important enough for someone high in the hierarchy).
  - hoodedmongoose 3y ago
    The open source model? If the away team can't get the home team to fulfill their requirements, they fork home teams code and take full ownership of it for their purposes. If they want to take updates from home team, they do so but have to handle the merging. If they want to, they can try to get home team to take their changes, but that's at home teams full discretion.
    - Others 3y ago
      How does that work when the thing you want to change is a complex micro-service with a public API? Do you want to spin up your own version of that service and own it forever? Even if you’re okay with that, you’ll not have the clients the older version has Open-source model works for libraries, or small changes in service code (where the maintenance burden is trivial). But it doesn’t work for complex services with high maintenance burdens
- bluedevilzn 3y ago
  Having worked at both amazon and Google, IMO Google does this significantly better by having OWNERS, away team member plus readability approver. This comes at the cost of much slower progress.
- fluoride1002 3y ago
  Before Away Teams became a thing there we had people "loaned" to teams to update our code for team projects. I remember during one of the code reviews I did for someone "loaned" to us involved him commenting out validation checks in critical systems that had legal and financial ramifications. When I asked him why he commented it out he said "Oh, my code didn't work with those checks in there so I removed them." I get that the code is complex and he doesn't have the full domain knowledge, but he didn't even ask why they were there and why certain use cases were not allowed for that part of the code.
  - ramraj07 3y ago
    This model only works if the away teams are the most competent people you have. Generally the “loaning” business is a cowardly way for teams to trade the worst engineers you have (bonus point if that engineer is someone who everyone praises but is secretly shit and only the manager knows that reality).
    - 8note 3y ago
      I've only gotten completely new hires as loans, where you also have to train them up to being useful
  - hiyer 3y ago
    I don't see how having Away Teams fixes this issue. The Away Team member is still free to write shitty code (or un-write good code, as in this case), and it is still the responsibility of the home team to catch that in code reviews.
grensley 3y ago
This has been my experience as well (as a dev). Curious if there's others in the same sort of situation, and I'd love to hear more thoughts. My general tips are: - Use as much as you can of what they already have. - Talk to them, but don't waste their time. - Follow their patterns. - Understand at the end of the day its their domain. - Choose your advice carefully. - Try to throw a little extra gratitude their way. Organizational visibility, code reviews, documentation, mentorship, whatever feels appropriate and not over-stepping. - Put extra effort into handoff documentation.
x86x87 3y ago
[dead]
jrd259 3y ago
I'm at Amazon and am the senior developer on my team. For the last few months an "away" team has been adding a feature to a service I own. It's only n of 1, but the process worked well. I did the high level design and ensured alignment on requirements. The away team did an excellent job of learning our code base. The new feature required a tricky refactoring of some of our oldest and most rotten code, which I did myself as it seemed too hard for an away team. I saw no sign at all of the away team compromising on quality to meet deadlines.
- gregw2 3y ago
  I am using a similar concept to scale my team's impact at a F100 company; making processes and ways for other teams to contribute code that is otherwise bottlenecked to/by us but ends up getting owned by us. It's worked for the last year or two semi-effectively, but I am loving this discussion for reminding me of the downsides/risks.

cjhowarddev 3y ago

I'm at Amazon now and my team is often the target of away team work as our system is seen as "critical" to other business unit growth. it's a farce. Interesting idea in theory, terrible in practice. The away team incentive is "launch as fast as possible at all costs" and that goes about as well as you'd expect. We constantly deal with shit code and loose rules. Given that many away teams work in different time zones AND Amazon gives them carte blanche to get their work done, the host team can't manage them effectively. So yeah. On one hand management is like "push back! Make them follow your standards!" while also saying "Don't dedicate time to away teams, you have your own deliverables." It's a major drain on the home team and can cause a lot of unnecessary stress. These opinions are my own, obviously, lol.

deleted 3y ago
[deleted]
billllll 3y ago
I think it really depends on the leadership. The worse case scenario is when the scenario you given: the "pet project," where a team goes nuts on a codebase they aren't maintaining, in which case it's effectively out-sourcing maintenance and tech debt. Ideally, the home team leadership has a lot of leverage to push back on work by the away team, which means away team work is slow but at least it gets done.
angarg12 3y ago
Fellow amazonian here. I have been on both sides of away teams and I agree that it sucks for everyone involved. You explains how it looks as the host teams. It also sucks for the guest team. * You have to work with an, at best, passive team, at worst, actively hostile team. * You need to work against very tight deadlines and a lot of pressure in a codebase that you know nothing about, and need to ramp up in quickly. * You spent weeks (months?) developing expertise in a system you will probably never work again. It is a terrible idea, but to be honest I can't think of another way to deal with the problem that it is trying to solve (mismatch between agendas of different teams).
- SoftTalker 3y ago
  Bubble the issue up the hierarchy until someone with authority over both teams can tell them what is more important?
  - angarg12 3y ago
    We resort to away teaming when what you describe doesn't work, usually by telling both teams that both goals are important, and they need to figure out how to deliver. Since away teaming is an accepted technique, it's usually considered as a way to solve these issues. I guess we would need to completely give away the concept, and just accept that some thing won't get done unless they are important enough to keep two distant teams aligned (in other words, they are important enough for someone high in the hierarchy).
- hoodedmongoose 3y ago
  The open source model? If the away team can't get the home team to fulfill their requirements, they fork home teams code and take full ownership of it for their purposes. If they want to take updates from home team, they do so but have to handle the merging. If they want to, they can try to get home team to take their changes, but that's at home teams full discretion.
  - Others 3y ago
    How does that work when the thing you want to change is a complex micro-service with a public API? Do you want to spin up your own version of that service and own it forever? Even if you’re okay with that, you’ll not have the clients the older version has Open-source model works for libraries, or small changes in service code (where the maintenance burden is trivial). But it doesn’t work for complex services with high maintenance burdens
bluedevilzn 3y ago
Having worked at both amazon and Google, IMO Google does this significantly better by having OWNERS, away team member plus readability approver. This comes at the cost of much slower progress.
fluoride1002 3y ago
Before Away Teams became a thing there we had people "loaned" to teams to update our code for team projects. I remember during one of the code reviews I did for someone "loaned" to us involved him commenting out validation checks in critical systems that had legal and financial ramifications. When I asked him why he commented it out he said "Oh, my code didn't work with those checks in there so I removed them." I get that the code is complex and he doesn't have the full domain knowledge, but he didn't even ask why they were there and why certain use cases were not allowed for that part of the code.
- ramraj07 3y ago
  This model only works if the away teams are the most competent people you have. Generally the “loaning” business is a cowardly way for teams to trade the worst engineers you have (bonus point if that engineer is someone who everyone praises but is secretly shit and only the manager knows that reality).
  - 8note 3y ago
    I've only gotten completely new hires as loans, where you also have to train them up to being useful
- hiyer 3y ago
  I don't see how having Away Teams fixes this issue. The Away Team member is still free to write shitty code (or un-write good code, as in this case), and it is still the responsibility of the home team to catch that in code reviews.

grensley 3y ago

This has been my experience as well (as a dev). Curious if there's others in the same sort of situation, and I'd love to hear more thoughts. My general tips are: - Use as much as you can of what they already have. - Talk to them, but don't waste their time. - Follow their patterns. - Understand at the end of the day its their domain. - Choose your advice carefully. - Try to throw a little extra gratitude their way. Organizational visibility, code reviews, documentation, mentorship, whatever feels appropriate and not over-stepping. - Put extra effort into handoff documentation.

x86x87 3y ago

[dead]

jrd259 3y ago

I'm at Amazon and am the senior developer on my team. For the last few months an "away" team has been adding a feature to a service I own. It's only n of 1, but the process worked well. I did the high level design and ensured alignment on requirements. The away team did an excellent job of learning our code base. The new feature required a tricky refactoring of some of our oldest and most rotten code, which I did myself as it seemed too hard for an away team. I saw no sign at all of the away team compromising on quality to meet deadlines.

gregw2 3y ago
I am using a similar concept to scale my team's impact at a F100 company; making processes and ways for other teams to contribute code that is otherwise bottlenecked to/by us but ends up getting owned by us. It's worked for the last year or two semi-effectively, but I am loving this discussion for reminding me of the downsides/risks.