8 ms·

Regarding Conway's Law: We have found that by changing our software/system architecture we have also inadvertently changed our organisation structure. - Inver

Regarding Conway's Law: We have found that by changing our software/system architecture we have also inadvertently changed our organisation structure. - Inverse Conway Law or just Roy's Law ;-) Before we had four cross functional teams, working on a single application, everyone felt responsible for it, worked overtime to fix bugs etc, we had good communication between the teams. But after we switched to microservices the teams became responsible for just a part of the system, their microservice(s). Whenever we had an outage, one team was left to fix it, the others just went home. They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together, some things got way worse in the organisation, all sparked by a 'simple' architectural change, moving to microservices.

BerislavLopac 8y ago
> They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together, some things got way worse in the organisation, all sparked by a 'simple' architectural change, moving to microservices. Honestly, this sounds like an improvement.
- badfrog 8y ago
  How is people not talking to each other an improvement?
  - jcadam 8y ago
    Well, from a systems standpoint, it means a given "problem" is isolated to a single service, and is therefore not impacting the other services or interrupting the work of the other teams. But culturally, it would be nice if people helped each other out from time to time...
    - marcosdumay 8y ago
      That's only true if any given problem is isolated on a single service, because while you are making intra-service problems much cheaper and faster to fix, you are also making inter-service problems almost impossible to fix.
      - jcadam 8y ago
        Good point. I've seen situations where nobody would take ownership of a bug and fix it - you just had teams pointing fingers at each other...
        BerislavLopac 8y ago
        This is precisely a symptom of unclear ownership responsibilities.
        AnimalMuppet 8y ago
        No, it can be an issue of it being unclear in which component the bug actually is.
        BerislavLopac 8y ago
        Which is a clear sign that your components are not sufficiently separate.
        AnimalMuppet 8y ago
        Um, no. The real world is not that simple. It would be nice if it were. Or perhaps your statement is correct, but in the real world components are never sufficiently separate. So, while your statement may be correct by definition, it is not useful.
        james_s_tayler 8y ago
        I think you really nailed it with this one. I can imagine there being a normal distribution of 'separateness' of software and the rare top tail-end of the distribution gets it perfectly right, most are in the middle somewhere between 'service oriented architecture' and 'ball of mud' and some are just at plain ball of mud.
        BerislavLopac 8y ago
  - vibrato 8y ago
    Dunbar's number. As humans, we only have room for so many relationships.
  - BerislavLopac 8y ago
    Oh, it is definitely a problem from a social/cultural standpoint. But from the point of software architecture (and, therefore, development organisation), too much (or even any) communication between teams working on discreet, separate units can become detrimental. It is perfectly fine that the people communicate, and even helping each other to improve tech skill should be encouraged; however, decisions about their respective products should be contained within each team, with clearly defined interfaces and usage documentation.
    - badfrog 8y ago
      > decisions about their respective products should be contained within each team, with clearly defined interfaces and usage documentation. In order to make those decisions and define the interfaces, you need to know a lot about how your software is going to be used. That will be much easier if you have good communication with the other teams and understand their goals and motivations.
      - BerislavLopac 8y ago
        I disagree. In my experience, direct coordination on interfaces tends to create unnecessary special cases (hey, can you add this field to your API, just for us?) which add complexity and make maintenance more difficult down the line. The main advantage of distributed system, and particularly microservices, is the ability to have each system completely independent: individual components can be written in different languages, running on different platforms and use completely independent internal components. Basically, it is just like using an external library, component or service: the authors provide documentation and interfaces, and you should be able to expect it to behave as advertised.
        badfrog 8y ago
        > In my experience, direct coordination on interfaces tends to create unnecessary special cases (hey, can you add this field to your API, just for us?) which add complexity and make maintenance more difficult down the line. If you just implement all requests directly, you're for sure going to end up with a horrible interface. You should approach API design the same way that UX/PM approaches UI and feature design: take the time to understand _why_ your partner teams/engineers are requesting certain changes and figure out the right interface to address their problems.
  - tonyedgecombe 8y ago
    Presumably if you don't like other people then it's an improvement. At a guess I'd say that covers about quarter of our industry.
    - badfrog 8y ago
      Probably we can agree that a company will be more productive if the engineers are learning from each other and generating ideas together? So if you've got a team of engineers who don't like working with people, it's probably in the company's best interest to set up a structure that explicitly encourages more communication.
- AnimalMuppet 8y ago
  This depends very much on the team size. Having a team of 20 people communicate in the way you describe below is insane overkill. Having a team of 100 do so may save everyones' sanity.
Shivetya 8y ago
I have seen new systems/software implemented just to isolate/remove parts of an organization. Worse I have seen it done when the existing system/software was just fine.
mlthoughts2018 8y ago
> “Before we had four cross functional teams, working on a single application, everyone felt responsible for it, worked overtime to fix bugs etc, we had good communication between the teams.” This actually sounds very dysfunctional, but with the type of positive PR spin that product / executive management wants, basically anyone who believes “cross-functional” is anything more than a buzzword. Would love to know what the engineers thought about working in that environment (which sounds like a monolithic, too-many-cooks, zero specialization situation likely negatively affecting career growth & routine skill building).
- badfrog 8y ago
  Aside from the working overtime part, what sounds dysfunctional?
  - mlthoughts2018 8y ago
    Types of bugs or failures are not separated into specialized areas, rather it’s one “cross-functional” unit. It’s like building a monolith class in software instead of following basic principles like Single Responsibility Principle and organizing workflows according to independent specializations. This part is often much worse than the overtime part, because it means you’re expected to sublimate your personal career goals in favor of whatever arbitrary thing needs done for the sake of the cross-functional unit. When I hear someone describe communication between cross-functional team members as “good” or “effective,” then I know it’s a big lie, and most probably it’s a disaster of overcommunication where product managers or non-tech leadership have a stranglehold on decision making when really engineering should be autonomous in these cases exactly according to independent specialization.
badfrog 8y ago
> They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together Was that accompanied by any growth in company size? I've found that this happens when a group grows past about 15 people even if the structure doesn't change.
Bizarro 8y ago
That reminds me of a place I used to work at, where initially we had DBAs embedded in the teams. They switched that and the DBAs were all grouped together and all hell broke loose. They were always have meetings, throwing out emails about they were dictating this and that, and had very little direct communications with the teams they were supposed to be supporting. I ended up leaving during the peak of all of this, but in an exit-interview, a director asked me about the problems this was causing.
ComodoHacker 8y ago
Could less overtimes be considered a positive outcome?
tempodox 8y ago
Moving to microservices is anything but a simple change. Your experience is one example of why microservices are not automatically a good idea. Normally, the advice is that they might be a good fit if you have isolated teams to begin with, and for different reasons.
scarejunba 8y ago
Haha, this is so similar to what happened where I work in SF that I feel you’re a coworker of mine.
organsnyder 8y ago
This outcome could be considered a feature of microservices: by abstracting the functionality into more tightly-contained units, failures are more isolated. Sounds like the organization needs to do other things to keep people from getting siloed, though that gets increasingly difficult at scale. Well-defined SLAs (along with monitoring and reporting of those SLAs) are also necessary so that microservice failures can be understood in the right context.
- meowface 8y ago
  This YC talk from Amazon's CTO on how they grew to a microservice model and team structure was really interesting: https://www.youtube.com/watch?v=adtuntQ8rh4 https://www.youtube.com/watch?v=adtuntQ8rh4
zemo 8y ago
it's almost like Bezos intentionally issued the microservices mandate at Amazon to discourage Amazon's engineers from unionizing.

BerislavLopac 8y ago

> They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together, some things got way worse in the organisation, all sparked by a 'simple' architectural change, moving to microservices. Honestly, this sounds like an improvement.

badfrog 8y ago
How is people not talking to each other an improvement?
- jcadam 8y ago
  Well, from a systems standpoint, it means a given "problem" is isolated to a single service, and is therefore not impacting the other services or interrupting the work of the other teams. But culturally, it would be nice if people helped each other out from time to time...
  - marcosdumay 8y ago
    That's only true if any given problem is isolated on a single service, because while you are making intra-service problems much cheaper and faster to fix, you are also making inter-service problems almost impossible to fix.
    - jcadam 8y ago
      Good point. I've seen situations where nobody would take ownership of a bug and fix it - you just had teams pointing fingers at each other...
      - BerislavLopac 8y ago
        This is precisely a symptom of unclear ownership responsibilities.
        AnimalMuppet 8y ago
        No, it can be an issue of it being unclear in which component the bug actually is.
        BerislavLopac 8y ago
        Which is a clear sign that your components are not sufficiently separate.
        AnimalMuppet 8y ago
        Um, no. The real world is not that simple. It would be nice if it were. Or perhaps your statement is correct, but in the real world components are never sufficiently separate. So, while your statement may be correct by definition, it is not useful.
        james_s_tayler 8y ago
        I think you really nailed it with this one. I can imagine there being a normal distribution of 'separateness' of software and the rare top tail-end of the distribution gets it perfectly right, most are in the middle somewhere between 'service oriented architecture' and 'ball of mud' and some are just at plain ball of mud.
        BerislavLopac 8y ago
- vibrato 8y ago
  Dunbar's number. As humans, we only have room for so many relationships.
- BerislavLopac 8y ago
  Oh, it is definitely a problem from a social/cultural standpoint. But from the point of software architecture (and, therefore, development organisation), too much (or even any) communication between teams working on discreet, separate units can become detrimental. It is perfectly fine that the people communicate, and even helping each other to improve tech skill should be encouraged; however, decisions about their respective products should be contained within each team, with clearly defined interfaces and usage documentation.
  - badfrog 8y ago
    > decisions about their respective products should be contained within each team, with clearly defined interfaces and usage documentation. In order to make those decisions and define the interfaces, you need to know a lot about how your software is going to be used. That will be much easier if you have good communication with the other teams and understand their goals and motivations.
    - BerislavLopac 8y ago
      I disagree. In my experience, direct coordination on interfaces tends to create unnecessary special cases (hey, can you add this field to your API, just for us?) which add complexity and make maintenance more difficult down the line. The main advantage of distributed system, and particularly microservices, is the ability to have each system completely independent: individual components can be written in different languages, running on different platforms and use completely independent internal components. Basically, it is just like using an external library, component or service: the authors provide documentation and interfaces, and you should be able to expect it to behave as advertised.
      - badfrog 8y ago
        > In my experience, direct coordination on interfaces tends to create unnecessary special cases (hey, can you add this field to your API, just for us?) which add complexity and make maintenance more difficult down the line. If you just implement all requests directly, you're for sure going to end up with a horrible interface. You should approach API design the same way that UX/PM approaches UI and feature design: take the time to understand _why_ your partner teams/engineers are requesting certain changes and figure out the right interface to address their problems.
- tonyedgecombe 8y ago
  Presumably if you don't like other people then it's an improvement. At a guess I'd say that covers about quarter of our industry.
  - badfrog 8y ago
    Probably we can agree that a company will be more productive if the engineers are learning from each other and generating ideas together? So if you've got a team of engineers who don't like working with people, it's probably in the company's best interest to set up a structure that explicitly encourages more communication.
AnimalMuppet 8y ago
This depends very much on the team size. Having a team of 20 people communicate in the way you describe below is insane overkill. Having a team of 100 do so may save everyones' sanity.

Shivetya 8y ago

I have seen new systems/software implemented just to isolate/remove parts of an organization. Worse I have seen it done when the existing system/software was just fine.

mlthoughts2018 8y ago

> “Before we had four cross functional teams, working on a single application, everyone felt responsible for it, worked overtime to fix bugs etc, we had good communication between the teams.” This actually sounds very dysfunctional, but with the type of positive PR spin that product / executive management wants, basically anyone who believes “cross-functional” is anything more than a buzzword. Would love to know what the engineers thought about working in that environment (which sounds like a monolithic, too-many-cooks, zero specialization situation likely negatively affecting career growth & routine skill building).

badfrog 8y ago
Aside from the working overtime part, what sounds dysfunctional?
- mlthoughts2018 8y ago
  Types of bugs or failures are not separated into specialized areas, rather it’s one “cross-functional” unit. It’s like building a monolith class in software instead of following basic principles like Single Responsibility Principle and organizing workflows according to independent specializations. This part is often much worse than the overtime part, because it means you’re expected to sublimate your personal career goals in favor of whatever arbitrary thing needs done for the sake of the cross-functional unit. When I hear someone describe communication between cross-functional team members as “good” or “effective,” then I know it’s a big lie, and most probably it’s a disaster of overcommunication where product managers or non-tech leadership have a stranglehold on decision making when really engineering should be autonomous in these cases exactly according to independent specialization.

badfrog 8y ago

> They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together Was that accompanied by any growth in company size? I've found that this happens when a group grows past about 15 people even if the structure doesn't change.

Bizarro 8y ago

That reminds me of a place I used to work at, where initially we had DBAs embedded in the teams. They switched that and the DBAs were all grouped together and all hell broke loose. They were always have meetings, throwing out emails about they were dictating this and that, and had very little direct communications with the teams they were supposed to be supporting. I ended up leaving during the peak of all of this, but in an exit-interview, a director asked me about the problems this was causing.

ComodoHacker 8y ago

Could less overtimes be considered a positive outcome?

tempodox 8y ago

Moving to microservices is anything but a simple change. Your experience is one example of why microservices are not automatically a good idea. Normally, the advice is that they might be a good fit if you have isolated teams to begin with, and for different reasons.

scarejunba 8y ago

Haha, this is so similar to what happened where I work in SF that I feel you’re a coworker of mine.

organsnyder 8y ago

This outcome could be considered a feature of microservices: by abstracting the functionality into more tightly-contained units, failures are more isolated. Sounds like the organization needs to do other things to keep people from getting siloed, though that gets increasingly difficult at scale. Well-defined SLAs (along with monitoring and reporting of those SLAs) are also necessary so that microservice failures can be understood in the right context.

meowface 8y ago
This YC talk from Amazon's CTO on how they grew to a microservice model and team structure was really interesting: https://www.youtube.com/watch?v=adtuntQ8rh4 https://www.youtube.com/watch?v=adtuntQ8rh4

zemo 8y ago

it's almost like Bezos intentionally issued the microservices mandate at Amazon to discourage Amazon's engineers from unionizing.