6 ms·

At scale software stops being a technology problem and becomes a people problem. And groups of humans don't scale. We naturally organize into hierarchies of sm

At scale software stops being a technology problem and becomes a people problem. And groups of humans don't scale. We naturally organize into hierarchies of small subgroups with limited interaction boundaries. Each subgroup will adopt slightly different methodologies, have different tooling preferences, will have integration and communication overhead with others, and so on. This can not be preventend, only mitigated. Which is perfectly fine. The important step is recognizing that software development is really not special [1], and that the limiting factor is the human element. Then you can build your organizational structures around these restrictions. [1] https://news.ycombinator.com/item?id=26713139 https://news.ycombinator.com/item?id=26713139

oblio 6y ago
This is part of the reason microservices are becoming reasonably popular. They facilitate scaling people. If you think about it one of the biggest projects we have, the military, scales through assembling near-identical pieces structurally (but not functionally, an artillery unit has a totally different role to an infantry unit).
- heisenbit 6y ago
  Microservices may help scaling but tend to come with a fixed amount of overhead which makes them more effort at a very low and low scale. They are also perfect as long as you got the overall architecture right and do not have to make cross-cutting changes because this turns a coordinating people in a team to coordinating people across teams problem.
  - oblio 6y ago
    Well, with the ideal technical leadership, you start off with a monolith and clearly delineate boundaries inside it. Then as the project grows chunks just get carved out. Though real life is much messier.
  - nly 6y ago
    Something that only works for huge sizes isn't scalable by definition. Scalable means it can scale up or down
    - kaliszad 6y ago
      Great point! Actually "scalable" in any direction is meaningless until you provide the scale, performance, reliability, cost and time requirements you have to consider while scaling. A single server for a typical web workload is just fine, if you don't need to be up more than ~ 99,9% of time, if you don't have more than 10s of thousands of users. Considering the availability, you could e.g. reboot into a smaller or larger VM. (I know, this is crazy to write on HN but a single virtual server is plenty reliable for most things if you are able to do a very fast recovery from backup to limit the extent of a disaster.) :-) If you don't consider "serverless", it really doesn't get much better than a single server considering cost. If you want more reliability/ availability and more performance, you probably have to take other approaches. E.g. more servers, services like S3/ lambda, maybe a mainframe (cluster) in certain situations etc.
    - ncallaway 6y ago
      I disagree to an extent. Very few solutions are scalable across an entire range of a problem space. Something being scalable, to me, means that it can cover a _broad_ range of the problem space, but doesn't imply it covers 100% of it. If we imagine that the "problem space" is managing software projects from 1 person up through 1,000,000 people and you told me that there was a solution that worked for the range of 50-1,000,000 people I would describe it as scalable, even if I wouldn't recommend it to a 5 person team.
  - josephg 6y ago
    Yes! Ironically, microservice architectures replicate the same set of advantages and challenges you have scaling teams. The abstractions (team / process boundaries) become much more rigid. But in exchange, within a team / service, change becomes easier because there's less to understand and navigate. How well the whole system functions depends deeply on how well those abstractions map the solution space. When you're doing something routine (eg most web development), lots of people have figured out reasonable ways of breaking the problem down already. So it works pretty well and you don't carry much risk that the abstractions you pick from the start will hurt you down the road. But when you're doing something more creative and exploratory - like a marketplace for VR, or making a new web browser or making something like Uber - well, then you take on abstraction risk. You will probably make mistakes in how you initially break up your teams and services. Executing well depends on your organizational ability to learn and refactor.
  - vsareto 6y ago
    And predicting cross-cutting concerns is at least as difficult as predicting all of your business changes, which is at least as difficult as predicting the future.
- _the_inflator 6y ago
  I see it differently. In my experience with 200+ devs, Microservices itself is a buzzword to a certain extend and allows for different team to avoid talking to each other: they build their "own" services, which are copies which leads to bloat.
  - pbronez 6y ago
    Yeah that’s not gonna work. I think you have to align incentives and the contractual level. For example, don’t pay by the hour, Pay for an API. Mandate that it’s documented in a machine-readable way. Require it to integrate with your centralized authentication and logging services. Mandate thorough, human-friendly documentation Programmatically validate as much of this as possible, add periodic human review (ie surveys, interviews). The goal is to treat each team as a vendor for their service. You hold them accountable for the product and pay them for outcomes.
- roguas 6y ago
  This, but typically a team of 20 has 1 monolith, afterwards team of 20 has 8 microservices. Equivalent of one battle squad now has to learn a bit of tanking, learn a bit of flying etc.
bob33212 6y ago
The sub groups are going to be somewhat homogeneous internally. So you will end up with a group who thinks leetcode profeciency is important in hiring and another group who thinks QA experience is more important and another group who thinks diversity is the most important. All those groups think they are right and better than the other groups and they don't work well together.
p0nce 6y ago
And who manages to do that?
deckard1 6y ago
This is all outlined by Fred Brooks in 1975. I've beat this dead horse before, but I suppose I'll beat it again. In the late '90s and early aughts everyone read Mythical Man-Month. There is even a law attributed to him, Brooks' law [1]. It really feels like no one internalized what he was actually saying. I suppose in a world of trends and fads, timeless advice is hard to recognize. > [...] the increased communication overhead will consume an ever-increasing quantity of the calendar time available. When n people have to communicate among themselves, as n increases, their output decreases and when it becomes negative the project is delayed further with every person added. Microservices, of course, exasperate this by actually increasing the communication required to make software. [1] https://en.wikipedia.org/wiki/The_Mythical_Man-Month https://en.wikipedia.org/wiki/The_Mythical_Man-Month
- eloff 6y ago
  > Microservices, of course, exasperate this by actually increasing the communication required to make software. I HATE microservices. I've never seen them used appropriately, although I imagine it can be done. But I think if you increase human communication by using microservices you're defeating the raison d'etre of microservices. That's not the fault of microservices - it's the fault of the humans using them badly or where they shouldn't. Story time. I once contracted to a fortune 500 doing a $100M total rewrite with literally 100 microservices all in their own git repo. It was such a disaster I contemplated shorting their stock. I don't know how the story ended, but I feel confident it didn't end well.
  - TheCoelacanth 6y ago
    Microservices can be okay, but they are greatly overused. They are a solution for very large organizations with a ton of teams. For 100 micro-services to make any sense at all, you need to have at least 200-300 developers. Total rewrites, on the other hand, are never anything other than a complete and unmitigated disaster. The only way that you can possibly make a $100M rewrite successful is to do it Ship of Theseus style, where the rewritten code is incorporated into the working and actively-used production software long before the full rewrite is complete. Even then, the full rewrite usually won't complete, so you need to make sure that project is still useful even if you only do a partial rewrite.
qznc 6y ago
One question which is open for me: Should groups be stable? Scrum and other methodologies assume stable teams. It takes a few sprints for a team to assign somewhat accurate story points to stories. It takes time for people to find their roles in a team. Whenever someone joins or leaves a team, it takes time to find a new equilibrium. So if you aim for peak performance teams, it takes months of stability to reach that peak. You better not change it then or it will take months again. In contrast, most organization have a constant need to reorganize to adapt to new business models, customers, product portfolio, etc. For consulting gigs which takes less than a few months, it makes no sense to aim for peak performance and stable teams. Instead you should aim for smooth transitions. In such an environment, more formal methodologies make sense because good documentation, unified code styles, and consistent processes make transitions more smooth. Of course, this isn't a black and white decision but more of a scale from stability to flexibility. There are small lifestyle companies which are very close to the stability extreme (the Sqlite team maybe?). Valve with its flat hierarchy might be an example for the flexibility extreme.

oblio 6y ago

This is part of the reason microservices are becoming reasonably popular. They facilitate scaling people. If you think about it one of the biggest projects we have, the military, scales through assembling near-identical pieces structurally (but not functionally, an artillery unit has a totally different role to an infantry unit).

heisenbit 6y ago
Microservices may help scaling but tend to come with a fixed amount of overhead which makes them more effort at a very low and low scale. They are also perfect as long as you got the overall architecture right and do not have to make cross-cutting changes because this turns a coordinating people in a team to coordinating people across teams problem.
- oblio 6y ago
  Well, with the ideal technical leadership, you start off with a monolith and clearly delineate boundaries inside it. Then as the project grows chunks just get carved out. Though real life is much messier.
- nly 6y ago
  Something that only works for huge sizes isn't scalable by definition. Scalable means it can scale up or down
  - kaliszad 6y ago
    Great point! Actually "scalable" in any direction is meaningless until you provide the scale, performance, reliability, cost and time requirements you have to consider while scaling. A single server for a typical web workload is just fine, if you don't need to be up more than ~ 99,9% of time, if you don't have more than 10s of thousands of users. Considering the availability, you could e.g. reboot into a smaller or larger VM. (I know, this is crazy to write on HN but a single virtual server is plenty reliable for most things if you are able to do a very fast recovery from backup to limit the extent of a disaster.) :-) If you don't consider "serverless", it really doesn't get much better than a single server considering cost. If you want more reliability/ availability and more performance, you probably have to take other approaches. E.g. more servers, services like S3/ lambda, maybe a mainframe (cluster) in certain situations etc.
  - ncallaway 6y ago
    I disagree to an extent. Very few solutions are scalable across an entire range of a problem space. Something being scalable, to me, means that it can cover a _broad_ range of the problem space, but doesn't imply it covers 100% of it. If we imagine that the "problem space" is managing software projects from 1 person up through 1,000,000 people and you told me that there was a solution that worked for the range of 50-1,000,000 people I would describe it as scalable, even if I wouldn't recommend it to a 5 person team.
- josephg 6y ago
  Yes! Ironically, microservice architectures replicate the same set of advantages and challenges you have scaling teams. The abstractions (team / process boundaries) become much more rigid. But in exchange, within a team / service, change becomes easier because there's less to understand and navigate. How well the whole system functions depends deeply on how well those abstractions map the solution space. When you're doing something routine (eg most web development), lots of people have figured out reasonable ways of breaking the problem down already. So it works pretty well and you don't carry much risk that the abstractions you pick from the start will hurt you down the road. But when you're doing something more creative and exploratory - like a marketplace for VR, or making a new web browser or making something like Uber - well, then you take on abstraction risk. You will probably make mistakes in how you initially break up your teams and services. Executing well depends on your organizational ability to learn and refactor.
- vsareto 6y ago
  And predicting cross-cutting concerns is at least as difficult as predicting all of your business changes, which is at least as difficult as predicting the future.
_the_inflator 6y ago
I see it differently. In my experience with 200+ devs, Microservices itself is a buzzword to a certain extend and allows for different team to avoid talking to each other: they build their "own" services, which are copies which leads to bloat.
- pbronez 6y ago
  Yeah that’s not gonna work. I think you have to align incentives and the contractual level. For example, don’t pay by the hour, Pay for an API. Mandate that it’s documented in a machine-readable way. Require it to integrate with your centralized authentication and logging services. Mandate thorough, human-friendly documentation Programmatically validate as much of this as possible, add periodic human review (ie surveys, interviews). The goal is to treat each team as a vendor for their service. You hold them accountable for the product and pay them for outcomes.
roguas 6y ago
This, but typically a team of 20 has 1 monolith, afterwards team of 20 has 8 microservices. Equivalent of one battle squad now has to learn a bit of tanking, learn a bit of flying etc.

bob33212 6y ago

The sub groups are going to be somewhat homogeneous internally. So you will end up with a group who thinks leetcode profeciency is important in hiring and another group who thinks QA experience is more important and another group who thinks diversity is the most important. All those groups think they are right and better than the other groups and they don't work well together.

p0nce 6y ago

And who manages to do that?

deckard1 6y ago

This is all outlined by Fred Brooks in 1975. I've beat this dead horse before, but I suppose I'll beat it again. In the late '90s and early aughts everyone read Mythical Man-Month. There is even a law attributed to him, Brooks' law [1]. It really feels like no one internalized what he was actually saying. I suppose in a world of trends and fads, timeless advice is hard to recognize. > [...] the increased communication overhead will consume an ever-increasing quantity of the calendar time available. When n people have to communicate among themselves, as n increases, their output decreases and when it becomes negative the project is delayed further with every person added. Microservices, of course, exasperate this by actually increasing the communication required to make software. [1] https://en.wikipedia.org/wiki/The_Mythical_Man-Month https://en.wikipedia.org/wiki/The_Mythical_Man-Month

eloff 6y ago
> Microservices, of course, exasperate this by actually increasing the communication required to make software. I HATE microservices. I've never seen them used appropriately, although I imagine it can be done. But I think if you increase human communication by using microservices you're defeating the raison d'etre of microservices. That's not the fault of microservices - it's the fault of the humans using them badly or where they shouldn't. Story time. I once contracted to a fortune 500 doing a $100M total rewrite with literally 100 microservices all in their own git repo. It was such a disaster I contemplated shorting their stock. I don't know how the story ended, but I feel confident it didn't end well.
- TheCoelacanth 6y ago
  Microservices can be okay, but they are greatly overused. They are a solution for very large organizations with a ton of teams. For 100 micro-services to make any sense at all, you need to have at least 200-300 developers. Total rewrites, on the other hand, are never anything other than a complete and unmitigated disaster. The only way that you can possibly make a $100M rewrite successful is to do it Ship of Theseus style, where the rewritten code is incorporated into the working and actively-used production software long before the full rewrite is complete. Even then, the full rewrite usually won't complete, so you need to make sure that project is still useful even if you only do a partial rewrite.

qznc 6y ago

One question which is open for me: Should groups be stable? Scrum and other methodologies assume stable teams. It takes a few sprints for a team to assign somewhat accurate story points to stories. It takes time for people to find their roles in a team. Whenever someone joins or leaves a team, it takes time to find a new equilibrium. So if you aim for peak performance teams, it takes months of stability to reach that peak. You better not change it then or it will take months again. In contrast, most organization have a constant need to reorganize to adapt to new business models, customers, product portfolio, etc. For consulting gigs which takes less than a few months, it makes no sense to aim for peak performance and stable teams. Instead you should aim for smooth transitions. In such an environment, more formal methodologies make sense because good documentation, unified code styles, and consistent processes make transitions more smooth. Of course, this isn't a black and white decision but more of a scale from stability to flexibility. There are small lifestyle companies which are very close to the stability extreme (the Sqlite team maybe?). Valve with its flat hierarchy might be an example for the flexibility extreme.