22 ms·

Software has diseconomies of scale, not economies of scale

162 points by henrik_w 10y ago

gwern 10y ago
> Suppose your developers write one bug a year which will slip through test and crash the users' machine. Suppose you know this, so in an effort to catch the bug you do more testing. In order to keep costs low on testing you need to test more software, so you do a bigger release with more changes - economies of scale thinking. That actually makes the testing harder but… Suppose you do one release a year. That release blue screens the machine. The user now sees every release you do crashes his machine. 100% of your releases screw up. If instead you release weekly, one release a year still crashes the machine but the user sees 51 releases a year which don’t. Less than 2% of your releases screw up. This example makes no sense. The user in either case still gets 1 crash per year. They're actually worse off in the many-releases case because in the annual update scenario, they can at least block off a few days to cope with the upgrade scenario (in the way that sysadmins schedule upgrades and downtime for the least costly times of years), but in the weekly release, they could be screwed over at any of 52 times a year at random, and knowing Murphy, it'll be at the worst time. '% of releases which crash' is irrelevant to anything.
- ZenoArrow 10y ago
  I agree, it makes no sense, not only from a planning perspective from an implementation perspective, specifically... "In order to keep costs low on testing you need to test more software, so you do a bigger release with more changes - economies of scale thinking." If you wanted to keep testing costs low, you wouldn't do bigger releases, you'd create automated tests. You may spend more effort up front on building the tests, but as long as you make the test components modular and target the tests at the right level of your application the 'cost' of testing will decrease over time.
  - AstralStorm 10y ago
    Yeah, sure, everything can be tested in an automated way. I can name a few projects that tried it. They were released with a hilarious amount of bugs. Also writing good automated tests requires a great test developer. The thing is, anyone with such credentials would be a great developer and as such, not a tester. Even if you go fully test-driven, which makes it much cheaper, the cost of test a lot development model is surprisingly high for any application of useful size. Just imagine trying to write even something a simple as MS Paint with good test coverage.
    - rco8786 10y ago
      I'm not sure what would be so special or difficult about writing unit tests for Ms Paint.
      - Retric 10y ago
        MS paint includes user selected faunts. That's an endless rabbit hole of complexity.
        ZenoArrow 10y ago
        Validation of new fonts is handled by Windows rather than Paint, I don't see why a new font that meets the criteria set out for Windows would cause any massive test burden for Paint.
        Retric 10y ago
        You're suggestion is to not test a massive chunk of functionality. That may be reasonable choice, but it's a long way from total test coverage.
        ZenoArrow 10y ago
        You target the functionality that resides the application, and its integration with other software. However, you don't need to test Windows functionality, that's for Windows developers to do. 'Total test coverage' is about testing the application under test, not every component of the OS it builds upon.
- AstralStorm 10y ago
  You mean they will block the one big release for a year? Or are you implying there will be a working hotfix soon after the release? What about security updates then? See, in a weekly model you can probably safely skip a cycle. Not so with a big release. If the release has some critical functionality broken, you might be waiting a long time for it to get fixed.
sp527 10y ago
This is a complete misinterpretation of the concept. The only economy of scale that matters in software is that (in theory, barring certain scaling considerations which are in any case becoming more abstracted out thanks to IaaS) the same program can be run to support 10s of millions of users. Each additional unit costs very little to produce (serve it up via a CDN, some amount of network calls, processing and data storage over the lifetime of a user).
- henrik_w 10y ago
  My read of the article is that he argues that software development doesn't have economy of scale - which I agree with.
  - sp527 10y ago
    Fair point but that's neither what the title says nor what he wants you to believe. This is clickbait. And technically what I described is an economy of scale wrt software development the same way the assembly line corresponds to car production. Software dev benefits from fast, cheap distribution of a product that can be duplicated easily because otherwise a developer would need to rewrite the software each time and deliver it by hand to users.
  - venomsnake 10y ago
    That is obviously true - double the codebase and you need four time the resources to maintain it.
  - rogersm 10y ago
    But the issue is not that this happens because this is software development. The reason is software is a job in which cost increase is linearly related to manpower. There are no silver bullets that can magically reduce the number of people needed. Also increasing number of people makes things even worse as communication problems increase, but I think all agree on that. But the main problem with the article is that milk is always the same: in a glass or in a tank car. Software is completely different: more functionality require more lines of code that increases people... and cost.
fizx 10y ago
By this guy's definition, housing construction has diseconomies of scale. You wouldn't believe how expensive a thousand story tall condominium complex would be!
draw_down 10y ago
Some great points in the article. Unsurprising that people here don't want to hear them.
nradov 10y ago
Small, frequent releases are a great idea for web applications and (to a lesser extent) mobile and desktop consumer apps where updates are automatically pushed through some sort of app store. But this totally doesn't work for software licensed to enterprise customers for on-premises use. Every upgrade you release means your customers now have to incur an expense to test your update with the rest of their environment, retrain their users, and roll out the update. Release too often and your customers will be unhappy. So the tolerance for defects goes way down.
- kasey_junk 10y ago
  There are lots of software models where you can't rerelease at all. The amount of formalism in your software development model is highly correlated with your release costs.
jondubois 10y ago
What the article says is true for now, but it doesn't mean it will always be true. Making and transporting large milk bottles is very efficient today (because it's all automated and there are well-tested processes in place), but it wasn't necessarily always like this. When people where still figuring out how to make glass bottles by hand (through glassblowing), bigger bottles were probably more challenging to make (more prone to flaws and breakage during transportation) than small bottles. So probably they just figured out what the optimal size was and just sold that one size. With software, it's the same thing, we don't currently have good tooling to make building scalable software easy. It's getting there, but it's not quite there yet. Once Docker, Swarm, Mesos and Kubernetes become more established, then we are likely to see the software industry behave more like an economy of scale. Once that happens, I think big corporations will see increased competition from small startups. Even people with basic programming knowledge will be able to create powerful, highly scalable enterprise-quality apps which scale to millions of users out of the box.
- wvenable 10y ago
  You seem to be confusing deploying software at scale rather than building software at scale. In fact, there are two different types of scale involved here. I can build a 1 page application and deploy it out to billion people. The article is talking about large software, not small software deployed to scale. As for tools to help build large software, we have them in spades and they will continue to improve. But some things still don't seem to scale. Tools help with managing 500 developers but not enough to really make 500 developers as effective as 50 developers on a smaller project.
  - greenshackle 10y ago
    More than two scales in fact, I can think of three: scale of firm, scale of distribution, scale of product. In most industries scale of firm goes hand-in-hand with scale of distribution. Software breaks the paradigm, so we have to be careful to say which scale we're talking about. Obviously, with distribution, software has insane economies of scale, since we can copy-paste our products nearly for free. That's why we can have small firms with a large distribution, unlike most industries. With scale of firm, we face some of the same diseconomies as other industries. Communication and coordination problems grow superlinearly with firm size. Effort and resources needed also grow superlinearly with product scale. That's also true of other engineering disciplines though. Making a tower twice as high is more than twice as hard. Part of it is the complexity inherent in the product, and part of it is that a more complex design needs a bigger team, so you run into the firm diseconomies of scale mentionned above.
musesum 10y ago
Article conflates distribution with production. Production would be a bigger cow. Deep learning benefits from scale: both in computation power and in a larger corpus. From the example: is the final product the cow, the milk, or the nutrition it provides? Is a software product the lines of code, the app to download, or the service it provides?
- HillaryBriss 10y ago
  Yes. It seems like the article's milk analogy is applied to the wrong thing: another pint of the same exact product. Maybe it's more apt to make the analogy with a pint of a different type of milk product, a type of milk with brand new features. I'm talking about real-world products like these which have appeared in the last 10-20 years: * Almond Milk * Cashew Milk * Coconut-based Milk products (not traditional coconut milk) * Soy-based Milk products (not traditional soy milk) * Lactose Free Milk * Grass Fed Cow Milk * Organic Cow Milk * Omega 3 Enriched Milk etc. We would not expect the marginal cost of these to be less than another pint of conventional regular cow's milk, and indeed, it isn't. That said, I suspect that the article is basically going in the right direction. I mean, it seems like additional features on a complex software project really do cost much much more, on a percentage basis, than a new kind of milk beverage does.
  - musesum 10y ago
    Interesting take on production. Multiple kinds of milk that use the same distribution medium of cartons and grocery stores. These products are decoupled from each other, so the development cost is comparatively linear. I agree with OP's premise that productivity diminishes with project complexity - a decades old problem addressed by Fred Brooks' Mythical Man-Month. But, a lot of the complexity is now wrapped by reusable components. It is now possible to write a component that is shared by 1000's of projects. Somewhat akin to a milk carton that can hold many kinds of milk.
ScottBurson 10y ago
There's a lot more that can be said about how things change as the scale of a software system grows. Every order of magnitude increase requires a new level of discipline. At 10^3 lines, you can do whatever you want -- all your function and variable names can be one or two letters, you don't need comments (or indeed any documentation), your functions don't need well-defined contracts because they only need to work in a few cases, etc. etc. At 10^4 lines, if you're smart you can still get away with making a mess, but it starts to become helpful to name things carefully, to add a few comments, to clear away dead code, to fuss a little over style and readability. At 10^5 lines, those things are not just helpful but necessary, and new things start to matter. It helps to think about your module boundaries and contracts more carefully. You need to minimize the preconditions of your contracts as much as practical -- meaning, make your functions handle all the corner cases you can -- because you can no longer mentally track all the restrictions and special cases. By 10^6 lines, architecture has become more important than coding. Clean interfaces are essential. Minimizing coupling is a major concern. It's easier to work on 10 10^5-line programs than one 10^6-line program, so the goal is to make the system behave, as much as possible, like a weakly interacting collection of subsystems. There's probably a book that explains all this much better than I can here, but perhaps this conveys the general idea.
- Waterluvian 10y ago
  The way you describe relevance at each order of magnitude is eye opening to me. It's exactly what we have been doing, but it was never explained so eloquently why we succeed because of that. Did you get this from somewhere or is it your wording?
  - ScottBurson 10y ago
    This is some combination of something I read once somewhere (Fred Brooks?? possibly) with my own experience over the years. (I didn't go past 10^6 lines because I've never worked on anything larger.)
    - Bombthecat 10y ago
heisenbit 10y ago
The argument that software development has diseconomies of scale is well know e.g. it has been incorporated a long time ago into the COCOMO estimation model. It would be interesting to reverse the question - where does software development really have some economies of scale? What situations does scaling the team or the chunk of software under consideration up make sense? - system testing? - roll-out? - bulk purchasing of licenses for development? - architecture in the sense leveraging consistent frameworks, naming etc. across a team? - infrastructure? - development time too long? (according to COCOMO there is an optimal time and beyond effort increases although slowly) - management of "hit by bus" risk
- Joeri 10y ago
  Building up an understanding of the domain. The bigger the solution, the more you can charge for it, the smaller the relative cost of understanding the problem space.
- yk 10y ago
  When I think about economy of scale and software development, all examples I come up with have the property that any gains are diminishing quickly. For example, a large system has a better build system, which catches a higher number of bugs, therefore the developers are more productive until the now larger project eats the added productivity due to higher complexity. On the other hand, what in traditional industries would be called production, that is producing the specific website for one request, has a rather absurd economy of scale. With a single server and a static site you can serve a sizable fraction of the world population, with negligible marginal costs of serving an additional user. Actually Metcalfe's law suggests that the marginal cost of serving an additional user is negative and hence we get the behemoths like Google and Facebook instead of competition of a few different corporations.
ucaetano 10y ago
Any production system has the same issues, they're usually called overhead or in some cases agency costs. This is captured well in the marginal productivity of labor: the returns on adding more labor decline as you add more labor. Talking about actual economies of scale, software has massive economies of scale. The marginal cost of serving one additional copy of software (in the old "software as a product" way) is close to zero.
vnchr 10y ago
We're transitioning from a monolith PHP app to a Node microservices architecture. The approach with microservices seems to take these scaling issue into account, allowing for narrow focus on each service during initial development and ongoing maintenance.
_pmf_ 10y ago
When developing product lines, you can enable tremendous leverage through good architecture over time. The problem is scaling with people, i.e. this is not something that can be accelerated by involving more people..
crazygringo 10y ago
This is ridiculous and misleading. Economy of scale has nothing to do with the price of milk in different-sized containers, nor does it have anything to do with software complexity. Economy of scale is about it being cheaper for a large farmer to produce a liter of milk than for a small farmer, because overhead costs don't increase linearly. Likewise, software has economies of scale because, on a per-user basis, it's far more expensive to support a 1st, or 5th, or 100th user than it is to support a 100,000th. The author is fine to note that software gets more complex and costly to maintain as it gets bigger, but that has nothing to do with economies of scale -- the author is completely confusing concepts here. Economies of scale are about the marginal cost per user, not the marginal cost per bugfix.
- kartickv 10y ago
  Maybe he shouldn't have used the term "economies of scale", instead using "economies of scope", which he does use. Do you agree with his point about diseconomies of scope?
  - threepipeproblm 10y ago
    I appreciate the thought process here, but to my reading the understanding of economies/diseconomies of scale is quite wrong. (a) The author correctly point out some diseconomies of scale in software, i.e. things that cost more when you do more of them. (b) The author fails to identify that economies of scale typically far outweigh the aforementioned diseconomies. The main error seems to be basing the argument on this statement -- "This happens because each time you add to software software work the marginal cost per unit increases" -- without considering the idea of dividing the expense by the number of users, which can increase dramatically for projects reaching a critical mass. To take a trivial example, most 10-employee businesses can't afford to pay for a 10,000 line software application. A company with 100 employees may need to write a 30,000 application due to increased complexity of their environment, but they can afford it because the project now involves 300 lines of code per employee rather than a 1,000. In short this author accounts for both the numerator and denominiator in the milk analogy that's up front, but then effectively ignores the denominator in the discussion of software costs. Of course this is why most programmers work for large organizations, at least relative to the average employee. It's also why a handful of large software projects make most of the money in the software business. I'm not happy about this, btw, but it is the case. To my reading the author's use of "economies of scope" and "economies of specialization" are even further off base. For example, the trend over the last 50 years or so has been towards increasing specialization (which again, benefits larger teams, although the app economy may have provided a brief counterpoint, the same forces are at work there).
siscia 10y ago
In my opinion the whole idea is a little flawed. Is not "software" that has "diseconomies of scale", is the design process. Then, because software has close to zero of marginal cost the design cost is all the one we end up paying. Imagine the same milk that the author is talking about, imagine to design the system that get the milk from the cow to 2 people every morning. Not a big deal, get a cow, a farmer and a truck mix all together and pretty much done. Now imagine to serve a supermarket, well you start to need something more than a cow a farmer and a truck. Now imagine to serve a whole city, what about a whole nations? What about the whole world? Simply design how to serve milk to a small supermarket is a problem, but since the marginal cost of the milk is not approaching zero, the cost of the design process will always be less that the cost of the milk (otherwise it wouldn't make financial sense), hence the whole idea of "economies of scale." To conclude I believe that the root causes of "diseconomies of scale" doesn't lie in the "software" part but in the "design" part.
- sly010 10y ago
  Said differently: Software == Farm Value == Milk
- im3w1l 10y ago
  Uh, what are you talking about? More cows, more farmers and more trucks (pack animals) is perfectly scalable, and how it worked until recently. It's only when you want a singular capitalist to get a cut of all these transactions, that you need to complicate things.
  - ajuc 10y ago
    The capitalist wins because he can sell the same milk cheaper, because then each farmer don't need own truck, and to bother with going around selling stuff to people. Capitalist will have a fleet of cars, sales team, etc optimized for cost efficiency. Cheaper food means for example - people who couldn't afford to go to vacations, or to buy a computer for their kids - can. So - designing the logistic chain is worth it, it provides value for customers. Same as making software. The design process/writing software on the other hand - is hard to do on big scale from scratch. I'm not aware of any big logistic network that started big.
poof131 10y ago
A few days ago I would have agreed whole heartedly. Now, after reading the article about the formation of AWS I’m not so sure.[1] It seems that Amazon (and others such as Google) have achieved economies of scale in development with internal services and well organized APIs. Certainly much of this is being exposed externally for profit, but it does seem to indicate that well designed software has economies of scale for the development of future software. Certainly one monolith application with a team of a thousand starting from scratch is a bad idea, but once internal APIs and services exist, these definitely seem to aid in the rapid development of future software products for a company, which seems like economies of scale to me. Bloated bureaucracies and bad processes can negatively impact any company in any industry, not just software. Some of the article’s logic doesn’t seem to differentiate software development from anything else, such as “working in the large increases risk”. So while a large monolith application is risky, building a hundred million widgets is risky too. Better to iterate and start with a prototype and expand, but the same goes for other industries too: better to prototype and market test your widget before mass production. Seems to me like the article is talking more about lean and agile and process in general than about economies of scale. [1] https://news.ycombinator.com/item?id=12022915 https://news.ycombinator.com/item?id=12022915
hyperpallium 10y ago
https://wikipedia.org/wiki/The_Mythical_Man-Month https://wikipedia.org/wiki/The_Mythical_Man-Month
hackaflocka 10y ago
This article is unoriginal click-bait. Software does have economies of scale. How much did a copy of Windows 7 or Office 2013 cost? Only about $100? That's because the more we produce/supply/consume the cheaper it gets, just like milk. The notion that there's an optimal amount of human capital needed for a project is nothing new. We've all heard of "Too many cooks spoil the broth."
kriro 10y ago
What he describes (a product growing in complexity) isn't a typical economics of scale type of product. I'd also argue that the general observation is true for all complex products. Even a car has relatively high R&D cost and only reaps the benefits of economies of scale (lower per unit production cost than the competition) once it is ready to ship. Unit 1 is going to be very resource intensive, the following units are what contributes to the scale (and like he said for software that scale is amazing). However maintenance cost in software are usually really high especially since there is a tendency to gradually "improve" the same code base instead of ripping it out and building a new one every n years (buy a new car).
wodenokoto 10y ago
This is usually not what is meant by economies of scale. Very few production lines get cheaper as they get more complex. But software, more than most products gets cheaper per unit, as you scale the number of units.
mehh 10y ago
bigger projects have bigger risks ... no shit sherlock

162 points by henrik_w 10y ago

gwern 10y ago

> Suppose your developers write one bug a year which will slip through test and crash the users' machine. Suppose you know this, so in an effort to catch the bug you do more testing. In order to keep costs low on testing you need to test more software, so you do a bigger release with more changes - economies of scale thinking. That actually makes the testing harder but… Suppose you do one release a year. That release blue screens the machine. The user now sees every release you do crashes his machine. 100% of your releases screw up. If instead you release weekly, one release a year still crashes the machine but the user sees 51 releases a year which don’t. Less than 2% of your releases screw up. This example makes no sense. The user in either case still gets 1 crash per year. They're actually worse off in the many-releases case because in the annual update scenario, they can at least block off a few days to cope with the upgrade scenario (in the way that sysadmins schedule upgrades and downtime for the least costly times of years), but in the weekly release, they could be screwed over at any of 52 times a year at random, and knowing Murphy, it'll be at the worst time. '% of releases which crash' is irrelevant to anything.

ZenoArrow 10y ago
I agree, it makes no sense, not only from a planning perspective from an implementation perspective, specifically... "In order to keep costs low on testing you need to test more software, so you do a bigger release with more changes - economies of scale thinking." If you wanted to keep testing costs low, you wouldn't do bigger releases, you'd create automated tests. You may spend more effort up front on building the tests, but as long as you make the test components modular and target the tests at the right level of your application the 'cost' of testing will decrease over time.
- AstralStorm 10y ago
  Yeah, sure, everything can be tested in an automated way. I can name a few projects that tried it. They were released with a hilarious amount of bugs. Also writing good automated tests requires a great test developer. The thing is, anyone with such credentials would be a great developer and as such, not a tester. Even if you go fully test-driven, which makes it much cheaper, the cost of test a lot development model is surprisingly high for any application of useful size. Just imagine trying to write even something a simple as MS Paint with good test coverage.
  - rco8786 10y ago
    I'm not sure what would be so special or difficult about writing unit tests for Ms Paint.
    - Retric 10y ago
      MS paint includes user selected faunts. That's an endless rabbit hole of complexity.
      - ZenoArrow 10y ago
        Validation of new fonts is handled by Windows rather than Paint, I don't see why a new font that meets the criteria set out for Windows would cause any massive test burden for Paint.
        Retric 10y ago
        You're suggestion is to not test a massive chunk of functionality. That may be reasonable choice, but it's a long way from total test coverage.
        ZenoArrow 10y ago
        You target the functionality that resides the application, and its integration with other software. However, you don't need to test Windows functionality, that's for Windows developers to do. 'Total test coverage' is about testing the application under test, not every component of the OS it builds upon.
AstralStorm 10y ago
You mean they will block the one big release for a year? Or are you implying there will be a working hotfix soon after the release? What about security updates then? See, in a weekly model you can probably safely skip a cycle. Not so with a big release. If the release has some critical functionality broken, you might be waiting a long time for it to get fixed.

sp527 10y ago

This is a complete misinterpretation of the concept. The only economy of scale that matters in software is that (in theory, barring certain scaling considerations which are in any case becoming more abstracted out thanks to IaaS) the same program can be run to support 10s of millions of users. Each additional unit costs very little to produce (serve it up via a CDN, some amount of network calls, processing and data storage over the lifetime of a user).

henrik_w 10y ago
My read of the article is that he argues that software development doesn't have economy of scale - which I agree with.
- sp527 10y ago
  Fair point but that's neither what the title says nor what he wants you to believe. This is clickbait. And technically what I described is an economy of scale wrt software development the same way the assembly line corresponds to car production. Software dev benefits from fast, cheap distribution of a product that can be duplicated easily because otherwise a developer would need to rewrite the software each time and deliver it by hand to users.
- venomsnake 10y ago
  That is obviously true - double the codebase and you need four time the resources to maintain it.
- rogersm 10y ago
  But the issue is not that this happens because this is software development. The reason is software is a job in which cost increase is linearly related to manpower. There are no silver bullets that can magically reduce the number of people needed. Also increasing number of people makes things even worse as communication problems increase, but I think all agree on that. But the main problem with the article is that milk is always the same: in a glass or in a tank car. Software is completely different: more functionality require more lines of code that increases people... and cost.

fizx 10y ago

By this guy's definition, housing construction has diseconomies of scale. You wouldn't believe how expensive a thousand story tall condominium complex would be!

draw_down 10y ago

Some great points in the article. Unsurprising that people here don't want to hear them.

nradov 10y ago

Small, frequent releases are a great idea for web applications and (to a lesser extent) mobile and desktop consumer apps where updates are automatically pushed through some sort of app store. But this totally doesn't work for software licensed to enterprise customers for on-premises use. Every upgrade you release means your customers now have to incur an expense to test your update with the rest of their environment, retrain their users, and roll out the update. Release too often and your customers will be unhappy. So the tolerance for defects goes way down.

kasey_junk 10y ago
There are lots of software models where you can't rerelease at all. The amount of formalism in your software development model is highly correlated with your release costs.

jondubois 10y ago

What the article says is true for now, but it doesn't mean it will always be true. Making and transporting large milk bottles is very efficient today (because it's all automated and there are well-tested processes in place), but it wasn't necessarily always like this. When people where still figuring out how to make glass bottles by hand (through glassblowing), bigger bottles were probably more challenging to make (more prone to flaws and breakage during transportation) than small bottles. So probably they just figured out what the optimal size was and just sold that one size. With software, it's the same thing, we don't currently have good tooling to make building scalable software easy. It's getting there, but it's not quite there yet. Once Docker, Swarm, Mesos and Kubernetes become more established, then we are likely to see the software industry behave more like an economy of scale. Once that happens, I think big corporations will see increased competition from small startups. Even people with basic programming knowledge will be able to create powerful, highly scalable enterprise-quality apps which scale to millions of users out of the box.

wvenable 10y ago
You seem to be confusing deploying software at scale rather than building software at scale. In fact, there are two different types of scale involved here. I can build a 1 page application and deploy it out to billion people. The article is talking about large software, not small software deployed to scale. As for tools to help build large software, we have them in spades and they will continue to improve. But some things still don't seem to scale. Tools help with managing 500 developers but not enough to really make 500 developers as effective as 50 developers on a smaller project.
- greenshackle 10y ago
  More than two scales in fact, I can think of three: scale of firm, scale of distribution, scale of product. In most industries scale of firm goes hand-in-hand with scale of distribution. Software breaks the paradigm, so we have to be careful to say which scale we're talking about. Obviously, with distribution, software has insane economies of scale, since we can copy-paste our products nearly for free. That's why we can have small firms with a large distribution, unlike most industries. With scale of firm, we face some of the same diseconomies as other industries. Communication and coordination problems grow superlinearly with firm size. Effort and resources needed also grow superlinearly with product scale. That's also true of other engineering disciplines though. Making a tower twice as high is more than twice as hard. Part of it is the complexity inherent in the product, and part of it is that a more complex design needs a bigger team, so you run into the firm diseconomies of scale mentionned above.

musesum 10y ago

Article conflates distribution with production. Production would be a bigger cow. Deep learning benefits from scale: both in computation power and in a larger corpus. From the example: is the final product the cow, the milk, or the nutrition it provides? Is a software product the lines of code, the app to download, or the service it provides?

HillaryBriss 10y ago
Yes. It seems like the article's milk analogy is applied to the wrong thing: another pint of the same exact product. Maybe it's more apt to make the analogy with a pint of a different type of milk product, a type of milk with brand new features. I'm talking about real-world products like these which have appeared in the last 10-20 years: * Almond Milk * Cashew Milk * Coconut-based Milk products (not traditional coconut milk) * Soy-based Milk products (not traditional soy milk) * Lactose Free Milk * Grass Fed Cow Milk * Organic Cow Milk * Omega 3 Enriched Milk etc. We would not expect the marginal cost of these to be less than another pint of conventional regular cow's milk, and indeed, it isn't. That said, I suspect that the article is basically going in the right direction. I mean, it seems like additional features on a complex software project really do cost much much more, on a percentage basis, than a new kind of milk beverage does.
- musesum 10y ago
  Interesting take on production. Multiple kinds of milk that use the same distribution medium of cartons and grocery stores. These products are decoupled from each other, so the development cost is comparatively linear. I agree with OP's premise that productivity diminishes with project complexity - a decades old problem addressed by Fred Brooks' Mythical Man-Month. But, a lot of the complexity is now wrapped by reusable components. It is now possible to write a component that is shared by 1000's of projects. Somewhat akin to a milk carton that can hold many kinds of milk.

ScottBurson 10y ago

There's a lot more that can be said about how things change as the scale of a software system grows. Every order of magnitude increase requires a new level of discipline. At 10^3 lines, you can do whatever you want -- all your function and variable names can be one or two letters, you don't need comments (or indeed any documentation), your functions don't need well-defined contracts because they only need to work in a few cases, etc. etc. At 10^4 lines, if you're smart you can still get away with making a mess, but it starts to become helpful to name things carefully, to add a few comments, to clear away dead code, to fuss a little over style and readability. At 10^5 lines, those things are not just helpful but necessary, and new things start to matter. It helps to think about your module boundaries and contracts more carefully. You need to minimize the preconditions of your contracts as much as practical -- meaning, make your functions handle all the corner cases you can -- because you can no longer mentally track all the restrictions and special cases. By 10^6 lines, architecture has become more important than coding. Clean interfaces are essential. Minimizing coupling is a major concern. It's easier to work on 10 10^5-line programs than one 10^6-line program, so the goal is to make the system behave, as much as possible, like a weakly interacting collection of subsystems. There's probably a book that explains all this much better than I can here, but perhaps this conveys the general idea.

Waterluvian 10y ago
The way you describe relevance at each order of magnitude is eye opening to me. It's exactly what we have been doing, but it was never explained so eloquently why we succeed because of that. Did you get this from somewhere or is it your wording?
- ScottBurson 10y ago
  This is some combination of something I read once somewhere (Fred Brooks?? possibly) with my own experience over the years. (I didn't go past 10^6 lines because I've never worked on anything larger.)
  - Bombthecat 10y ago

heisenbit 10y ago

The argument that software development has diseconomies of scale is well know e.g. it has been incorporated a long time ago into the COCOMO estimation model. It would be interesting to reverse the question - where does software development really have some economies of scale? What situations does scaling the team or the chunk of software under consideration up make sense? - system testing? - roll-out? - bulk purchasing of licenses for development? - architecture in the sense leveraging consistent frameworks, naming etc. across a team? - infrastructure? - development time too long? (according to COCOMO there is an optimal time and beyond effort increases although slowly) - management of "hit by bus" risk

Joeri 10y ago
Building up an understanding of the domain. The bigger the solution, the more you can charge for it, the smaller the relative cost of understanding the problem space.
yk 10y ago
When I think about economy of scale and software development, all examples I come up with have the property that any gains are diminishing quickly. For example, a large system has a better build system, which catches a higher number of bugs, therefore the developers are more productive until the now larger project eats the added productivity due to higher complexity. On the other hand, what in traditional industries would be called production, that is producing the specific website for one request, has a rather absurd economy of scale. With a single server and a static site you can serve a sizable fraction of the world population, with negligible marginal costs of serving an additional user. Actually Metcalfe's law suggests that the marginal cost of serving an additional user is negative and hence we get the behemoths like Google and Facebook instead of competition of a few different corporations.

ucaetano 10y ago

Any production system has the same issues, they're usually called overhead or in some cases agency costs. This is captured well in the marginal productivity of labor: the returns on adding more labor decline as you add more labor. Talking about actual economies of scale, software has massive economies of scale. The marginal cost of serving one additional copy of software (in the old "software as a product" way) is close to zero.

vnchr 10y ago

We're transitioning from a monolith PHP app to a Node microservices architecture. The approach with microservices seems to take these scaling issue into account, allowing for narrow focus on each service during initial development and ongoing maintenance.

_pmf_ 10y ago

When developing product lines, you can enable tremendous leverage through good architecture over time. The problem is scaling with people, i.e. this is not something that can be accelerated by involving more people..

crazygringo 10y ago

This is ridiculous and misleading. Economy of scale has nothing to do with the price of milk in different-sized containers, nor does it have anything to do with software complexity. Economy of scale is about it being cheaper for a large farmer to produce a liter of milk than for a small farmer, because overhead costs don't increase linearly. Likewise, software has economies of scale because, on a per-user basis, it's far more expensive to support a 1st, or 5th, or 100th user than it is to support a 100,000th. The author is fine to note that software gets more complex and costly to maintain as it gets bigger, but that has nothing to do with economies of scale -- the author is completely confusing concepts here. Economies of scale are about the marginal cost per user, not the marginal cost per bugfix.

kartickv 10y ago
Maybe he shouldn't have used the term "economies of scale", instead using "economies of scope", which he does use. Do you agree with his point about diseconomies of scope?
- threepipeproblm 10y ago
  I appreciate the thought process here, but to my reading the understanding of economies/diseconomies of scale is quite wrong. (a) The author correctly point out some diseconomies of scale in software, i.e. things that cost more when you do more of them. (b) The author fails to identify that economies of scale typically far outweigh the aforementioned diseconomies. The main error seems to be basing the argument on this statement -- "This happens because each time you add to software software work the marginal cost per unit increases" -- without considering the idea of dividing the expense by the number of users, which can increase dramatically for projects reaching a critical mass. To take a trivial example, most 10-employee businesses can't afford to pay for a 10,000 line software application. A company with 100 employees may need to write a 30,000 application due to increased complexity of their environment, but they can afford it because the project now involves 300 lines of code per employee rather than a 1,000. In short this author accounts for both the numerator and denominiator in the milk analogy that's up front, but then effectively ignores the denominator in the discussion of software costs. Of course this is why most programmers work for large organizations, at least relative to the average employee. It's also why a handful of large software projects make most of the money in the software business. I'm not happy about this, btw, but it is the case. To my reading the author's use of "economies of scope" and "economies of specialization" are even further off base. For example, the trend over the last 50 years or so has been towards increasing specialization (which again, benefits larger teams, although the app economy may have provided a brief counterpoint, the same forces are at work there).

siscia 10y ago

In my opinion the whole idea is a little flawed. Is not "software" that has "diseconomies of scale", is the design process. Then, because software has close to zero of marginal cost the design cost is all the one we end up paying. Imagine the same milk that the author is talking about, imagine to design the system that get the milk from the cow to 2 people every morning. Not a big deal, get a cow, a farmer and a truck mix all together and pretty much done. Now imagine to serve a supermarket, well you start to need something more than a cow a farmer and a truck. Now imagine to serve a whole city, what about a whole nations? What about the whole world? Simply design how to serve milk to a small supermarket is a problem, but since the marginal cost of the milk is not approaching zero, the cost of the design process will always be less that the cost of the milk (otherwise it wouldn't make financial sense), hence the whole idea of "economies of scale." To conclude I believe that the root causes of "diseconomies of scale" doesn't lie in the "software" part but in the "design" part.

sly010 10y ago
Said differently: Software == Farm Value == Milk
im3w1l 10y ago
Uh, what are you talking about? More cows, more farmers and more trucks (pack animals) is perfectly scalable, and how it worked until recently. It's only when you want a singular capitalist to get a cut of all these transactions, that you need to complicate things.
- ajuc 10y ago
  The capitalist wins because he can sell the same milk cheaper, because then each farmer don't need own truck, and to bother with going around selling stuff to people. Capitalist will have a fleet of cars, sales team, etc optimized for cost efficiency. Cheaper food means for example - people who couldn't afford to go to vacations, or to buy a computer for their kids - can. So - designing the logistic chain is worth it, it provides value for customers. Same as making software. The design process/writing software on the other hand - is hard to do on big scale from scratch. I'm not aware of any big logistic network that started big.

poof131 10y ago

A few days ago I would have agreed whole heartedly. Now, after reading the article about the formation of AWS I’m not so sure.[1] It seems that Amazon (and others such as Google) have achieved economies of scale in development with internal services and well organized APIs. Certainly much of this is being exposed externally for profit, but it does seem to indicate that well designed software has economies of scale for the development of future software. Certainly one monolith application with a team of a thousand starting from scratch is a bad idea, but once internal APIs and services exist, these definitely seem to aid in the rapid development of future software products for a company, which seems like economies of scale to me. Bloated bureaucracies and bad processes can negatively impact any company in any industry, not just software. Some of the article’s logic doesn’t seem to differentiate software development from anything else, such as “working in the large increases risk”. So while a large monolith application is risky, building a hundred million widgets is risky too. Better to iterate and start with a prototype and expand, but the same goes for other industries too: better to prototype and market test your widget before mass production. Seems to me like the article is talking more about lean and agile and process in general than about economies of scale. [1] https://news.ycombinator.com/item?id=12022915 https://news.ycombinator.com/item?id=12022915

hyperpallium 10y ago

https://wikipedia.org/wiki/The_Mythical_Man-Month https://wikipedia.org/wiki/The_Mythical_Man-Month

hackaflocka 10y ago

This article is unoriginal click-bait. Software does have economies of scale. How much did a copy of Windows 7 or Office 2013 cost? Only about $100? That's because the more we produce/supply/consume the cheaper it gets, just like milk. The notion that there's an optimal amount of human capital needed for a project is nothing new. We've all heard of "Too many cooks spoil the broth."

kriro 10y ago

What he describes (a product growing in complexity) isn't a typical economics of scale type of product. I'd also argue that the general observation is true for all complex products. Even a car has relatively high R&D cost and only reaps the benefits of economies of scale (lower per unit production cost than the competition) once it is ready to ship. Unit 1 is going to be very resource intensive, the following units are what contributes to the scale (and like he said for software that scale is amazing). However maintenance cost in software are usually really high especially since there is a tendency to gradually "improve" the same code base instead of ripping it out and building a new one every n years (buy a new car).

wodenokoto 10y ago

This is usually not what is meant by economies of scale. Very few production lines get cheaper as they get more complex. But software, more than most products gets cheaper per unit, as you scale the number of units.

mehh 10y ago

bigger projects have bigger risks ... no shit sherlock