10 ms·

It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production

It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features. If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level. There is no software development process which reliably produces software that works at scale without doing it small, and medium sized, first, and fixing what goes wrong before you go big.

solatic 11mo ago
That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen. Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.
- dustingetz 11mo ago
  we get paid to add to it, we don’t get paid to take away
  - cjfd 11mo ago
    Now there is your problem. It is only true in the context of grave incompetence, though. I have worked on tickets with 'remove' in the title.
- Jtsummers 11mo ago
  Designing or intending a system to be used at massive scale is not the same as building and deploying it so that it only initially runs at that massive scale. That's just a recipe for disaster, "We don't even know if we can handle 100 users, let's now force 1 million people to use the system simultaneously." Even WhatsApp couldn't handle hundreds of millions of users on the day it was first released, nor did it attempt to. You build out slowly and make sure things work, at least if you're competent and sane.
  - solatic 11mo ago
    Sure, but if you did a good job, the gradual deployment can go relatively quickly and smoothly, which is how $FAANG roll out new features and products to very large audiences. The actual rollout is usually a bit of an implementation detail of what first needed to be architected to handle that larger scale.
    - vlovich123 11mo ago
      You get certain big pieces correct maybe but you’d be surprised how many mistakes get made. For example, I had designed the billing system for a large distributed product that the engineer ended up implementing not as described in the spec which fell down fairly quickly with even a modicum of growth.
    - eru 11mo ago
      Well, Google got good at large scale rollouts, because they are doing large scale rollouts all the time. _And_ most of the time, the system they are rolling out is a small iteration from the last system they rolled out: the new GMail servers look almost exactly like the last GMail servers, but they have on extra feature flag you can turn on (and which is disabled by default) or have one bug fixed. That's a very different challenge from rolling out a brand new system once.
    - coliveira 11mo ago
      The issue with FAANG is that they already have the infrastructure to make these large scale deployments. So any new system - by necessity - needs to conform to that large scale architecture.
- paulsutter 11mo ago
  You have to design for scale AND deploy gradually
  - rossdavidh 11mo ago
    Yes, absolutely. Knowing that it will need to get big eventually is important, but not at all the same as deploying at scale initially.
- golemiprague 11mo ago
  [dead]
- lelandbatey 11mo ago
  Gradual growth =/= many tacked on features. Many tacked on features =/= technical debt. Technical debt =/= "everybody is afraid and frozen." Those are merely often correlated, but not required. Whatsapp is a terrible example because it's barely a product; Whatsapp is mostly a free offering of goodwill riding on the back of actual products like Facebook Ads. A great example would be a product like Salesforce, SAP, or Microsoft Dynamics. Those products are forced to grow and change and adapt and scale, to massive numbers doing tons of work, all while being actual products and being software systems. I think such products act as stark rebukes of what you've described.
- jimbokun 11mo ago
  Yes, it can be very difficult to add “scale” after the fact, once you already have a lot of data persisted in a certain way.
- otterley 11mo ago
  Software is a component of a product, if not the product itself. Treating software like a product, besides being the underlying truth, also means it makes sense to manage it like one. Technical debt isn’t usually the problem people think it is. When it does become a problem, it’s best to think of it in product-like terms. Does it make the product less useful for its intended purpose? Does it make maintenance or repair inconvenient or costly? Or does it make it more difficult or even impossible to add competitive features or improvements? Taking a product evaluation approach to the question can help you figure out what the right response is. Sometimes it’s no response at all.
  - jfreds 11mo ago
    Took me way too long to learn this. It still makes me sad to leave projects “imperfect” and not fiddle in my free time sometimes
  - YetAnotherNick 11mo ago
    The discussion is not about the product where you can just remove the stuff. The thread was testing in small setting and then moving to oddball setting. If it is required to cover oddball settings, it makes sense to know and plan for oddball setting.
- mekoka 11mo ago
  > Gradual growth inevitably results in loads of technical debt. Why is this stated as though it's some de facto software law? The argument is not whether it's possible to waterfall a massive software system. It clearly is possible, but the failure ratios have historically been sufficiently uncomfortable to give rise to entirely different (and evidently more successful) project development philosophies, especially when promoters were more sensitive to the massive sums involved (which in my opinion also helps explains why so many wasteful government examples). The lean startup did not appear in a vacuum. Do things that don't scale did not become a motto in these parts without reason. In case some are still confused about the historical purpose of these benign sounding advices, no, they weren't originally addressed at entrepreneurs aiming to run "lifestyle" businesses.
  - jappgar 11mo ago
    It is a law. The law of entropy. Try as you might, you cannot fight entropy eternally, as mistakes in this fight will accumulate and overpower you. It's the natural process of aging we see in every lifeform. The way life continues on despite this law is through reproduction. If you bud off independent organisms, an ecosystem can gain "eternal" life. The cost is that you must devote much of your energy to effective reproduction. In software, this means embracing rewrites. The people who push against rewrites and claim they're not necessary are just as delusional as those who think they can live forever.
    - cjfd 11mo ago
      You don't understand very much about entropy. This reasoning is very, very, very sloppy.
      - jappgar 11mo ago
        Now I remember why I stopped commenting here.
      - ebcode 11mo ago
        low-effort comment with ad hominem and zero rationale. fairly toxic.
- philipallstar 11mo ago
  > Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. This isn't technical debt, necessarily. Technical debt is a specific thing. You probably mean "an underlying design that doesn't perfectly map to what ended up being the requirements". But then the world moves on (what if a regulation is added that ruins your perfect structure anyway?) and you can't just wish for perfect requirements. Or not in software that interacts directly with the real world, anyway.
- fatbird 11mo ago
  There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism. Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine. The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."
the_duke 11mo ago
The accounting, legal and business process requirements are vastly different at different scales, different jurisdictions, different countries, etc. There's a crazy amount of complexity and customizability in systems like ERPs for multinational corporations (SAP, Oracle). When you start with a small town, you'll have to throw most of everything away when moving to a different scale. That's true for software systems in general. If major requirements are bolted on after the fact, instead of designed into the system from the beginning, you usually end up with an unmaintainable mess.
- rossdavidh 11mo ago
  Knowing that the rules for your first small deployment are not the same as the rules for everywhere, is valuable for designing well. Trying to implement all of those sets of rules in your initial deployment, is not a good idea. There is a general principle that you shouldn't code the abstraction until you've coded for the concrete example 2 or 3 times, because otherwise you won't make the right abstraction. Looking ahead is not the same as starting with the whole enchilada for your initial deployment.
OtherShrezzing 11mo ago
While I think this is good advice in general, I don’t think your statement that “there is no process to create scalable software” holds true. The uk gov development service reliably implements huge systems over and over again, and those systems go out to tens of millions from day 1. As a rule of thumb, the parts of the uk govt digital suite that suck are the parts the development service haven’t been assigned to yet. The Swift banking org launches reliable features to hundreds of millions of users. There’s honestly loads of instances of organisations reliably implementing robust and scalable software without starting with tens of users.
- sam_lowry_ 11mo ago
  SWIFT? Hold my beer. SWIFT did not launch anything substantial since its startup days in early 70-ies. Moreover, their core tech did not evolve that far from that era, and the 70-ies tech bros are still there through their progeniture. Here's an anecdote: The first messaging system built by SWIFT was text-based, somewhat similar to ASN.1. The next one used XML, as it was the fad of the day. Unfortunately, neither SWIFT nor the banks could handle 2-3 orders of magnitude increase in payload size in their ancient systems. Yes, as engineers, you would think compressing XML would solve the problem and you would by right. Moreover, XML Infoset already existed, and it defined compression as a function of the XML Schema, so it was somewhat more deterministic even though not more efficient than LZMA. But the suits decided differently. At one of the SIBOS conferences they abbreviate XML tags, and did it literally on paper and without thinking about back-and-forth translation, dupes, etc. And this is how we landed with ISO20022 abberviations that we all know and love: Ccy for Currency, Pmt for Payment, Dt for Date, etc.
  - deleted 11mo ago
    [deleted]
  - noname120 11mo ago
    Harder to audit when every payload needs to be decompressed to be inspected
    - 11mo ago
Izikiel43 11mo ago
What works at small scale possibly won't work at a huge scale.
- skywhopper 11mo ago
  But what hasn’t even been tried at a small scale definitely won’t work at a huge scale.
- rossdavidh 11mo ago
  Which is absolutely true, and a reason to try at medium scale second. But what doesn't work at small scale, almost certainly won't work at huge scale.
hinkley 11mo ago
The dominant factor is: there is a human who understands the entire system. That is vastly easier to achieve by making a small, successful system, which gets buy in from both users and builders to the extent that the former pay sufficient money for the latter to be invested in understanding the entire system and then growing it and keeping up with the changes. Occasionally a moon shot program can overcome all of that inertia, but the “90% of all projects fail” is definitely overrepresented in large projects. And the Precautionary Principle says you shouldn’t because the consequences are so high.
- dominicrose 11mo ago
  This works for Clojure, git and even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it. But these things are meant to be used by technical people. The non-technical people I know might want to use Linux but stay on Windows or choose Mac OS because it's more straightforward. I use Windows+WSL at work even though I would like to use a native Linux distribution. I know someone who created a MUD game (text online game) and said to him I wanted to make one with a browser client. He said something we could translate as "Good, you can have all the newbies." Not only was he right that a MUD should be played with a MUD client like tintin++, but making a good browser client is harder than it seems and that's time not spent making content for the game or improving the engine. My point is that he was un uncomprimising person who refused adding layers to a project because they would come at a cost which isn't only time or dollars but also things like motivation and focus.
  - wickedsight 11mo ago
    > ... even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it. I really wonder what will happen to Linux once Linus is no longer involved.
    - inemesitaffia 11mo ago
      Greg KH
chrisweekly 11mo ago
See also Gall's Law: "All complex systems that work evolved from simpler systems that worked"
shagie 11mo ago
> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level. At a large box retail chain (15 states, ~300 stores) I worked on a project to replace the POS system. The original plan had us getting everything working (Ha!) and then deploying it out to stores and then ending up with the two oddball "stores". The company cafeteria and surplus store were technically stores in that they had all the same setup and processes but were odd. When the team that I was on was brought into this project, we flipped that around and first deployed to those two several months ahead of the schedule to deploy to the regular stores. In particular, the surplus store had a few dozen transactions a day. If anything broke, you could do reconciliation by hand. The cafeteria had single register transaction volume that surpassed a surplus store on most any other day. Furthermore, all of its transactions were payroll deductions (swipe your badge rather than credit card or cash). This meant that if anything went wrong there we weren't in trouble with PCI and could debit and credit accounts. Ultimately, we made our deadline to get things out to stores. We did have one nasty bug that showed up in late October (or was it early November?) with repackaging counts (if a box of 6 was $24 and if purchased as a single item it was $4.50 ... but if you bought 6 single items it was "repackaged" to cost $24 rather than $27) which interacted with a BOGO sale. That bug resulted in absurd receipts with sales and discounts (the receipt showed you spent $10,000 but were discounted $9,976 ... and then the GMs got alerts that the store was not able to make payroll because of a $9,976 discount ... one of the devs pulled an all nighter to fix that one and it got pushed to the stores ). I shudder to think about what would have happened if we had tried to push the POS system out to customer facing stores where the performance issues in the cafeteria where worked out first or if we had to reconcile transactions to hunt down incorrect tax calculations.
- einpoklum 11mo ago
  You could have, in principle, implemented the new system to be able to run in "dummy mode" alongside the existing system at regular stores, so that you see that it produces the 'same' results in terms of what the existing system is able to provide. Which is to say, there is more than one approach to gradual deployment.
hintymad 11mo ago
> https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 This is what https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 advocates too: start small, modularize, and then scale. The example of Tesla's mega factory was particular enticing.
nostrademons 11mo ago
Came here to say this. I still think that Linus Torvalds has the most profound advice to building a large, highly successful software system: "Nobody should start to undertake a large project. You start with a small trivial project, and you should never expect it to get large. If you do, you'll just overdesign and generally think it is more important than it likely is at that stage. Or worse, you might be scared away by the sheer size of the work you envision. So start small, and think about the details. Don't think about some big picture and fancy design. If it doesn't solve some fairly immediate need, it's almost certainly over-designed. And don't expect people to jump in and help you. That's not how these things work. You need to get something half-way useful first, and then others will say "hey, that almost works for me", and they'll get involved in the project." -- Linux Times, October 2004.
- ambicapter 11mo ago
  This is a really dense paragraph of lifetime-accumulated wisdom in that single quote.
- tsimionescu 11mo ago
  I don't think this applies in any way to companies contracted to build a massive system for a government with a clear need. Linus is talking about growing a greenfield open-source project, which may or may not ever be used by anyone. In contrast, if your purpose is "we need to manage our country's accounting without pen and paper", that's a clear need for a massive system. Starting work on this by designing a system that can solve accounting for a small firm is not the right way to go. Instead, you have to design with the end-goal in mind, since that's what you were paid for. But, you don't launch your system to the entire country at once: you first use this system designed for a country in a small shop, to make sure it actually handles the small scale well, before gradually rolling out to more and more people.
  - mrweasel 11mo ago
    > for a government with a clear need. There's your problem. The needs are never clear, not on massive systems. Governments will write a spec, companies will read the spec, offer to implement it as written, knowing full well that it won't work. Then they charge exorbitant fees to modify the system after launch, so that it will actually full fill business needs. The Danish government is famous for sucking at buying massive IT systems. * Specs for new tax system: 6000 page, tax laws not included. That's basically impossible to implement and it predictably failed. The version that worked: Implement just the basics to collect TV license fees. The build from there. * System to calculate the value of people home, I think we're at round five (rumors has it that one system worked, but was scrapped because it showed that most home are massively overvalued and it do terrible things to the tax collection in the municipalities). * New case management system for the police, failed, development never restarted. One suggested solution was to have the police hire a handful of the best developers in the country and have them produce smaller deliverable over a number of year. The money wasted could have funded 10 world class developers for ~30-50 years.
BrenBarn 11mo ago
Yes. Also the same applies to companies. There should not be companies that are growing to $100 million revenue while losing money on a gamble that they will eventually get big enough to succeed. Good first, big later.
- SchemaLoad 11mo ago
  $100M maybe. But pretty much all tech needs an initial investment before you can start making profit. It takes a lot of development before you can get a product that anyone would want to pay for.
creer 11mo ago
The very specific example you chose: payroll, shows how it can be difficult to incrementally step from small to huge. As you grow from town to national, you will run into all the disadvantages without really hitting the advantages. I feel that incremental does help you move from one level to one just a few above. But only if there are enough customers at these starting levels exactly. When developing for towns, you will have all small random subsets of the variations imposed by year after year of legal changes BUT small sales. You will have to implement niche variations in arbitrary aspects for all the towns you have to support AND you will not have the customer size on which to amortize this work. Each new customer will bring a new arbitrary set of legal aspects to be met. Each new customer may be arbitrarily difficult to support. By the time you reach national, you will have already covered most of the historical legal quirks - but that will have been done in one kludgy manner after another - and then you will hit one more set of legal quirks at the level of national organizations (some of them will have their very own laws). You will now have a very large budget to finalize things but you will be burdened by an illogical software base. So I agree that you will need experience and subject matter experts that have worked at the various levels. BUT, now that you have this experience you now know the degree of flexibility that is required (you know where and what needs to be variable and quirk-friendly and how far the quirks can go = "any size") as well as size-related issues (mailing, transaction, user support volume) and you can now plan for all this AS YOU restart a new development from scratch. Because at this new "master" level you need both systematic flexibility AND relience at size. Payroll is exactly the kind of topic where "adding features" will be "fun" - I mean bewildering - while you learn, but probably economically difficult to manage, until it kills you "as you climb up"? You will be killed by a large software project that can afford to hire out a bunch of your subject matter specialists (or hires new ones) and uses them in a "from scratch" project. If you are lucky, this large project will be from the same company but only if you are lucky. Now. AFTER you have done the one top level project - for one country -, you will probably be in a good situation to sell service to all kinds of organizations. Because you now have a system in which you can implement ridiculous quirks without breaking everything. And if you have done the job just right, you can onboard smaller customers (towns) economically enough that they can afford your solution. That's different from where you deploy your solution first. Sure, deploy a national-design solution first at a subset of the target employees - although that does impose more requirements still: now you need to coexist with the legacy solutions. Which would be another hard to meet handicap when developing for towns first.
jiggawatts 11mo ago
You will never get to the moon by making a faster and faster bus. I see a lot of software with that initial small scale "baked into it" at every level of its design, from the database engine choice, schema, concurrency handling, internal architecture, and even the form design and layout. The best-engineered software I've seen (and written) always started at the maximum scale, with at least a plan for handling future feature extensions. As a random example, the CommVault backup software was developed in AT&T to deal with their enormous distributed scale, and it was the only decently scalable backup software I had ever used. It was a serious challenge with its competitors to run a mere report of last night's backup job status! I also see a lot of "started small, grew too big" software make hundreds of silly little mistakes throughout, such as using drop-down controls for selecting users or groups. Works great for that mom & pop corner store customer with half a dozen accounts, fails miserably at orgs with half a million. Ripping that out and fixing it can be a decidedly non-trivial piece of work. Similarly, cardinality in the database schema has really irritating exceptions that only turn up at the million or billion row scale and can be obscenely difficult to fix later. An example I'm familiar with is that the ISBN codes used to "uniquely" identify books are almost, but not quite unique. There are a handful of duplicates, and yes, they turn up in real libraries. This means that if you used these as a primary key somewhere... bzzt... start over from the beginning with something else! There is no way to prepare for this if you start with indexing the book on your own bookshelf. Whatever you cook up will fail at scale and will need a rethink.
- rmunn 11mo ago
  Counterpoint: the idea that your project will be the one to scale up to the millions of users/requests/etc is hubris. Odds are, your project won't scale past a scale of 10,000 to 100,000. Designing every project to scale to the millions from the beginning often leads to overengineering, adding needless complexity when a simpler solution would have worked better. Naturally, that advice doesn't hold if you know ahead of time that the project is going to be deployed at massive scale. In which case, go ahead and implement your database replication, load balancing, and failover from the start. But if you're designing an app for internal use at your company of 500, well, feel free to just use SQLite as your database. You won't ever run into the problems of scale in this app, and single-file databases have unique advantages when your scale is small. Basically: know when huge scale is likely, and when it's immensely UNlikely. Design accordingly.
chrsw 11mo ago
That sounds like the way nature handles growth and complexity: slowly and over long time scales. Assume there will be failures, don't die and keep trying. When you bite off too much complexity at once you end up not shipping anything or building something brittle.
eru 11mo ago
> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level. You could also try to buy some off-the-shelf solutions? Making payroll, even for very large organisations, isn't exactly a new problem. As a corollary I would also suggest: subsidiarity. > Subsidiarity is a principle of social organization that holds that social and political issues should be dealt with at the most immediate or local level that is consistent with their resolution. (from https://en.wikipedia.org/wiki/Subsidiarity https://en.wikipedia.org/wiki/Subsidiarity) If you solve more problems more locally, you don't need that many people at the national level, thus making payroll there is easier.
- tsimionescu 11mo ago
  I think you'll find that is exactly what people do. However, payroll solutions are highly customized for every individual company and even business unit. You don't buy a payroll software in a box, deploy it, and now you have payroll. Instead, you pay a payroll software company, they come in and get information about your payroll systems, and then they roll out their software on some of your systems and work with you to make sure their customizations worked etc. There's rarely any truly "off-the-shelf" software in B2B transactions, especially the type of end-user solutions that also interact with legal systems. Also, governments are typically at least an order of magnitude larger than the largest companies operating in their countries, in terms of employees. So sure, the government of Liechtenstein has fewer employees than Google overall, but the US government certainly does not, and even Liechtenstein probably has way more government employees than Google employees in their country.
duxup 11mo ago
I work at a small shop, I'm a big advocate of giving customers the 0.1 version and then talking it out what they want. It's often not exactly what they asked for at the start ... but it often is better in the end. It's hard to hit the target right the first time.
bryanhogan 11mo ago
You just need: Plan -> Implement -> Test -> Repeat Whether you are creating software, games or whatever, these iterations are foundational. How these steps look like in detail of course depends on the project itself.
deleted 11mo ago
[deleted]
roeles 11mo ago
Not saying you're wrong, but I wonder what is the differentiating factor for software? We can build huge things like airliners, massive bridges and buildings without starting small. Incremental makes less sense to me when you want to go to mars. Would you propose to write the software for such a mission in an incremental fashion too? Yet for software systems it is sometimes proposed as the best way.
- cheepin 11mo ago
  All of the things you mentioned are designed and tested incrementally. Furthermore software has been used on Mars missions in the past, and that software was also developed incrementally. It’s proposed as the best way because it’s a way that has a track record
  - roeles 11mo ago
    > All of the things you mentioned are designed and tested incrementally. In a different way that what is proposed in this thread. We don't build a small bridge and grow it. We build small bridges, develop a theory for building bridges and use that to design the big bridge. I don't know of any theory of computing that would help us design a "big" program at once.
- deleted 11mo ago
  [deleted]
- Ensorceled 11mo ago
  > We can build huge things like airliners, massive bridges and buildings without starting small. We did start small with all of those things. We developed rigorous disciplines around engineering, architecture, material sciences. And people died along the way in the thousands[0][1] People are still dying from those failures; The Boeing 737 MAX 9 crash was only two years ago. > Incremental makes less sense to me when you want to go to mars. This is yet another reason why a manned Mars mission will be exceedingly dangerous NOT a strike against incremental development and deployment. [0] https://en.wikipedia.org/wiki/List_of_building_and_structure_collapses https://en.wikipedia.org/wiki/List_of_building_and_structure... [1] https://en.wikipedia.org/wiki/List_of_accidents_and_incidents_involving_commercial_aircraft https://en.wikipedia.org/wiki/List_of_accidents_and_incident...
Cthulhu_ 11mo ago
That's the ideal, but a lot of these big problems can't start small because the problem they have is already big. A lot of government IT programs are set up to replace existing software and -processes, often combining a lot of legacy software's jobs and the manual labor involved. If you have something like a tax office or payroll, they need to integrate decades of legislation and rules. It's doable, but you need to understand the problem (which at those scales is almost impossible to fit in one person's head) and more importantly have diligent processes and architecture to slowly build up and deploy the software. tl;dr it's hard. I have no experience in anything that scale, I've been at the edges of large organizations (e.g. consumer facing front-ends) for most of my career.
WJW 11mo ago
While I like the "start small and expand" strategy better than the "big project upfront", this trades project size for project length and often that is no better: - It gives outside leadership types many more opportunities to add requirements later. This is nice is they are things missed in the original design, but it can also lead to massive scope creep. - A big enough project that gets done the "start small and expand" way can easily grow into a decade-plus project. For an extreme example, see the multi-decade project by the Indian rail company to gradually replace all its railways to standard gauge. It works fine if you have the organisational backing for a long duration, but the constant knowledge leaks from people leaving, retiring, getting promoted, etc can be a real problem for a project like that. Especially in fields where the knowledge is the product, like in software. - Not every project can feasibly start small.
base698 11mo ago
> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system. Gall’s law wins again.
ozim 11mo ago
There is no solution because these projects are not failing because of technical reasons. They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions. Then you have "important people" who want to be important and want to have an opinion on font size and that some button should be 12px to the right because they are "important" it doesn't matter for the project but they have to assert the dominance. You have 2 or 3 companies working on a project? Great! now they will be throwing stuff over the fence to limit their own cost and blame others while trying to get away with as least work done cashing most money possible. That is how sausage is made. Coming up with "reasonable approach" is not the solution because as soon as you get different suppliers, different departments you end up with power/money struggle.
- ethbr1 11mo ago
  Political corruption is like environmental radiation: a viable fix is never 'just get rid of political corruption'*. It's an environmental constant that needs to be handled by an effective approach. That said, parent's size- and scope-iterative approach also helps with corruption, because corruption metastasizes in the time between {specification} and {deliverable}. Shrink that, by tying incremental payments to working systems at smaller scales, and you shrink the blast radius for failure. That said, there are myriad other problems the approach creates (encouraging architectures that won't scale to the final system, promoting duct taped features on top of an existing system, vendor-to-vendor transitions if the system builder changes, etc). But on the whole, the pros outweigh the cons... for projects controlled by a political process (either public or private). That's why military procurement has essentially landed on spiral development (i.e. iterative demonstrated risk burn-down) as a meta-framework. * Limit political corruption, to the extent possible in a cost efficient manner, sure
- throw0101c 11mo ago
  > They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions. Not (necessarily) wrong, but if you start small, Important People may not want to bother with something that is Unimportant and may leave things alone so something useful and working can get going. If you starting with an Important project then Important People will start circling it right away.
throw0101c 11mo ago
> I'm afraid that the solution is: build something small, and use it in production before you add more features. Gall's Law: > A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[8] * https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law
patrick451 11mo ago
Imagine if the only way to build a skyscraper was to start with a dollhouse and keep tacking extensions and pieces onto it until. Imagine if the only way to build a bridge across San Francisco bay was to start with pop sickle sticks.
mathattack 11mo ago
I do get concerned when the solution is to be more strict on the waterfall process. I used to believe there were some worlds in which waterfalls are better: where requirements are well know in advance and set in stone. I’ve since come to realize neither of those assumptions is ever true.
hermitcrab 11mo ago
>It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features. I think that is true for a lot of projects. But I'm not sure it is realistic to incrementally develop a control system for a nuclear reactor or an air traffic control system.

solatic 11mo ago

That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen. Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.

dustingetz 11mo ago
we get paid to add to it, we don’t get paid to take away
- cjfd 11mo ago
  Now there is your problem. It is only true in the context of grave incompetence, though. I have worked on tickets with 'remove' in the title.
Jtsummers 11mo ago
Designing or intending a system to be used at massive scale is not the same as building and deploying it so that it only initially runs at that massive scale. That's just a recipe for disaster, "We don't even know if we can handle 100 users, let's now force 1 million people to use the system simultaneously." Even WhatsApp couldn't handle hundreds of millions of users on the day it was first released, nor did it attempt to. You build out slowly and make sure things work, at least if you're competent and sane.
- solatic 11mo ago
  Sure, but if you did a good job, the gradual deployment can go relatively quickly and smoothly, which is how $FAANG roll out new features and products to very large audiences. The actual rollout is usually a bit of an implementation detail of what first needed to be architected to handle that larger scale.
  - vlovich123 11mo ago
    You get certain big pieces correct maybe but you’d be surprised how many mistakes get made. For example, I had designed the billing system for a large distributed product that the engineer ended up implementing not as described in the spec which fell down fairly quickly with even a modicum of growth.
  - eru 11mo ago
    Well, Google got good at large scale rollouts, because they are doing large scale rollouts all the time. _And_ most of the time, the system they are rolling out is a small iteration from the last system they rolled out: the new GMail servers look almost exactly like the last GMail servers, but they have on extra feature flag you can turn on (and which is disabled by default) or have one bug fixed. That's a very different challenge from rolling out a brand new system once.
  - coliveira 11mo ago
    The issue with FAANG is that they already have the infrastructure to make these large scale deployments. So any new system - by necessity - needs to conform to that large scale architecture.
paulsutter 11mo ago
You have to design for scale AND deploy gradually
- rossdavidh 11mo ago
  Yes, absolutely. Knowing that it will need to get big eventually is important, but not at all the same as deploying at scale initially.
golemiprague 11mo ago
[dead]
lelandbatey 11mo ago
Gradual growth =/= many tacked on features. Many tacked on features =/= technical debt. Technical debt =/= "everybody is afraid and frozen." Those are merely often correlated, but not required. Whatsapp is a terrible example because it's barely a product; Whatsapp is mostly a free offering of goodwill riding on the back of actual products like Facebook Ads. A great example would be a product like Salesforce, SAP, or Microsoft Dynamics. Those products are forced to grow and change and adapt and scale, to massive numbers doing tons of work, all while being actual products and being software systems. I think such products act as stark rebukes of what you've described.
jimbokun 11mo ago
Yes, it can be very difficult to add “scale” after the fact, once you already have a lot of data persisted in a certain way.
otterley 11mo ago
Software is a component of a product, if not the product itself. Treating software like a product, besides being the underlying truth, also means it makes sense to manage it like one. Technical debt isn’t usually the problem people think it is. When it does become a problem, it’s best to think of it in product-like terms. Does it make the product less useful for its intended purpose? Does it make maintenance or repair inconvenient or costly? Or does it make it more difficult or even impossible to add competitive features or improvements? Taking a product evaluation approach to the question can help you figure out what the right response is. Sometimes it’s no response at all.
- jfreds 11mo ago
  Took me way too long to learn this. It still makes me sad to leave projects “imperfect” and not fiddle in my free time sometimes
- YetAnotherNick 11mo ago
  The discussion is not about the product where you can just remove the stuff. The thread was testing in small setting and then moving to oddball setting. If it is required to cover oddball settings, it makes sense to know and plan for oddball setting.
mekoka 11mo ago
> Gradual growth inevitably results in loads of technical debt. Why is this stated as though it's some de facto software law? The argument is not whether it's possible to waterfall a massive software system. It clearly is possible, but the failure ratios have historically been sufficiently uncomfortable to give rise to entirely different (and evidently more successful) project development philosophies, especially when promoters were more sensitive to the massive sums involved (which in my opinion also helps explains why so many wasteful government examples). The lean startup did not appear in a vacuum. Do things that don't scale did not become a motto in these parts without reason. In case some are still confused about the historical purpose of these benign sounding advices, no, they weren't originally addressed at entrepreneurs aiming to run "lifestyle" businesses.
- jappgar 11mo ago
  It is a law. The law of entropy. Try as you might, you cannot fight entropy eternally, as mistakes in this fight will accumulate and overpower you. It's the natural process of aging we see in every lifeform. The way life continues on despite this law is through reproduction. If you bud off independent organisms, an ecosystem can gain "eternal" life. The cost is that you must devote much of your energy to effective reproduction. In software, this means embracing rewrites. The people who push against rewrites and claim they're not necessary are just as delusional as those who think they can live forever.
  - cjfd 11mo ago
    You don't understand very much about entropy. This reasoning is very, very, very sloppy.
    - jappgar 11mo ago
      Now I remember why I stopped commenting here.
    - ebcode 11mo ago
      low-effort comment with ad hominem and zero rationale. fairly toxic.
philipallstar 11mo ago
> Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. This isn't technical debt, necessarily. Technical debt is a specific thing. You probably mean "an underlying design that doesn't perfectly map to what ended up being the requirements". But then the world moves on (what if a regulation is added that ruins your perfect structure anyway?) and you can't just wish for perfect requirements. Or not in software that interacts directly with the real world, anyway.
fatbird 11mo ago
There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism. Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine. The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."

the_duke 11mo ago

The accounting, legal and business process requirements are vastly different at different scales, different jurisdictions, different countries, etc. There's a crazy amount of complexity and customizability in systems like ERPs for multinational corporations (SAP, Oracle). When you start with a small town, you'll have to throw most of everything away when moving to a different scale. That's true for software systems in general. If major requirements are bolted on after the fact, instead of designed into the system from the beginning, you usually end up with an unmaintainable mess.

rossdavidh 11mo ago
Knowing that the rules for your first small deployment are not the same as the rules for everywhere, is valuable for designing well. Trying to implement all of those sets of rules in your initial deployment, is not a good idea. There is a general principle that you shouldn't code the abstraction until you've coded for the concrete example 2 or 3 times, because otherwise you won't make the right abstraction. Looking ahead is not the same as starting with the whole enchilada for your initial deployment.

OtherShrezzing 11mo ago

While I think this is good advice in general, I don’t think your statement that “there is no process to create scalable software” holds true. The uk gov development service reliably implements huge systems over and over again, and those systems go out to tens of millions from day 1. As a rule of thumb, the parts of the uk govt digital suite that suck are the parts the development service haven’t been assigned to yet. The Swift banking org launches reliable features to hundreds of millions of users. There’s honestly loads of instances of organisations reliably implementing robust and scalable software without starting with tens of users.

sam_lowry_ 11mo ago
SWIFT? Hold my beer. SWIFT did not launch anything substantial since its startup days in early 70-ies. Moreover, their core tech did not evolve that far from that era, and the 70-ies tech bros are still there through their progeniture. Here's an anecdote: The first messaging system built by SWIFT was text-based, somewhat similar to ASN.1. The next one used XML, as it was the fad of the day. Unfortunately, neither SWIFT nor the banks could handle 2-3 orders of magnitude increase in payload size in their ancient systems. Yes, as engineers, you would think compressing XML would solve the problem and you would by right. Moreover, XML Infoset already existed, and it defined compression as a function of the XML Schema, so it was somewhat more deterministic even though not more efficient than LZMA. But the suits decided differently. At one of the SIBOS conferences they abbreviate XML tags, and did it literally on paper and without thinking about back-and-forth translation, dupes, etc. And this is how we landed with ISO20022 abberviations that we all know and love: Ccy for Currency, Pmt for Payment, Dt for Date, etc.
- deleted 11mo ago
  [deleted]
- noname120 11mo ago
  Harder to audit when every payload needs to be decompressed to be inspected
  - 11mo ago

Izikiel43 11mo ago

What works at small scale possibly won't work at a huge scale.

skywhopper 11mo ago
But what hasn’t even been tried at a small scale definitely won’t work at a huge scale.
rossdavidh 11mo ago
Which is absolutely true, and a reason to try at medium scale second. But what doesn't work at small scale, almost certainly won't work at huge scale.

hinkley 11mo ago

The dominant factor is: there is a human who understands the entire system. That is vastly easier to achieve by making a small, successful system, which gets buy in from both users and builders to the extent that the former pay sufficient money for the latter to be invested in understanding the entire system and then growing it and keeping up with the changes. Occasionally a moon shot program can overcome all of that inertia, but the “90% of all projects fail” is definitely overrepresented in large projects. And the Precautionary Principle says you shouldn’t because the consequences are so high.

dominicrose 11mo ago
This works for Clojure, git and even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it. But these things are meant to be used by technical people. The non-technical people I know might want to use Linux but stay on Windows or choose Mac OS because it's more straightforward. I use Windows+WSL at work even though I would like to use a native Linux distribution. I know someone who created a MUD game (text online game) and said to him I wanted to make one with a browser client. He said something we could translate as "Good, you can have all the newbies." Not only was he right that a MUD should be played with a MUD client like tintin++, but making a good browser client is harder than it seems and that's time not spent making content for the game or improving the engine. My point is that he was un uncomprimising person who refused adding layers to a project because they would come at a cost which isn't only time or dollars but also things like motivation and focus.
- wickedsight 11mo ago
  > ... even Linux. It seems there's a human who understands the entire system and decides what's allowed to be added to it. I really wonder what will happen to Linux once Linus is no longer involved.
  - inemesitaffia 11mo ago
    Greg KH

chrisweekly 11mo ago

See also Gall's Law: "All complex systems that work evolved from simpler systems that worked"

shagie 11mo ago

> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level. At a large box retail chain (15 states, ~300 stores) I worked on a project to replace the POS system. The original plan had us getting everything working (Ha!) and then deploying it out to stores and then ending up with the two oddball "stores". The company cafeteria and surplus store were technically stores in that they had all the same setup and processes but were odd. When the team that I was on was brought into this project, we flipped that around and first deployed to those two several months ahead of the schedule to deploy to the regular stores. In particular, the surplus store had a few dozen transactions a day. If anything broke, you could do reconciliation by hand. The cafeteria had single register transaction volume that surpassed a surplus store on most any other day. Furthermore, all of its transactions were payroll deductions (swipe your badge rather than credit card or cash). This meant that if anything went wrong there we weren't in trouble with PCI and could debit and credit accounts. Ultimately, we made our deadline to get things out to stores. We did have one nasty bug that showed up in late October (or was it early November?) with repackaging counts (if a box of 6 was $24 and if purchased as a single item it was $4.50 ... but if you bought 6 single items it was "repackaged" to cost $24 rather than $27) which interacted with a BOGO sale. That bug resulted in absurd receipts with sales and discounts (the receipt showed you spent $10,000 but were discounted $9,976 ... and then the GMs got alerts that the store was not able to make payroll because of a $9,976 discount ... one of the devs pulled an all nighter to fix that one and it got pushed to the stores ). I shudder to think about what would have happened if we had tried to push the POS system out to customer facing stores where the performance issues in the cafeteria where worked out first or if we had to reconcile transactions to hunt down incorrect tax calculations.

einpoklum 11mo ago
You could have, in principle, implemented the new system to be able to run in "dummy mode" alongside the existing system at regular stores, so that you see that it produces the 'same' results in terms of what the existing system is able to provide. Which is to say, there is more than one approach to gradual deployment.

hintymad 11mo ago

> https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 This is what https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 advocates too: start small, modularize, and then scale. The example of Tesla's mega factory was particular enticing.

nostrademons 11mo ago

Came here to say this. I still think that Linus Torvalds has the most profound advice to building a large, highly successful software system: "Nobody should start to undertake a large project. You start with a small trivial project, and you should never expect it to get large. If you do, you'll just overdesign and generally think it is more important than it likely is at that stage. Or worse, you might be scared away by the sheer size of the work you envision. So start small, and think about the details. Don't think about some big picture and fancy design. If it doesn't solve some fairly immediate need, it's almost certainly over-designed. And don't expect people to jump in and help you. That's not how these things work. You need to get something half-way useful first, and then others will say "hey, that almost works for me", and they'll get involved in the project." -- Linux Times, October 2004.

ambicapter 11mo ago
This is a really dense paragraph of lifetime-accumulated wisdom in that single quote.
tsimionescu 11mo ago
I don't think this applies in any way to companies contracted to build a massive system for a government with a clear need. Linus is talking about growing a greenfield open-source project, which may or may not ever be used by anyone. In contrast, if your purpose is "we need to manage our country's accounting without pen and paper", that's a clear need for a massive system. Starting work on this by designing a system that can solve accounting for a small firm is not the right way to go. Instead, you have to design with the end-goal in mind, since that's what you were paid for. But, you don't launch your system to the entire country at once: you first use this system designed for a country in a small shop, to make sure it actually handles the small scale well, before gradually rolling out to more and more people.
- mrweasel 11mo ago
  > for a government with a clear need. There's your problem. The needs are never clear, not on massive systems. Governments will write a spec, companies will read the spec, offer to implement it as written, knowing full well that it won't work. Then they charge exorbitant fees to modify the system after launch, so that it will actually full fill business needs. The Danish government is famous for sucking at buying massive IT systems. * Specs for new tax system: 6000 page, tax laws not included. That's basically impossible to implement and it predictably failed. The version that worked: Implement just the basics to collect TV license fees. The build from there. * System to calculate the value of people home, I think we're at round five (rumors has it that one system worked, but was scrapped because it showed that most home are massively overvalued and it do terrible things to the tax collection in the municipalities). * New case management system for the police, failed, development never restarted. One suggested solution was to have the police hire a handful of the best developers in the country and have them produce smaller deliverable over a number of year. The money wasted could have funded 10 world class developers for ~30-50 years.

BrenBarn 11mo ago

Yes. Also the same applies to companies. There should not be companies that are growing to $100 million revenue while losing money on a gamble that they will eventually get big enough to succeed. Good first, big later.

SchemaLoad 11mo ago
$100M maybe. But pretty much all tech needs an initial investment before you can start making profit. It takes a lot of development before you can get a product that anyone would want to pay for.

creer 11mo ago

The very specific example you chose: payroll, shows how it can be difficult to incrementally step from small to huge. As you grow from town to national, you will run into all the disadvantages without really hitting the advantages. I feel that incremental does help you move from one level to one just a few above. But only if there are enough customers at these starting levels exactly. When developing for towns, you will have all small random subsets of the variations imposed by year after year of legal changes BUT small sales. You will have to implement niche variations in arbitrary aspects for all the towns you have to support AND you will not have the customer size on which to amortize this work. Each new customer will bring a new arbitrary set of legal aspects to be met. Each new customer may be arbitrarily difficult to support. By the time you reach national, you will have already covered most of the historical legal quirks - but that will have been done in one kludgy manner after another - and then you will hit one more set of legal quirks at the level of national organizations (some of them will have their very own laws). You will now have a very large budget to finalize things but you will be burdened by an illogical software base. So I agree that you will need experience and subject matter experts that have worked at the various levels. BUT, now that you have this experience you now know the degree of flexibility that is required (you know where and what needs to be variable and quirk-friendly and how far the quirks can go = "any size") as well as size-related issues (mailing, transaction, user support volume) and you can now plan for all this AS YOU restart a new development from scratch. Because at this new "master" level you need both systematic flexibility AND relience at size. Payroll is exactly the kind of topic where "adding features" will be "fun" - I mean bewildering - while you learn, but probably economically difficult to manage, until it kills you "as you climb up"? You will be killed by a large software project that can afford to hire out a bunch of your subject matter specialists (or hires new ones) and uses them in a "from scratch" project. If you are lucky, this large project will be from the same company but only if you are lucky. Now. AFTER you have done the one top level project - for one country -, you will probably be in a good situation to sell service to all kinds of organizations. Because you now have a system in which you can implement ridiculous quirks without breaking everything. And if you have done the job just right, you can onboard smaller customers (towns) economically enough that they can afford your solution. That's different from where you deploy your solution first. Sure, deploy a national-design solution first at a subset of the target employees - although that does impose more requirements still: now you need to coexist with the legacy solutions. Which would be another hard to meet handicap when developing for towns first.

jiggawatts 11mo ago

You will never get to the moon by making a faster and faster bus. I see a lot of software with that initial small scale "baked into it" at every level of its design, from the database engine choice, schema, concurrency handling, internal architecture, and even the form design and layout. The best-engineered software I've seen (and written) always started at the maximum scale, with at least a plan for handling future feature extensions. As a random example, the CommVault backup software was developed in AT&T to deal with their enormous distributed scale, and it was the only decently scalable backup software I had ever used. It was a serious challenge with its competitors to run a mere report of last night's backup job status! I also see a lot of "started small, grew too big" software make hundreds of silly little mistakes throughout, such as using drop-down controls for selecting users or groups. Works great for that mom & pop corner store customer with half a dozen accounts, fails miserably at orgs with half a million. Ripping that out and fixing it can be a decidedly non-trivial piece of work. Similarly, cardinality in the database schema has really irritating exceptions that only turn up at the million or billion row scale and can be obscenely difficult to fix later. An example I'm familiar with is that the ISBN codes used to "uniquely" identify books are almost, but not quite unique. There are a handful of duplicates, and yes, they turn up in real libraries. This means that if you used these as a primary key somewhere... bzzt... start over from the beginning with something else! There is no way to prepare for this if you start with indexing the book on your own bookshelf. Whatever you cook up will fail at scale and will need a rethink.

rmunn 11mo ago
Counterpoint: the idea that your project will be the one to scale up to the millions of users/requests/etc is hubris. Odds are, your project won't scale past a scale of 10,000 to 100,000. Designing every project to scale to the millions from the beginning often leads to overengineering, adding needless complexity when a simpler solution would have worked better. Naturally, that advice doesn't hold if you know ahead of time that the project is going to be deployed at massive scale. In which case, go ahead and implement your database replication, load balancing, and failover from the start. But if you're designing an app for internal use at your company of 500, well, feel free to just use SQLite as your database. You won't ever run into the problems of scale in this app, and single-file databases have unique advantages when your scale is small. Basically: know when huge scale is likely, and when it's immensely UNlikely. Design accordingly.

chrsw 11mo ago

That sounds like the way nature handles growth and complexity: slowly and over long time scales. Assume there will be failures, don't die and keep trying. When you bite off too much complexity at once you end up not shipping anything or building something brittle.

eru 11mo ago

tsimionescu 11mo ago
I think you'll find that is exactly what people do. However, payroll solutions are highly customized for every individual company and even business unit. You don't buy a payroll software in a box, deploy it, and now you have payroll. Instead, you pay a payroll software company, they come in and get information about your payroll systems, and then they roll out their software on some of your systems and work with you to make sure their customizations worked etc. There's rarely any truly "off-the-shelf" software in B2B transactions, especially the type of end-user solutions that also interact with legal systems. Also, governments are typically at least an order of magnitude larger than the largest companies operating in their countries, in terms of employees. So sure, the government of Liechtenstein has fewer employees than Google overall, but the US government certainly does not, and even Liechtenstein probably has way more government employees than Google employees in their country.

duxup 11mo ago

I work at a small shop, I'm a big advocate of giving customers the 0.1 version and then talking it out what they want. It's often not exactly what they asked for at the start ... but it often is better in the end. It's hard to hit the target right the first time.

bryanhogan 11mo ago

You just need: Plan -> Implement -> Test -> Repeat Whether you are creating software, games or whatever, these iterations are foundational. How these steps look like in detail of course depends on the project itself.

deleted 11mo ago

[deleted]

roeles 11mo ago

Not saying you're wrong, but I wonder what is the differentiating factor for software? We can build huge things like airliners, massive bridges and buildings without starting small. Incremental makes less sense to me when you want to go to mars. Would you propose to write the software for such a mission in an incremental fashion too? Yet for software systems it is sometimes proposed as the best way.

cheepin 11mo ago
All of the things you mentioned are designed and tested incrementally. Furthermore software has been used on Mars missions in the past, and that software was also developed incrementally. It’s proposed as the best way because it’s a way that has a track record
- roeles 11mo ago
  > All of the things you mentioned are designed and tested incrementally. In a different way that what is proposed in this thread. We don't build a small bridge and grow it. We build small bridges, develop a theory for building bridges and use that to design the big bridge. I don't know of any theory of computing that would help us design a "big" program at once.
deleted 11mo ago
[deleted]
Ensorceled 11mo ago
> We can build huge things like airliners, massive bridges and buildings without starting small. We did start small with all of those things. We developed rigorous disciplines around engineering, architecture, material sciences. And people died along the way in the thousands[0][1] People are still dying from those failures; The Boeing 737 MAX 9 crash was only two years ago. > Incremental makes less sense to me when you want to go to mars. This is yet another reason why a manned Mars mission will be exceedingly dangerous NOT a strike against incremental development and deployment. [0] https://en.wikipedia.org/wiki/List_of_building_and_structure_collapses https://en.wikipedia.org/wiki/List_of_building_and_structure... [1] https://en.wikipedia.org/wiki/List_of_accidents_and_incidents_involving_commercial_aircraft https://en.wikipedia.org/wiki/List_of_accidents_and_incident...

Cthulhu_ 11mo ago

That's the ideal, but a lot of these big problems can't start small because the problem they have is already big. A lot of government IT programs are set up to replace existing software and -processes, often combining a lot of legacy software's jobs and the manual labor involved. If you have something like a tax office or payroll, they need to integrate decades of legislation and rules. It's doable, but you need to understand the problem (which at those scales is almost impossible to fit in one person's head) and more importantly have diligent processes and architecture to slowly build up and deploy the software. tl;dr it's hard. I have no experience in anything that scale, I've been at the edges of large organizations (e.g. consumer facing front-ends) for most of my career.

WJW 11mo ago

While I like the "start small and expand" strategy better than the "big project upfront", this trades project size for project length and often that is no better: - It gives outside leadership types many more opportunities to add requirements later. This is nice is they are things missed in the original design, but it can also lead to massive scope creep. - A big enough project that gets done the "start small and expand" way can easily grow into a decade-plus project. For an extreme example, see the multi-decade project by the Indian rail company to gradually replace all its railways to standard gauge. It works fine if you have the organisational backing for a long duration, but the constant knowledge leaks from people leaving, retiring, getting promoted, etc can be a real problem for a project like that. Especially in fields where the knowledge is the product, like in software. - Not every project can feasibly start small.

base698 11mo ago

> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system. Gall’s law wins again.

ozim 11mo ago

There is no solution because these projects are not failing because of technical reasons. They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions. Then you have "important people" who want to be important and want to have an opinion on font size and that some button should be 12px to the right because they are "important" it doesn't matter for the project but they have to assert the dominance. You have 2 or 3 companies working on a project? Great! now they will be throwing stuff over the fence to limit their own cost and blame others while trying to get away with as least work done cashing most money possible. That is how sausage is made. Coming up with "reasonable approach" is not the solution because as soon as you get different suppliers, different departments you end up with power/money struggle.

ethbr1 11mo ago
Political corruption is like environmental radiation: a viable fix is never 'just get rid of political corruption'*. It's an environmental constant that needs to be handled by an effective approach. That said, parent's size- and scope-iterative approach also helps with corruption, because corruption metastasizes in the time between {specification} and {deliverable}. Shrink that, by tying incremental payments to working systems at smaller scales, and you shrink the blast radius for failure. That said, there are myriad other problems the approach creates (encouraging architectures that won't scale to the final system, promoting duct taped features on top of an existing system, vendor-to-vendor transitions if the system builder changes, etc). But on the whole, the pros outweigh the cons... for projects controlled by a political process (either public or private). That's why military procurement has essentially landed on spiral development (i.e. iterative demonstrated risk burn-down) as a meta-framework. * Limit political corruption, to the extent possible in a cost efficient manner, sure
throw0101c 11mo ago
> They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions. Not (necessarily) wrong, but if you start small, Important People may not want to bother with something that is Unimportant and may leave things alone so something useful and working can get going. If you starting with an Important project then Important People will start circling it right away.

throw0101c 11mo ago

> I'm afraid that the solution is: build something small, and use it in production before you add more features. Gall's Law: > A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[8] * https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law

patrick451 11mo ago

Imagine if the only way to build a skyscraper was to start with a dollhouse and keep tacking extensions and pieces onto it until. Imagine if the only way to build a bridge across San Francisco bay was to start with pop sickle sticks.

mathattack 11mo ago

I do get concerned when the solution is to be more strict on the waterfall process. I used to believe there were some worlds in which waterfalls are better: where requirements are well know in advance and set in stone. I’ve since come to realize neither of those assumptions is ever true.

hermitcrab 11mo ago

>It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features. I think that is true for a lot of projects. But I'm not sure it is realistic to incrementally develop a control system for a nuclear reactor or an air traffic control system.