9 ms·

The disproportionate influence of early tech decisions

184 points by kiyanwang 4y ago

Nomentatus 4y ago
"path dependence" - nobody has said the words that are the proper rubric for this discussion, so I thought I would. If you want to find some of the previous bibliography on this subject, them's the magic words.
agentultra 4y ago
Early tech decisions can become myopic in the long run. If you're not careful and admit that you made quick decisions early on these early decisions can become entrenched as gospel. This can alienate later, more experienced developers and become a factor in burnout as they are forced to deal with, "this is how we do things around here." One way to deal with tech debt is to take advantage of the experience and skill of your developers and embrace change. You might not want to be switching databases every time someone suggests, "X is better than Y," for sure! Inertia has its uses. But you don't want to stagnate either. Conclusion of tfa is solid: the truth is somewhere in the middle. The keyword for me is sufficient: the local minimum that meets all of our requirements. If we need to be able to be able to ship changes faster and it takes weeks to get changes merged, something is not sufficient in our stack/process/etc.
- synu 4y ago
  It can be helpful to write down the context of why you made your decision, so that if/when you revisit it later it’s more clear. ADRs are one way to do this: https://adr.github.io/madr/ https://adr.github.io/madr/
  - agentultra 4y ago
    Great advice. I also use learning milestones to manage our novelty budget and record our findings/decisions/context/etc. I don't know if this is formalized or written down anywhere. The idea is that when your team wants to try something and learn from it you record down your thoughts and ideas into a document. You then have a triggering condition in that document for when to review it later. When you do the review you record what you learned and what decisions you make as a result. Over time your teams build up little libraries of these learnings and it can help build organizational continuity. People come and go, leaving teams or the company, joining other teams, etc. It's a good practice to avoid having to re-transmit what your organization has learned through re-telling folk-lore.
  - troelsSteegin 4y ago
    Short of something structured, any format that captures key assumptions and intentions is good. The problem we are solving is X. The "pre-mortem" is the idea is that if we fail, it would be because of Y. Then in your design or decision, you address Y as a contingency. The collateral idea for code would be "good until" - this is good until (we expect) some threshold.
  - hinkley 4y ago
    I've run into at least a few people who can't understand why my answer changes when the context changes. I said we were going to do A because of C, D and E. Why are you acting like it's a gotcha when we find out E doesn't apply anymore and now I agree B is a better option? Did you think we were joking when we said, "It depends"?
- BurningFrog 4y ago
  One solution is to design things so swapping out one component isn't harder than it needs to be. Easier said than done, but still quite doable with modular design, which you should do anyway.
  - 8n4vidtmkvmk 4y ago
    that's also how we end up with an excessive number of microservices
- hinkley 4y ago
  Some of the most uselessly toxic behavior is defending X now because it was critical to our survival 3 years ago. This world is absolutely rife with ideas, actions and substances that are appropriate to a situation but are detrimental or even deadly once that moment has passed. Oxy because of a bone graft is not the same as Oxy because it's a Tuesday. We aren't questioning your wisdom because you did this thing three years ago. We are questioning your wisdom because you won't let us kill it right now.
- dilyevsky 4y ago
  There’s a flip side to this when people who come later totally underestimate the problem space and difficulty when trying to “fix” things.
  - agentultra 4y ago
    Definitely and highlights another reason why it’s important to document these things for future maintainers and developers.
    - dilyevsky 4y ago
      Good in theory but in practice nobody is going to have the time to document every one of hundreds (or thousands) little decisions with potentially large implications while trying to keep the product afloat. Even for those they do, in my experience, nobody bothers to read them...
      - agentultra 4y ago
        Can’t help people who don’t want to help themselves.
        dilyevsky 4y ago
        It’s true but it’s worth keeping in mind that there are some underwater rocks that OGs didn’t tell anyone bc they had better things to do
- aledalgrande 4y ago
  Had to deal with this in the past, where junior engs made choices on the initial architecture and by the time I joined they were in management/high roles. They refused any change that was too complicated for them to understand and never tried to improve their knowledge. Meantime our team was suffering under the load of technical debt vs non technical POs wanting to dish out features experiments fast. > this is how we do things around here resonates so much
andsoitis 4y ago
“Software has inertia.”
throw1234651234 4y ago
"Cloud provider: AWS originally, and with so much custom infrastructure built on its APIs, unlikely to ever change." Who cares? It literally doesn't matter unless you are doing something very, very specific with ML. As far as line-of-business projects go, every service you need has a direct equivalent. Also, NO ONE EVER SWITCHES CLOUDS, this is a TERRAFORM MYTH. Terraform being USEFUL in Cloud transition is a MYTH. I have direct experience with this where a corp started competing with Amazon (once Amazon went into groceries) and had to move to Azure. This is the ONLY instance where it made sense. "The underlying database: Mongo originally begot more Mongo, but switching any database anywhere is unusual." RDS is best 90% of the time. Other than that, just keep it modular. Definitely no MVC front ends married to the BE - API always.
- Rafert 4y ago
  People do switch clouds, but sure it's never trivial. https://about.gitlab.com/blog/2019/05/02/gitlab-journey-from-azure-to-gcp/ https://about.gitlab.com/blog/2019/05/02/gitlab-journey-from... for example.
  - throw1234651234 4y ago
    Thanks for the link! I also know a national grocery store that switches from AWS to Azure once Amazons started competing with them directly. With that said though, I think it's very, very rare and I hate how people use it as an argument to justify Terraform, which largely doesn't help because all the APIs are different enough to still require manual adjustments.
thinkingkong 4y ago
Its pretty easy to find flaws or old parts in any stack. The trick is figuring out which ones need replacing, in which order, with an ROI you can justify. Most of the time you can’t because that same time is better used for other things. Its not to say it’s not possible. Just that most ic’s, managers, pms, etc dont build careers on proactive technical debt.
- deleted 4y ago
  [deleted]
chubot 4y ago
I'm a bit surprised that Stripe uses Mongo? For financial transactions? Is it the primary backing store or a cache? I always thought of Mongo as the DB with questionable durability and performance claims ... i.e. the "quick and dirty" DB. And also under-featured compared to relational DBs. But I haven't used it so maybe that is not accurate.
- gqewogpdqa 4y ago
  MongoDB is used by enterprises all over the world. It's true that it's the easiest DB to get started with, and it's also true that in the distant past (pre 3.0, back in ~2015), they had some issues. But they now run the financial backends of companies as varied as banks and coinbase and... https://www.mongodb.com/who-uses-mongodb https://www.mongodb.com/who-uses-mongodb
  - jandrewrogers 4y ago
    MongoDB 3.x was broken too[0]. Their reputation for consistently shipping databases with fundamental design defects is well-deserved. Data loss was a hallmark of that product across multiple major versions. Maybe they've suddenly turned a new leaf in the last couple years but I wouldn't want to bet my business on it. Most of the companies I know that use it, use it for data that doesn't matter such that data loss isn't a big deal. [0] https://jepsen.io/analyses/mongodb-3-4-0-rc3 https://jepsen.io/analyses/mongodb-3-4-0-rc3
    - docmechanic 4y ago
      I wrote for the technical writing team at MongoDB when this "security issue" made the news. The precise problem was whether authentication was turned off/on by default for the default database, I believe, during installation. Product management initially chose this configuration to make it quicker for evaluators to get something up and running as quick as possible. The assumption was that no developer would actually use that in production. Further, the documentation clearly stated that this default configuration option was not secure. Blaming that data loss on MongoDB rather than the noob dev who deployed that evaluation configuration is as erroneous as the logic used by the noob dev. We hear it from developers all the time, so let me repeat it: User error. It's a thing.
KerrAvon 4y ago
AWS is early? I was thinking about all the crap Unix did in the 1970’s that we’re still stuck with. The tty subsystem, for one example.
- Jtsummers 4y ago
  It was an early decision in Stripe's history, which is the example system in the article.
nine_k 4y ago
Heh. Read about cosmic microwave background: it's a print of some microscopic disturbances in the baby-age Universe, now permeating all of it. It's just how this world works. Tiny early events grow to colossal scale as the thing that experienced it grows, be it a code base, a company's culture, biological structures, all the way up. No matter what, we have to put up with that, and tread lightly when anything of consequence depends on us. For which, of course, there's never enough time.
jitl 4y ago
This article is timely for me! I think about this a lot while working on Notion — and I’m drafting an RFC for a change that future engineers will either praise or curse. Right now the eng team is ~150, so we’re exiting the period when an experienced senior engineer (ie me) can easily make a profound tech stack change. Our database (Postgres) and backend stack (Express/Typescript) are already quite solidified. There is still time to change our front-end stack, although React is probably a given, we can change our API contract layer, our state management pattern, and possibly our client-side replicated ORM. It’s tricky to weigh continued speed today versus long-term speed. I think a lot about the tradeoff of “invest in infra improvements below the interface line” and “change the interface now, while there’s still time” when it comes to our ORM/state manager. We have a bunch of interesting challenges across all layers (see https://www.notion.so/blog/data-model-behind-notion https://www.notion.so/blog/data-model-behind-notion) so it’s not obvious what changes today will really change the constraints on our solutions tomorrow. My current philosophy is that we need to add new abstraction layers now if there’s a chance we need them, before detangling later becomes an insurmountable lift.
- majormajor 4y ago
  Do you need (more?) new abstraction layers or do you need extremely clear un-abstracted separations of concern? In the cases of trying to build abstraction layers in advance "in case we need them" that I've worked on, they rarely are sufficiently right-shaped for what the future feature need ends up being. Having more direct access to the primitives usually works better in terms of avoiding "the business logic is split across three layers of service calls" type of situations.
  - jitl 4y ago
    We share the same record types (primitives?) across the stack - but because of that sharing, those types are hard to evolve, especially when it comes to their semantics. Too much code interprets then directly, which ossifies those primitives. How to enforce separation of concern when such data structure is shared liberally? > the business logic is split across three layers of service calls we are thankfully very far from this world.
helsinkiandrew 4y ago
Early decisions in every field have disproportionate influence, not because they were the 'right' or the best choice but because its much easier not to change, or its easier to improve the existing choice (making the current database faster) than jumping to another. a) The politicians who drafted the US constitution probably have had more impact than most presidents since. b) Train tickets, timetables, having different class carriages, ticket offices and train guards is mostly unchanged from when introduced by GWR in the 1840s c) Car peddles, typewriter keyboards, pizza recipes. etc
- photochemsyn 4y ago
  The US Constitution is a good example. While it's a politically controversial opinion, if the framers had extended 'indentured servitude' status to all slaves in the colonies at the time (a common status for many European migrants in the Middle Colonies, generally involving a ten-year period of forced labor before freedom was granted), it's entirely possible that the US Civil War could have been avoided. Some argue that southern slave states would never have gone along with this, however. The distribution of free laborers, indentured servants, and chattel slaves across the American colonies in the 17th century is a very interesting history that's typically neglected in education: https://www.oercommons.org/authoring/6734-slavery-and-indentured-servitude-in-the-american-c/view https://www.oercommons.org/authoring/6734-slavery-and-indent... Another example might be how the plantation system took hold in the American South but not in the New England Colonies.
  - rrrrrrrrrrrryan 4y ago
    America may not have become the absolute economic powerhouse that it became if it wasn't for the 2+ centuries of uncompensated labor, but IIRC America was the only nation that had to fight a war to end slavery. Most nations simply bought the slaves from the slaveowners and then freed them, which might have actually been cheaper than fighting the Civil War.
    - bpodgursky 4y ago
      The south wasn't a meaningful contributor to the US's economic powerhouse status. That's the whole reason they lost the civil war — it was an economic backwater with almost no factories, and what useful goods they traded abroad were fairly easily replaced (so no European country bothered trying to help them).
michaelfeathers 4y ago
Often this dynamic is called path-dependence too.
deleted 4y ago
[deleted]
hosh 4y ago
This is why I like playing Go (the game). The decisions made early in the game has huge effects on the rest of the game. Lingering effects of dead formations (aji) comes back to haunt and shape future decisions. Playing Go exercises this, and helps develops the intuition on the effects of architecture, outside of the game. I am again also reminded of Christopher Alexander’s ideas in living architecture.
- naet 4y ago
  Go is the most beautiful game. Chess is the dominant game where I live, and it is fun to play quick blitz games and look for tactics, but the feeling is nothing like a game of Go. As far as early decisions, one of the most important in Go is knowing when to rethink your plans. You may place a stone with the intention of defending it, but if the board state changes, it might end up serving better as a local influencer or a sacrificial stone instead. To that extent, your early decisions are key, but also you have to remain flexible and attentive to the evolving game as moves are played, adapting your early decisions to unexpected later complications. Playing a good Go game feels almost zen or enlightening; you pick your moves with careful determination, but your plans flow like water around the fluctuating board state. There's a reason there are so many Go "proverbs", something in the game mechanics makes it a perfect metaphor for life or the universe.
  - Nomentatus 4y ago
    Or it's wildly discouraging as you discover that 'cause you're human, you "tunnel" to parts of the board too much and you just can't stop it. AI plays very differently than any human, by not-tunnelling. Associative meat-minds don't seem to work that way.
photochemsyn 4y ago
> "And this fails the other way too, where major believers in academic-level correctness agonize over details to such a degree that projects never ship, and sometimes never even start." Anxiety paralysis. This is also why many people aren't too enthusiastic about the hottest new thing, and would prefer to stick with say, a language like C for low-level development. Experts in the field trying to push the boundaries aren't necessarily good examples for everyone else to follow.
redwood 4y ago
I like to describe this kind of thing via the analogy of the big bang: the little ripples in the plasma, after expansion, led to the galaxies that we see today, the smallest part of early state have MASSIVE implications later on
bestinterest 4y ago
I feel another point is that early tech decisions have a huge impact on the type of people you will have to hire. For example, Ruby attracts ruthless productivity and Go attracts people who prefer longer standing apps with less dependencies and maintainability. I make it sound like Go may be the 'better' choice here but that is not the case, as the author mentions, it's a balance.
- deleted 4y ago
  [deleted]
3pt14159 4y ago
Some early tech decisions matter, but most don't matter as much as people think they do. Other than the OS, I haven't really seen it be a complete deal breaker for anyone that's willing to put in the work in the first five years. Postgres not handling things well? Slow migration to DynamoDB or to sharding. Python falling over? Roll the hot codepaths in Cython or horizontally scale. Node doesn't have great data science libraries? No worries, build a couple of small python data science services and get on with life. The only reason I think the OS choice matters at all is because I've never seen someone move from .NET on Windows to Linux or visa versa. Could be done, but people generally don't do it. So your hiring pool is going to look different. Generally more scrapy on the linux side, generally more careful on the Windows side, but even there that's mostly about culture and priorities. Both stacks can handle basically anything.
- barking_biscuit 4y ago
  >The only reason I think the OS choice matters at all is because I've never seen someone move from .NET on Windows to Linux or visa versa. Could be done, but people generally don't do it. Done it myself. Seen it being done all the time in my org. We're migrating large amounts of legacy .NET Framework apps running on Windows on EC2 to .NET Core / .NET apps running in our K8s cluster using Linux containers.
FigmentEngine 4y ago
Inertia is a thing, especially around mental models and ways of working. Its hard to change an early decision, because many other parts and people have subsequently made decisions ontop of that. Like the ship of theseus, replacing parts is possible, but the replacement needs to fit in the gap, and the inertia means even if you replace everything, its always a ship.
mikewarot 4y ago
Even so, it's still better to start something and make mistakes (thus gaining experience) rather than getting stuck in analysis paralysis. Technical debt is real, and has interest rate that would make a loan shark blush, keep that in mind, and you should be just fine.
Clubber 4y ago
>these were known to be very imperfect implementations even when they were originally added, but they were added anyway in the name of velocity, with an implicit assumption baked in that they’d be shored up and improved later I've heard this a million times but don't remember ever seeing it actually done, at least on full coverage. This is why it's critically important to get the right first team of developers. A team who also knows this strategy is rarely implemented. Don't feel too bad, it's probably in every live codebase on the planet, just a matter of degree.
JoshCole 4y ago
In counterfactual regret minimization and in reinforcement learning this shows up too. The solution there tends to discount early iterations with something like a weighted average. It makes convergence to the true values happen faster. A similar thing happens in k mean clustering but is more horrifying because you can get arbitrary wrong solutions rather than just taking longer to converge. The solution there is to construct a sampling distribution when making your initial decision - instead of making a decision according to the principle of indifference to instead make a decision after getting more informed about the problem. This doesn't mean you don't have bad solutions, but it gives a bounding on just how bad those solutions can be. It happens in other places too, I'm sure, because this isn't really a software problem - this is a fundamental problem. Uniformed choices are uninformed but influence the choices that come after them. Certain heuristics like delaying your decision to the last responsible moment can help to avoid making your decisions with insufficient information, but it is important to realize the type of decision you are making. Are you in a case where getting things wrong just means you go slower but eventually converge? Or at you in a case where you can't reverse course and will have your outcomes determined by the decision you make without any guarantee that it will turn out well and no simple path backward. I hear people talk about this with terms like "one way doors" versus "two way doors". In the latter case you are much better off thinking more deeply through the problem. We're not really solving new fundamental problems. We're solving fundamental problems in a new context.
TeeMassive 4y ago
I worked for a company that opted for Windows Embedded back in the early 2000s, probably inspired by Tektronix scopes due to them also making scientific instruments. They boomed. They must have had good relationships with Microsoft because they produced kernel patches just for them. Anyway, 15 years later of "it works now, why change it", they start a new project to rewrite the cash cow. It was a Phoenix Project; and it also had a name associated with rebirth, as most of these projects do. Turns out, Windows Embedded is still not a good idea 20 years later, and they couldn't change their code because it was so tightly coupled with Windows. The project took 250% more time than budgeted and even then it was a MVP. A competitor decided to do the same product but in Yocto Linux using modern tech and remade the cash cow product in less than their budgeted time. The interface was snappy, there was barely no bugs, etc. Software entropy is real, and I've seen it in many startups that got bought out after they boomed. Then the entropy is left for those who weren't there during booming phase as a poisoned gift. I like to call it tech inflation.
- robocat 4y ago
  To me, your story is “Linux good, Windows bad” and you have created a narrative around that. Anecdotally, I have seen multiple successful greenfields rewrites on Windows. I am sure we could find plenty of examples of unsuccessful rewrites from Windows onto Linux. That is: your cause/effect analysis is probably incorrect. Rewrites are hard, and the OS is probably only a minor role. Disclaimer: I am a long time Windows hater, but Windows has been used to put food on my table.
  - TeeMassive 4y ago
    It's more about the right tool for the job. Also, was your greenfield projects in Windows Embedded specifically?
robertlagrant 4y ago
It still blows my mind that Facebook invented a better PHP runtime instead of carving off bits of functionality into other technologies.
- jitl 4y ago
  They did that too.
sporkland 4y ago
This has always felt like the catch 22/anthropic principal of software development to me: 1. Early on you favor decisions that enable rapid iteration and growth without being able to put in much deep thought around long term productivity. Most of these tend to be fine in the long run but there's probably at least 1 or 2 that will haunt the company for the rest of its days. Companies that don't take these "short cuts" will probably kill the business. 2. Companies that survive largely have their architecture dictated by #1 and are fairly constrained in how radically they can change the decisions in #1. So most of us end up working on a product which has some pretty critical architectural flaws that are very hard to repair. The only companies I've seen that do work their way out of it are the printing presses (google, linkedin, yahoo, amazon, etc) that are able to invest in very strong platforms and move their applications onto them, and even then it's a long drawn out process.
- HeyLaughingBoy 4y ago
  There's a way out of this, but it requires discipline and besides, most companies don't want to do it. Prototype. Deliberately build "one to throw away." Iterate fast and figure out what you need. When you've learned the important lessons about architecture, structure and collaborations, make a note of those lessons in a design document and then build the "real" product with proper engineering, but with a lot less risk. I worked for an organization that did this and it results in good quality software, but you need buy-in at all levels of management or you run into "why did your team just waste 6 months on throw-away work?" questions.
  - anthonypasq 4y ago
    arent you just describing the business from #1 that does this method and fails because they have no money and no customers?
    - n42 4y ago
      It’s important to time this well and start doing it as soon as you have your feet on the ground, and the hard part is to hit the ground running. Too early and you crash, too late and you grind to a halt edit: missed “6 months”, I live in another world — a week to prototype and throw it out. spending 6 months to prototype anything and you’re biting too big
ggm 4y ago
I have taken projects of significance to me from MySQL to Postgres. It was certainly a short-term cost but the upside justified it. (first class IPv6 objects, JSON) I have tried to do the language port thing Several times. I never succeeded in bringing others with me (Perl to python) but that said, others I work with have done Perl -> Java and C -> Perl -> C and Perl -> Java -> Rust so I think language may be one of the "tractable" problems in many cases. We tried AWS for low-end k8s and it was too complex. GCP/GKE was just better presented. It has the same complexity but gcloud/kubectl seemed to work "better" for us, which led directly to helm outcomes we liked. I think the AWS experience we had could be called at best "toe in the water" maybe try before you buy works? The one which has proved the most sticky is VM/Cloud provider, for routing. If you are in a world which cares about BGP, the one you pick that works, you probably stay with. We partnered with CloudFlare for some stuff, and Linode for other stuff. I do wish at times we'd gone digitalocean, or fastly, or even Akamai (which is behind Linode apparently) -but the BGP surface we have, is the one we want to maintain because we understand it. Maybe it's not that different to AWS vs Azure vs GCP. Also DNS provider. You tend to stick to the underlying NS and related registrar inside your chosen namespace registry. Also Cryptography TA. Well.. once you move to letsencrypt you stop moving I guess. I do think the DNA baked in early tends to persist. If you do monolithic solutions, you aren't walking to microservices without a huge conversation. If you went XML/SOAP you would be asking the cost/benefit of JSON given JSON and XML can be considered strictly equivalent for data content. If you went apache and mod_<something> then moving to nginx or caddy incurs massive cost for what? But if you wind up behind dead/legacy platform (apache/mod_python) that can suck. Even just moving off wordpress costs. Moving off one CI to another CI costs. I don't know any CI which works all the time, they all seem to incur opex burdens We moved self-hosted gitlab to k8s gitlab to k8s in GCP gitlab and that went ok. Maybe when the abstraction moves to the package, change in how the package is presented matters less?
taeric 4y ago
Is this any different to any other technology? One could easily point to the rapidly diminishing advantage of AC over DC for power. ICE over electric cars. I'm sure I could extend this list.
langsoul-com 4y ago
Also it depends on how fast the software/company is growing. A slowly company could reasonably decide and reactor their stack. Whist one that moves fast won't want to risk blowing up everything, nor want to spare resources on something that customers don't notice until it all blows up.
kazinator 4y ago
Sometimes the early decisions are obviously the right ones in the space, which is why they are early. E.g. eyeglasses having nose pads, and arms that rest on the ears. Attempts to replace pneumatic tires with airless have been failing for over 100 years.
smasher164 4y ago
Not to stretch the analogy too far, but isn’t this true of any non-trivial dynamical system? They are highly sensitive to initial conditions.
simonw 4y ago
One thing I realized working on a large company codebase is that if you are the first person to do something you have an additional moral responsibility to do it well. In large codebases, the default answer to "how do I do X?" should always be to do it the way it has been done before. Consistency is hugely valuable. As a result, both junior and senior engineers, when faced with a thing they need to, will start by checking to see if that thing has been done elsewhere in the codebase already. This means that if you are the FIRST person to do something, it's on you to set a really great example - because it will inevitably be copied many times over in the future.
picsao 4y ago

184 points by kiyanwang 4y ago

"path dependence" - nobody has said the words that are the proper rubric for this discussion, so I thought I would. If you want to find some of the previous bibliography on this subject, them's the magic words.

agentultra 4y ago

Early tech decisions can become myopic in the long run. If you're not careful and admit that you made quick decisions early on these early decisions can become entrenched as gospel. This can alienate later, more experienced developers and become a factor in burnout as they are forced to deal with, "this is how we do things around here." One way to deal with tech debt is to take advantage of the experience and skill of your developers and embrace change. You might not want to be switching databases every time someone suggests, "X is better than Y," for sure! Inertia has its uses. But you don't want to stagnate either. Conclusion of tfa is solid: the truth is somewhere in the middle. The keyword for me is sufficient: the local minimum that meets all of our requirements. If we need to be able to be able to ship changes faster and it takes weeks to get changes merged, something is not sufficient in our stack/process/etc.

synu 4y ago
It can be helpful to write down the context of why you made your decision, so that if/when you revisit it later it’s more clear. ADRs are one way to do this: https://adr.github.io/madr/ https://adr.github.io/madr/
- agentultra 4y ago
  Great advice. I also use learning milestones to manage our novelty budget and record our findings/decisions/context/etc. I don't know if this is formalized or written down anywhere. The idea is that when your team wants to try something and learn from it you record down your thoughts and ideas into a document. You then have a triggering condition in that document for when to review it later. When you do the review you record what you learned and what decisions you make as a result. Over time your teams build up little libraries of these learnings and it can help build organizational continuity. People come and go, leaving teams or the company, joining other teams, etc. It's a good practice to avoid having to re-transmit what your organization has learned through re-telling folk-lore.
- troelsSteegin 4y ago
  Short of something structured, any format that captures key assumptions and intentions is good. The problem we are solving is X. The "pre-mortem" is the idea is that if we fail, it would be because of Y. Then in your design or decision, you address Y as a contingency. The collateral idea for code would be "good until" - this is good until (we expect) some threshold.
- hinkley 4y ago
  I've run into at least a few people who can't understand why my answer changes when the context changes. I said we were going to do A because of C, D and E. Why are you acting like it's a gotcha when we find out E doesn't apply anymore and now I agree B is a better option? Did you think we were joking when we said, "It depends"?
BurningFrog 4y ago
One solution is to design things so swapping out one component isn't harder than it needs to be. Easier said than done, but still quite doable with modular design, which you should do anyway.
- 8n4vidtmkvmk 4y ago
  that's also how we end up with an excessive number of microservices
hinkley 4y ago
Some of the most uselessly toxic behavior is defending X now because it was critical to our survival 3 years ago. This world is absolutely rife with ideas, actions and substances that are appropriate to a situation but are detrimental or even deadly once that moment has passed. Oxy because of a bone graft is not the same as Oxy because it's a Tuesday. We aren't questioning your wisdom because you did this thing three years ago. We are questioning your wisdom because you won't let us kill it right now.
dilyevsky 4y ago
There’s a flip side to this when people who come later totally underestimate the problem space and difficulty when trying to “fix” things.
- agentultra 4y ago
  Definitely and highlights another reason why it’s important to document these things for future maintainers and developers.
  - dilyevsky 4y ago
    Good in theory but in practice nobody is going to have the time to document every one of hundreds (or thousands) little decisions with potentially large implications while trying to keep the product afloat. Even for those they do, in my experience, nobody bothers to read them...
    - agentultra 4y ago
      Can’t help people who don’t want to help themselves.
      - dilyevsky 4y ago
        It’s true but it’s worth keeping in mind that there are some underwater rocks that OGs didn’t tell anyone bc they had better things to do
aledalgrande 4y ago
Had to deal with this in the past, where junior engs made choices on the initial architecture and by the time I joined they were in management/high roles. They refused any change that was too complicated for them to understand and never tried to improve their knowledge. Meantime our team was suffering under the load of technical debt vs non technical POs wanting to dish out features experiments fast. > this is how we do things around here resonates so much

andsoitis 4y ago

“Software has inertia.”

throw1234651234 4y ago

"Cloud provider: AWS originally, and with so much custom infrastructure built on its APIs, unlikely to ever change." Who cares? It literally doesn't matter unless you are doing something very, very specific with ML. As far as line-of-business projects go, every service you need has a direct equivalent. Also, NO ONE EVER SWITCHES CLOUDS, this is a TERRAFORM MYTH. Terraform being USEFUL in Cloud transition is a MYTH. I have direct experience with this where a corp started competing with Amazon (once Amazon went into groceries) and had to move to Azure. This is the ONLY instance where it made sense. "The underlying database: Mongo originally begot more Mongo, but switching any database anywhere is unusual." RDS is best 90% of the time. Other than that, just keep it modular. Definitely no MVC front ends married to the BE - API always.

Rafert 4y ago
People do switch clouds, but sure it's never trivial. https://about.gitlab.com/blog/2019/05/02/gitlab-journey-from-azure-to-gcp/ https://about.gitlab.com/blog/2019/05/02/gitlab-journey-from... for example.
- throw1234651234 4y ago
  Thanks for the link! I also know a national grocery store that switches from AWS to Azure once Amazons started competing with them directly. With that said though, I think it's very, very rare and I hate how people use it as an argument to justify Terraform, which largely doesn't help because all the APIs are different enough to still require manual adjustments.

thinkingkong 4y ago

Its pretty easy to find flaws or old parts in any stack. The trick is figuring out which ones need replacing, in which order, with an ROI you can justify. Most of the time you can’t because that same time is better used for other things. Its not to say it’s not possible. Just that most ic’s, managers, pms, etc dont build careers on proactive technical debt.

deleted 4y ago
[deleted]

chubot 4y ago

I'm a bit surprised that Stripe uses Mongo? For financial transactions? Is it the primary backing store or a cache? I always thought of Mongo as the DB with questionable durability and performance claims ... i.e. the "quick and dirty" DB. And also under-featured compared to relational DBs. But I haven't used it so maybe that is not accurate.

gqewogpdqa 4y ago
MongoDB is used by enterprises all over the world. It's true that it's the easiest DB to get started with, and it's also true that in the distant past (pre 3.0, back in ~2015), they had some issues. But they now run the financial backends of companies as varied as banks and coinbase and... https://www.mongodb.com/who-uses-mongodb https://www.mongodb.com/who-uses-mongodb
- jandrewrogers 4y ago
  MongoDB 3.x was broken too[0]. Their reputation for consistently shipping databases with fundamental design defects is well-deserved. Data loss was a hallmark of that product across multiple major versions. Maybe they've suddenly turned a new leaf in the last couple years but I wouldn't want to bet my business on it. Most of the companies I know that use it, use it for data that doesn't matter such that data loss isn't a big deal. [0] https://jepsen.io/analyses/mongodb-3-4-0-rc3 https://jepsen.io/analyses/mongodb-3-4-0-rc3
  - docmechanic 4y ago
    I wrote for the technical writing team at MongoDB when this "security issue" made the news. The precise problem was whether authentication was turned off/on by default for the default database, I believe, during installation. Product management initially chose this configuration to make it quicker for evaluators to get something up and running as quick as possible. The assumption was that no developer would actually use that in production. Further, the documentation clearly stated that this default configuration option was not secure. Blaming that data loss on MongoDB rather than the noob dev who deployed that evaluation configuration is as erroneous as the logic used by the noob dev. We hear it from developers all the time, so let me repeat it: User error. It's a thing.

KerrAvon 4y ago

AWS is early? I was thinking about all the crap Unix did in the 1970’s that we’re still stuck with. The tty subsystem, for one example.

Jtsummers 4y ago
It was an early decision in Stripe's history, which is the example system in the article.

nine_k 4y ago

Heh. Read about cosmic microwave background: it's a print of some microscopic disturbances in the baby-age Universe, now permeating all of it. It's just how this world works. Tiny early events grow to colossal scale as the thing that experienced it grows, be it a code base, a company's culture, biological structures, all the way up. No matter what, we have to put up with that, and tread lightly when anything of consequence depends on us. For which, of course, there's never enough time.

jitl 4y ago

This article is timely for me! I think about this a lot while working on Notion — and I’m drafting an RFC for a change that future engineers will either praise or curse. Right now the eng team is ~150, so we’re exiting the period when an experienced senior engineer (ie me) can easily make a profound tech stack change. Our database (Postgres) and backend stack (Express/Typescript) are already quite solidified. There is still time to change our front-end stack, although React is probably a given, we can change our API contract layer, our state management pattern, and possibly our client-side replicated ORM. It’s tricky to weigh continued speed today versus long-term speed. I think a lot about the tradeoff of “invest in infra improvements below the interface line” and “change the interface now, while there’s still time” when it comes to our ORM/state manager. We have a bunch of interesting challenges across all layers (see https://www.notion.so/blog/data-model-behind-notion https://www.notion.so/blog/data-model-behind-notion) so it’s not obvious what changes today will really change the constraints on our solutions tomorrow. My current philosophy is that we need to add new abstraction layers now if there’s a chance we need them, before detangling later becomes an insurmountable lift.

majormajor 4y ago
Do you need (more?) new abstraction layers or do you need extremely clear un-abstracted separations of concern? In the cases of trying to build abstraction layers in advance "in case we need them" that I've worked on, they rarely are sufficiently right-shaped for what the future feature need ends up being. Having more direct access to the primitives usually works better in terms of avoiding "the business logic is split across three layers of service calls" type of situations.
- jitl 4y ago
  We share the same record types (primitives?) across the stack - but because of that sharing, those types are hard to evolve, especially when it comes to their semantics. Too much code interprets then directly, which ossifies those primitives. How to enforce separation of concern when such data structure is shared liberally? > the business logic is split across three layers of service calls we are thankfully very far from this world.

helsinkiandrew 4y ago

Early decisions in every field have disproportionate influence, not because they were the 'right' or the best choice but because its much easier not to change, or its easier to improve the existing choice (making the current database faster) than jumping to another. a) The politicians who drafted the US constitution probably have had more impact than most presidents since. b) Train tickets, timetables, having different class carriages, ticket offices and train guards is mostly unchanged from when introduced by GWR in the 1840s c) Car peddles, typewriter keyboards, pizza recipes. etc

photochemsyn 4y ago
The US Constitution is a good example. While it's a politically controversial opinion, if the framers had extended 'indentured servitude' status to all slaves in the colonies at the time (a common status for many European migrants in the Middle Colonies, generally involving a ten-year period of forced labor before freedom was granted), it's entirely possible that the US Civil War could have been avoided. Some argue that southern slave states would never have gone along with this, however. The distribution of free laborers, indentured servants, and chattel slaves across the American colonies in the 17th century is a very interesting history that's typically neglected in education: https://www.oercommons.org/authoring/6734-slavery-and-indentured-servitude-in-the-american-c/view https://www.oercommons.org/authoring/6734-slavery-and-indent... Another example might be how the plantation system took hold in the American South but not in the New England Colonies.
- rrrrrrrrrrrryan 4y ago
  America may not have become the absolute economic powerhouse that it became if it wasn't for the 2+ centuries of uncompensated labor, but IIRC America was the only nation that had to fight a war to end slavery. Most nations simply bought the slaves from the slaveowners and then freed them, which might have actually been cheaper than fighting the Civil War.
  - bpodgursky 4y ago
    The south wasn't a meaningful contributor to the US's economic powerhouse status. That's the whole reason they lost the civil war — it was an economic backwater with almost no factories, and what useful goods they traded abroad were fairly easily replaced (so no European country bothered trying to help them).

michaelfeathers 4y ago

Often this dynamic is called path-dependence too.

deleted 4y ago

[deleted]

hosh 4y ago

This is why I like playing Go (the game). The decisions made early in the game has huge effects on the rest of the game. Lingering effects of dead formations (aji) comes back to haunt and shape future decisions. Playing Go exercises this, and helps develops the intuition on the effects of architecture, outside of the game. I am again also reminded of Christopher Alexander’s ideas in living architecture.

naet 4y ago
Go is the most beautiful game. Chess is the dominant game where I live, and it is fun to play quick blitz games and look for tactics, but the feeling is nothing like a game of Go. As far as early decisions, one of the most important in Go is knowing when to rethink your plans. You may place a stone with the intention of defending it, but if the board state changes, it might end up serving better as a local influencer or a sacrificial stone instead. To that extent, your early decisions are key, but also you have to remain flexible and attentive to the evolving game as moves are played, adapting your early decisions to unexpected later complications. Playing a good Go game feels almost zen or enlightening; you pick your moves with careful determination, but your plans flow like water around the fluctuating board state. There's a reason there are so many Go "proverbs", something in the game mechanics makes it a perfect metaphor for life or the universe.
- Nomentatus 4y ago
  Or it's wildly discouraging as you discover that 'cause you're human, you "tunnel" to parts of the board too much and you just can't stop it. AI plays very differently than any human, by not-tunnelling. Associative meat-minds don't seem to work that way.

photochemsyn 4y ago

> "And this fails the other way too, where major believers in academic-level correctness agonize over details to such a degree that projects never ship, and sometimes never even start." Anxiety paralysis. This is also why many people aren't too enthusiastic about the hottest new thing, and would prefer to stick with say, a language like C for low-level development. Experts in the field trying to push the boundaries aren't necessarily good examples for everyone else to follow.

redwood 4y ago

I like to describe this kind of thing via the analogy of the big bang: the little ripples in the plasma, after expansion, led to the galaxies that we see today, the smallest part of early state have MASSIVE implications later on

bestinterest 4y ago

I feel another point is that early tech decisions have a huge impact on the type of people you will have to hire. For example, Ruby attracts ruthless productivity and Go attracts people who prefer longer standing apps with less dependencies and maintainability. I make it sound like Go may be the 'better' choice here but that is not the case, as the author mentions, it's a balance.

deleted 4y ago
[deleted]

3pt14159 4y ago

Some early tech decisions matter, but most don't matter as much as people think they do. Other than the OS, I haven't really seen it be a complete deal breaker for anyone that's willing to put in the work in the first five years. Postgres not handling things well? Slow migration to DynamoDB or to sharding. Python falling over? Roll the hot codepaths in Cython or horizontally scale. Node doesn't have great data science libraries? No worries, build a couple of small python data science services and get on with life. The only reason I think the OS choice matters at all is because I've never seen someone move from .NET on Windows to Linux or visa versa. Could be done, but people generally don't do it. So your hiring pool is going to look different. Generally more scrapy on the linux side, generally more careful on the Windows side, but even there that's mostly about culture and priorities. Both stacks can handle basically anything.

barking_biscuit 4y ago
>The only reason I think the OS choice matters at all is because I've never seen someone move from .NET on Windows to Linux or visa versa. Could be done, but people generally don't do it. Done it myself. Seen it being done all the time in my org. We're migrating large amounts of legacy .NET Framework apps running on Windows on EC2 to .NET Core / .NET apps running in our K8s cluster using Linux containers.

FigmentEngine 4y ago

Inertia is a thing, especially around mental models and ways of working. Its hard to change an early decision, because many other parts and people have subsequently made decisions ontop of that. Like the ship of theseus, replacing parts is possible, but the replacement needs to fit in the gap, and the inertia means even if you replace everything, its always a ship.

mikewarot 4y ago

Even so, it's still better to start something and make mistakes (thus gaining experience) rather than getting stuck in analysis paralysis. Technical debt is real, and has interest rate that would make a loan shark blush, keep that in mind, and you should be just fine.

Clubber 4y ago

>these were known to be very imperfect implementations even when they were originally added, but they were added anyway in the name of velocity, with an implicit assumption baked in that they’d be shored up and improved later I've heard this a million times but don't remember ever seeing it actually done, at least on full coverage. This is why it's critically important to get the right first team of developers. A team who also knows this strategy is rarely implemented. Don't feel too bad, it's probably in every live codebase on the planet, just a matter of degree.

JoshCole 4y ago

In counterfactual regret minimization and in reinforcement learning this shows up too. The solution there tends to discount early iterations with something like a weighted average. It makes convergence to the true values happen faster. A similar thing happens in k mean clustering but is more horrifying because you can get arbitrary wrong solutions rather than just taking longer to converge. The solution there is to construct a sampling distribution when making your initial decision - instead of making a decision according to the principle of indifference to instead make a decision after getting more informed about the problem. This doesn't mean you don't have bad solutions, but it gives a bounding on just how bad those solutions can be. It happens in other places too, I'm sure, because this isn't really a software problem - this is a fundamental problem. Uniformed choices are uninformed but influence the choices that come after them. Certain heuristics like delaying your decision to the last responsible moment can help to avoid making your decisions with insufficient information, but it is important to realize the type of decision you are making. Are you in a case where getting things wrong just means you go slower but eventually converge? Or at you in a case where you can't reverse course and will have your outcomes determined by the decision you make without any guarantee that it will turn out well and no simple path backward. I hear people talk about this with terms like "one way doors" versus "two way doors". In the latter case you are much better off thinking more deeply through the problem. We're not really solving new fundamental problems. We're solving fundamental problems in a new context.

TeeMassive 4y ago

I worked for a company that opted for Windows Embedded back in the early 2000s, probably inspired by Tektronix scopes due to them also making scientific instruments. They boomed. They must have had good relationships with Microsoft because they produced kernel patches just for them. Anyway, 15 years later of "it works now, why change it", they start a new project to rewrite the cash cow. It was a Phoenix Project; and it also had a name associated with rebirth, as most of these projects do. Turns out, Windows Embedded is still not a good idea 20 years later, and they couldn't change their code because it was so tightly coupled with Windows. The project took 250% more time than budgeted and even then it was a MVP. A competitor decided to do the same product but in Yocto Linux using modern tech and remade the cash cow product in less than their budgeted time. The interface was snappy, there was barely no bugs, etc. Software entropy is real, and I've seen it in many startups that got bought out after they boomed. Then the entropy is left for those who weren't there during booming phase as a poisoned gift. I like to call it tech inflation.

robocat 4y ago
To me, your story is “Linux good, Windows bad” and you have created a narrative around that. Anecdotally, I have seen multiple successful greenfields rewrites on Windows. I am sure we could find plenty of examples of unsuccessful rewrites from Windows onto Linux. That is: your cause/effect analysis is probably incorrect. Rewrites are hard, and the OS is probably only a minor role. Disclaimer: I am a long time Windows hater, but Windows has been used to put food on my table.
- TeeMassive 4y ago
  It's more about the right tool for the job. Also, was your greenfield projects in Windows Embedded specifically?

robertlagrant 4y ago

It still blows my mind that Facebook invented a better PHP runtime instead of carving off bits of functionality into other technologies.

jitl 4y ago
They did that too.

sporkland 4y ago

This has always felt like the catch 22/anthropic principal of software development to me: 1. Early on you favor decisions that enable rapid iteration and growth without being able to put in much deep thought around long term productivity. Most of these tend to be fine in the long run but there's probably at least 1 or 2 that will haunt the company for the rest of its days. Companies that don't take these "short cuts" will probably kill the business. 2. Companies that survive largely have their architecture dictated by #1 and are fairly constrained in how radically they can change the decisions in #1. So most of us end up working on a product which has some pretty critical architectural flaws that are very hard to repair. The only companies I've seen that do work their way out of it are the printing presses (google, linkedin, yahoo, amazon, etc) that are able to invest in very strong platforms and move their applications onto them, and even then it's a long drawn out process.

HeyLaughingBoy 4y ago
There's a way out of this, but it requires discipline and besides, most companies don't want to do it. Prototype. Deliberately build "one to throw away." Iterate fast and figure out what you need. When you've learned the important lessons about architecture, structure and collaborations, make a note of those lessons in a design document and then build the "real" product with proper engineering, but with a lot less risk. I worked for an organization that did this and it results in good quality software, but you need buy-in at all levels of management or you run into "why did your team just waste 6 months on throw-away work?" questions.
- anthonypasq 4y ago
  arent you just describing the business from #1 that does this method and fails because they have no money and no customers?
  - n42 4y ago
    It’s important to time this well and start doing it as soon as you have your feet on the ground, and the hard part is to hit the ground running. Too early and you crash, too late and you grind to a halt edit: missed “6 months”, I live in another world — a week to prototype and throw it out. spending 6 months to prototype anything and you’re biting too big

ggm 4y ago

I have taken projects of significance to me from MySQL to Postgres. It was certainly a short-term cost but the upside justified it. (first class IPv6 objects, JSON) I have tried to do the language port thing Several times. I never succeeded in bringing others with me (Perl to python) but that said, others I work with have done Perl -> Java and C -> Perl -> C and Perl -> Java -> Rust so I think language may be one of the "tractable" problems in many cases. We tried AWS for low-end k8s and it was too complex. GCP/GKE was just better presented. It has the same complexity but gcloud/kubectl seemed to work "better" for us, which led directly to helm outcomes we liked. I think the AWS experience we had could be called at best "toe in the water" maybe try before you buy works? The one which has proved the most sticky is VM/Cloud provider, for routing. If you are in a world which cares about BGP, the one you pick that works, you probably stay with. We partnered with CloudFlare for some stuff, and Linode for other stuff. I do wish at times we'd gone digitalocean, or fastly, or even Akamai (which is behind Linode apparently) -but the BGP surface we have, is the one we want to maintain because we understand it. Maybe it's not that different to AWS vs Azure vs GCP. Also DNS provider. You tend to stick to the underlying NS and related registrar inside your chosen namespace registry. Also Cryptography TA. Well.. once you move to letsencrypt you stop moving I guess. I do think the DNA baked in early tends to persist. If you do monolithic solutions, you aren't walking to microservices without a huge conversation. If you went XML/SOAP you would be asking the cost/benefit of JSON given JSON and XML can be considered strictly equivalent for data content. If you went apache and mod_<something> then moving to nginx or caddy incurs massive cost for what? But if you wind up behind dead/legacy platform (apache/mod_python) that can suck. Even just moving off wordpress costs. Moving off one CI to another CI costs. I don't know any CI which works all the time, they all seem to incur opex burdens We moved self-hosted gitlab to k8s gitlab to k8s in GCP gitlab and that went ok. Maybe when the abstraction moves to the package, change in how the package is presented matters less?

taeric 4y ago

Is this any different to any other technology? One could easily point to the rapidly diminishing advantage of AC over DC for power. ICE over electric cars. I'm sure I could extend this list.

langsoul-com 4y ago

Also it depends on how fast the software/company is growing. A slowly company could reasonably decide and reactor their stack. Whist one that moves fast won't want to risk blowing up everything, nor want to spare resources on something that customers don't notice until it all blows up.

kazinator 4y ago

Sometimes the early decisions are obviously the right ones in the space, which is why they are early. E.g. eyeglasses having nose pads, and arms that rest on the ears. Attempts to replace pneumatic tires with airless have been failing for over 100 years.

smasher164 4y ago

Not to stretch the analogy too far, but isn’t this true of any non-trivial dynamical system? They are highly sensitive to initial conditions.

simonw 4y ago

One thing I realized working on a large company codebase is that if you are the first person to do something you have an additional moral responsibility to do it well. In large codebases, the default answer to "how do I do X?" should always be to do it the way it has been done before. Consistency is hugely valuable. As a result, both junior and senior engineers, when faced with a thing they need to, will start by checking to see if that thing has been done elsewhere in the codebase already. This means that if you are the FIRST person to do something, it's on you to set a really great example - because it will inevitably be copied many times over in the future.

picsao 4y ago