9 ms·

Software Design Is Knowledge Building

419 points by signa11 2y ago

chrisweekly 2y ago
Great article, +1 Insightful. > the ultimate goal of software design should be (organizational) knowledge building
smikhanov 2y ago
Good article, both in spirit and factually. One thing to add: the author talks about reviving a system as “slow and difficult process”, and it is. However, the concrete example described is not worthy of hand-wringing of this kind: a system that could have been built by a single competent engineer in 6 months (inevitably of alpha quality, at best), could be resurrected by a competent team of several programmers and brought to, say, beta quality, while keeping the lights on their alpha system on in how long? Let’s say, 9-12 months. No biggie, really. Most companies routinely discard man-years of programmer’s effort, so those 9-12 months are likely just a blip in the lifetime of that firm.
uludag 2y ago
I find this article spot on and resonates with what I've experienced. The article mentions Zach Tellman's newsletter "Explaining Software Design" (https://explaining.software/ https://explaining.software/) which I highly recommend reading. I have found his works to provide deep insight into the process of software design.
nosefurhairdo 2y ago
I've handed off a few services I built with minimal oversight or documentation. The receiving teams have been able to make changes without my involvement and everyone is happy. I believe the only reason I've been successful in this is because I agonize over simplicity. There are times during the development of any project where one might be tempted to hack around an issue, or commit the ugly code that seems to work. These are the rough edges that inheritors of a codebase use as evidence that a blank slate would be preferable. They're also the bits where the underlying business logic becomes murky. My goal is for the code to be so clear that documentation would feel redundant. This approach of course takes more time and requires that your management trusts you and is willing to compromise on timelines. It's extremely rewarding if you can sell it and deliver.
- MrMcCall 2y ago
  Documentation is useful, when done well. The code is always the authority, and there are very many ways the correct logic can be constructed. How it is constructed is the difference between good enough and excellent. And getting "compromise on timelines" is a most sublime political art. It requires the combination of both a humble, competent manager and an established, successful engineer worthy of trust. Congratulations on your success on those two varied fronts!
  - nosefurhairdo 2y ago
    Thank you! I am indeed very fortunate to have a great manager.
  - chrsig 2y ago
    I think it depends on who the documentation is intended for. I often let public facing documentation be the source of truth for expected behavior unless it's infeasible to coerce the system to that behavior. If the latter does occur, the documentation gets updated. If the question is what does the software actually do, then of course the code, toolchain, and runtime are the authority.
    - MrMcCall 2y ago
      Good point. I was only speaking to targeting other developers.
  - t43562 2y ago
    It needs to answer the "why" question most IMO. I can read code and see "how" and perhaps guess "what" but the "why" is missing. It also doesn't have to be that detailed - just a one line comment at the top of a file saying why it's there and what for can make an immense difference to the time it takes to understand code. Class comments are great too if they have in them everything that's NOT in a ChatGPT summary :-). i.e. I can paste code into ChatGPT myself to get a summary if I really wanted to so I don't need that - but I need all the things it doesn't tell you which is basically why the class exists and what it's intended for. The lower down the hierarchy it goes the less comments matter IMO.
softwaredoug 2y ago
I see star, lone-wolf ICs get too out in front of their teams all the time. It usually doesn’t end well. The star IC could objectively be building the right thing (like a state of the art recommendation system). But, like the article says, requirements change, bugs need to be fixed, the team needs to adjust the implementation and eventually the team reimplements the thing to within their capabilities. It’s more than the usual software maintenance too-It’s the entire operation of a piece of software. Scaling it out, being on call for it, adding monitoring, alerting and logging. Inter-operating with other software in the company. Developing libraries and services for other developers to consume. Security. Understanding and deploying the dependencies of the software. And more. The clever recsys in my example is only the tiny kernel of the actual challenge of delivering this to users. Its the complex care and feeding of a live service that matters.
- physicles 2y ago
  According to the article, the mistake wasn’t that X10 got too far out in front of the team (the initiative to build svc was supposed to be a lone effort), it was Org’s failure to orchestrate a hand-off from X10 to team.
brettgriffin 2y ago
There's some really interesting stuff in here, but I think, given the example, it is burying the lede: organizations systemically underestimate the total cost of ownership of a service. By, like, orders of magnitude. In this example, it isn't entirely clear if this service ('saas middleware') is deeply integrated with org's core competency or value. But I'll assume it isn't. They do not understand the service domain well enough and cannot staff or motivate the people to build and maintain it correctly. This is exactly why SaaS exists and is so ubiquitous. You're just not going to be able to build something as good or better for less in the long run. If they properly understood the cost of building and maintaining this, compared to the (probably) insignificant increase in enterprise value to the org, they would have probably would have just RIF'd these spare engineers and just pay the SaaS provider. I deal with the internals of many engineering teams across companies of all sizes, and sure as shit, every. single. one. of them has multiple of these internal failed creations. I just don't think people truly understand how much of a liability these systems are orgs. But yeah, once they made the first mistake, the rest of the blog pretty much hits the nail on the head.
- dambi0 2y ago
  I think the article is trying to say more than that the development and ownership of projects is often underestimated. It’s attempting to explain why that is the case. Because the cost of theory building is misunderstood. I don’t think the lede has been buried at all.
  - brettgriffin 2y ago
    The whole scenario only exists because of the axiom introduced between points 3 and 5: > 3 ...ORG spends an egregious amount of money on middleware SaaS > 4 ...executive figures they should be able to replace SaaS with in-house system > 5 ...manager tasks one of ORG ’s finest engineers with the job of building it If in Point 4 it was determined this was a low value, high TCO project with many replacements, the stud engineer doesn't work on the project, no events past this point occur. If point 3 was that they had an opportunity to build a flagship product/feature in their wheelhouse and drastically grow market share, nothing past 6 and or 7 happens. Like I said, there are interesting things here about knowledge transfer, but the root cause seems to be missed from the analysis. Maybe there's some other real world scenarios where teams of critical software are getting replaced whole sale and remain confused, but I'm not convinced most of these issues would come up in a situation that wasn't the one described in 3-5.
gatinsama 2y ago
> the mental model that allows the designer to map a subset of the world (the domain) to and from the system, and not the system itself, is the primary product of the software design activity This is spot on. I was never able to put it in such precise words. This theory has Brook's Law as a corollary: "Adding manpower to a late software project makes it later." Because the developers need time to develop this mental model before they can meaningfully contribute to the codebase.
tekchip 2y ago
This is a general problem. Why's and how's. Or in project management, and what should be associated documentation, process and procedures. In this case code is procedure or how. The steps to do the thing. Which is great but it's hard to make meaningful and useful changes without understanding the Why, process, does the thing needing to be done and why, process, are the procedures or code doing the things they are in the way they are. Presumably for code you would get enough why/process via comments but that seems unlikely. Perhaps coding needs to take some other tools from project management or something? Knowledge sharing/transfer is hard.
harrall 2y ago
I feel that the real problem is a lot of people don’t care about collecting requirements. It’s one of my favorite parts of the process. People just want to build the app that they want to build. I’ve talked to engineers who just say “I don’t really care until we can start coding.” I got into engineering because I like building things that are useful.
- avg_dev 2y ago
  Where I live, most software developers are not legally allowed to call themselves software engineers (engineering is licensed by regulatory body). Still, many do call themselves that and so do their employers. But in my view, "I don't really care until we can start coding" is not actually engineering at all.
  - worik 2y ago
    > developers are not legally allowed to call themselves software engineers How about "Solution Architect"? I think a collection of nonsense job titles for computer programmers would be fun...
    - Nevermark 2y ago
      Every Solution Architect should have at least seven Framework Framers and two Data Plumbers supporting them, to give the title its proper gravitas.
  - harrall 2y ago
    I’m not sure why you are making a point about engineering certification. It’s not like physical products are immune to this problem. I could list you a billion poorly designed products that don’t seem to meet the correct requirements. At the end of the day, some people just like to build stuff without understanding who they are building for. It could be because they like engineering. It could be because they think they will make money because “people will come if you build it.” Both strategies make poor solutions. When it should be “the users have these specific problems and the product should make their life easier.”
avg_dev 2y ago
Yes, artifacts (PRs, tickets, commit messages) should have good context (documentation) associated with them, and yes, simplicity is very important, and yes, some tech debt is always going to be incurred, but most importantly, not understanding a "legacy" system that is already running in production, and then modifying it significantly, is not going to lead to good results. I think it's as simple as that.
siscia 2y ago
Most of our white collar jobs are about knowledge sharing and synchronization between people. And surprisingly this is an aspect in which I see very very little progress. The most we have are tools like confluence or Jira that are actually quite bad in my opinion. The bad part is how knowledge is shared. At the moment is just formatted text with a questionable search. LLMs I believe can help in synthesize what knowledge is there and what is missing. Moreover it would be possible to ask what is missing or what could be improved. And it would be possible to continuously test the knowledge base, asking the model question about the topic and checking the answer. I am working on a prototype and it is looking great. If someone is interested, please let me know.
- mdgrech23 2y ago
  knowledge is power and people don't always want to share. Maybe it's more reflective of my company culture but I've seen knowledge effectively hoarded and used strategically as a weapon at times.
  - siscia 2y ago
    Of course, but at least in my personal case is more about the lack of tooling.
  - nicce 2y ago
    It is visible everywhere. Some people hoard knowledge so that they stay important in the company. Some people hoard knowledge so that they can get more money from bug bounties. It is almost always about personal gain.
  - ozim 2y ago
    Of course there is no upside for spending time updating documentation unless it actually is part of your job description or there is legal requirement for company. If you put knowledge in wiki, no one will read it and they will keep asking about stuff anyway. Then if you put it there and keep it up to date you open yourself to a bunch of attacks from unhappy coworkers who might use it as a weapon nagging that you did not do good job or find some gaps they can nag about.
picometer 2y ago
This is a well-referenced essay, drawing the on writing of David Parnas [1], Peter Naur [2], and Zach Tellman [3]. As software developers we’re intimately familiar with these ideas. But the industry still treats it as “folk knowledge”, despite decades of academic work and systemization attempts like the original Agile. We really need more connective work, relating the theoretical ideas to the observed behavior of real-life software projects, and to the subsequent damage and dysfunction. I liked this essay because it scratches that itch for me. But we need this work to go beyond personal blogs/newsletters/dev.to articles. It needs to be recognized & accepted as formal “scientific” knowledge, and to be seen and grokked by industry and corporate leadership. [1] https://dl.acm.org/doi/pdf/10.5555/257734.257788 https://dl.acm.org/doi/pdf/10.5555/257734.257788 [2] https://pages.cs.wisc.edu/~remzi/Naur.pdf https://pages.cs.wisc.edu/~remzi/Naur.pdf [3] https://explaining.software/ https://explaining.software/
- physicles 2y ago
  I suspect systemization would require quantifying some of the variables involved, which include things like - The size and complexity of the code base (for some definition of size and complexity) - The quality of the code and docs (for some definition of quality) - The skill and experience of the people involved In four years in a big tech role, my team twice inherited and had to modify a code base without any input from the original authors. One was a quagmire, the other was a resounding success: - The first was a media player control that we had to update to support a new COM interface and have a new UI. We decided that it was too complicated, and nobody understood it, so we’d reimplement it from scratch. One year later it mostly worked, but still had bugs and performance issues that the original version didn’t have. In hindsight, I suspect it would’ve been cheaper to try to revive the original code base. - The second was a music database for an app running on a mobile device. Our current one was based on the version of SQL available, but some principal engineers on another team suggested replacing it with a custom in-memory database that already shipped in another device. We argued that the original authors had left and the code was unwieldy; they argued that “it’s just code, we can read it” and its performance was known to be better. They did the work to revive it and successfully integrated it into our app. Wild success. The flip side of “it’s impossible to revive a dead system” is “don’t rewrite a working system from scratch”. Absent more research, the only way to correctly guess which situation you’re actually in is to have tons of experience.
mfld 2y ago
I can't resist to point out that, in theory, there are at least two other options avoid wasting many resources via the failing new teams: 1. Get the original dev to explain his theories (keep employees longer or engage them as consultants) 2. Make and get a "diary" of the original devs theory building. In this story, and in probably many places, the business environment however supports the explained outcome.
boricj 2y ago
The focus of this article is on the big political project that fails (the SVC), but the part that resonates most with me is the small forgotten project that lives (the SaaS). Over the years, I've built a number of contraptions under a similar set of circumstances: a technical problem usually created by an organizational issue suddenly appears that is both severe enough to threaten a project yet falls outside the core business, so it needs to be fixed both yesterday and on the cheap. Inevitably, I get saddled with it and produce a kludge that is equally effective and cursed before going back to business as usual. More than once I've learned to my horror that years later the thing is not only unmaintained yet still in place, but its usage expanded to the point where it became load-bearing, because the underlying organizational issue was never solved. In a manner of speaking, it is the opposite situation as described in the article: a complete lack of software design that somehow manages to survive in spite of a lack of knowledge building.
binary_slinger 2y ago
Also related: https://blog.codinghorror.com/commandos-infantry-and-police/ https://blog.codinghorror.com/commandos-infantry-and-police/ I work a lot in the transition area between commando and infantry aka X_10 and TEAM. I’ve also found myself on TEAM++ coming in to replace TEAM. It is difficult to explain to customers that SVC was built on a set of assumptions which I turn informed the design. Once the assumptions changed then design typically needs to change as well.
- t43562 2y ago
  This is a major issue. Why the code is what it is. You need a history of the assumptions so that new developers can know what's legacy and what isn't. I've never yet had a set of requirements that didn't change.
  - CRConrad 2y ago
    > You need a history of the assumptions so that new developers can know what's legacy and what isn't. Isn't this (at least in part (and perhaps only approximately)) what Architecture Decision Records are for?
gervwyk 2y ago
Totally resonated with me re software lifecycle etc. Great article. I know this is not what the article is about. But perhaps exec should have spent resources and time trying to increase revenue instead of cutting cost marginally, and creating an expensive system down the road. Derailing team focus. Build vs buy… Build is almost always not cheaper. Many other reasons to build though.
jt2190 2y ago
> Knowing that [program] revival [i.e. bringing a new development team “up to speed” by having them learn the model] is a plausible future need has powerful consequences for our work. I’m not sure that most developers are willing to revive software, based on my observation that very few read anything at all, especially the source code. Instead I see a lot of adjusting the input and output of the existing program by adding a new layer. This new code is totally understood by the new dev, and they can modify/maintain it easily without worrying that they broke the existing system. It also usually duplicates something that already exists inside the system. As the process repeats more and more layers are added. I think a few lucky teams have developed a culture that encourages learning the existing code. (Popular web frameworks comes to mind as an example.)
Mawr 2y ago
> The problem is that TEAM members don’t have enough elements to build a satisfactory mental model of SVC. They need to go by a mix of the client’s interpretation of what the system should be, and what they can tell from the code that the system actually is. These views can be disconnected and contradictory. The code may tell the what and the how, but it doesn’t tell the why. Only X10 could say what was a functional requirement, what a technical necessity, what a whim, what an accident. The team has to resort to reverse engineering, extrapolating, and guessing. Hence, write down your thought process, mental model, and assumptions alongside the code. Tip: Call the process "writing documentation" instead of "commenting code".
deskr 2y ago
> The program should preferably be discarded, and the new team should be given the opportunity to resolve the problem from scratch. Yeah right. "We don't know how it works so we're going to ditch it and create it again." It works in some cases but by no means should that be the default.
- datadrivenangel 2y ago
  This is the right answer, except that when software has been so neglected that this is the best answer, individuals within the organization that neglected to maintain the software will realize that this is their only chance to get bugs fixed or new features added, and so the requirements will expand until the re-write probably dies.
deskr 2y ago
I have another suggestion which I'm sure played a large part in this. SVC was an unwanted child. It wasn't their "product". One employee was tasked to write it to save paying money to a "seemingly innocuous middleware SaaS". To anyone in ORG working on it, it was a dead end. No one wanted to own it and perhaps no one did. A team was asked to add features to it. Doing the ground work of actually understanding SVC had many negative consequences: * It would take a very long time, making managers not happy. It would be largely a wasted effort, since no further work was then needed on SVG. * If you became an expert on SVG, it would be yours to keep and no one wanted that.
contingencies 2y ago
I imagine that experienced cross-disciplinary designers would concur that all complex design is knowledge building, which is why documenting design decisions is important. You tend to learn this lesson when maintaining projects of nontrivial complexity over a longer period.
gr3ml1n 2y ago
This is an interesting argument for the (definitely common) pattern the author describes in the intro. A more cynical take (that I'm inclined towards is): the median software developer is simply not very good. X10 was a good developer; the people on TEAM and TEAM++ were not.
- tresil 2y ago
  This is absolutely my take as well. I see the points that the author brought up as additional contributing factors. However, the leading reason for this “phenomena” is that many companies are brimming with individuals (including managers) that are simply not competent or motivated enough to meet the demands of this profession. In the author’s story, this is probably why X10 decided to leave the company, because they were tired of working with incapable co-workers.
nextworddev 2y ago
Kind of a roundabout way of saying that you can’t evolve a program without fully understanding the codebase
- googamooga 2y ago
  Not only the codebase, but also the domain, the stakeholders, the history of requirements, etc etc etc.
wwarner 2y ago
This is really why AI is going to hit s/w development so hard. It's not merely going to make code easier to write, it's going to be a massive knowledge repository that takes a team from initial conception, through product design and finally engineering design and coding.
- n_ary 2y ago
  No, LLM needs excellent communicator. It can statistically spit out knowledge but someone has to embed that knowledge. Given how vague and contradictory most requirements are and how complete and excruciatingly detailed prompts must be, LLMs will be useful to generate prototypes faster to check assumptions of the lost knowledge, nothing more or less.
  - namaria 2y ago
    The real trouble with LLMs is that they emulate knowledge so well. People assume they can depend on it to know things, but it is not reliable at all. A lot of traps are being laid in code by people trusting the output or behavior of LLMs.
  - wwarner 2y ago
    Yes, this what i’m saying; the development process will be about writing and talking into a new tool, and then with that recorded information generating summaries, mocks prototypes and code. Definitely people would be involved. What I’m pointing out is that LLMs are natural tools for summarizing and synthesizing domain expertise, which can be naturally applied to the product development process. If an LLM based tool can be a great personal assistant, it can also be a great knowledge repository for an organization.
namaria 2y ago
People often assume that code is knowledge. They want "self explanatory" or "well documented code". Companies and managers often treat developers as interchangeable. But therein lies the mistake. Knowledge exists in mental models and team structures. Small components and systems can be understood by a person, but team structure also embodies knowledge of larger systems. People will need complementary mental models to understand a large system together. That's why adding manpower to a late project makes it later. That's why maintenance is hard and handover is harder. That's why systems devolve into big balls of mud. Because companies and managers do not respect the fact that you need people and teams who have good mental models and that mental models take time to build and share. No amount or quality of code can make up for this fact. And simulacra of ownership - having "product owners" or whatever - won't cut it. You need people to own their systems, understand them deeply. Moving people around, churn, treating developers as interchangeable, substituting rituals for deep work, accumulating 'technical debt' (deferred work as in ship now and think later) etc are all detrimental to building and sharing sound mental models.
- dominicrose 2y ago
  I've seen code like this: - 10 lines - every line has a different author in git blame - there are conditional branches inside conditional branches - last but not least: all conditional branches have the same behavior! How does it end up like this? Why doesn't the last commiter just delete everything and write it in a single line instead?
  - namaria 2y ago
    Treating code as a means to an end, which is what I think you're describing, is just as bad as treating it as an end in itself. It should be neither an opaque black box nor a transparent 'bicycle of the mind'. In my opinion it cannot be lumped all in one category. It is an aspect of software systems. It may have many features, depending on where it is and how it is used. Code can embody knowledge, but it is not the embodiment of knowledge. It can express functionality but it is not a functional component of a system. I think aspect is the best description: when you look at a system from the source code, you see some of it. Not a projection of the system over a set of dimensions as some people seem to treat it. It is not a textual description of the system. It is the part of the system you can see when you come at it from that side.
pjbster 2y ago
Even if the organistion is fully signed up to the knowledge building philosphy, it can still be derailed if the staff aren't up to scratch. Around 12 years ago, my employer tasked me with building a quote engine for a new product they wanted to sell online. The engine needed to produce 4 additional quotes (2 lower, 2 higher) to either prevent potential walk-aways or to offer upsells to capture potential additional revenue. And it struck me at the time that this sort of hand-wavey selling tactic would be just the sort of thing that was likely to change so I put all this logic into a pure function - pass in an original quote request and the function returns a collection of alternative quote requests. And, sure enough, a couple of years later, the business decides to change the approach and offer 1 lower quote and 3 upsells. And they give the job of implementing this to another developer. I was still at the company and was known as the original developer (my name was in a comment at the top of the file, for a start) so I was asked to review the code changes. I was surprised to learn that the other dev had left the pure function untouched and had, instead, written a bunch of new logic to generate the alternative quotes. Not based on the original quote request but on the collection of alternative quotes returned from the original function. Furthermore, this new logic was placed in main procedure - mama's finest spaghetti in the making, right there. So I rejected the change and told the dev where to put the actual logic. Then I waited for the re-review request to come in. What happened instead is that the code went live anyway - the dev had simply re-raised the review request and assigned it to another dev who rubber stamped it. Looking back, I don't think all the documentation in the world would have prevented this behaviour. A better approach would be for the company to pass the changes to the original developer and to pair with another dev - like Fred Brooks' Chief Programmer plus Assistant recommendation. I was never approached for an end of year review for this developer and I left the company before them. It's not personal but I'll resign on the spot if a company I work for employs this developer in future.
t43562 2y ago
The problem I have is that only people who "know" already tend to accept this insight. i.e. explaining it to those who make decisions doesn't work that well because it doesn't fit their mental model very well - they don't understand that a large part of their asset is sitting in people's heads. That guy wants 5k more.....? Pay it. The cost of hiring someone new and training them up and relearning it all will be far higher. Don't force people back to the office, be relaxed about everything and keep the knowledge. At the same time make sure other people are learning it so you do have replacements. At the same time make sure you have proper tests so you can work with code you don't understand yet if someone leaves. At the same time try to gather all the correct and uptodate documentation and explanations somewhere.
lincpa 2y ago
[dead]
red_admiral 2y ago
Part of the story that is mentioned but not discussed is that X_10 had a fixed and tight schedule. Under that constraint you expect some compromises for writing the mental model down in a way others can learn, because that model presumably lived in her head. It might have been possible to say, great job you delivered on time, now you have as much time as you need to write your mental model down. On full pay of course, and we won't count that against you as non-technical work in the next performance review. That increases the upfront cost of SVC but is an investment that pays back interest as soon as someone else has to fix anything.
Peterthomos 2y ago
[dead]
beryilma 2y ago
> A dead program may continue to be used for execution in a computer and to produce useful results. The actual state of death becomes visible when demands for modifications of the program cannot be intelligently answered. I totally believe this. I see zombie programs at my employer, including the ones that are currently making a profit, that just don't know yet that they are dead. The Jira model of non-ownership software development being a leading cause in my opinion...

419 points by signa11 2y ago

Great article, +1 Insightful. > the ultimate goal of software design should be (organizational) knowledge building

Good article, both in spirit and factually. One thing to add: the author talks about reviving a system as “slow and difficult process”, and it is. However, the concrete example described is not worthy of hand-wringing of this kind: a system that could have been built by a single competent engineer in 6 months (inevitably of alpha quality, at best), could be resurrected by a competent team of several programmers and brought to, say, beta quality, while keeping the lights on their alpha system on in how long? Let’s say, 9-12 months. No biggie, really. Most companies routinely discard man-years of programmer’s effort, so those 9-12 months are likely just a blip in the lifetime of that firm.

uludag 2y ago

I find this article spot on and resonates with what I've experienced. The article mentions Zach Tellman's newsletter "Explaining Software Design" (https://explaining.software/ https://explaining.software/) which I highly recommend reading. I have found his works to provide deep insight into the process of software design.

nosefurhairdo 2y ago

I've handed off a few services I built with minimal oversight or documentation. The receiving teams have been able to make changes without my involvement and everyone is happy. I believe the only reason I've been successful in this is because I agonize over simplicity. There are times during the development of any project where one might be tempted to hack around an issue, or commit the ugly code that seems to work. These are the rough edges that inheritors of a codebase use as evidence that a blank slate would be preferable. They're also the bits where the underlying business logic becomes murky. My goal is for the code to be so clear that documentation would feel redundant. This approach of course takes more time and requires that your management trusts you and is willing to compromise on timelines. It's extremely rewarding if you can sell it and deliver.

MrMcCall 2y ago
Documentation is useful, when done well. The code is always the authority, and there are very many ways the correct logic can be constructed. How it is constructed is the difference between good enough and excellent. And getting "compromise on timelines" is a most sublime political art. It requires the combination of both a humble, competent manager and an established, successful engineer worthy of trust. Congratulations on your success on those two varied fronts!
- nosefurhairdo 2y ago
  Thank you! I am indeed very fortunate to have a great manager.
- chrsig 2y ago
  I think it depends on who the documentation is intended for. I often let public facing documentation be the source of truth for expected behavior unless it's infeasible to coerce the system to that behavior. If the latter does occur, the documentation gets updated. If the question is what does the software actually do, then of course the code, toolchain, and runtime are the authority.
  - MrMcCall 2y ago
    Good point. I was only speaking to targeting other developers.
- t43562 2y ago
  It needs to answer the "why" question most IMO. I can read code and see "how" and perhaps guess "what" but the "why" is missing. It also doesn't have to be that detailed - just a one line comment at the top of a file saying why it's there and what for can make an immense difference to the time it takes to understand code. Class comments are great too if they have in them everything that's NOT in a ChatGPT summary :-). i.e. I can paste code into ChatGPT myself to get a summary if I really wanted to so I don't need that - but I need all the things it doesn't tell you which is basically why the class exists and what it's intended for. The lower down the hierarchy it goes the less comments matter IMO.

softwaredoug 2y ago

I see star, lone-wolf ICs get too out in front of their teams all the time. It usually doesn’t end well. The star IC could objectively be building the right thing (like a state of the art recommendation system). But, like the article says, requirements change, bugs need to be fixed, the team needs to adjust the implementation and eventually the team reimplements the thing to within their capabilities. It’s more than the usual software maintenance too-It’s the entire operation of a piece of software. Scaling it out, being on call for it, adding monitoring, alerting and logging. Inter-operating with other software in the company. Developing libraries and services for other developers to consume. Security. Understanding and deploying the dependencies of the software. And more. The clever recsys in my example is only the tiny kernel of the actual challenge of delivering this to users. Its the complex care and feeding of a live service that matters.

physicles 2y ago
According to the article, the mistake wasn’t that X10 got too far out in front of the team (the initiative to build svc was supposed to be a lone effort), it was Org’s failure to orchestrate a hand-off from X10 to team.

brettgriffin 2y ago

There's some really interesting stuff in here, but I think, given the example, it is burying the lede: organizations systemically underestimate the total cost of ownership of a service. By, like, orders of magnitude. In this example, it isn't entirely clear if this service ('saas middleware') is deeply integrated with org's core competency or value. But I'll assume it isn't. They do not understand the service domain well enough and cannot staff or motivate the people to build and maintain it correctly. This is exactly why SaaS exists and is so ubiquitous. You're just not going to be able to build something as good or better for less in the long run. If they properly understood the cost of building and maintaining this, compared to the (probably) insignificant increase in enterprise value to the org, they would have probably would have just RIF'd these spare engineers and just pay the SaaS provider. I deal with the internals of many engineering teams across companies of all sizes, and sure as shit, every. single. one. of them has multiple of these internal failed creations. I just don't think people truly understand how much of a liability these systems are orgs. But yeah, once they made the first mistake, the rest of the blog pretty much hits the nail on the head.

dambi0 2y ago
I think the article is trying to say more than that the development and ownership of projects is often underestimated. It’s attempting to explain why that is the case. Because the cost of theory building is misunderstood. I don’t think the lede has been buried at all.
- brettgriffin 2y ago
  The whole scenario only exists because of the axiom introduced between points 3 and 5: > 3 ...ORG spends an egregious amount of money on middleware SaaS > 4 ...executive figures they should be able to replace SaaS with in-house system > 5 ...manager tasks one of ORG ’s finest engineers with the job of building it If in Point 4 it was determined this was a low value, high TCO project with many replacements, the stud engineer doesn't work on the project, no events past this point occur. If point 3 was that they had an opportunity to build a flagship product/feature in their wheelhouse and drastically grow market share, nothing past 6 and or 7 happens. Like I said, there are interesting things here about knowledge transfer, but the root cause seems to be missed from the analysis. Maybe there's some other real world scenarios where teams of critical software are getting replaced whole sale and remain confused, but I'm not convinced most of these issues would come up in a situation that wasn't the one described in 3-5.

gatinsama 2y ago

> the mental model that allows the designer to map a subset of the world (the domain) to and from the system, and not the system itself, is the primary product of the software design activity This is spot on. I was never able to put it in such precise words. This theory has Brook's Law as a corollary: "Adding manpower to a late software project makes it later." Because the developers need time to develop this mental model before they can meaningfully contribute to the codebase.

tekchip 2y ago

This is a general problem. Why's and how's. Or in project management, and what should be associated documentation, process and procedures. In this case code is procedure or how. The steps to do the thing. Which is great but it's hard to make meaningful and useful changes without understanding the Why, process, does the thing needing to be done and why, process, are the procedures or code doing the things they are in the way they are. Presumably for code you would get enough why/process via comments but that seems unlikely. Perhaps coding needs to take some other tools from project management or something? Knowledge sharing/transfer is hard.

harrall 2y ago

I feel that the real problem is a lot of people don’t care about collecting requirements. It’s one of my favorite parts of the process. People just want to build the app that they want to build. I’ve talked to engineers who just say “I don’t really care until we can start coding.” I got into engineering because I like building things that are useful.

avg_dev 2y ago
Where I live, most software developers are not legally allowed to call themselves software engineers (engineering is licensed by regulatory body). Still, many do call themselves that and so do their employers. But in my view, "I don't really care until we can start coding" is not actually engineering at all.
- worik 2y ago
  > developers are not legally allowed to call themselves software engineers How about "Solution Architect"? I think a collection of nonsense job titles for computer programmers would be fun...
  - Nevermark 2y ago
    Every Solution Architect should have at least seven Framework Framers and two Data Plumbers supporting them, to give the title its proper gravitas.
- harrall 2y ago
  I’m not sure why you are making a point about engineering certification. It’s not like physical products are immune to this problem. I could list you a billion poorly designed products that don’t seem to meet the correct requirements. At the end of the day, some people just like to build stuff without understanding who they are building for. It could be because they like engineering. It could be because they think they will make money because “people will come if you build it.” Both strategies make poor solutions. When it should be “the users have these specific problems and the product should make their life easier.”

avg_dev 2y ago

Yes, artifacts (PRs, tickets, commit messages) should have good context (documentation) associated with them, and yes, simplicity is very important, and yes, some tech debt is always going to be incurred, but most importantly, not understanding a "legacy" system that is already running in production, and then modifying it significantly, is not going to lead to good results. I think it's as simple as that.

siscia 2y ago

Most of our white collar jobs are about knowledge sharing and synchronization between people. And surprisingly this is an aspect in which I see very very little progress. The most we have are tools like confluence or Jira that are actually quite bad in my opinion. The bad part is how knowledge is shared. At the moment is just formatted text with a questionable search. LLMs I believe can help in synthesize what knowledge is there and what is missing. Moreover it would be possible to ask what is missing or what could be improved. And it would be possible to continuously test the knowledge base, asking the model question about the topic and checking the answer. I am working on a prototype and it is looking great. If someone is interested, please let me know.

mdgrech23 2y ago
knowledge is power and people don't always want to share. Maybe it's more reflective of my company culture but I've seen knowledge effectively hoarded and used strategically as a weapon at times.
- siscia 2y ago
  Of course, but at least in my personal case is more about the lack of tooling.
- nicce 2y ago
  It is visible everywhere. Some people hoard knowledge so that they stay important in the company. Some people hoard knowledge so that they can get more money from bug bounties. It is almost always about personal gain.
- ozim 2y ago
  Of course there is no upside for spending time updating documentation unless it actually is part of your job description or there is legal requirement for company. If you put knowledge in wiki, no one will read it and they will keep asking about stuff anyway. Then if you put it there and keep it up to date you open yourself to a bunch of attacks from unhappy coworkers who might use it as a weapon nagging that you did not do good job or find some gaps they can nag about.

picometer 2y ago

This is a well-referenced essay, drawing the on writing of David Parnas [1], Peter Naur [2], and Zach Tellman [3]. As software developers we’re intimately familiar with these ideas. But the industry still treats it as “folk knowledge”, despite decades of academic work and systemization attempts like the original Agile. We really need more connective work, relating the theoretical ideas to the observed behavior of real-life software projects, and to the subsequent damage and dysfunction. I liked this essay because it scratches that itch for me. But we need this work to go beyond personal blogs/newsletters/dev.to articles. It needs to be recognized & accepted as formal “scientific” knowledge, and to be seen and grokked by industry and corporate leadership. [1] https://dl.acm.org/doi/pdf/10.5555/257734.257788 https://dl.acm.org/doi/pdf/10.5555/257734.257788 [2] https://pages.cs.wisc.edu/~remzi/Naur.pdf https://pages.cs.wisc.edu/~remzi/Naur.pdf [3] https://explaining.software/ https://explaining.software/

physicles 2y ago
I suspect systemization would require quantifying some of the variables involved, which include things like - The size and complexity of the code base (for some definition of size and complexity) - The quality of the code and docs (for some definition of quality) - The skill and experience of the people involved In four years in a big tech role, my team twice inherited and had to modify a code base without any input from the original authors. One was a quagmire, the other was a resounding success: - The first was a media player control that we had to update to support a new COM interface and have a new UI. We decided that it was too complicated, and nobody understood it, so we’d reimplement it from scratch. One year later it mostly worked, but still had bugs and performance issues that the original version didn’t have. In hindsight, I suspect it would’ve been cheaper to try to revive the original code base. - The second was a music database for an app running on a mobile device. Our current one was based on the version of SQL available, but some principal engineers on another team suggested replacing it with a custom in-memory database that already shipped in another device. We argued that the original authors had left and the code was unwieldy; they argued that “it’s just code, we can read it” and its performance was known to be better. They did the work to revive it and successfully integrated it into our app. Wild success. The flip side of “it’s impossible to revive a dead system” is “don’t rewrite a working system from scratch”. Absent more research, the only way to correctly guess which situation you’re actually in is to have tons of experience.

mfld 2y ago

I can't resist to point out that, in theory, there are at least two other options avoid wasting many resources via the failing new teams: 1. Get the original dev to explain his theories (keep employees longer or engage them as consultants) 2. Make and get a "diary" of the original devs theory building. In this story, and in probably many places, the business environment however supports the explained outcome.

boricj 2y ago

The focus of this article is on the big political project that fails (the SVC), but the part that resonates most with me is the small forgotten project that lives (the SaaS). Over the years, I've built a number of contraptions under a similar set of circumstances: a technical problem usually created by an organizational issue suddenly appears that is both severe enough to threaten a project yet falls outside the core business, so it needs to be fixed both yesterday and on the cheap. Inevitably, I get saddled with it and produce a kludge that is equally effective and cursed before going back to business as usual. More than once I've learned to my horror that years later the thing is not only unmaintained yet still in place, but its usage expanded to the point where it became load-bearing, because the underlying organizational issue was never solved. In a manner of speaking, it is the opposite situation as described in the article: a complete lack of software design that somehow manages to survive in spite of a lack of knowledge building.

binary_slinger 2y ago

Also related: https://blog.codinghorror.com/commandos-infantry-and-police/ https://blog.codinghorror.com/commandos-infantry-and-police/ I work a lot in the transition area between commando and infantry aka X_10 and TEAM. I’ve also found myself on TEAM++ coming in to replace TEAM. It is difficult to explain to customers that SVC was built on a set of assumptions which I turn informed the design. Once the assumptions changed then design typically needs to change as well.

t43562 2y ago
This is a major issue. Why the code is what it is. You need a history of the assumptions so that new developers can know what's legacy and what isn't. I've never yet had a set of requirements that didn't change.
- CRConrad 2y ago
  > You need a history of the assumptions so that new developers can know what's legacy and what isn't. Isn't this (at least in part (and perhaps only approximately)) what Architecture Decision Records are for?

gervwyk 2y ago

Totally resonated with me re software lifecycle etc. Great article. I know this is not what the article is about. But perhaps exec should have spent resources and time trying to increase revenue instead of cutting cost marginally, and creating an expensive system down the road. Derailing team focus. Build vs buy… Build is almost always not cheaper. Many other reasons to build though.

jt2190 2y ago

> Knowing that [program] revival [i.e. bringing a new development team “up to speed” by having them learn the model] is a plausible future need has powerful consequences for our work. I’m not sure that most developers are willing to revive software, based on my observation that very few read anything at all, especially the source code. Instead I see a lot of adjusting the input and output of the existing program by adding a new layer. This new code is totally understood by the new dev, and they can modify/maintain it easily without worrying that they broke the existing system. It also usually duplicates something that already exists inside the system. As the process repeats more and more layers are added. I think a few lucky teams have developed a culture that encourages learning the existing code. (Popular web frameworks comes to mind as an example.)

Mawr 2y ago

> The problem is that TEAM members don’t have enough elements to build a satisfactory mental model of SVC. They need to go by a mix of the client’s interpretation of what the system should be, and what they can tell from the code that the system actually is. These views can be disconnected and contradictory. The code may tell the what and the how, but it doesn’t tell the why. Only X10 could say what was a functional requirement, what a technical necessity, what a whim, what an accident. The team has to resort to reverse engineering, extrapolating, and guessing. Hence, write down your thought process, mental model, and assumptions alongside the code. Tip: Call the process "writing documentation" instead of "commenting code".

deskr 2y ago

> The program should preferably be discarded, and the new team should be given the opportunity to resolve the problem from scratch. Yeah right. "We don't know how it works so we're going to ditch it and create it again." It works in some cases but by no means should that be the default.

datadrivenangel 2y ago
This is the right answer, except that when software has been so neglected that this is the best answer, individuals within the organization that neglected to maintain the software will realize that this is their only chance to get bugs fixed or new features added, and so the requirements will expand until the re-write probably dies.

deskr 2y ago

I have another suggestion which I'm sure played a large part in this. SVC was an unwanted child. It wasn't their "product". One employee was tasked to write it to save paying money to a "seemingly innocuous middleware SaaS". To anyone in ORG working on it, it was a dead end. No one wanted to own it and perhaps no one did. A team was asked to add features to it. Doing the ground work of actually understanding SVC had many negative consequences: * It would take a very long time, making managers not happy. It would be largely a wasted effort, since no further work was then needed on SVG. * If you became an expert on SVG, it would be yours to keep and no one wanted that.

contingencies 2y ago

I imagine that experienced cross-disciplinary designers would concur that all complex design is knowledge building, which is why documenting design decisions is important. You tend to learn this lesson when maintaining projects of nontrivial complexity over a longer period.

gr3ml1n 2y ago

This is an interesting argument for the (definitely common) pattern the author describes in the intro. A more cynical take (that I'm inclined towards is): the median software developer is simply not very good. X10 was a good developer; the people on TEAM and TEAM++ were not.

tresil 2y ago
This is absolutely my take as well. I see the points that the author brought up as additional contributing factors. However, the leading reason for this “phenomena” is that many companies are brimming with individuals (including managers) that are simply not competent or motivated enough to meet the demands of this profession. In the author’s story, this is probably why X10 decided to leave the company, because they were tired of working with incapable co-workers.

nextworddev 2y ago

Kind of a roundabout way of saying that you can’t evolve a program without fully understanding the codebase

googamooga 2y ago
Not only the codebase, but also the domain, the stakeholders, the history of requirements, etc etc etc.

wwarner 2y ago

This is really why AI is going to hit s/w development so hard. It's not merely going to make code easier to write, it's going to be a massive knowledge repository that takes a team from initial conception, through product design and finally engineering design and coding.

n_ary 2y ago
No, LLM needs excellent communicator. It can statistically spit out knowledge but someone has to embed that knowledge. Given how vague and contradictory most requirements are and how complete and excruciatingly detailed prompts must be, LLMs will be useful to generate prototypes faster to check assumptions of the lost knowledge, nothing more or less.
- namaria 2y ago
  The real trouble with LLMs is that they emulate knowledge so well. People assume they can depend on it to know things, but it is not reliable at all. A lot of traps are being laid in code by people trusting the output or behavior of LLMs.
- wwarner 2y ago
  Yes, this what i’m saying; the development process will be about writing and talking into a new tool, and then with that recorded information generating summaries, mocks prototypes and code. Definitely people would be involved. What I’m pointing out is that LLMs are natural tools for summarizing and synthesizing domain expertise, which can be naturally applied to the product development process. If an LLM based tool can be a great personal assistant, it can also be a great knowledge repository for an organization.

namaria 2y ago

People often assume that code is knowledge. They want "self explanatory" or "well documented code". Companies and managers often treat developers as interchangeable. But therein lies the mistake. Knowledge exists in mental models and team structures. Small components and systems can be understood by a person, but team structure also embodies knowledge of larger systems. People will need complementary mental models to understand a large system together. That's why adding manpower to a late project makes it later. That's why maintenance is hard and handover is harder. That's why systems devolve into big balls of mud. Because companies and managers do not respect the fact that you need people and teams who have good mental models and that mental models take time to build and share. No amount or quality of code can make up for this fact. And simulacra of ownership - having "product owners" or whatever - won't cut it. You need people to own their systems, understand them deeply. Moving people around, churn, treating developers as interchangeable, substituting rituals for deep work, accumulating 'technical debt' (deferred work as in ship now and think later) etc are all detrimental to building and sharing sound mental models.

dominicrose 2y ago
I've seen code like this: - 10 lines - every line has a different author in git blame - there are conditional branches inside conditional branches - last but not least: all conditional branches have the same behavior! How does it end up like this? Why doesn't the last commiter just delete everything and write it in a single line instead?
- namaria 2y ago
  Treating code as a means to an end, which is what I think you're describing, is just as bad as treating it as an end in itself. It should be neither an opaque black box nor a transparent 'bicycle of the mind'. In my opinion it cannot be lumped all in one category. It is an aspect of software systems. It may have many features, depending on where it is and how it is used. Code can embody knowledge, but it is not the embodiment of knowledge. It can express functionality but it is not a functional component of a system. I think aspect is the best description: when you look at a system from the source code, you see some of it. Not a projection of the system over a set of dimensions as some people seem to treat it. It is not a textual description of the system. It is the part of the system you can see when you come at it from that side.

pjbster 2y ago

Even if the organistion is fully signed up to the knowledge building philosphy, it can still be derailed if the staff aren't up to scratch. Around 12 years ago, my employer tasked me with building a quote engine for a new product they wanted to sell online. The engine needed to produce 4 additional quotes (2 lower, 2 higher) to either prevent potential walk-aways or to offer upsells to capture potential additional revenue. And it struck me at the time that this sort of hand-wavey selling tactic would be just the sort of thing that was likely to change so I put all this logic into a pure function - pass in an original quote request and the function returns a collection of alternative quote requests. And, sure enough, a couple of years later, the business decides to change the approach and offer 1 lower quote and 3 upsells. And they give the job of implementing this to another developer. I was still at the company and was known as the original developer (my name was in a comment at the top of the file, for a start) so I was asked to review the code changes. I was surprised to learn that the other dev had left the pure function untouched and had, instead, written a bunch of new logic to generate the alternative quotes. Not based on the original quote request but on the collection of alternative quotes returned from the original function. Furthermore, this new logic was placed in main procedure - mama's finest spaghetti in the making, right there. So I rejected the change and told the dev where to put the actual logic. Then I waited for the re-review request to come in. What happened instead is that the code went live anyway - the dev had simply re-raised the review request and assigned it to another dev who rubber stamped it. Looking back, I don't think all the documentation in the world would have prevented this behaviour. A better approach would be for the company to pass the changes to the original developer and to pair with another dev - like Fred Brooks' Chief Programmer plus Assistant recommendation. I was never approached for an end of year review for this developer and I left the company before them. It's not personal but I'll resign on the spot if a company I work for employs this developer in future.

t43562 2y ago

The problem I have is that only people who "know" already tend to accept this insight. i.e. explaining it to those who make decisions doesn't work that well because it doesn't fit their mental model very well - they don't understand that a large part of their asset is sitting in people's heads. That guy wants 5k more.....? Pay it. The cost of hiring someone new and training them up and relearning it all will be far higher. Don't force people back to the office, be relaxed about everything and keep the knowledge. At the same time make sure other people are learning it so you do have replacements. At the same time make sure you have proper tests so you can work with code you don't understand yet if someone leaves. At the same time try to gather all the correct and uptodate documentation and explanations somewhere.

lincpa 2y ago

[dead]

red_admiral 2y ago

Part of the story that is mentioned but not discussed is that X_10 had a fixed and tight schedule. Under that constraint you expect some compromises for writing the mental model down in a way others can learn, because that model presumably lived in her head. It might have been possible to say, great job you delivered on time, now you have as much time as you need to write your mental model down. On full pay of course, and we won't count that against you as non-technical work in the next performance review. That increases the upfront cost of SVC but is an investment that pays back interest as soon as someone else has to fix anything.

Peterthomos 2y ago

[dead]

beryilma 2y ago

> A dead program may continue to be used for execution in a computer and to produce useful results. The actual state of death becomes visible when demands for modifications of the program cannot be intelligently answered. I totally believe this. I see zombie programs at my employer, including the ones that are currently making a profit, that just don't know yet that they are dead. The Jira model of non-ownership software development being a leading cause in my opinion...