4 ms·

The thing about churn (of developers/contributors) rings so true to me from my career, and I think is really under-appreciated. In "the industry", or just among

The thing about churn (of developers/contributors) rings so true to me from my career, and I think is really under-appreciated. In "the industry", or just among "people who make things in teams" (including open source). > Constant churn in a software development team, both among the programmers and designers, is absolutely devastating. It is the death knell for a software project. Makes deadlines meaningless. It turns software into a disposable, single-use product like a paper towel. Anything that increases team member churn threatens the very viability of the project and the software it’s creating. From an essay that is worth reading in full, "Theory-building and why employee churn is lethal to software companies": https://www.baldurbjarnason.com/2022/theory-building/ https://www.baldurbjarnason.com/2022/theory-building/ I think it's exactly right about the importance of "mental models" and "theory building", and if we could keep that in mind we could build better more maintainable software. ("More maintainable" over the long-term I think is almost synonymous with "has easy to grasp mental models that are sufficient to guide your work with the system")

thrwaway011023 4y ago
Churn is worse now that we have split everything into microservices that often use their own libraries, frameworks and languages. That adds a lot of overhead when employees inherit a pet project
- ericmcer 4y ago
  Really true, ownership is a huge part of microservices. I have been on teams that have adopted orphaned services, then been called into meetings to advise on use of a service I am barely aware of. It is a really easy way to make your employees overwhelmed, but you can always hire someone else to take ownership once they quit.
- Pet_Ant 4y ago
  I feel like microservices are specifically chosen because they work best with lots of churn. You can throw away the code. You only need to learn only the code in your service. It is an architecture (often) chose not for it's technical merits but rather how it works in high-churn environments.
  - wmichelin 4y ago
    The same exact results can be yielded by OOP interfaces and implementations, so I unfortunately don't feel this is a valid argument for microservices.
    - Spivak 4y ago
      What you’re describing is just microservices but with different message passing. So I think this is overall very pro-microservices.
      - namaria 4y ago
        All of this is just structured programming with extra steps
    - blindhippo 4y ago
      The point for "microservices" to me is to separate the components of an integrated system to achieve scaling factors independent of other related but not directly coupled parts of the system. Most systems in my experience don't really benefit from separation of the entire application domain into a set of independent systems. Most use cases don't benefit from decoupling a request/response in a series of asynchronous calls across 20 different APIs. So, most systems can and should be implemented as a simple abstracted interface inside a single application stack. But then, no one ever gets promoted/hired/funded to build the most optimal solution.
    - KronisLV 4y ago
      > The same exact results can be yielded by OOP interfaces and implementations, so I unfortunately don't feel this is a valid argument for microservices. I'm not sure about this, there's so much going on in most software, a lot of the ruin comes from the dependencies around your code, like libraries and frameworks. Let's take web development as an example. Suppose you need to query the database in some legacy project - well, it doesn't have a framework that supports proper logging of each query/parameter for debugging, it doesn't have any codegen capabilities, it has a wonky connection pool implementation. And yet, if you want to work within the same codebase, you have to use it. What could take a bit of codegen and an hour of work now can take close to a day, all because you're pigeonholed into using what's already there for "consistency". Suppose instead you now want to have a task queue. Well, instead of RabbitMQ or something of the like being used, instead you have to work with what's present - a bunch of pre-existing abstractions and a DB centric implementation (with arguably poor documentation) that the business requirements have largely outgrown, so now you'll need to hack around the existing code, hope that you break nothing there, all while attempting to create something passable (which may or may not be viable with this approach, depending on the use case). And doing logging? Well, maybe the team didn't do structured logging or log shipping, so you have to be very frugal with the logs, or spend time setting up Logrotate or an alternative for that monolithic app, just so you can turn on some additional logging and make sure that the eventual rollout actually works. Well, what about the actual database or maybe tests? Be prepared that the test framework will be somewhat out of date, some of the test utilities won't quite fit what you'll try to do and perhaps your current ORM and DB choices won't really work well with the idea of letting integration tests test actual DB statements and migrations against a DB instance that would be setup for the tests and torn down automatically afterwards (say, a PostgreSQL/MariaDB container somewhere, as opposed to MS SQL or Oracle), so you won't be able to test every schema migration that easily. You might even run into a situation where you need to run a relatively recent runtime (say JDK 11) but the project is stuck on an older release (JDK 8), whereas migrating everything isn't going to be done for now. So you can't do what you need now, because the libraries don't support your runtime. Worse yet if the monolithic project has dozens of business use cases, whereas the bit of code that's preventing the migration is needed for just one or two of them. I won't even get into instrumentation and scaling parts of the total solution up or down, or things like configuration, since that'd make this needlessly long. I think that microservices indeed are a good choice for mitigating some of these issues, by untying your hands - the impact of issues is lessened, to a per-service basis. An individual service might therefore rot and this rot could be addressed separately, without slowing everything else down. Even to the point where the old implementation can be thrown out and rewritten, if that's ever needed. However, microservices come at a possibly huge operational and complexity cost otherwise, to the point where they might either make the project late, hard to work with, or unfeasible altogether. And it's hard to know which approach will be the best for any given situation. Starting out with monoliths makes sense, because most smaller projects will never need to be split up, but when you need to do that, you will really need to. Modular monoliths (e.g. feature flags for enabled/disabled bits of functionality) can be an intermediate step. In an ideal world, programming language module systems would be enough. We don't live in a perfect world.
- z3t4 4y ago
  One idea of Micro services is that you can rewrite the entire service. There are more combinations of libraries, frameworks and languages then there are programmers. Most programmers however has a favourite stack that they are 10x more productive in.
dustbitying 4y ago
on the other hand, there's a strong incentive for group-organizations, specially profit-driven and "the show MUST go on" kinds of organizations to never depend on a single (or a small sub-group) of people to keep things going; lest somebody important gets hit by the proverbial bus. IMO, this is also part of the challenges that come with dealing in software (understood as some sort of novel "substance"); in great part we (as a culture/civilization) haven't figured out how to deal with software (and in general, with digital artifacts) in the best way.
- jffhn 4y ago
  Preventing people from being productive by fear of the bus factor, is like throwing them yourself under the bus.
  - bitwize 4y ago
    Well, it's how business works. Maximal productivity of individual developers is not necessarily as valuable to the business as consistent forward progress that's resilient to risks, especially human risks like cranky developers quitting. As much as Office Space is supposed to be a pastiche of corporate toxicity, Initech was perhaps more honest than most real-world companies. They had banners everywhere reading "Is it good for the COMPANY?" In the real world the same principle applies but you have to keep it in mind yourself. Any time corporate makes a decision which you think of as foolish because it hampers or disadvantages the line workers, ask yourself... Is it good for the COMPANY?
TremendousJudge 4y ago
That's a great essay, thank you. I have been thinking this exact same thing for over a year now, it's nice to see it written out so clearly.
- simongray 4y ago
  He paraphrases a classic paper by Peter Naur which is worth reading in its entirety.
hot_gril 4y ago
Unpredictable churn from layoffs is probably bad, but I view churn in general as an almost necessary thing for software dev. Every great thing I've done on a SWE team has been something the previous person in my position wouldn't do, and every bad thing I've done was redone by someone after me. The truly well-designed and maintainable systems survive multiple owners, so there's a trend towards better. And it's a bad sign for a system to rely on someone.
autokad 4y ago
is this on the wrong article... if so please delete
- jrochkind1 4y ago
  The quote I quoted is quoted at the end of the very short OP, I took it from there.
lo_zamoyski 4y ago
A large part of the value is not the software, but the knowledge built around the software, and that knowledge must be known and understood by someone (the developers). This includes everything from domain knowledge, the history of why things are the way they are, what failed and what succeeded, the technical idiosyncrasies, etc. The software is in a sense secondary, certainly secondary to business goals as its entire purpose is to serve business goals. Software is a means and an instrument, not an end. But perverse incentives make churn attractive. Switching jobs is often the best way to raise your income, for example.
- ozim 4y ago
  Now we have to take this argument and explain it to people wanting to get rid of developers and use low code or chat gpt.
  - datavirtue 4y ago
    I just tell them to go right ahead.
  - weatherlite 4y ago
    Who will operate the low code and chat gpt though ? It's not clear to me at all that the humans can be totally skipped in the near future. Maybe reduce their headcount but even that one is debatable (if devs become more productive they will get more tasks, create more companies etc).

thrwaway011023 4y ago

Churn is worse now that we have split everything into microservices that often use their own libraries, frameworks and languages. That adds a lot of overhead when employees inherit a pet project

ericmcer 4y ago
Really true, ownership is a huge part of microservices. I have been on teams that have adopted orphaned services, then been called into meetings to advise on use of a service I am barely aware of. It is a really easy way to make your employees overwhelmed, but you can always hire someone else to take ownership once they quit.
Pet_Ant 4y ago
I feel like microservices are specifically chosen because they work best with lots of churn. You can throw away the code. You only need to learn only the code in your service. It is an architecture (often) chose not for it's technical merits but rather how it works in high-churn environments.
- wmichelin 4y ago
  The same exact results can be yielded by OOP interfaces and implementations, so I unfortunately don't feel this is a valid argument for microservices.
  - Spivak 4y ago
    What you’re describing is just microservices but with different message passing. So I think this is overall very pro-microservices.
    - namaria 4y ago
      All of this is just structured programming with extra steps
  - blindhippo 4y ago
    The point for "microservices" to me is to separate the components of an integrated system to achieve scaling factors independent of other related but not directly coupled parts of the system. Most systems in my experience don't really benefit from separation of the entire application domain into a set of independent systems. Most use cases don't benefit from decoupling a request/response in a series of asynchronous calls across 20 different APIs. So, most systems can and should be implemented as a simple abstracted interface inside a single application stack. But then, no one ever gets promoted/hired/funded to build the most optimal solution.
  - KronisLV 4y ago
    > The same exact results can be yielded by OOP interfaces and implementations, so I unfortunately don't feel this is a valid argument for microservices. I'm not sure about this, there's so much going on in most software, a lot of the ruin comes from the dependencies around your code, like libraries and frameworks. Let's take web development as an example. Suppose you need to query the database in some legacy project - well, it doesn't have a framework that supports proper logging of each query/parameter for debugging, it doesn't have any codegen capabilities, it has a wonky connection pool implementation. And yet, if you want to work within the same codebase, you have to use it. What could take a bit of codegen and an hour of work now can take close to a day, all because you're pigeonholed into using what's already there for "consistency". Suppose instead you now want to have a task queue. Well, instead of RabbitMQ or something of the like being used, instead you have to work with what's present - a bunch of pre-existing abstractions and a DB centric implementation (with arguably poor documentation) that the business requirements have largely outgrown, so now you'll need to hack around the existing code, hope that you break nothing there, all while attempting to create something passable (which may or may not be viable with this approach, depending on the use case). And doing logging? Well, maybe the team didn't do structured logging or log shipping, so you have to be very frugal with the logs, or spend time setting up Logrotate or an alternative for that monolithic app, just so you can turn on some additional logging and make sure that the eventual rollout actually works. Well, what about the actual database or maybe tests? Be prepared that the test framework will be somewhat out of date, some of the test utilities won't quite fit what you'll try to do and perhaps your current ORM and DB choices won't really work well with the idea of letting integration tests test actual DB statements and migrations against a DB instance that would be setup for the tests and torn down automatically afterwards (say, a PostgreSQL/MariaDB container somewhere, as opposed to MS SQL or Oracle), so you won't be able to test every schema migration that easily. You might even run into a situation where you need to run a relatively recent runtime (say JDK 11) but the project is stuck on an older release (JDK 8), whereas migrating everything isn't going to be done for now. So you can't do what you need now, because the libraries don't support your runtime. Worse yet if the monolithic project has dozens of business use cases, whereas the bit of code that's preventing the migration is needed for just one or two of them. I won't even get into instrumentation and scaling parts of the total solution up or down, or things like configuration, since that'd make this needlessly long. I think that microservices indeed are a good choice for mitigating some of these issues, by untying your hands - the impact of issues is lessened, to a per-service basis. An individual service might therefore rot and this rot could be addressed separately, without slowing everything else down. Even to the point where the old implementation can be thrown out and rewritten, if that's ever needed. However, microservices come at a possibly huge operational and complexity cost otherwise, to the point where they might either make the project late, hard to work with, or unfeasible altogether. And it's hard to know which approach will be the best for any given situation. Starting out with monoliths makes sense, because most smaller projects will never need to be split up, but when you need to do that, you will really need to. Modular monoliths (e.g. feature flags for enabled/disabled bits of functionality) can be an intermediate step. In an ideal world, programming language module systems would be enough. We don't live in a perfect world.
z3t4 4y ago
One idea of Micro services is that you can rewrite the entire service. There are more combinations of libraries, frameworks and languages then there are programmers. Most programmers however has a favourite stack that they are 10x more productive in.

dustbitying 4y ago

on the other hand, there's a strong incentive for group-organizations, specially profit-driven and "the show MUST go on" kinds of organizations to never depend on a single (or a small sub-group) of people to keep things going; lest somebody important gets hit by the proverbial bus. IMO, this is also part of the challenges that come with dealing in software (understood as some sort of novel "substance"); in great part we (as a culture/civilization) haven't figured out how to deal with software (and in general, with digital artifacts) in the best way.

jffhn 4y ago
Preventing people from being productive by fear of the bus factor, is like throwing them yourself under the bus.
- bitwize 4y ago
  Well, it's how business works. Maximal productivity of individual developers is not necessarily as valuable to the business as consistent forward progress that's resilient to risks, especially human risks like cranky developers quitting. As much as Office Space is supposed to be a pastiche of corporate toxicity, Initech was perhaps more honest than most real-world companies. They had banners everywhere reading "Is it good for the COMPANY?" In the real world the same principle applies but you have to keep it in mind yourself. Any time corporate makes a decision which you think of as foolish because it hampers or disadvantages the line workers, ask yourself... Is it good for the COMPANY?

TremendousJudge 4y ago

That's a great essay, thank you. I have been thinking this exact same thing for over a year now, it's nice to see it written out so clearly.

simongray 4y ago
He paraphrases a classic paper by Peter Naur which is worth reading in its entirety.

hot_gril 4y ago

Unpredictable churn from layoffs is probably bad, but I view churn in general as an almost necessary thing for software dev. Every great thing I've done on a SWE team has been something the previous person in my position wouldn't do, and every bad thing I've done was redone by someone after me. The truly well-designed and maintainable systems survive multiple owners, so there's a trend towards better. And it's a bad sign for a system to rely on someone.

autokad 4y ago

is this on the wrong article... if so please delete

jrochkind1 4y ago
The quote I quoted is quoted at the end of the very short OP, I took it from there.

lo_zamoyski 4y ago

A large part of the value is not the software, but the knowledge built around the software, and that knowledge must be known and understood by someone (the developers). This includes everything from domain knowledge, the history of why things are the way they are, what failed and what succeeded, the technical idiosyncrasies, etc. The software is in a sense secondary, certainly secondary to business goals as its entire purpose is to serve business goals. Software is a means and an instrument, not an end. But perverse incentives make churn attractive. Switching jobs is often the best way to raise your income, for example.

ozim 4y ago
Now we have to take this argument and explain it to people wanting to get rid of developers and use low code or chat gpt.
- datavirtue 4y ago
  I just tell them to go right ahead.
- weatherlite 4y ago
  Who will operate the low code and chat gpt though ? It's not clear to me at all that the humans can be totally skipped in the near future. Maybe reduce their headcount but even that one is debatable (if devs become more productive they will get more tasks, create more companies etc).