5 ms·

> What keeps the software alive are the programmers who have an accurate mental model (theory) of how it is built and works. That mental model can only be learn

> What keeps the software alive are the programmers who have an accurate mental model (theory) of how it is built and works. That mental model can only be learned by having worked on the project while it grew or by working alongside somebody who did, who can help you absorb the theory. Replace enough of the programmers, and their mental models become disconnected from the reality of the code, and the code dies. That dead code can only be replaced by new code that has been ‘grown’ by the current programmers. and > Team stability is vital for software development. Each team needs to be composed of a majority of first-generation developers. How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... To take this seriously, internal churn (even if just someone being promoted into management, or specializing on expertise in a small part of the original project, or switching projects when they get bored) is almost as costly as external churn. Five, ten years later how much is their domain knowledge still that relevant to the people working on the code day to day today? It could be harmful if they've been out of the loop and make outdated suggestions, even! Assuming and desiring your employees will stay in the exact same role indefinitely seems like a huge cause of churn.

ElevenLathe 4y ago
I've worked in incident/crisis management for several big, complicated, grown-via-acquisition companies with a tangle of internal systems that were never quite fully integrated. When TSHTF and we need some expertise from the original builders or operators of a system, it's a lot nicer to have them on another team than at another company. If they still work for us, somebody can make the case to them (or their boss) that borrowing them for a few hours to fix it is in everyone's interest. A lot of times they are in some kind of management position now and they kind of smile and say "Wow, I haven't thought about that system in a long time. Sure, I can take a look." and it's even kind of a fun diversion from whatever budget meeting they were in. If they work for somebody else, the answer is often "sorry, I'm busy" or just a hangup when you explain who you are and why you are calling.
codazoda 4y ago
I don’t wholesale agree with the writing but it does mention working with someone who worked on the project and can share the theory. In that case, a new dev is hired, and you instill all the knowledge of the system in the new dev. Once you’ve trained them enough, you can move on. New devs will adapt and create new theory and solutions. Rinse and repeat.
- majormajor 4y ago
  That's that "second gen" vs "first gen" thing but it actually leans in hard to the "no movement" right after introducing that concept - the "Each team needs to be composed of a majority of first-generation developers" quote is from that paragraph after the distinction. If you try to fix the theory knowledge/documentation/practice problems purely through insisting the "first gen" stays where they are, they're going to leave you for more interesting work, and then you're in exactly the situation you wanted to avoid. The only way out is "more effective practice of theory and knowledge transfer" but all the author seems to have to say about that is "This is a team management issue that can only be addressed by a manager who understands the fundamental nature of software development. There is no magic development process that will prevent these problems. It takes skill." E.g. that's the real hard part yet they completely punt on trying to dig into it.
- MarkMarine 4y ago
  That kind of misses what is being said in this article. The first generation programmers support a constrained population of second generation developers, and over time the second generation developers have written enough code that they become the first generation. A big component of what this article is saying is basically that knowledge transfer in the way you're describing doesn't work, and if you dig down to the theory this article is based on, new devs adapt and create their own theory and solutions was specifically called out as a problem. In case 1 of this paper, it's described as second generation programmers submitting patches to the first generation programmers that don't make use of the structure of the program, and add things that are trivially done by the existing systems. Then, a third, disconnected generation of programmers takes over the maintenance from the second generation, and they've added so much cruft to the system that the original powerful architecture can still be seen, but isn't in use because of all the additions. [0] https://pages.cs.wisc.edu/~remzi/Naur.pdf https://pages.cs.wisc.edu/~remzi/Naur.pdf
GalenErso 4y ago
> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... By documenting your work and/or mentoring a junior employee to pass on your tribal knowledge.
naravara 4y ago
An employer shouldn't want that for you either. I think what this ends up being is an argument for good documentation and onboarding so you can transmit the "theory" to other people. Reasonable amounts of churn are manageable if you do a good job of acculturating new people into the team. This has actually been one of my concerns about the switchover to WFH. I know a lot of that acculturation used to happen by osmosis and people working next to each other. I think my team doesn't do as good of a job maintaining it when we're all working separately. We're a lot more efficient now, but I think we haven't figured out a good way to share and transmit context like we used to and I worry what happens if we hit a period of flux in our company. I worry our team culture is a little more fragile now.
- jpadkins 4y ago
  > I know a lot of that acculturation used to happen by osmosis and people working next to each other. I used to think this too, but I think we need to be cautious of magical mechanisms for knowledge transference. I think the real mechanisms are code reviews, discussions of issues+code+project, consults / shared problem solving, design reviews. Just sitting next to someone doesn't work, there needs to be a shared task or problem for knowledge transfer to happen. I think all of these shared tasks happen with remote workers just as effectively, as long as you have good team comm hygiene.
  - majormajor 4y ago
    The biggest difference between remote and in-person is that many of those things can happen more easily without being scheduled. The organizational management skill bar you need to hit as a team is lower when the communication friction is much lower. Talking to your teammate who sits nearby in person is both lower friction and higher bandwidth than sending them a chat method. "Adhoc conversations" don't magically lead to knowledge transfer, the probability of any adhoc convo being long-term-meaningful is low, but they greatly increase the cumulative probability.
wpietri 4y ago
> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... To me this is the a huge incentive for building strong teams. Nobody should be indispensable; it's a disservice both to the company and the person. Pair programming, collective code ownership, intentional cross-training, good automated tests, continuous deployment, and good automated monitoring have all been very helpful to me there. And vacations! Everybody should take long vacations where they are out of contact. By managerial force if necessary.
- nickfromseattle 4y ago
  +1 on long vacations. My team is in Europe with 38 days of PTO and while it offends my American capitalistic sensibilities, it's created so much resilience because everyone is always on vacation, we've built our company ops to reflect this reality.
  - satvikpendem 4y ago
    Like Netflix's chaos monkey but for people; unplug anyone from the system and the system should still work.
    - grahamplace 4y ago
      At least one CTO has actually tried exactly that: https://danlebrero.com/2021/06/30/cto-dairy-lucky-lotto-chaos-engineering-for-teams/ https://danlebrero.com/2021/06/30/cto-dairy-lucky-lotto-chao...
  - antihipocrat 4y ago
    Is having longer PTO in your business a detriment or neutral/net gain to overall productivity in your experience?
  - yoyohello13 4y ago
    Kind of funny that being allowed to live life outside of work is offensive to American sensibilities.
banannaise 4y ago
Lead moves on to something else, everyone moves up a role, add a new junior. Repeat as often as necessary. Alternatively: just promote and pay your people as they develop, even if they're in the "same role".
- _qdwr 4y ago
  Because when you don't, your folks write reviews like this: https://www.glassdoor.com/Reviews/Employee-Review-Inspire-RVW60109261.htm https://www.glassdoor.com/Reviews/Employee-Review-Inspire-RV...
hinkley 4y ago
I worked on a project that was paying the bills while another team was getting to work with new cool stuff. What I learned from that project is that being indispensable sucks. That lesson has recurred in my career either for me, or when I got wiser, for other people. It's not enough to write reasonable code that 'works' or document it as it is now, there's a whole unmeasured and thus under-encouraged aspect around maintaining a legible edit history. I often am trying to get into the head of the bug author to figure out why they thought this was a feature in order to preserve functionality while fixing bugs. They aren't always around, and they aren't always fun to talk to. If you have the history, the person is free to work on other things. To gift them to people they mentor, or otherwise re-home them. If you don't have that, if it's all in their head, they have to leave to get away from the project. If I can figure out what you're doing I can figure out how to change one or two conditional blocks to fix a problem. Your code stays intact. If there's no history anyway, and the code is sketchy, I'm far more likely to take the code away from you by rewriting it. We can't have black boxes on the team. That's a ball-hog, and ball-hogs get benched. If you don't show your work, then you're going to find you have fewer and fewer interesting responsibilities. Because "how we got here" is also a "why" and we ask Why all the time.
- hedora 4y ago
  I agree, but am often outnumbered by people who enable "squash commits on merge" and that simultaneously add many process barriers against actually merging pull requests.
  - pnt12 4y ago
    I defend atomic PRs and squashing merge commits. It's simple, makes reviewing easier and works for 90% of the cases. Maybe I'm missing something, but I find it painful to rebase into a neat history, and my colleagues would probably hate me to police them around to do it as well.
  - Andys 4y ago
    PRs should be small units of change. PRs should be the edit history. If your process involves one PR per feature this makes them large and unwieldy, hard to review, hard to test, and more likely hard to rollback.
feoren 4y ago
> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... Grow a garden, not a tree. Let's disambiguate the word "project" from the word "system" (not "product" -- that's sales). Let's call a project a time- and scope- limited effort to create or modify some system(s) in your suite (your garden). I think it's a common mistake to mix these two up. If projects just drag on forever, then yes, employees can feel stuck. By this definition, I certainly would not want to work on one project forever. What about working on one system forever? The problem there is that to stay busy, you're forcing attention on a system that might not need attention. At the end of a project, a system should be fulfilling its purpose, and it should do so indefinitely, forever. That purpose may expire, but the system shouldn't. There should be no urgency to keep messing with a working system. So let's work in one garden forever. Systems will grow; we'll prune and re-plant them; we'll give the healthy ones the barest maintenance it needs; we'll give the dying ones the sweet release of death; we'll shape and braid them together. All your institutional knowledge about how System A works will be great when you're writing System F that depends on System A, or when we're splitting System A into Systems A1 and A2. I would absolutely love to work at a company where I can work in one garden for the rest of my life. That's my dream job. Sidebar: do not make the mistake of assuming "system" means "microservice"! That's only one of many ways to define system boundaries.
- majormajor 4y ago
  I agree with this - while at a single less-than-a-few-thousand-people-company I'd generally call that a single "garden" - and think it's a good extension on the original article. The author, though, focuses very narrowly on the "code" level instead of the "components of the system" level. If you can grow people into being the ones who know how the components of the system work together at a layer above how the code for any given component works, you can have a retention plan and keep institutional knowledge. (Or even then move people into a layer above that, as the system grows, etc). You still won't retain everyone, some people want to jump between gardens, but you will be able to retain a LOT of people, and it's much better than telling people they have to own all their individual lines of code forever.
- 4y ago
dathinab 4y ago
> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... The first project is maybe a bad example but if we say the "nth" project instead then: - not always changing members doesn't mean never changing, companies should just not want you to jump between projects all the time (except maybe as junior programmer) and you shouldn't want this either as it's hard to build up proper deep expertise this way (instead of a lot of shallow expertise) - if you are still in the same company and the company is healthy then it should be fine if you get asked questions about your previous project for some time after you switched (through not arbitrary, just as an "emergency" thing) The problem is that the concept of carrier and seniority is seriously messed up in the software industry. Often shallow knowledge about a lot of things is more valued then actual expertise in the things needed. Similar you will find a surprisingly lot of people believing that a senior engineer which doesn't try to get into team lead/people managing positions is doing so because they "surly are incompetent and shouldn't be hired" (IMHO completely rediculus but I have still seen it often enough, just most times in a less directly phrased way). But in the end senior software development skills and team lead skills are distinct skill sets. Implicitly pressuring someone who is good at solving technical problems into a team lead position (or the other way around) is just a pretty stupid decision from a human resource utilization POV. (This depends a bit on the definition of team lead and yes team leads also need to have technical skills but most important they need to have the skill to listen to their team and not get stuck on their opinions, etc.).

ElevenLathe 4y ago

I've worked in incident/crisis management for several big, complicated, grown-via-acquisition companies with a tangle of internal systems that were never quite fully integrated. When TSHTF and we need some expertise from the original builders or operators of a system, it's a lot nicer to have them on another team than at another company. If they still work for us, somebody can make the case to them (or their boss) that borrowing them for a few hours to fix it is in everyone's interest. A lot of times they are in some kind of management position now and they kind of smile and say "Wow, I haven't thought about that system in a long time. Sure, I can take a look." and it's even kind of a fun diversion from whatever budget meeting they were in. If they work for somebody else, the answer is often "sorry, I'm busy" or just a hangup when you explain who you are and why you are calling.

codazoda 4y ago

I don’t wholesale agree with the writing but it does mention working with someone who worked on the project and can share the theory. In that case, a new dev is hired, and you instill all the knowledge of the system in the new dev. Once you’ve trained them enough, you can move on. New devs will adapt and create new theory and solutions. Rinse and repeat.

majormajor 4y ago
That's that "second gen" vs "first gen" thing but it actually leans in hard to the "no movement" right after introducing that concept - the "Each team needs to be composed of a majority of first-generation developers" quote is from that paragraph after the distinction. If you try to fix the theory knowledge/documentation/practice problems purely through insisting the "first gen" stays where they are, they're going to leave you for more interesting work, and then you're in exactly the situation you wanted to avoid. The only way out is "more effective practice of theory and knowledge transfer" but all the author seems to have to say about that is "This is a team management issue that can only be addressed by a manager who understands the fundamental nature of software development. There is no magic development process that will prevent these problems. It takes skill." E.g. that's the real hard part yet they completely punt on trying to dig into it.
MarkMarine 4y ago
That kind of misses what is being said in this article. The first generation programmers support a constrained population of second generation developers, and over time the second generation developers have written enough code that they become the first generation. A big component of what this article is saying is basically that knowledge transfer in the way you're describing doesn't work, and if you dig down to the theory this article is based on, new devs adapt and create their own theory and solutions was specifically called out as a problem. In case 1 of this paper, it's described as second generation programmers submitting patches to the first generation programmers that don't make use of the structure of the program, and add things that are trivially done by the existing systems. Then, a third, disconnected generation of programmers takes over the maintenance from the second generation, and they've added so much cruft to the system that the original powerful architecture can still be seen, but isn't in use because of all the additions. [0] https://pages.cs.wisc.edu/~remzi/Naur.pdf https://pages.cs.wisc.edu/~remzi/Naur.pdf

GalenErso 4y ago

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... By documenting your work and/or mentoring a junior employee to pass on your tribal knowledge.

naravara 4y ago

An employer shouldn't want that for you either. I think what this ends up being is an argument for good documentation and onboarding so you can transmit the "theory" to other people. Reasonable amounts of churn are manageable if you do a good job of acculturating new people into the team. This has actually been one of my concerns about the switchover to WFH. I know a lot of that acculturation used to happen by osmosis and people working next to each other. I think my team doesn't do as good of a job maintaining it when we're all working separately. We're a lot more efficient now, but I think we haven't figured out a good way to share and transmit context like we used to and I worry what happens if we hit a period of flux in our company. I worry our team culture is a little more fragile now.

jpadkins 4y ago
> I know a lot of that acculturation used to happen by osmosis and people working next to each other. I used to think this too, but I think we need to be cautious of magical mechanisms for knowledge transference. I think the real mechanisms are code reviews, discussions of issues+code+project, consults / shared problem solving, design reviews. Just sitting next to someone doesn't work, there needs to be a shared task or problem for knowledge transfer to happen. I think all of these shared tasks happen with remote workers just as effectively, as long as you have good team comm hygiene.
- majormajor 4y ago
  The biggest difference between remote and in-person is that many of those things can happen more easily without being scheduled. The organizational management skill bar you need to hit as a team is lower when the communication friction is much lower. Talking to your teammate who sits nearby in person is both lower friction and higher bandwidth than sending them a chat method. "Adhoc conversations" don't magically lead to knowledge transfer, the probability of any adhoc convo being long-term-meaningful is low, but they greatly increase the cumulative probability.

wpietri 4y ago

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... To me this is the a huge incentive for building strong teams. Nobody should be indispensable; it's a disservice both to the company and the person. Pair programming, collective code ownership, intentional cross-training, good automated tests, continuous deployment, and good automated monitoring have all been very helpful to me there. And vacations! Everybody should take long vacations where they are out of contact. By managerial force if necessary.

nickfromseattle 4y ago
+1 on long vacations. My team is in Europe with 38 days of PTO and while it offends my American capitalistic sensibilities, it's created so much resilience because everyone is always on vacation, we've built our company ops to reflect this reality.
- satvikpendem 4y ago
  Like Netflix's chaos monkey but for people; unplug anyone from the system and the system should still work.
  - grahamplace 4y ago
    At least one CTO has actually tried exactly that: https://danlebrero.com/2021/06/30/cto-dairy-lucky-lotto-chaos-engineering-for-teams/ https://danlebrero.com/2021/06/30/cto-dairy-lucky-lotto-chao...
- antihipocrat 4y ago
  Is having longer PTO in your business a detriment or neutral/net gain to overall productivity in your experience?
- yoyohello13 4y ago
  Kind of funny that being allowed to live life outside of work is offensive to American sensibilities.

banannaise 4y ago

Lead moves on to something else, everyone moves up a role, add a new junior. Repeat as often as necessary. Alternatively: just promote and pay your people as they develop, even if they're in the "same role".

_qdwr 4y ago
Because when you don't, your folks write reviews like this: https://www.glassdoor.com/Reviews/Employee-Review-Inspire-RVW60109261.htm https://www.glassdoor.com/Reviews/Employee-Review-Inspire-RV...

hinkley 4y ago

I worked on a project that was paying the bills while another team was getting to work with new cool stuff. What I learned from that project is that being indispensable sucks. That lesson has recurred in my career either for me, or when I got wiser, for other people. It's not enough to write reasonable code that 'works' or document it as it is now, there's a whole unmeasured and thus under-encouraged aspect around maintaining a legible edit history. I often am trying to get into the head of the bug author to figure out why they thought this was a feature in order to preserve functionality while fixing bugs. They aren't always around, and they aren't always fun to talk to. If you have the history, the person is free to work on other things. To gift them to people they mentor, or otherwise re-home them. If you don't have that, if it's all in their head, they have to leave to get away from the project. If I can figure out what you're doing I can figure out how to change one or two conditional blocks to fix a problem. Your code stays intact. If there's no history anyway, and the code is sketchy, I'm far more likely to take the code away from you by rewriting it. We can't have black boxes on the team. That's a ball-hog, and ball-hogs get benched. If you don't show your work, then you're going to find you have fewer and fewer interesting responsibilities. Because "how we got here" is also a "why" and we ask Why all the time.

hedora 4y ago
I agree, but am often outnumbered by people who enable "squash commits on merge" and that simultaneously add many process barriers against actually merging pull requests.
- pnt12 4y ago
  I defend atomic PRs and squashing merge commits. It's simple, makes reviewing easier and works for 90% of the cases. Maybe I'm missing something, but I find it painful to rebase into a neat history, and my colleagues would probably hate me to police them around to do it as well.
- Andys 4y ago
  PRs should be small units of change. PRs should be the edit history. If your process involves one PR per feature this makes them large and unwieldy, hard to review, hard to test, and more likely hard to rollback.

feoren 4y ago

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... Grow a garden, not a tree. Let's disambiguate the word "project" from the word "system" (not "product" -- that's sales). Let's call a project a time- and scope- limited effort to create or modify some system(s) in your suite (your garden). I think it's a common mistake to mix these two up. If projects just drag on forever, then yes, employees can feel stuck. By this definition, I certainly would not want to work on one project forever. What about working on one system forever? The problem there is that to stay busy, you're forcing attention on a system that might not need attention. At the end of a project, a system should be fulfilling its purpose, and it should do so indefinitely, forever. That purpose may expire, but the system shouldn't. There should be no urgency to keep messing with a working system. So let's work in one garden forever. Systems will grow; we'll prune and re-plant them; we'll give the healthy ones the barest maintenance it needs; we'll give the dying ones the sweet release of death; we'll shape and braid them together. All your institutional knowledge about how System A works will be great when you're writing System F that depends on System A, or when we're splitting System A into Systems A1 and A2. I would absolutely love to work at a company where I can work in one garden for the rest of my life. That's my dream job. Sidebar: do not make the mistake of assuming "system" means "microservice"! That's only one of many ways to define system boundaries.

majormajor 4y ago
I agree with this - while at a single less-than-a-few-thousand-people-company I'd generally call that a single "garden" - and think it's a good extension on the original article. The author, though, focuses very narrowly on the "code" level instead of the "components of the system" level. If you can grow people into being the ones who know how the components of the system work together at a layer above how the code for any given component works, you can have a retention plan and keep institutional knowledge. (Or even then move people into a layer above that, as the system grows, etc). You still won't retain everyone, some people want to jump between gardens, but you will be able to retain a LOT of people, and it's much better than telling people they have to own all their individual lines of code forever.
4y ago

dathinab 4y ago

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever... The first project is maybe a bad example but if we say the "nth" project instead then: - not always changing members doesn't mean never changing, companies should just not want you to jump between projects all the time (except maybe as junior programmer) and you shouldn't want this either as it's hard to build up proper deep expertise this way (instead of a lot of shallow expertise) - if you are still in the same company and the company is healthy then it should be fine if you get asked questions about your previous project for some time after you switched (through not arbitrary, just as an "emergency" thing) The problem is that the concept of carrier and seniority is seriously messed up in the software industry. Often shallow knowledge about a lot of things is more valued then actual expertise in the things needed. Similar you will find a surprisingly lot of people believing that a senior engineer which doesn't try to get into team lead/people managing positions is doing so because they "surly are incompetent and shouldn't be hired" (IMHO completely rediculus but I have still seen it often enough, just most times in a less directly phrased way). But in the end senior software development skills and team lead skills are distinct skill sets. Implicitly pressuring someone who is good at solving technical problems into a team lead position (or the other way around) is just a pretty stupid decision from a human resource utilization POV. (This depends a bit on the definition of team lead and yes team leads also need to have technical skills but most important they need to have the skill to listen to their team and not get stuck on their opinions, etc.).