7 ms·

Institutional memory and reverse smuggling

150 points by adulau 15y ago

siavosh 15y ago
This is sad, scary, and as far as I've seen an inherent part of any long running engineering project. As software developers, we have a hard time understanding the rationale of some of the code we wrote last week, let alone someone else's code from a year ago (I don't want to imagine any of my company's code running 30 years from now!). In a recent series of design meetings I was running for a large project, I started to tackle some of these problems by emailing out the final decisions we had made at the end of each meeting. The following week we'd still have problems in understanding/remembering the chain of arguments that led to the final decisions, and we'd lose some time re-tracing our steps. So I then adjusted to not only summarizing the decisions at the end of each meeting, but noting step by step the arguments/debates that led to it. Then at the beginning of each meeting I'd review them to the group. This worked, but it was an awful lot of note taking. I almost wonder that now that more and more of these discussions are done through work email, IM, version controlled documentation (and code), the job of future archaeologists will be much more manageable (though still hard and fraught with risk). Maybe more and more tools will develop to piece together a project's evolution, rationale, and implicit assumptions and decisions.
- zdw 15y ago
  The problem with going through people's old email and such is the signal/noise is extremely low, and often of a timely nature, and may not represent the final outcome, especially if work has been done on a system after the fact. What we need is testable documentation. I do IT work most of the time, and document using a mashup of Markdown and text snippets in a specific format for network documentation, all kept in version control. I then have a motley collection of scripts and tools that run against my text files and do simple testing like pinging all the hosts mentioned, checking DNS entries, etc. Frequently I'll write up docs before I start work, then by the end of work, all the tests will past. It's TDD applied to network design.
  - abecedarius 15y ago
    Right. Here's my script for this sort of thing: https://github.com/darius/tush https://github.com/darius/tush
- bane 15y ago
  I almost wonder that now that more and more of these discussions are done through work email, IM, version controlled documentation (and code), the job of future archaeologists will be much more manageable (though still hard and fraught with risk). Maybe more and more tools will develop to piece together a project's evolution, rationale, and implicit assumptions and decisions. It's funny, Wave was actually a really fabulous tool for exactly this.
- roguecoder 15y ago
  We aim for self-documenting code. If there is a decision that absolutely can not be reflected in the code itself, a comment goes with the code as close to where the decision is implemented as possible. We have a 10-year-old code base that is doing well and continues to support rapid, robust and maintainable development. The learning curve is higher than if the documentation were external to the code base and accurate diagrams existed, but unlike those artifacts the code will always be right about what the code is doing.
bostonvaulter2 15y ago
Side note: The scrolling on this page is horribly broken on Android. I can't even read the whole article because I can't scroll down. After loading the article on m laptop I must say that it is a fascinating article and will change how I think of documentation. For example, soon my company will be making a change that will render the old wiki url's incorrect, the problem is that many older documents will still refer to the old url's. I think we should have some sort of list of the old wiki url's (there is an even older set of url's as well) to help future archeologists in piecing together the information.
- Mz 15y ago
  Side note: The scrolling on this page is horribly broken on Android. I can't even read the whole article because I can't scroll down. I'm on a netbook and it was pretty bad. But, yes, excellent and thought provoking article.
- karamazov 15y ago
  Out of curiosity, why don't you have the old url just point to the new one?
- rubyrescue 15y ago
  hi - any ideas how to fix it? wrttn.in is my site and i can make a change to the css or rendering if necessary...
  - swang 15y ago
    It scrolls very slowly on my orig iPad. Also iOS scroll to top feature seems broken
  - evmar 15y ago
    (I am a Chrome developer.) I glanced at your scroll.css and saw a bunch of rules that attempt to fiddle with the overflow/position attributes. This sort of thing confuses all browsers. Why not just use a normal scrollbar? It's fine to change the way it looks it if you want using the -webkit-scrollbar stuff you've done, but hacking out the normal web page scroll bar confuses everything. There are, for example, optimization opportunities in web rendering engines that only hit when the page is "normal" (which probably explains the poor performance other people are encountering). (I noticed even on desktop Chrome the scrolling isn't smooth, unlike other pages. Probably for the same reason.)
Mz 15y ago
I've been in the same job only five years and I have newer people coming to my desk asking "what do they mean by that?" because these emails get sent out with instructions that don't really make sense with current processes. I can infer what folks above me are trying to tell me to do, but the phrase used is a case of institutional memory and the process described/referenced is obsolete. No one has bothered to update the phrasing to fit the current situation and newer employees then don't get a detailed explanation of what it means (what the goal or purpose is) so it is slowly turning into gibberish. Over a longer scale, I can well imagine this can get much uglier. And I have an entry level job, not some high paid job with super specialized training and all that.
- a_a_r_o_n 15y ago
  This is like a family of 1st, 2nd and 3rd generation immigrants. The 1st generation (the managers sending down gibberish instructions) use language that most people don't understand and refer to people, places and customs from the old country that have no meaning to anyone here. The 2nd generation (you) are responsible for keeping communication going in both directions, and making substitutions and translations in both directions, sometimes changing words and orders into something that makes sense even though it wasn't at all what was asked for.
  - losethos 15y ago
    Sometimes, I say naming things is the most overlooked programming skill there is. Being well-read probably helps you to use standard names. (Probably makes you a copy-monkey, though. I just made that word up.) A bit of wisdom is... at some point an entirely fresh word is useful. The alternative is encumbered words that mislead and make things worse. I shamelessly groom names in LoseThos by renaming globally across files. I behave as though I have no users. I get pissed because I get no emails. I guess there was no reason for a new word. "Copycat" exists. However, they speak of "Aping" someone as copying. I wanted to capture some of that, possibly throwing-in planet of the apes. The standards NAZIs would prefer "copycat". "Surrender-monkey" is a good word. I'm a surrender monkey and you're a copy-monkey-planet-of-the-apes-unix-nazi-heathen. God says... C:\LoseThos\www.losethos.com\text\WEALTH.TXT scarce any thing but some miserable pasture, just sufficient to keep alive a few straggling, half-starved cattle; the farm, though much overstocked in proportion to what would be necessary for its complete cultivation, being very frequently overstocked in proportion to its actual produce. A portion of this waste land, however, after having been pastured in this wretched manner for six or seven years together, may be ploughed up, when it will yield, perhaps, a poor crop or two of bad oats, or of som
losethos 15y ago
LoseThos was written entirely by me on my time, except the file compression routine which I wrote at Ticketmaster. The portion I kept was what I got from a magazine.
DanBC 15y ago
In theory some quality systems (ISO 900x etc) should help with institutional memory. Bob has a job; he performs a series of tasks to do that job; he writes down what he does for each task, including what information (also product) comes in (and who from) and what information (also product) goes out (and who to), also including all the documentation he needs to complete for each task. Someone else edits that puts it in the company QA manual. In future new employees just read that documentation, and maybe do a bit of shadowing of existing employees. Unfortunately, that's not what happens. What happens is that Bob has a job, but Charles (Bob's boss) writes a bunch of stuff that he thinks Bob should be doing. And he doesn't write sentences like "Set the machine for 190 degree Celsius. Place widgets in machine when it's at temperature. Leave widgets in machine for four hours." but he writes abstract ambiguous business speak that people can't use to actually do the job. Then Edwin (Charles's boss) adds another bunch of stuff. Then that gets edited into a god-awful mish-mash. The people who did the software (with the "help" of Edwin) add some stuff about information. Bob gets to work to an impenetrable document that bears little relation to his actual work flow.
- bane 15y ago
  You forgot the last parts: Charles acts aghast when Bob discusses with him the lack of actual direction and information in the document, they have an argument where Charles pronounces that if Bob were an expert in his field he should just know what to do anyway. Later, Bob presents his work to Charles, who pronounces it "great!". So they move up to present it to Edwin for final sign-off, where Edwin pronounces it "crap! not what it's supposed to be at all!". Then takes the work, sells it anyway, pockets the bonus, trickles a couple percent down to Charles and Bob gets fired for incompetence three days before Christmas.
- groby_b 15y ago
  You seem to miss the part in the original post where he states that things are running just fine. Everybody knows Bob has to set the machine to 190 degrees. What they lack is the knowledge why it's 190 degrees. That's a bit harder to just pin on management. Anybody who's worked on a project for a long time has seen that drain of knowledge. Alice, who wrote subsystem XYZ, leaves for greener pastures. Bianca and Christy worked on it with her, but they were juniors when they first started. Some practices, they just adopted without thinking about the why. Now XYZ needs to be ported to a new environment, call it dotWEB. The question why things were done that way inevitably arises. Unfortunately, Alice works for a competitor - talking to her would violate all sorts of security rules. That is the problem he's talking about. And like the original author, I'm not sure there's an easy to fix this. Not even if everybody including the bosses is supremely competent.
retroafroman 15y ago
Original link and discussion here: http://news.ycombinator.com/item?id=3311015 http://news.ycombinator.com/item?id=3311015
16s 15y ago
I've seen memory drains during down-turns in the economy. The company will offer the oldest, most-experienced, highest paid employees a buy-out or they'll offer them five years extra pension/retirement if they leave now. Mgt figures that consultants can be hired if they ever need X, Y or Z done again at a fraction of the cost of the employees. Many senior employees accept the offers, and then later mgt realizes that 95% of their Fortran or Cobol (or name your old programming language/product) experience has vanished. Thank god the software is mostly in maintenance mode, but client problems do still arise and what once took 15 mins to fix, now requires a week and phone calls to retirees begging them to help fix a client problem. They'll pay them loads of money too just for a few minutes of work. Consultants turn out to lack the domain specific knowledge required to fix the problem. Bottom line is that sometimes, mgt is so focused on hitting short-term financial goals, that they lose site of the big, long-term picture. Business needs less 20 something MBAs who only see this quarterly report and more managers who understand that business is a decades long proposition and that senior employees are key to maintaining an edge and passing along knowledge.
- yummyfajitas 15y ago
  It's easy to blame the popular meme of "short sighted management", but they aren't necessarily as short sighted as you think. Often management gets rid of the old guard because the old guard actively prevents the business from improving it's processes. I know someone (maybe multiple people) who works at a large financial institution. A particular department there is running on ancient systems using ancient processes, most of which can and should be properly outsourced to $BIGVENDOR. The fed agrees, and is constantly hitting them with MRIA's (Matters Requiring Immediate Attention). But right now there are many managers of 50-100 employees who will become redundant when 49 of their employees are replaced by a few servers in NJ. These people tend to passive aggressively slow down the replacement, hoping to delay it until they retire. There are other people who have stable jobs, based not so much on competence as the fact that they hold lots of institutional memory. They don't want to help build a system that will make their knowledge obsolete. When management fires these people, it doesn't do so because it doesn't understand their value. It does so because they are actively hindering the company from moving forward. Sometimes this process fails as you describe. But when properly managed, this process results in a streamlined department half the size it was before, processes properly documented, everything automated, where people are fighting to improve efficiency from 99.7% to 99.9%.
JabavuAdams 15y ago
Can we learn anything from legal discovery processes?
mikerg87 15y ago
Lack of a succession plan is plaguing many industries. I am in the category of grey hair, no hair, written a book. When I die, the code dies with me
jwatte 15y ago
I'm sure they saved money in the short run by going with the cheapest option each time they tried to manage their documentation the last time around. On the other hand -- at the time, they didn't know which parts would be valuable, and which parts would be wasted effort, so it probably was the right choice. However, if a major incident were to happen around that plant, could the company tell the emergency responders what they would need to know?

150 points by adulau 15y ago

This is sad, scary, and as far as I've seen an inherent part of any long running engineering project. As software developers, we have a hard time understanding the rationale of some of the code we wrote last week, let alone someone else's code from a year ago (I don't want to imagine any of my company's code running 30 years from now!). In a recent series of design meetings I was running for a large project, I started to tackle some of these problems by emailing out the final decisions we had made at the end of each meeting. The following week we'd still have problems in understanding/remembering the chain of arguments that led to the final decisions, and we'd lose some time re-tracing our steps. So I then adjusted to not only summarizing the decisions at the end of each meeting, but noting step by step the arguments/debates that led to it. Then at the beginning of each meeting I'd review them to the group. This worked, but it was an awful lot of note taking. I almost wonder that now that more and more of these discussions are done through work email, IM, version controlled documentation (and code), the job of future archaeologists will be much more manageable (though still hard and fraught with risk). Maybe more and more tools will develop to piece together a project's evolution, rationale, and implicit assumptions and decisions.

zdw 15y ago
The problem with going through people's old email and such is the signal/noise is extremely low, and often of a timely nature, and may not represent the final outcome, especially if work has been done on a system after the fact. What we need is testable documentation. I do IT work most of the time, and document using a mashup of Markdown and text snippets in a specific format for network documentation, all kept in version control. I then have a motley collection of scripts and tools that run against my text files and do simple testing like pinging all the hosts mentioned, checking DNS entries, etc. Frequently I'll write up docs before I start work, then by the end of work, all the tests will past. It's TDD applied to network design.
- abecedarius 15y ago
  Right. Here's my script for this sort of thing: https://github.com/darius/tush https://github.com/darius/tush
bane 15y ago
I almost wonder that now that more and more of these discussions are done through work email, IM, version controlled documentation (and code), the job of future archaeologists will be much more manageable (though still hard and fraught with risk). Maybe more and more tools will develop to piece together a project's evolution, rationale, and implicit assumptions and decisions. It's funny, Wave was actually a really fabulous tool for exactly this.
roguecoder 15y ago
We aim for self-documenting code. If there is a decision that absolutely can not be reflected in the code itself, a comment goes with the code as close to where the decision is implemented as possible. We have a 10-year-old code base that is doing well and continues to support rapid, robust and maintainable development. The learning curve is higher than if the documentation were external to the code base and accurate diagrams existed, but unlike those artifacts the code will always be right about what the code is doing.

bostonvaulter2 15y ago

Side note: The scrolling on this page is horribly broken on Android. I can't even read the whole article because I can't scroll down. After loading the article on m laptop I must say that it is a fascinating article and will change how I think of documentation. For example, soon my company will be making a change that will render the old wiki url's incorrect, the problem is that many older documents will still refer to the old url's. I think we should have some sort of list of the old wiki url's (there is an even older set of url's as well) to help future archeologists in piecing together the information.

Mz 15y ago
Side note: The scrolling on this page is horribly broken on Android. I can't even read the whole article because I can't scroll down. I'm on a netbook and it was pretty bad. But, yes, excellent and thought provoking article.
karamazov 15y ago
Out of curiosity, why don't you have the old url just point to the new one?
rubyrescue 15y ago
hi - any ideas how to fix it? wrttn.in is my site and i can make a change to the css or rendering if necessary...
- swang 15y ago
  It scrolls very slowly on my orig iPad. Also iOS scroll to top feature seems broken
- evmar 15y ago
  (I am a Chrome developer.) I glanced at your scroll.css and saw a bunch of rules that attempt to fiddle with the overflow/position attributes. This sort of thing confuses all browsers. Why not just use a normal scrollbar? It's fine to change the way it looks it if you want using the -webkit-scrollbar stuff you've done, but hacking out the normal web page scroll bar confuses everything. There are, for example, optimization opportunities in web rendering engines that only hit when the page is "normal" (which probably explains the poor performance other people are encountering). (I noticed even on desktop Chrome the scrolling isn't smooth, unlike other pages. Probably for the same reason.)

Mz 15y ago

I've been in the same job only five years and I have newer people coming to my desk asking "what do they mean by that?" because these emails get sent out with instructions that don't really make sense with current processes. I can infer what folks above me are trying to tell me to do, but the phrase used is a case of institutional memory and the process described/referenced is obsolete. No one has bothered to update the phrasing to fit the current situation and newer employees then don't get a detailed explanation of what it means (what the goal or purpose is) so it is slowly turning into gibberish. Over a longer scale, I can well imagine this can get much uglier. And I have an entry level job, not some high paid job with super specialized training and all that.

a_a_r_o_n 15y ago
This is like a family of 1st, 2nd and 3rd generation immigrants. The 1st generation (the managers sending down gibberish instructions) use language that most people don't understand and refer to people, places and customs from the old country that have no meaning to anyone here. The 2nd generation (you) are responsible for keeping communication going in both directions, and making substitutions and translations in both directions, sometimes changing words and orders into something that makes sense even though it wasn't at all what was asked for.
- losethos 15y ago
  Sometimes, I say naming things is the most overlooked programming skill there is. Being well-read probably helps you to use standard names. (Probably makes you a copy-monkey, though. I just made that word up.) A bit of wisdom is... at some point an entirely fresh word is useful. The alternative is encumbered words that mislead and make things worse. I shamelessly groom names in LoseThos by renaming globally across files. I behave as though I have no users. I get pissed because I get no emails. I guess there was no reason for a new word. "Copycat" exists. However, they speak of "Aping" someone as copying. I wanted to capture some of that, possibly throwing-in planet of the apes. The standards NAZIs would prefer "copycat". "Surrender-monkey" is a good word. I'm a surrender monkey and you're a copy-monkey-planet-of-the-apes-unix-nazi-heathen. God says... C:\LoseThos\www.losethos.com\text\WEALTH.TXT scarce any thing but some miserable pasture, just sufficient to keep alive a few straggling, half-starved cattle; the farm, though much overstocked in proportion to what would be necessary for its complete cultivation, being very frequently overstocked in proportion to its actual produce. A portion of this waste land, however, after having been pastured in this wretched manner for six or seven years together, may be ploughed up, when it will yield, perhaps, a poor crop or two of bad oats, or of som

losethos 15y ago

LoseThos was written entirely by me on my time, except the file compression routine which I wrote at Ticketmaster. The portion I kept was what I got from a magazine.

DanBC 15y ago

In theory some quality systems (ISO 900x etc) should help with institutional memory. Bob has a job; he performs a series of tasks to do that job; he writes down what he does for each task, including what information (also product) comes in (and who from) and what information (also product) goes out (and who to), also including all the documentation he needs to complete for each task. Someone else edits that puts it in the company QA manual. In future new employees just read that documentation, and maybe do a bit of shadowing of existing employees. Unfortunately, that's not what happens. What happens is that Bob has a job, but Charles (Bob's boss) writes a bunch of stuff that he thinks Bob should be doing. And he doesn't write sentences like "Set the machine for 190 degree Celsius. Place widgets in machine when it's at temperature. Leave widgets in machine for four hours." but he writes abstract ambiguous business speak that people can't use to actually do the job. Then Edwin (Charles's boss) adds another bunch of stuff. Then that gets edited into a god-awful mish-mash. The people who did the software (with the "help" of Edwin) add some stuff about information. Bob gets to work to an impenetrable document that bears little relation to his actual work flow.

bane 15y ago
You forgot the last parts: Charles acts aghast when Bob discusses with him the lack of actual direction and information in the document, they have an argument where Charles pronounces that if Bob were an expert in his field he should just know what to do anyway. Later, Bob presents his work to Charles, who pronounces it "great!". So they move up to present it to Edwin for final sign-off, where Edwin pronounces it "crap! not what it's supposed to be at all!". Then takes the work, sells it anyway, pockets the bonus, trickles a couple percent down to Charles and Bob gets fired for incompetence three days before Christmas.
groby_b 15y ago
You seem to miss the part in the original post where he states that things are running just fine. Everybody knows Bob has to set the machine to 190 degrees. What they lack is the knowledge why it's 190 degrees. That's a bit harder to just pin on management. Anybody who's worked on a project for a long time has seen that drain of knowledge. Alice, who wrote subsystem XYZ, leaves for greener pastures. Bianca and Christy worked on it with her, but they were juniors when they first started. Some practices, they just adopted without thinking about the why. Now XYZ needs to be ported to a new environment, call it dotWEB. The question why things were done that way inevitably arises. Unfortunately, Alice works for a competitor - talking to her would violate all sorts of security rules. That is the problem he's talking about. And like the original author, I'm not sure there's an easy to fix this. Not even if everybody including the bosses is supremely competent.

retroafroman 15y ago

Original link and discussion here: http://news.ycombinator.com/item?id=3311015 http://news.ycombinator.com/item?id=3311015

16s 15y ago

I've seen memory drains during down-turns in the economy. The company will offer the oldest, most-experienced, highest paid employees a buy-out or they'll offer them five years extra pension/retirement if they leave now. Mgt figures that consultants can be hired if they ever need X, Y or Z done again at a fraction of the cost of the employees. Many senior employees accept the offers, and then later mgt realizes that 95% of their Fortran or Cobol (or name your old programming language/product) experience has vanished. Thank god the software is mostly in maintenance mode, but client problems do still arise and what once took 15 mins to fix, now requires a week and phone calls to retirees begging them to help fix a client problem. They'll pay them loads of money too just for a few minutes of work. Consultants turn out to lack the domain specific knowledge required to fix the problem. Bottom line is that sometimes, mgt is so focused on hitting short-term financial goals, that they lose site of the big, long-term picture. Business needs less 20 something MBAs who only see this quarterly report and more managers who understand that business is a decades long proposition and that senior employees are key to maintaining an edge and passing along knowledge.

yummyfajitas 15y ago
It's easy to blame the popular meme of "short sighted management", but they aren't necessarily as short sighted as you think. Often management gets rid of the old guard because the old guard actively prevents the business from improving it's processes. I know someone (maybe multiple people) who works at a large financial institution. A particular department there is running on ancient systems using ancient processes, most of which can and should be properly outsourced to $BIGVENDOR. The fed agrees, and is constantly hitting them with MRIA's (Matters Requiring Immediate Attention). But right now there are many managers of 50-100 employees who will become redundant when 49 of their employees are replaced by a few servers in NJ. These people tend to passive aggressively slow down the replacement, hoping to delay it until they retire. There are other people who have stable jobs, based not so much on competence as the fact that they hold lots of institutional memory. They don't want to help build a system that will make their knowledge obsolete. When management fires these people, it doesn't do so because it doesn't understand their value. It does so because they are actively hindering the company from moving forward. Sometimes this process fails as you describe. But when properly managed, this process results in a streamlined department half the size it was before, processes properly documented, everything automated, where people are fighting to improve efficiency from 99.7% to 99.9%.

JabavuAdams 15y ago

Can we learn anything from legal discovery processes?

mikerg87 15y ago

Lack of a succession plan is plaguing many industries. I am in the category of grey hair, no hair, written a book. When I die, the code dies with me

jwatte 15y ago

I'm sure they saved money in the short run by going with the cheapest option each time they tried to manage their documentation the last time around. On the other hand -- at the time, they didn't know which parts would be valuable, and which parts would be wasted effort, so it probably was the right choice. However, if a major incident were to happen around that plant, could the company tell the emergency responders what they would need to know?