13 ms·
Ask HN: How does your company keep track of lessons learned?
As a single developer, I would usually just keep a .txt file with interesting things that I've learned throughout my tenure. It's a lot harder to replicate that efficiency for a large team; especially a distributed team. How do you effectively share knowledge with your co-workers? What have you seen work/fail?
- eru 7y agoI used to work for Google as an SRE a while ago. They have pretty good procedures for keeping track of lessons learned. The book (https://landing.google.com/sre/sre-book/toc/ https://landing.google.com/sre/sre-book/toc/) goes into some detail.
- petr25102018 7y agoI have seen post mortems or "common problems" in knowledge bases, but outside of that the lessons learned were either addressed directly (improved process, fixed code, improved docs) or only in the heads of people involved:)
- dboreham 7y agoIn "the circular filing cabinet"?
- ragebol 7y agoVertically archived
- itake 7y agoI work at a company with ~1k engineers. Every every system failure, we email the entire org a postmortem google doc describing what went wrong, why, and what we are doing to prevent it from happening again. Postmortems are also as their own JIRA project.
- hwj 7y agoHats off. What industry are you working in? (I guess it's not web development).
- rb808 7y agoThat sounds great but if I was mailed postmortem docs every day I'd soon start forwarding them to spam folder. Do you know how many people go through them?
- itake 7y agoThey happen about 3-5x month. I typically read the once that are most relevant to my role.
- thyagoquintas 7y agoWe have implemented an collaborative wiki (wikimedia, like a copy of wikipedia but just opened for internal usage) in 2009, in a company with 200 employee. We engaged people to write everything that they think is good for the others.
- crobertsbmw 7y agoHow often do people read that stuff?
- bluGill 7y agoHow often do people maintain it? My experience with internal wikis is they start off good, but a few years latter everything is obsolete: it is a good things nobody reads it because what was true last year isn't anymore.
- thyagoquintas 7y agoWe create a department with two employees to incentivize and review the articles (correct typo error, writing mistakes and text enhancements). They are hierarchically located in the internal communication area, so the internal weekly e-mail about the news of the company was linked in the Wiki. All the changes that are happening the products, or events, awards and other news from the company were input in this wiki. They got a lot of view incentives from the weekly news of the company. The idea is to document all the projects, steps, actions and mistakes, and all the things that happen in the company. [Translated from the sustainability report from 2009 - page 31 - GiiMiiPedia] GiiMiiPedia is a free encyclopedia built on the Wiki model (the same as Wikipedia) whose purpose is to transfer Know how. This initiative encourages employees of the organization to contribute knowledge that can be useful in different contexts, optimizing the time of employees. This year, the GiiMiiPedia layout underwent some modifications, with new sessions included, such as: market news, videos and subjects of interest from all areas. Sorry the report is in Portuguese, we are from Brazil :) http://www.daiichisankyo.com.br/site/wp-content/uploads/2019/04/relatorio_daiichisankyo_PORTUGUES_web.pdf http://www.daiichisankyo.com.br/site/wp-content/uploads/2019...
- boardmad 7y agoemail
- b3lvedere 7y agoVery fragmented all over the corporate network in all kinds of weird tools and applications. They did make effort to standardize everything, but nobody seems to care.
- 3dfan 7y agoWe have a directory lessons_learned/ with *.md files. It's a git repo and everyone can pull/push directly to master. I skim the commits once a week. It's basically plain text files. But the md extension triggers some nice eye candy in vim and other browsers. I think we will keep this structure forever. Maybe we will (additionally) serve the files over http at some point. Maybe we might even add edit / search / push functions over http, but for now I have not planned that. I have seen CMS come and go. And I'm tired of it. Text files are forever.
- ascherj 7y agoThis is a really cool idea. Thanks for the inspiration.
- MrGilbert 7y ago> I have seen CMS come and go. And I'm tired of it. Text files are forever. My website uses a "Apache-DirectoryListing"-like asp.net core frontend + a backend markdown parser. If I weren't that lazy, I'm pretty sure I could have written a module for apache or nginx with an integrated md parser. Maybe there is something already.
- davidgl 7y agoWe push text files into our git repo, that is hosted on a local GitLab. Gives us the nice visuals, with good support for inline code/xml/data/whatever, and searchability, and you can add new files, or edit existing ones, directly in the WebUI
- 7y ago
- 2rsf 7y agoKeep and share are two different things. You can email or have a small meeting sharing results, but keeping a searchable, updated, database is hard.
- ericalexander0 7y agoDaily notes posted in slack where a bot logs to elastic search. Notes reviewed in our retros and the useful/actionable is logged in a wiki and or jira. Wiki notes are reviewed in our annual retro.
- happywolf 7y ago<sarcasm> My company is awesome in the sense to make sure it doesn't forget, it repeats the same mistakes every couple of years </sarcasm>
- vidanay 7y agoWe keep ours in a manilla folder labeled "Business plan 2021"
- mattcrail 7y agoOne of the first thing my co-founder and I did when we started our new project is create a table in Notion with all the previous lessons we had from our last company, and have been adding to it as we create fresh, new mistakes. Using tagging to keep it organised and readily available so when we start adding more people everyone has ready access to it.
- Damorian 7y agoWe have a checklist system. Depending on which files are on your commit or which database objects are modified, you're presented with a checklist specific to the things you're changing. When something goes really wrong, the outcome of the post mortem often includes adding to or changing the checklist for the affected files/filetypes/team/whatever. It's not a perfect system, but it beats reading our long, confusing wiki pages.
- admerox 7y agoI like this system a lot. It convert postmoterns into actionable checklist for next project. My only concern is that the checklist might get too big. But definitely, better than expecting people to go to wikis to avoid past mistakes
- jve 7y agoCan you tell me more - how/where do you present checklist before commit?
- Damorian 7y agoWe use in house project tracking software to manage development, bugs, documentation, Etc. that it's built into. These checklists are verified by another developer as part of code review before the branch is allowed to merge to the release/main branch. There's only a couple of "generic" checks (one for security things, another that's automated that enforces coding standards), at least for our team. A small bug fix may only have 2-4 checklist options, but a larger enhancement may touch enough things that there may be 20-30 checklist options.
- pdexter 7y agoIs this automated?
- weekay 7y agoDeveloping a postmortem culture is important to share the lessons learnt from production. For eg. Documenting and sharing the lessons learnt from a SRE perspective , a google doc would suffice. Some pointers at https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-... Have seen Wiki being setup and fail and go stale very quickly. Lots of knowledge and learnings are tribal knowledge. Sharing tribal knowledge is effective in person or in a non wiki mode especially through brown bags or chapter meetings etc., Challenge is not everyone has the time or the energy / enthusiasm to be talking in front of a wide audience. Never seen one solution working effectively. You need to figure out the best approach based on the team culture and how the organisation is setup. For eg., some teams are hesitant to share knowledge and learnings with other groups - Conways law comes into play. End of the day it is not up to the company to track lessons learnt. It is the job or becomes the job of the person supporting production to a large extent to maintain it for making it easier to do their job. That being said , that knowledge leaves when that person moves jobs & the cycle continues
- HenryBemis 7y agoSimilar to that; Post Implementation Reviews (PIRs), then we adapt the relevant documents (processes, procedures, policies, PM methodology, etc.). Usually on the projects the Sponsor is high enough that can approve/influence the changes in the aforemention documentation. On the side, we have a Wiki where every project has its own page (with user access in place because some semi-sensitive info can be found there).
- asplake 7y agoReminds of the ironic naming of the "Lessons learned meeting". If no change is made to what people actually do, it's very hard to say that anything has really been learned, and I would put that first.
- ing33k 7y agowe are using a service called https://www.getoutline.com/ https://www.getoutline.com/ + for being OSS
- ZeroCool2u 7y agoThis is cool. I like the focus on UI performance. Seems a lot like Space actually.
- unixhero 7y agoLooks similar to Bookstack http://www.bookstackapp.com http://www.bookstackapp.com
- bilekas 7y ago> How does your company keep track of lessons learned. The dev overall, across the globe: i find retrospectives after a sprint cycle really good actually, it's a good place to call up where improvements can be made too. On a personal level: When my mess up/mistake causes grief for someone else, I make damn sure I learn from that.
- slumdev 7y agoBluntly: We don't bother. We don't even have reliable mechanisms for collecting feedback from customers. I'm leaving soon.
- gwbas1c 7y agoWe don't! One of the reasons why some people become valuable in long-tenure positions is because of the lessons learned. At a certain point, no one is going to read through every page in the wiki / archive / man pages / whatever is popular this year. That's where onboarding and process come in: Management needs to make sure that lessons drive improving the process, that newcomers are onboarded with lessons from the past, and that everyone continues to follow the processes. Now, jokes aside, in my company, the new owners decided they didn't like the people we were outsourcing with, and decided to replace them with their own outsourcing center. Now everyone's re-learning lessons that are probably tracked in our various wikis, repositories, ect. But, the newcomers want to run things their own way. That's why a few long-tenured people are important.
- Psyladine 7y agoInstitutional knowledge. So, so crucial to culture and process.
- jack_codes 7y agoWe have Confluence pages that no one reads but it's nice we go through the exercise, I guess.
- larrik 7y agoI can never find anything in Confluence unless I had the foresight to link to it from the main page.
- jack_codes 7y agoSame here - the default search seems woefully inadequate at least as far as I use it. If I do a partial search sometimes I get back docs I want but if I type in more specificity (more through muscle memory while typing, not necessarily trying to narrow down the results) then what I'm searching for disappears from the list even though it's more precise.
- kache_ 7y agoConfluence is absolutely terrible. Their accessibility user experience is abysmal. It was designed for people who operate user interfaces with their mouses rather than their keyboards. Not only that, but it seems to be quite unreliable. Seems every other day some random component is not working, like uploading screenshots.
- thewebcount 7y agoCouldn't agree more! Confluence is a steaming pile. Worse, it steals key strokes used by the OS. If I want to search the page, I do Cmd-F and instead of bringing up the browser's search page function, it brings up its own broken "search everything except what I need" function. It's the worst.
- deleted 7y ago[deleted]
- EamonnMR 7y agoWriting something in Confluence is about as useful as writing it on paper and leaving it in a desk drawer.
- qxxx 7y agowe have a wiki in microsoft teams, which no one ever reads...
- leandot 7y agoGithub repo with TILs (Today I Learned / Things I Learned)
- mariocesar 7y agoWhen I was working in a team, I did Bug's Anniversaries celebrations. It actually boost moral. February 2th. when I wipe out a production database because the ansible playbooks had hadcorded settings. Since then we use settings repositories and confirmation dialogs when playbooks will run in production. September 2. They day we realise we are unable to restore old backups because of media paths are not related to the data, so when we move all data from different servers with lost all user's images forever. Since then images are prefixed with the db record id where it belongs, later we added metadata to S3 to add extra stuff like user_id, object_id, company_id, etc. so we keep urls clean September 10. Inbox carnaval: we have an small hack that added users to BCC to send newsletter, with the time all users where receiving emails 2 times, then 3, then 4, then 10, then 2 again. It was a threading issue where the variable of the BCC was set to global in certain cases and it appended to the list instead of starting again ... 2 full weeks into that. Python3 and typing was the way to fix it
- xolox 7y agoAt my employer we have two ways: 1. For large issues visible to customers an incident report is shared inside the company. These are written for general consumption and so lack any technically interesting aspects (they're "dumbed-down" a lot). 2. Technical "lessons learned" are curated in a Sphinx based documentation website that I started but which is starting to see more and more contributions from other tech heads in the company. We used to have a wiki but it ossified after years of no contributions. Personally I didn't like the MoinMoin wiki engine that much but this is just personal taste of course. I started setting up the Sphinx site to encourage knowledge retention despite turnover - I kept explaining the same things again and again. Now I just share a URL when such questions come up :-).
- unixhero 7y agoGreat! What's so good about using Sphinx for this?
- xolox 7y agoHonestly I think this is personal preference more than anything else, so I'm not sure how useful my answer is, but I thought I should at least try to answer your question :-). I have quite a few years of experience in using Sphinx to create high quality technical documentation for open source Python projects, which explains why I chose Sphinx for this endeavor in the first place, but that's not an answer to your question. The following dichotomy that I feel exists may (?) be more useful: * Wikis tend to grow organically without much restructuring. I don't think this is fundamental to any specific wiki system, more of an emergent behavior, if you will. Enough discipline can surely avoid this, but wiki systems don't exactly make it easy - I haven't seen many "refactoring" tools in wiki engines, though they probably exist (global rename that updates references, moving of sections of text between documents, etc). I have definitely seen quite a few wikis devolve into the documentation equivalent of a "big ball of mud". * Sphinx being based on a simple directory structure of text files on a local filesystem in reStructuredText format makes it much easier for me to "refactor mercilessly" in order to adjust the structure of the documentation so it keeps making sense as it evolves (e.g. things as simple as "grep -r" and "find | xargs | sed -i" or equivalents built into editors). To make sure no references were harmed there is "sphinx-build -nW". I've also grown to appreciate the value of generated content. For example we have a dozen software projects and another dozen internal web services. I cataloged both as a directory of simple CSV files that contain details like repository locations, programming languages and frameworks used, type of release management process used, etc. During the documentation build these CSV files are rendered into multiple output formats, for example an overview table that lists the most relevant high level details of each project / service and separately from that the more detailed information about each project / service. The data only needs to be entered once (DRY), but can be rendered any number of times.
- quaffapint 7y agoWe have a handbook based on Hugo for the tech folks thats editable by anyone and bloomfire for the general masses.
- fierarul 7y agoAt company level lessons learned turn into internal rules. Too flexible holiday time results in more explicit rules about what is and what isn't allowed. The endgame is you become too bureaucratic after enough creative employees.
- phlhar 7y agoI work voluntarily at a university radio station. We are a small team of three technicians, but the team changes quite often. I have been there for four years now, which is already quite long as people come and go as they finish their studies. We keep track of everything in a dokuwiki. Often I catch myself not wanting to document stuff, but it is essential for coming technicians to have a place where they can look stuff up. That was the place to learn about how the infrastructure works for me when I was new. The second tool we use is openproject for ticket tracking. We don't delete old tickets, so if a problem comes up again it is likely that there is already a solution for it documented in the ticket system.
- meddlepal 7y agoWe put it in a write-only wiki.
- nickthemagicman 7y agoMy company DXC just laid off or let go 99% of my team. I'm the only one left. There's no lessons learned here only the short term bottom line. And the developers are simply a commodity.
- gfs78 7y agoProfessional services companies business is about selling/billing hours. Those involved in the sales are seen as an asset, those involved in providing the service are seen as a cost. They don´t care about the latter. DXC, Accenture, Capgemini, Indra, etc...they are all the same.
- Vaslo 7y agoUsually through tribal knowledge contained in experienced employees who “retire” when they get too old and expensive
- haxplorer 7y agoWe use a mix of confluence and slack groups. We have one slack group per type of learning (RCAs for failures, best practices in engineering / manufacturing / logistics, etc., Top of the mind thoughts, interesting reads). We use confluence to do detailed documentation or writing, and slack to share and drive visibility.
- aklemm 7y agoOral histories of each firing. /s
- hirako2000 7y agoWe talk about it, some deserve to know, some don't. Some better know, some better not. Non written education is probably the most effective way to communicate and maintain important information. Writings leave it up to the authors' ability to know where and how to communicate...
- rod_rodriguez 7y agoWe are building a single open-source repository consisting of markdown good practices from lessons learned and share-code repository. https://github.com/pragmatismo-io/pragmatismo-io-framework/tree/master/docs/pt-br https://github.com/pragmatismo-io/pragmatismo-io-framework/t... (Currently available in Portuguese)
- jrhusney 7y agoYou know that feeling of "we always complain about the same things, but it never gets any better?" Your question is at the heart of making it possible to escape stagnation and actually evolve the way a team works together. Sharing knowledge isn't just a matter of tooling, but a matter of principle. Because we want the knowledge we share to not just float out there as a "lesson", we want people to use the lessons and act differently what we're actually talking about here is governance (i.e. law!). This might sound heavy. It's not. It's just a change in orientation from "I'm passively sharing this lesson we learned" to "this lesson we learned changes the way we act." There are 3 things to do to shift from "lessons learned" to working agreements: 1. Capture knowledge in the pattern "when this happens, our team will act this way" 2. Adopt a workflow for formally adopting a working agreement – could be a majority vote, consensus vote, etc. 3. Keep that knowledge someplace the team can browse, search, and update (e.g. Confluence, Notion, Google Drive, etc.) If you do this, something magical happens: you'll begin to evolve your knowledge over time. Have a working agreement that didn't quite cover a corner case? Update it! Have a working agreement that was too restrictive? Nuke it! It's no less shift in magnitude as when humanity switched from oral tradition to the written word. And guess what? The written word works much better when you're operating remotely. Our remote team has been operating this way for nearly 5 years at Parabol. It's a common pattern that at the end of every retrospective we have a new working agreement we'd like to adopt. We've even come up with a Slack-based async workflow for adopting them: https://www.parabol.co/blog/async-decision-making-slack https://www.parabol.co/blog/async-decision-making-slack
- kirubakaran 7y agoI use a Histre notebook shared with my team https://histre.com/ https://histre.com/
- thu2111 7y agoThe answers here are breaking down into two categories: 1. General business lessons, to which companies generally don't summarise or track them. 2. DevOps outage post-mortems, which competent companies generally have some sort of process around. I've never seen a rigorous post-mortem culture in tech outside of DevOps/SRE. I guess there are a few reasons. One is probably that the DevOps/SRE space is very amenable to encoding lessons learned in scripts of various kinds, so it's actually useful to do a post-mortem exercise because the outcomes are very small, very concrete and will be somewhat actionable. Things like "errors in parsing this file shouldn't cause the server to blow up" are easily corrected and a process (unit test) put in place to formally encode that institutional knowledge. In regular software development there's way less reflection. This is partly because the tooling is much less home grown and malleable. Lessons are learned and they are encoded, but it happens slowly and through the mechanism of library and language design. It's generally not something you do within a single company but rather, it's an emergent consensus across the whole industry. Additionally, this is harder because lessons learned are often ambiguous or subjective. For instance I learned the lesson, many years ago, that dynamic typing leads to more mistakes than static typing. But you see many programmers still who prefer dynamically typed languages and dispute this sort of conclusion. In the business world there's virtually never any kind of "lessons learned" repository or process. At most you get something like a formalised interview process, but even then, those are usually baked into a company from day one or never adopted at all. I've heard of very few large companies that adopted a more rigorous approach to hiring than the one they previously used. It does happen but it's rare. At the executive/CEO level lessons learned get recorded in the form of strategy talks given at fancy conferences, if at all. Often abstracted or vague to the point of uselessness, any insight that is present gets forgotten immediately by the audience who are mostly there because it's easier than doing real work. These lessons learned are things like "innovation is key to the customer experience", which is a genuine learning in a sense (usually from observing the wreckage of firms that went up against a competent tech company). But it's not really useful in the sense of being actionable by normal employees.
- MisterOctober 7y agowe have a lesson_log channel [with standard entry format] in slack -- for all Slack's shortcomings, this works great, as the whole team can see lessons learned from every department
- konschubert 7y agoMaybe I’m a pessimist but I don’t really think lessons are learned by companies, lessons are learned by individuals. Postmortems are important to drive home what went wrong, but newcomers won’t read them all. That’s why you need people with experience and people with tenure. What companies sometimes do: They encode lessons into rules, which tend to survive turnover. But that comes with its own set of problems, where you end up having a lot of rules who’s reason is forgotten.
- ltbarcly3 7y agoThis has been my experience. I've never worked at a place where the wiki was up to date, or where institutional knowledge was managed in a way that wasn't "ask this person about this, ask that person about that". I think attempts to document things "so we don't forget them" are basically a waste of time for an organization. Keeping notes for yourself can be handy, though.
- lowercased 7y agoI mostly agree with the 'waste of time' except... I did work one place where it was expected that you'd update docs/wikis/etc, and that was part of the deliverable. It wasn't perfect, and there was always some stuff that was out of date, but when you found something out of date, you were expected to confirm it and update the doc as 'out of date' - ideally link to the updated page if it was possible. This was not a perfect system - nothing is - but the culture of the group just baked this behavior in to the timeline for 'done'. When the majority of people had that expectation, it seemed to work decently. In 25 years of professional work, that's the only place I can remember seeing it done that disciplined. And... they were only around 100 folks total - not sure it could necessarily scale as they grew.
- henrik_w 7y agoI agree with keeping notes for yourself. I think the key benefit is from writing the notes more than the resulting document. The act of writing it down helps you learn/remember the lessons I think. I have been keeping records of the trickiest bugs I have come across for over 15 years [1]. I believe this has helped me internalize the lessons I wanted to learn from them, so I (mostly) won't repeat them. Also, when you leave the company you keep your own notes. [1] https://henrikwarne.com/2016/06/16/18-lessons-from-13-years-of-tricky-bugs/ https://henrikwarne.com/2016/06/16/18-lessons-from-13-years-...
- lwh 7y agoDiscuss fervently, make some tickets, writeup in Confluence. Then act surprised when it happens two years from now.
- unfunco 7y agoAt a previous job, we had an internal Wiki with a page titled: "Cow in a ditch" which was specifically for this purpose. Otherwise, it's mostly Confluence now, but no specific page of lessons learned, instead, those lessons are dotted around in individual documentation pages.
- bjornlouser 7y agoWe have an alumni network that any developer can ping via email. The group is comprised of former standout employees who were paid a small retainer at the time of departure in return for their commitment to provide missing details and lore about design choices, missteps, and implementation.
- bjornlouser 7y agoactually, I'm unemployed... but in the future when I am running things, this is how it will be done.
- cushychicken 7y agoWhat a clever comment double tap XD "...and what do you do for work, Mr. Lebowski?" "I'm unemployed."
- yawz 7y agoInteresting approach. Do you have a formal method to selecting them?
- harrisreynolds 7y agoI've started using Webase [1] for this. Just create a simple data model to structure anything you want to retain and everything is stored in one easy place. [1] https://www.webase.com https://www.webase.com
- kamarg 7y agoIs the retainer for a specific amount of time? I know my knowledge of systems I no longer work on fades pretty quickly once I no longer am actively working on it.
- TopHand 7y agoI established a wiki where I use to work. I was quite efficient about keeping it up to date and well categorized. I found it quite useful for not having to re-work out issues I had already solved. It was easy to search and correctable. I gave the people I worked with full access to it. Most of my colleagues ignored it. A small handful utilized it as much as I did. Some would even go back and edit articles and make it much more readable. I found it to be a very useful tool which increased my productivity immensely. I was somewhat disappointed by the lack of use it got from the majority of the people I worked with. On the other hand they wouldn't utilize the version control system I maintained for us either.
- mrsakasaka 7y agoSounds Enterprise
- rb808 7y agoUsually new Tests(Unit or Integration) or monitoring.
- jblakey 7y agoI set up Confluence and used it a lot for "how to do this" and "how to do that" notes for myself. It was helpful to me, and I figure that it MIGHT be helpful to other people. It doesn't HURT to have the information searchable, and with a GUI on it for the less-UNIXy folk. Some people use it, most don't.
- bryanmgreen 7y agoFor an organization, I think it's best to have a company handbook that covers all your standard practices and culture. And if there are significant failures in policy, you can just put a note after the revised policy that says "we tried X, it didn't work because of Y.
- thewebcount 7y agoOne thing I started doing a few years ago is keeping a debug diary. This helps me avoid mistakes I've made in the past, or at least fix them more quickly the next time they happen. I've actually put the more interesting parts of my diary onto our Wiki and encourage others to add their stories. I think I got the idea for a debug diary from HackerNews years ago.
- cborenstein 7y agoDisclosure: I am a founder at bytebase.io, currently in closed beta. We're building Bytebase with this kind of use case in mind. Wikis - we found wikis are too heavyweight and formal to be used consistently for recording learnings. Slack - in our experience, Slack makes capturing learnings easy but organizing and keeping track of learnings difficult. Our goal with BB is to make recording a learning as convenient as writing a Slack message AND to make organizing and keeping track of these learnings similarly easy. You can write bytes directly in Bytebase or save them from Slack. Would love any feedback or ideas. Email me (cara@bytebase.io) to get access to the closed beta with HN in the subject line.
- sailfast 7y agoPossible to use this type of service with another webhook / API a la Hubot for other services? (Say, Mattermost, for example?) I was tinkering with an NLU-style chat bot (Rasa framework) that allowed for efficient querying of this kind of data via Github Wikis (and a mapping of where this stuff might sit), but quick storage is also a piece that would need to be fixed as lessons were learned. Definitely a problem area for us that is not just solved with auto-responder regex.
- cborenstein 7y agoThanks for your question! We don't support integrations other than Slack yet but plan to add more soon as well as an API. Mind shooting me an email so I can learn more about what your team needs?
- awinter-py 7y agotacit knowledge is hard enough for a person to describe, much harder to do in a company context IMO if the full answer to a question doesn't exist in a single brain, you'll have a hard time reconstructing what really happened without a challenger-level investigation companies only solve problems temporarily
- zzaner 7y agoWe used to try and do this in Confluence. Felt good to know that stuff is neatly documented somewhere, but since no one ever knows where that was, it was of little value and few ever read it. People still tapped on shoulders and repeated the same mistakes. It baffles me that an established company like Atlassian can’t get something as fundamental as search right. I can’t even find the content I myself created at times. We have since switched to Nuclino (https://www.nuclino.com https://www.nuclino.com) and so far are having a better experience. It's as feature-packed, but the basics work as expected and are a lot more user-friendly. Re-establishing a proper documentation culture in the team is still a challenge, but that’s not something a tool can solve.
- znpy 7y ago> It baffles me that an established company like Atlassian can’t get something as fundamental as search right. I can’t even find the content I myself created at times. i agree, indexing and searching could be improved a lot. their editing experience extremely good though.
- jkingsbery 7y agoDocumentation can be nice (it's something to point someone to if in a design review it becomes clear we're about to repeat a past mistake), but the best way to improve based on lessons learned is to bake them into your team's mechanisms. To describe an example at a previous company from the one I'm at now: I used to work for a startup building a mobile phone network (technically, a Mobile Virtual Network Operator, MVNO, if you care about those distinctions - the point is that phone calls went through our infrastructure). It was very easy in the process of changing someone's account (porting in a phone number, changing a plan, or something like that) for the different IDs from different systems (phone number, phone's serial number, SIM's serial number, billing system ID number) to get out of sync. So, we could have documented how to avoid this, and the document would have sat there with no one reading it. Instead we created a nightly job that went through all our accounts and verified that ID numbers were "as expected." It would output to a slack channel with whatever breakages occurred for us to look into the next day. This program also served as documentation - I could look at it to understand what IDs should match to which systems. My current employer follows the same sort of learn-by-building-mechanisms approach, but at a much larger scale.
- Pamar 7y agoI have a question too, but I will put it here instead/before of making an Ask HN entry: Is there any intersection between "keeping track of lessons learned" and "agile methodologies" or are the two completely unrelated/orthogonal ?
- perlgeek 7y agoAgile methodologies have a regular meeting for lessons learned, the retrospective. You still have to take care to actually record that knowledge / keep track of it though.
- znpy 7y agoI've worked at a company that as seen a HUGE amount of turnover. The things I have learnt are: - companies don't learn, people do - having an internal wiki/kb helps a lot IF it's structured/indexed well enough that you can actually find information - in an ideal world, no project should be considered done if documentation is not written It's all about people. People learn and can recall lessons learned. Otherwise you rely on the good faith and will of the next person to actually go through all the documentation that has been left from the people before. This person might not have all the will, or it might not have the time.
- techslave 7y agovia the post mortem process
- donatj 7y agoWe have had a couple wikis over the years. They're rarely updated and even more rarely cited. It really hasn't worked out.
- sailfast 7y agoWikis work pretty well for this, provided you use them. If you can't be bothered to put together documentation as you build the software (sometimes I can't be bothered), you should at least make sure to document as you troubleshoot later so you don't keep making the same mistake. We store these as "Flight Rules" for our application (or error signatures, etc - whatever you want to call them) which provides the team a single location to start their search when things go belly up. That way, when you run your post-mortem (you run these, right?) you have a place to store the error notes which eventually builds up into a really useful document. Lastly, I'd say having a team norm that when one person does something the others should also be able to test it (and therefore have the right instructions to do so) is a good one for continuity. EDIT: COuple other things that have worked: - Checked-in ".dot" file GraphViz context diagrams alongside your repos are nice and easy to update - Creating decision documents for a quick run-through of options with your technical team is a great way to run an effective process while also creating a searchable artifact for later, which is great for context / lessons learned.
- Waterluvian 7y agoAs I mature as a developer I'm learning what many of you probably know: effective long term curation of knowledge is really frickin hard. It's the hardest part of my job, especially because it's so subtly important and easily missed. The end product rarely breaks because of a short term lapse in documentation. I don't have a better answer yet other than making it a personal point of pride that my docs are always up to date and well-organized.
- zikani_03 7y agoWe have a git repo (called the devops repo) and we use sphinx to generate the html. We try to document lessons, post-mortems, decisions and tools we plan to evaluate. Unfortunately, folks don't really read the docs (and I've learned from this thread that we're not alone ;) Been thinking about this problem and thought to embed something like a quiz in the docs to make them interactive, yet still static - something like howtographql.com. Yet to try that approach, though.
- Balanceinfinity 7y agoRealistically, the way we do it is we have a team of supervisors who have been around long enough and have seen everything that - as a group - we are the institutional knowledge. We are pretty effective at communicating with each other to bring the institutional knowledge to bear on a problem - we could be better at training our subordinates so that they have access to this institutional knowledge, but the truth is that the training is expensive and not usually well received. Generally, I don't think efforts to accumulate institutional knowledge on a website work bear fruit - no one really wants to update the website, both because it's thankless, but also because of access time. It is much faster to tap the institutional knowledge in management by sending an email than by paging through the results of a search. For written institutional knowledge to have real value, the access time has to be small, which means someone has taken real care in curating the knowledge so it's easily accessed. Finally, we have the Brian problem. Brian was the person most likely to update our internal websites - unfortunately Brian wasn't very good and had some poor ideas regarding lessons learned - by adding them to the websites, his bad ideas were passed on to younger team members who didn't know better.
- lutorm 7y agoIt doesn't sound exactly like what you have in mind, but we have a DFIUA program ("Don't Fuck It Up Again..." ;-) where people do postmortems on serious misses and report it out to the software team.
- SergeAx 7y ago1. We are writing detailed postmortems for incidens of medium scale and larger. It is a must read for all the newbies. Postmorteams are always linked to issues to prevent similar situations in the future. 2. We are diligently linking our issues into a hypertext mesh: what is related to what, what was blocked by what, what was decomposed from what. We are using milestones and epics. 3. All the commits in all the codebases are linked to issues. There is a rule on a server side that forbids pushing non-linked commits. There is a single exception for firefighting code changes, when there is no time to write a ticket first, but those commits are marked with a special signs and author should create an issue and link to commit when the problem is solved. 4. Documentation and API specs are laying in the same or adjacent repos and are changing according to same rules. 5. So, every line of code is linked to the corresponding issue via commit and then to other issues and commits in other repos via hypertext mesh. When your code is clean, it mostly self-documented and becomes a knowledge itself.
- kvz 7y agoWe’re small but sharing post mortems publicly on our transloadit blog helps to a) be transparent towards customers b) have a reference for future team mates / scenarios c) make sure we really understand the issue. Once something works again, the brain is all to eager to accept any kind of summary and move on. If you share your post mortems, you’re forced to look deeply at all the assumptions and preconceptions they are built upon, and often (always) that process reveals new insight and fixes. It can be painful and time consuming, but it pays dividends.
- Justsignedup 7y ago- we're still in business. We learned something. - anything written down and not enforced is almost the same as nothing.
- emarsden 7y agoMy work concerns safety in high-hazard industries, where lessons learned analysis (also called operational experience feedback) is a very important organizational process. It's also well-known to be difficult, not so much related to the technical tools used for recording/sharing incidents and lessons, but mainly in terms of organizational culture (learning, blame, psychological safety and speaking up). Some references from this area: https://risk-engineering.org/learning-incidents-accidents/ https://risk-engineering.org/learning-incidents-accidents/ https://risk-engineering.org/barriers-learning-experience/ https://risk-engineering.org/barriers-learning-experience/ https://risk-engineering.org/concept/psychological-safety https://risk-engineering.org/concept/psychological-safety
- meristem 7y agoKeeping track is the initial part of the effort. Getting the information from ‘existing’ to ‘integrated’ requires a system easy enough to use ( add Bare-bones description of issue, add comments or post mortems, format in ways that highlight core problem/solution) and robust enough to have hooks everywhere an engineer might need to see it. The negative side of integration is the person with knowledge integrates it in their practice and ‘forgets’ it is new knowledge for everyone else, thus not propagating the new lesson learned.
- jvanderbot 7y agoFrom what I can tell, we keep the best people around, or maintain good ties with them as they are leaving to ensure knowledge transfer. Many come back because they find the work environment to be unmatched elsewhere.
- p2t2p 7y agoWe have a kss - knowledge sharing system, that looks at patterns in code changes on pull requests and add comments if needed. Anyone can add a rule
- dragonsky67 7y agoWe have an active process whereby staff with more experience try to actively pass on that experience, spending large amounts of time explaining that the "new" initiative has been tried before and detailing what happened and why it failed. Of course nobody takes any notice.
- Kaivo 7y agoWe haven't started using it yet but we've been thinking about using the Architecture Decision Record pattern as presented in the ThoughtWorks Technology Radar [1]. The basic idea is document decisions with a specific structure and keep it close to the code. The thing is any time we can answer "why", it's a form of decision that can be documented somehow. Since it's close to the code, while coding, any search will also land on those decision if the same terms are used. There are several tools to help with that as presented here [2] and here [3]. [1] https://www.thoughtworks.com/radar/techniques/lightweight-architecture-decision-records https://www.thoughtworks.com/radar/techniques/lightweight-ar... [2] https://adr.github.io/ https://adr.github.io/ [3] https://github.com/joelparkerhenderson/architecture_decision_record https://github.com/joelparkerhenderson/architecture_decision...
- soonnow 7y agoWe, at some point, had a Knowledge Sharing event every week. The goal was just go round the table and every developer would share something they learned that week. As I was the senior developer at the time I would usually bring something that I learned or from CS, like a nifty algorithm or how Hashmaps are implemented in C#. Other people would share some bugs or unexpected behaviour. I liked it a lot and I felt that the team did as well. Sadly we stopped doing it, as there was a reorganisation that impacted our ability to do it.
- 0x4164 7y agoTelegram bot with options button. As the telegram bot are processed by my coordinator website, almost every message sent by any user to the bot are saved in my co's database. There are 2 types of options, the public and private. Public type will be saved in db and tell the whole bot's user about information someone sent, with or without further feedback from other bot users. The private type will be saved in db and only visible to several user specified. I though it is very versatile for report, lesson learned and etc.
- codegladiator 7y agoOnly by repeating it frequently over span of months.