4 ms·
Ask HN: How do teams remember why infrastructure decisions were made?
On teams I’ve worked with, we often run into systems where nobody really knows why certain configs, services, or architectural choices exist. Docs are outdated, Slack history is messy, and the people who made the decision are often gone. When something breaks, we end up rediscovering the same context over and over.
Is this just inevitable on long-lived systems, or do experienced teams have a better way of preserving this kind of context?
- toomuchtodo 11mo agoADR records. Store as markdown file(s) in the repo. https://adr.github.io/ https://adr.github.io/ https://github.com/adr/madr https://github.com/adr/madr
- curious_sre 11mo agoThat makes sense. In your experience, do ADRs actually get revisited during incidents or onboarding, or do they mostly exist as reference docs that people forget about?
- toomuchtodo 11mo agoRarely useful for incidents in my experience, mostly useful for onboarding those new to a codebase or part of a system. It’s a mechanism to preserve institutional knowledge that would otherwise evaporate due to not being represented as code; formalized documentation. The benefit of this information existing in markdown files is this can also be used with LLMs and RAG if getting a natural language interface to the knowledge might be relevant to your enterprise.
- curious_sre 11mo agoThat’s helpful context, thanks. In practice, when incidents happen, where do people actually go to reconstruct the “why”? Is it mostly Slack archaeology and asking senior engineers, or is there something else that works better?
- toomuchtodo 11mo agoAs you said, chat logs, logging source of truth, and any context that can be provided by subject matter experts on the system(s) in question. Google's SRE resources on this topic are somewhat helpful, consider reviewing and evaluating for your environment. https://sre.google/workbook/postmortem-culture/ https://sre.google/workbook/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/
- gardenhedge 11mo agoGood but needs to be driven by a head of architecture type role incentivizing the practice. Otherwise it's a patchwork of records and largely ignored.
- d--b 11mo agoAh I wanted to make a product to solve this problem. I posted here to see if anyone thought it was worth solving, but nobody seemed to care. The way I wanted to do this is to create dashboards that would serve at the same time as infrastructure diagrams for documentation and live health monitoring. Right now, most documentation solutions aren't used on a daily basis so become out of date, because people don't think about it when making changes and fixes. And monitoring solutions only show you charts of things you're supposed to already know. They're very technically-oriented, and not business-logically oriented, if that makes sense. Like they'll tell you that process x is running on machine m, and that it's running out of ram, but nothing will tell you that process y that depends on x's outputs is going to fail as well.
- mmarian 11mo agoAfter I read your comment, I remembered I posted about it too, and came to the same conclusion ^_^
- mmarian 11mo agoConfluence pages, ADRs in github. Not perfect though.
- gus_massa 11mo agoI send an email to the team, that has a few notes. It's actually an email for me in the future, and the notes are in case the future me forgot the details.
- curious_sre 11mo agoThanks all, this has been really helpful. One recurring theme seems to be that anything manual eventually gets ignored. Out of curiosity, has anyone seen any system that captures context automatically, without relying on people to write it down?