8 ms·

Etsy’s experiment with immutable documentation

187 points by telotortium 8y ago

munchbunny 8y ago
This is a really interesting idea and I feel like it has the potential to make the common usages much easier to ramp up on, but I'm skeptical because I run into conceptual completeness problems of the "how do I think about this?" sort much more than "how do I do this?" problems. Blog posts like this are always optimistic and celebratory. Sometimes it's warranted, sometimes it isn't. I'd love to hear how people on the ground at Etsy actually feel about it.
- tetha 8y ago
  Our documentation is split into around 2 parts due to this. We got runbooks/ - these are hard about how. And they are prone to some degree of decay, I won't lie about that. Thus, runbooks/ is encouraged to be changed, extended, modified, clarified. runbooks/ isn't anything pretty, runbooks/ is the oil- and coffee stained notebook with more loose paper than bound paper somewhere on top of a machine and we're all going to die if you lose runbooks/. And on the other hand, we have doc/. doc/ doesn't change without a lot of discussion, and doc/ defines hard standards. doc/ won't generally help you with an immediate problem. doc/ explains why things are defined the way they are - or if this is just an arbitrary naming convention.
- wgerard 8y ago
  > I run into conceptual completeness problems of the "how do I think about this?" sort much more than "how do I do this?" problems. While I agree with you, the latter problem is particularly frustrating to encounter. Solving the conceptual problems feel more like an exploration or a learning experience, and I rarely feel like time spent solving those problems is wasted even if it's very meandering. Solving the "how do I do this?" problem rarely feels like more than a very isolated learning experience, and more often it feels like banging your head against a wall. Doubly-so when the documentation is obsolete and leads you in the wrong direction. Learning the basics of media compression is interesting and worth spending time on. But I'm going to hate myself if I have to spend 6 hours figuring out how to convert a file using ffmpeg.
  - munchbunny 8y ago
    Sounds like we're talking about different levels of conceptual abstraction. Here's an example: I need to generate a report where I need to cross-reference data from A and B. I need to know where A is going, where B is going, how to modify the schema of the events going into B to add a specific piece of data, where we can process inputs from A, how to do the join, and how to add the report into the dashboard. On the one hand, I could ask the surface level question "how do I generate the report for ____?" but really there's a whole lot to unwrap, and what I need is the information to do the unwrapping. I don't need a course on statistics, I just need to understand our specific architecture.
    - wgerard 8y ago
      I think I see what you mean, though maybe I'm misunderstanding still. The answer (from my experience at Etsy, anyway) to questions like: > how to modify the schema of the events going into B to add a specific piece of data, was "Run schema_modifier", the docs for which were frequently obsolete. Figuring out how to "run schema_modifier" was usually a very frustrating experience in that particular instance, whereas unwrapping the actual problem was usually less frustrating.
- snarf21 8y ago
  I personally feel that the problem with most documentation is that it always focuses on the how/what. That will always decay and be wrong. The code is the only truth in the end. The thing I suggest here is that if you are going to document the function, it should only be the WHY. The how can change and the documentation doesn't really decay. If you are making a new function you have to start with the why and if you can't then you shouldn't be adding a new interface. This also helps build a better picture of how the pieces interface together.
  - mason55 8y ago
    This is my most common feedback when I review docs that people have written. The other is when someone write “if you need to do X” with no information about how you’d know if you need to do X
gregknicholson 8y ago
The article is both a why-doc and a how-doc. I do agree that this is a useful distinction. I'd have called how-docs and FYIs "howto", as I think that term would be recognised more clearly, but that's a quibble.
- ebiester 8y ago
  Internal documentation has a handful of patterns. There's the cookbook patern, which corresponds to a "how-to." There's the "readme" pattern, which is meant to conceptually get you started on a project. There's the "architecture" pattern, which is a conceptual design and talks about the rejected alternatives - this is not used everywhere but is a key deliverable in both outsourced and waterfall applications. There is the "small chunk of information that is hard to organize" pattern that has multiple solutions, be it FAQs or Etsy's bot. Immutable documentation, at its essence, isn't new. Wikis are in many ways immutable documentation due to the diff. The interesting thing here is trying to organize the "small bits of hard to organize" information around slack. More often, however, the real problem is that we need to properly organize this documentation at the point the organization becomes more evident. For a tool, that should be obvious: --help should be obvious. It's not true in all cases.
  - lazyasciiart 8y ago
    > Internal documentation has a handful of patterns. This is a great way to articulate something I've been trying to talk to people about for a while - is this something you've thought of for yourself, or do you know of anywhere that talks about these patterns?
    - mr_tristan 8y ago
      I've noticed the concept of _information architecture_ is being used in technical communication more frequently these days, borrowed from graphic design. Which covers a bit more ground then patterns, including topics like search, labeling, and taxonomy. Sure, getting everyone to maintain docs with their features is useful. But given enough time, you'll end up with so many "close but not really close" concepts in documents it's very hard to stay on top of without someone putting some organization into play.
gtirloni 8y ago
Reminded me of Infobot and the countless knowledgebase bots it inspired. https://en.wikipedia.org/wiki/Infobot https://en.wikipedia.org/wiki/Infobot
rootedbox 8y ago
You should never break interface. You should expand interface. If you do that.. then.. You never edit docs. You only add to docs.
- mikekchar 8y ago
  Works great for a year or two ;-) Never refactoring your interfaces means that you can never improve the design as you gain new information. It means you can never change to something more appropriate if the underlying use cases change. It means that you can't easily consolidate common functionality in a different place as new similar interfaces are constructed. In other words, you are locking yourself in to a brittle system that will degrade into a mess of spaghetti that nobody can understand. There are, of course, also downsides to refactoring. No simplistic rule of thumb will tell you your best approach.
  - screaminghawk 8y ago
    You can have the best of both worlds by versioning your endpoints
    - nradov 8y ago
      Supporting and maintaining old endpoint versions is expensive. Especially if the old endpoint shares a back end data store with the new endpoint. You can get locked into a bad data model and unable to move forward. If you can do it then great, but it's just not always technically feasible or cost effective.
      - beart 8y ago
        If you are only ever adding and never refactoring, aren't you already locked in? I guess supporting one piece of technical debt is probably going to be cheaper than having to support a few legacy versions on top of the current version. However, the legacy version can at least be end-of-lifed eventually, whereas you would have to support all legacy features for the life of the product with the 'only add' method.
cat199 8y ago
or alternately, treat the documentation as code and don't accept code changes which are not properly documented..
- snoman 8y ago
  This is it. I would barely consider random snips about a thing/keyword, scattered over time, to be documentation. Certainly, a problem was likely solved with this FYI system... but it wasn't a documentation problem.
  - GrinningFool 8y ago
    Discoverability is often a major documentation shortcoming. They solved task-oriented discovery of 'how' snippets. Someone can search based on what they want to do, and not based on a specific command. I also prefer the 'documentation is code' approach, but it doesn't solve the same problem.
- jhayward 8y ago
  That's a nice idea but hard to make work, unless you know of some way to write automated tests and do regressions against docs.
  - cat199 8y ago
    well, automatically ensuring the documentation is correct is hard/impossible, yes, but it's simple to check if code change X impacts Y usage, and Y usage is documented, yet code change X has no accompanying documentation changes, so code change X is 'incomplete' due to lack of accompanying documentation.. see also: BSD-derived os's commit history, etc.
- kariluoma 8y ago
  I've been fascinated recently with out-of-line documentation, I was inspired by this project: https://github.com/adobe/hyde https://github.com/adobe/hyde A more robust implementation would parse the code to object-code, then emit a document(s) for the developer to fill in. The killer feature of such a system is that non-code changes (whitespace, etc.) would not trigger a need to re-document, but seemingly benign changes that affect distant parts of code (or a -O4 being added to the Makefile) would trigger a round of documentation
swalsh 8y ago
I think the part about this that is truly successful, is the use of slack significantly lowers the transaction cost of adding the documentation. In a wiki, where there is a certain level of quality expected, you might make the decision to just move on with your life once you find the answer you need, as you've already wasted enough time. With this, however, it's so cheap to leave a breadcrumb it's just polite to do so.
- gowld 8y ago
  That's sad to read, since Wiki was originally invented to create a simple user-editable document, to solve the problem of "just move on with your life once you find the answer you need, as you've already wasted enough time"
  - darkerside 8y ago
    This is just the next iteration following the same pattern down its logical path. Assuredly at some point in the future, we'll find another faster and more convenient way to convert real-time answers into documentation, and we'll seize it.
- adjkant 8y ago
  Quip has a great interface for documentation/instructions and easily leaving comments, would highly recommend it. https://quip.com/ https://quip.com/ Disclaimer: Nothing to do with them, just at a company that uses it and finds it very good at this part of docs
- DEADBEEFC0FFEE 8y ago
  Interesting that you used the verb wasted, and not invested.
- forapurpose 8y ago
  > the use of slack significantly lowers the transaction cost of adding the documentation. In a wiki, where there is a certain level of quality expected ... The wikis I've used also have slow editing interfaces because they have separate read and edit modes. Does anyone know of a modeless wiki, which probably means one that allows you to edit-in-place? The problem with modes is that to edit something on a sizeable page you must, 1. Load the page in read mode 2. Read/search to locate the spot on the page where your edit goes, read the context around it, and plan the edit. 3. Load the page in edit mode 4. Read/search to locate the spot on the page where your edit goes 5. Type your edit 6. Reload the page in preview mode (a form of read mode) 7. Read/search to locate your edit and verify it 8-?. Repeat steps 5-7 until your edit is correct. N. Reload the page in read mode. Compare to a modeless wiki: 1. Load the page in read mode 2. Read/search to locate the spot on the page where your edit goes, read the context around it, and plan the edit. 3. Edit until satisfied
BerislavLopac 8y ago
There is also Stack Overflow for Enterprise: https://stackoverflow.com/enterprise https://stackoverflow.com/enterprise
- TarpitCarnivore 8y ago
  This kind of falls into the same problem as wikis though. It requires people to A. remember it's there and B. be actively monitoring it for questions. We have one at my location and I constantly forget it's a thing. With something like this tool it puts it right in a primary communication tool.
- devhead 8y ago
  yikes, not at the enterprise price point. that's well over six figures a year; might as well hire a technical writer to sit with all your team members.
- CleanShirt 8y ago
  Genuine question, is the internal search any good? I can’t remember ever going straight to stackoverflow so I’m wondering how many people actually use Stack Overflow’s search compared to finding the most relevant answer via google’s index? This would be lost for an enterprise version.
farazzz 8y ago
I feel like searching for the solution may become time-consuming because you have to search the right set of keywords to find the answer, which may or may not be an intuitive set
roblabla 8y ago
Why isn't the how-docs tested? I don't work in ops or anything like that, but it seems to me that it'd be the best way to ensure everything stays up to date. In Rust (and many other languages), when you document a function, you usually add a code snippet that explains how to use this function. That code snippet will be treated as a unit test and ran when testing the functionality. In this way, documentation is always up to date (assuming the tests are ran). Why can't tool documentation be the same? The tool documentation usually contains a sample tool invocation. All we need to do is run the tool in a controlled environment, and ensure that the invocation runs successfully. When a tool gets updated, run the "documentation tests" to ensure all the commands in the docs still run successfully. If they don't, then update the docs. Surely, there are tools out there that already do this? Does anyone know?
- pytester 8y ago
  This is kind of what I have been doing with this testing framework: https://github.com/hitchdev/hitchstory https://github.com/hitchdev/hitchstory It's designed so that you can write tests that are the basis of the "how to" documentation - with a small script and some jinja2 you can generate readable documentation in the form of markdown from the test: Original test (behavioral example) : https://github.com/crdoconnor/strictyaml/blob/master/hitch/story/email-url.story https://github.com/crdoconnor/strictyaml/blob/master/hitch/s... Example docs generated from that test : https://hitchdev.com/strictyaml/using/alpha/scalar/email-and-url/ https://hitchdev.com/strictyaml/using/alpha/scalar/email-and... It's also designed so that the testing framework can be programmed to rewrite the test automatically for simple code changes (e.g. changing the wording of that exception). This makes building tests much quicker and much, much less tedious and perfectly in sync docs almost becomes a side effect. Essentially "how to" documentation and example driven behavioral tests are the same thing for all projects. It's just that most people don't realize and don't treat them as such.
kaycebasques 8y ago
Cool! One of the big picture value propositions of technical writers IMO is that we're shepherds of organizational knowledge. For example, if someone sends an email to me asking how to get a particular task done in Chrome DevTools (which I write the docs for), I will ask their question on Stack Overflow, provide an answer for it, and then send the link back to the asker. It's a similar amount of work, but I'm making the knowledge public, rather than one-off. What's really innovative about Etsy's idea IMO is that they have found a way to make tribal knowledge shared on an ephemeral platform like Slack more permanent. My big issue with real-time chat up to this point has been that all the beautiful knowledge that you share on the platform isn't recorded in a permanent way. I acknowledge that a lot of people prefer the real-time messaging format, so it's really cool that Etsy has figured out a way to get the best of both worlds.
dguo 8y ago
In the specific example given here of "dbconnect", I'd consider a different approach: incorporating the "how" knowledge into the CLI itself. For example, running the command without any arguments could trigger an interactive guide that asks the user questions. "What do you want to do?" o Get a list of databases o Connect to a database "Do you want to connect to dev or prod?" o dev o prod o I don't know (this option could link the user to documentation on the different environments; maybe the user is a student on his or her first internship and has never even heard of the concept) "What's the shard number?" "Which side?" The CLI could output the equivalent, non-interactive command before performing the requested action, making it a teaching opportunity as well. Maybe the "I don't know" option or this approach in general is overkill, but I generally think tools should be smarter. You don't need as much documentation if your tools are inherently capable of guiding you to the right answer.
- bhhaskin 8y ago
  Seems like a lot of overhead and more things to maintain. On top of that it doesn't teach people how to use the tool. Kinda like 600 hours of coding saved me 30 hours of documentation.
  - dguo 8y ago
    I did mention that it could output the equivalent command to teach the user what they can run in the future to bypass the interactive prompt. Point taken, but I don't think it should take that much effort to implement. Some CLI frameworks already have built-in methods for interactivity.
- rabidrat 8y ago
  That's quite a bit more work than adding a reacmoji to 100 chars which were typed into Slack anyway. We all want better tools with fantastically helpful interfaces, but we also want functioning MVPs yesterday. I like the Etsy approach because it fits into our existing JIT dev/doc workflow.
  - dguo 8y ago
chatmasta 8y ago
Why not just add a comment system to the doc page?
TheOtherHobbes 8y ago
It took me a while to work out why this doesn't feel like a full solution. I can imagine it working for simple one-line commands and their switches. I can also imagine it failing to work for any application that requires multi-line code blocks and/or more detailed how-to examples. For example - as a hypothetical, imagine you change your API from Oauth to Oauth2, and you have to provide examples of new standard code, perhaps in different languages. That's very much a how-doc and not a why-doc problem, so supposedly it doesn't belong in the Wiki. But I really can't see the changes being explainable in a fyi. Of course you can say "The new code is in repository X instead of X-1." But if there are other questions and fyi's around Oauth2 - and there will be - there's no guarantee that important fact is going to be at the top of the read list. For complex breaking changes that read list is going to get very long. Simpler changes are going to have the same issues on a smaller scale. So this isn't really immutable documentation - it's more of a database-hosted command cheat sheet. That's a useful thing in itself, but I don't think it's a complete solution for documentation.
habosa 8y ago
At Firebase (and at Google) I have done a _ton_ of work at making our "how docs" consistently correct with limited maintenance time. Consider a page like this: https://firebase.google.com/docs/firestore/manage-data/add-data https://firebase.google.com/docs/firestore/manage-data/add-d... It has 12 "how" sections of code snippets, each with 12 language options. That means 144 code snippets need to be correct for this page to be 100% correct! And we don't auto-generate any of these snippets, since auto-generating idiomatic code is a rabbit hole that's consumed many a good developer. So we, like Etsy, have come up with a separation of concerns between How and Why that works for us. The text on the page is maintained by a tech writer. The code on the page is a series of blocks that look like this: {% includecode snippet="android_firestore_write_doc" %} We have a system that resolves these, at publish time, to match snippets on GitHub. For example: https://github.com/firebase/snippets-android/blob/5f94890d7ba329a8a4c25f0ec8b42d4b9d781bc0/firestore/app/src/main/java/com/google/example/firestore/DocSnippets.java#L318-L336 https://github.com/firebase/snippets-android/blob/5f94890d7b... These snippets can be maintained with all the wonderful tools that exist in the world for maintaining code! We don't need to think of them as just docs. So that "snippets-android" repo has continuous build, a robot that updates dependencies, PRs/Issues for third-party contributions, etc. What we've found is that this system allows us to maintain many thousands of code snippets with only a handful of people watching over them.
- itcmcgrath 8y ago
  I can't upvote this enough. It also means it is easier for all internal people (not just tech writers or engineers) to make modifications (such as fixing typos, grammar, or adding missing context) safely and without needing to set up complicated tooling/tests or annoy people beyond review diffs+approval.
- dancek 8y ago
  I noticed the same system in the Google Cloud python samples repository. It's actually quite useful to have the full code available and to easily see which parts are displayed in docs. Feature request, though: include a url in the source code that takes you to the snippet's usages in documentation. Something like // [START https://codesamples.google/firebase/set_document https://codesamples.google/firebase/set_document ] or just the url as a separate comment.
desc 8y ago
This feels incredibly broken to me. The 'how' should not change often, because it enforces rapid change on anything which depends upon it. Any system inevitably acquires dependencies. Maybe it's fine to break those dependencies if you're building some public website where you can always have the latest version of the code running, but if you've got customers who depend on having a stable build which was initially branched three years ago and you still need to build, run and test that, you damn well better have a consistent and stable mechanism for dealing with that codebase which spans multiple years. Your ops and QA people will hate you otherwise. Our own approach has been to pull out the coarse-grained operations and have an 'environment' script which examines the version it's working with in order to spit out relevant details. The 'porcelain' scripts are never allowed to change their API in any breaking fashion, and their behaviour is adapted based on the environment's output.
- rhizome 8y ago
  DDD: Documentation-driven development. With apologies to Fred Brooks, write documentation first, then write tests for the functionality and syntax described in the documentation, then the code. I would speculate that doing straight docs instead of something like Cucumber would be more productive.
tchaffee 8y ago
I'm surprised no one has yet mentioned how well automated functional tests can document the how. I constantly go back to my automated tests to understand how to do something I've since forgotten but that the code supports.
siliconc0w 8y ago
Maybe the docs are binary and code aware, so the doc system can pull up the binary's code in source control and find all the possible doc strings. Developers can't add/change flags without also updating these. Maybe the docs are history/execution aware so they can list the top N ways the binary is executed. Maybe the docs are also team aware so they list the top N ways the binaries have been executed by your team. So you say, "Here at CoolTechCompany, we built a tool to help control our widgets. Please see the `cli widget` command. And everything kinda works automatically.
jacobsimon 8y ago
This doesn't seem like a solution to me but rather a symptom of a much deeper problem in the way that we write and document code.

187 points by telotortium 8y ago

This is a really interesting idea and I feel like it has the potential to make the common usages much easier to ramp up on, but I'm skeptical because I run into conceptual completeness problems of the "how do I think about this?" sort much more than "how do I do this?" problems. Blog posts like this are always optimistic and celebratory. Sometimes it's warranted, sometimes it isn't. I'd love to hear how people on the ground at Etsy actually feel about it.

tetha 8y ago
Our documentation is split into around 2 parts due to this. We got runbooks/ - these are hard about how. And they are prone to some degree of decay, I won't lie about that. Thus, runbooks/ is encouraged to be changed, extended, modified, clarified. runbooks/ isn't anything pretty, runbooks/ is the oil- and coffee stained notebook with more loose paper than bound paper somewhere on top of a machine and we're all going to die if you lose runbooks/. And on the other hand, we have doc/. doc/ doesn't change without a lot of discussion, and doc/ defines hard standards. doc/ won't generally help you with an immediate problem. doc/ explains why things are defined the way they are - or if this is just an arbitrary naming convention.
wgerard 8y ago
> I run into conceptual completeness problems of the "how do I think about this?" sort much more than "how do I do this?" problems. While I agree with you, the latter problem is particularly frustrating to encounter. Solving the conceptual problems feel more like an exploration or a learning experience, and I rarely feel like time spent solving those problems is wasted even if it's very meandering. Solving the "how do I do this?" problem rarely feels like more than a very isolated learning experience, and more often it feels like banging your head against a wall. Doubly-so when the documentation is obsolete and leads you in the wrong direction. Learning the basics of media compression is interesting and worth spending time on. But I'm going to hate myself if I have to spend 6 hours figuring out how to convert a file using ffmpeg.
- munchbunny 8y ago
  Sounds like we're talking about different levels of conceptual abstraction. Here's an example: I need to generate a report where I need to cross-reference data from A and B. I need to know where A is going, where B is going, how to modify the schema of the events going into B to add a specific piece of data, where we can process inputs from A, how to do the join, and how to add the report into the dashboard. On the one hand, I could ask the surface level question "how do I generate the report for ____?" but really there's a whole lot to unwrap, and what I need is the information to do the unwrapping. I don't need a course on statistics, I just need to understand our specific architecture.
  - wgerard 8y ago
    I think I see what you mean, though maybe I'm misunderstanding still. The answer (from my experience at Etsy, anyway) to questions like: > how to modify the schema of the events going into B to add a specific piece of data, was "Run schema_modifier", the docs for which were frequently obsolete. Figuring out how to "run schema_modifier" was usually a very frustrating experience in that particular instance, whereas unwrapping the actual problem was usually less frustrating.
snarf21 8y ago
I personally feel that the problem with most documentation is that it always focuses on the how/what. That will always decay and be wrong. The code is the only truth in the end. The thing I suggest here is that if you are going to document the function, it should only be the WHY. The how can change and the documentation doesn't really decay. If you are making a new function you have to start with the why and if you can't then you shouldn't be adding a new interface. This also helps build a better picture of how the pieces interface together.
- mason55 8y ago
  This is my most common feedback when I review docs that people have written. The other is when someone write “if you need to do X” with no information about how you’d know if you need to do X

gregknicholson 8y ago

The article is both a why-doc and a how-doc. I do agree that this is a useful distinction. I'd have called how-docs and FYIs "howto", as I think that term would be recognised more clearly, but that's a quibble.

ebiester 8y ago
Internal documentation has a handful of patterns. There's the cookbook patern, which corresponds to a "how-to." There's the "readme" pattern, which is meant to conceptually get you started on a project. There's the "architecture" pattern, which is a conceptual design and talks about the rejected alternatives - this is not used everywhere but is a key deliverable in both outsourced and waterfall applications. There is the "small chunk of information that is hard to organize" pattern that has multiple solutions, be it FAQs or Etsy's bot. Immutable documentation, at its essence, isn't new. Wikis are in many ways immutable documentation due to the diff. The interesting thing here is trying to organize the "small bits of hard to organize" information around slack. More often, however, the real problem is that we need to properly organize this documentation at the point the organization becomes more evident. For a tool, that should be obvious: --help should be obvious. It's not true in all cases.
- lazyasciiart 8y ago
  > Internal documentation has a handful of patterns. This is a great way to articulate something I've been trying to talk to people about for a while - is this something you've thought of for yourself, or do you know of anywhere that talks about these patterns?
  - mr_tristan 8y ago
    I've noticed the concept of _information architecture_ is being used in technical communication more frequently these days, borrowed from graphic design. Which covers a bit more ground then patterns, including topics like search, labeling, and taxonomy. Sure, getting everyone to maintain docs with their features is useful. But given enough time, you'll end up with so many "close but not really close" concepts in documents it's very hard to stay on top of without someone putting some organization into play.

gtirloni 8y ago

Reminded me of Infobot and the countless knowledgebase bots it inspired. https://en.wikipedia.org/wiki/Infobot https://en.wikipedia.org/wiki/Infobot

rootedbox 8y ago

You should never break interface. You should expand interface. If you do that.. then.. You never edit docs. You only add to docs.

mikekchar 8y ago
Works great for a year or two ;-) Never refactoring your interfaces means that you can never improve the design as you gain new information. It means you can never change to something more appropriate if the underlying use cases change. It means that you can't easily consolidate common functionality in a different place as new similar interfaces are constructed. In other words, you are locking yourself in to a brittle system that will degrade into a mess of spaghetti that nobody can understand. There are, of course, also downsides to refactoring. No simplistic rule of thumb will tell you your best approach.
- screaminghawk 8y ago
  You can have the best of both worlds by versioning your endpoints
  - nradov 8y ago
    Supporting and maintaining old endpoint versions is expensive. Especially if the old endpoint shares a back end data store with the new endpoint. You can get locked into a bad data model and unable to move forward. If you can do it then great, but it's just not always technically feasible or cost effective.
    - beart 8y ago
      If you are only ever adding and never refactoring, aren't you already locked in? I guess supporting one piece of technical debt is probably going to be cheaper than having to support a few legacy versions on top of the current version. However, the legacy version can at least be end-of-lifed eventually, whereas you would have to support all legacy features for the life of the product with the 'only add' method.

cat199 8y ago

or alternately, treat the documentation as code and don't accept code changes which are not properly documented..

snoman 8y ago
This is it. I would barely consider random snips about a thing/keyword, scattered over time, to be documentation. Certainly, a problem was likely solved with this FYI system... but it wasn't a documentation problem.
- GrinningFool 8y ago
  Discoverability is often a major documentation shortcoming. They solved task-oriented discovery of 'how' snippets. Someone can search based on what they want to do, and not based on a specific command. I also prefer the 'documentation is code' approach, but it doesn't solve the same problem.
jhayward 8y ago
That's a nice idea but hard to make work, unless you know of some way to write automated tests and do regressions against docs.
- cat199 8y ago
  well, automatically ensuring the documentation is correct is hard/impossible, yes, but it's simple to check if code change X impacts Y usage, and Y usage is documented, yet code change X has no accompanying documentation changes, so code change X is 'incomplete' due to lack of accompanying documentation.. see also: BSD-derived os's commit history, etc.
kariluoma 8y ago
I've been fascinated recently with out-of-line documentation, I was inspired by this project: https://github.com/adobe/hyde https://github.com/adobe/hyde A more robust implementation would parse the code to object-code, then emit a document(s) for the developer to fill in. The killer feature of such a system is that non-code changes (whitespace, etc.) would not trigger a need to re-document, but seemingly benign changes that affect distant parts of code (or a -O4 being added to the Makefile) would trigger a round of documentation

swalsh 8y ago

I think the part about this that is truly successful, is the use of slack significantly lowers the transaction cost of adding the documentation. In a wiki, where there is a certain level of quality expected, you might make the decision to just move on with your life once you find the answer you need, as you've already wasted enough time. With this, however, it's so cheap to leave a breadcrumb it's just polite to do so.

gowld 8y ago
That's sad to read, since Wiki was originally invented to create a simple user-editable document, to solve the problem of "just move on with your life once you find the answer you need, as you've already wasted enough time"
- darkerside 8y ago
  This is just the next iteration following the same pattern down its logical path. Assuredly at some point in the future, we'll find another faster and more convenient way to convert real-time answers into documentation, and we'll seize it.
adjkant 8y ago
Quip has a great interface for documentation/instructions and easily leaving comments, would highly recommend it. https://quip.com/ https://quip.com/ Disclaimer: Nothing to do with them, just at a company that uses it and finds it very good at this part of docs
DEADBEEFC0FFEE 8y ago
Interesting that you used the verb wasted, and not invested.
forapurpose 8y ago
> the use of slack significantly lowers the transaction cost of adding the documentation. In a wiki, where there is a certain level of quality expected ... The wikis I've used also have slow editing interfaces because they have separate read and edit modes. Does anyone know of a modeless wiki, which probably means one that allows you to edit-in-place? The problem with modes is that to edit something on a sizeable page you must, 1. Load the page in read mode 2. Read/search to locate the spot on the page where your edit goes, read the context around it, and plan the edit. 3. Load the page in edit mode 4. Read/search to locate the spot on the page where your edit goes 5. Type your edit 6. Reload the page in preview mode (a form of read mode) 7. Read/search to locate your edit and verify it 8-?. Repeat steps 5-7 until your edit is correct. N. Reload the page in read mode. Compare to a modeless wiki: 1. Load the page in read mode 2. Read/search to locate the spot on the page where your edit goes, read the context around it, and plan the edit. 3. Edit until satisfied

BerislavLopac 8y ago

There is also Stack Overflow for Enterprise: https://stackoverflow.com/enterprise https://stackoverflow.com/enterprise

TarpitCarnivore 8y ago
This kind of falls into the same problem as wikis though. It requires people to A. remember it's there and B. be actively monitoring it for questions. We have one at my location and I constantly forget it's a thing. With something like this tool it puts it right in a primary communication tool.
devhead 8y ago
yikes, not at the enterprise price point. that's well over six figures a year; might as well hire a technical writer to sit with all your team members.
CleanShirt 8y ago
Genuine question, is the internal search any good? I can’t remember ever going straight to stackoverflow so I’m wondering how many people actually use Stack Overflow’s search compared to finding the most relevant answer via google’s index? This would be lost for an enterprise version.

farazzz 8y ago

I feel like searching for the solution may become time-consuming because you have to search the right set of keywords to find the answer, which may or may not be an intuitive set

roblabla 8y ago

Why isn't the how-docs tested? I don't work in ops or anything like that, but it seems to me that it'd be the best way to ensure everything stays up to date. In Rust (and many other languages), when you document a function, you usually add a code snippet that explains how to use this function. That code snippet will be treated as a unit test and ran when testing the functionality. In this way, documentation is always up to date (assuming the tests are ran). Why can't tool documentation be the same? The tool documentation usually contains a sample tool invocation. All we need to do is run the tool in a controlled environment, and ensure that the invocation runs successfully. When a tool gets updated, run the "documentation tests" to ensure all the commands in the docs still run successfully. If they don't, then update the docs. Surely, there are tools out there that already do this? Does anyone know?

pytester 8y ago
This is kind of what I have been doing with this testing framework: https://github.com/hitchdev/hitchstory https://github.com/hitchdev/hitchstory It's designed so that you can write tests that are the basis of the "how to" documentation - with a small script and some jinja2 you can generate readable documentation in the form of markdown from the test: Original test (behavioral example) : https://github.com/crdoconnor/strictyaml/blob/master/hitch/story/email-url.story https://github.com/crdoconnor/strictyaml/blob/master/hitch/s... Example docs generated from that test : https://hitchdev.com/strictyaml/using/alpha/scalar/email-and-url/ https://hitchdev.com/strictyaml/using/alpha/scalar/email-and... It's also designed so that the testing framework can be programmed to rewrite the test automatically for simple code changes (e.g. changing the wording of that exception). This makes building tests much quicker and much, much less tedious and perfectly in sync docs almost becomes a side effect. Essentially "how to" documentation and example driven behavioral tests are the same thing for all projects. It's just that most people don't realize and don't treat them as such.

kaycebasques 8y ago

Cool! One of the big picture value propositions of technical writers IMO is that we're shepherds of organizational knowledge. For example, if someone sends an email to me asking how to get a particular task done in Chrome DevTools (which I write the docs for), I will ask their question on Stack Overflow, provide an answer for it, and then send the link back to the asker. It's a similar amount of work, but I'm making the knowledge public, rather than one-off. What's really innovative about Etsy's idea IMO is that they have found a way to make tribal knowledge shared on an ephemeral platform like Slack more permanent. My big issue with real-time chat up to this point has been that all the beautiful knowledge that you share on the platform isn't recorded in a permanent way. I acknowledge that a lot of people prefer the real-time messaging format, so it's really cool that Etsy has figured out a way to get the best of both worlds.

dguo 8y ago

In the specific example given here of "dbconnect", I'd consider a different approach: incorporating the "how" knowledge into the CLI itself. For example, running the command without any arguments could trigger an interactive guide that asks the user questions. "What do you want to do?" o Get a list of databases o Connect to a database "Do you want to connect to dev or prod?" o dev o prod o I don't know (this option could link the user to documentation on the different environments; maybe the user is a student on his or her first internship and has never even heard of the concept) "What's the shard number?" "Which side?" The CLI could output the equivalent, non-interactive command before performing the requested action, making it a teaching opportunity as well. Maybe the "I don't know" option or this approach in general is overkill, but I generally think tools should be smarter. You don't need as much documentation if your tools are inherently capable of guiding you to the right answer.

bhhaskin 8y ago
Seems like a lot of overhead and more things to maintain. On top of that it doesn't teach people how to use the tool. Kinda like 600 hours of coding saved me 30 hours of documentation.
- dguo 8y ago
  I did mention that it could output the equivalent command to teach the user what they can run in the future to bypass the interactive prompt. Point taken, but I don't think it should take that much effort to implement. Some CLI frameworks already have built-in methods for interactivity.
rabidrat 8y ago
That's quite a bit more work than adding a reacmoji to 100 chars which were typed into Slack anyway. We all want better tools with fantastically helpful interfaces, but we also want functioning MVPs yesterday. I like the Etsy approach because it fits into our existing JIT dev/doc workflow.
- dguo 8y ago

chatmasta 8y ago

Why not just add a comment system to the doc page?

TheOtherHobbes 8y ago

It took me a while to work out why this doesn't feel like a full solution. I can imagine it working for simple one-line commands and their switches. I can also imagine it failing to work for any application that requires multi-line code blocks and/or more detailed how-to examples. For example - as a hypothetical, imagine you change your API from Oauth to Oauth2, and you have to provide examples of new standard code, perhaps in different languages. That's very much a how-doc and not a why-doc problem, so supposedly it doesn't belong in the Wiki. But I really can't see the changes being explainable in a fyi. Of course you can say "The new code is in repository X instead of X-1." But if there are other questions and fyi's around Oauth2 - and there will be - there's no guarantee that important fact is going to be at the top of the read list. For complex breaking changes that read list is going to get very long. Simpler changes are going to have the same issues on a smaller scale. So this isn't really immutable documentation - it's more of a database-hosted command cheat sheet. That's a useful thing in itself, but I don't think it's a complete solution for documentation.

habosa 8y ago

At Firebase (and at Google) I have done a _ton_ of work at making our "how docs" consistently correct with limited maintenance time. Consider a page like this: https://firebase.google.com/docs/firestore/manage-data/add-data https://firebase.google.com/docs/firestore/manage-data/add-d... It has 12 "how" sections of code snippets, each with 12 language options. That means 144 code snippets need to be correct for this page to be 100% correct! And we don't auto-generate any of these snippets, since auto-generating idiomatic code is a rabbit hole that's consumed many a good developer. So we, like Etsy, have come up with a separation of concerns between How and Why that works for us. The text on the page is maintained by a tech writer. The code on the page is a series of blocks that look like this: {% includecode snippet="android_firestore_write_doc" %} We have a system that resolves these, at publish time, to match snippets on GitHub. For example: https://github.com/firebase/snippets-android/blob/5f94890d7ba329a8a4c25f0ec8b42d4b9d781bc0/firestore/app/src/main/java/com/google/example/firestore/DocSnippets.java#L318-L336 https://github.com/firebase/snippets-android/blob/5f94890d7b... These snippets can be maintained with all the wonderful tools that exist in the world for maintaining code! We don't need to think of them as just docs. So that "snippets-android" repo has continuous build, a robot that updates dependencies, PRs/Issues for third-party contributions, etc. What we've found is that this system allows us to maintain many thousands of code snippets with only a handful of people watching over them.

itcmcgrath 8y ago
I can't upvote this enough. It also means it is easier for all internal people (not just tech writers or engineers) to make modifications (such as fixing typos, grammar, or adding missing context) safely and without needing to set up complicated tooling/tests or annoy people beyond review diffs+approval.
dancek 8y ago
I noticed the same system in the Google Cloud python samples repository. It's actually quite useful to have the full code available and to easily see which parts are displayed in docs. Feature request, though: include a url in the source code that takes you to the snippet's usages in documentation. Something like // [START https://codesamples.google/firebase/set_document https://codesamples.google/firebase/set_document ] or just the url as a separate comment.

desc 8y ago

This feels incredibly broken to me. The 'how' should not change often, because it enforces rapid change on anything which depends upon it. Any system inevitably acquires dependencies. Maybe it's fine to break those dependencies if you're building some public website where you can always have the latest version of the code running, but if you've got customers who depend on having a stable build which was initially branched three years ago and you still need to build, run and test that, you damn well better have a consistent and stable mechanism for dealing with that codebase which spans multiple years. Your ops and QA people will hate you otherwise. Our own approach has been to pull out the coarse-grained operations and have an 'environment' script which examines the version it's working with in order to spit out relevant details. The 'porcelain' scripts are never allowed to change their API in any breaking fashion, and their behaviour is adapted based on the environment's output.

rhizome 8y ago
DDD: Documentation-driven development. With apologies to Fred Brooks, write documentation first, then write tests for the functionality and syntax described in the documentation, then the code. I would speculate that doing straight docs instead of something like Cucumber would be more productive.

tchaffee 8y ago

I'm surprised no one has yet mentioned how well automated functional tests can document the how. I constantly go back to my automated tests to understand how to do something I've since forgotten but that the code supports.

siliconc0w 8y ago

Maybe the docs are binary and code aware, so the doc system can pull up the binary's code in source control and find all the possible doc strings. Developers can't add/change flags without also updating these. Maybe the docs are history/execution aware so they can list the top N ways the binary is executed. Maybe the docs are also team aware so they list the top N ways the binaries have been executed by your team. So you say, "Here at CoolTechCompany, we built a tool to help control our widgets. Please see the `cli widget` command. And everything kinda works automatically.

jacobsimon 8y ago

This doesn't seem like a solution to me but rather a symptom of a much deeper problem in the way that we write and document code.