5 ms·

Ask HN: How do FAANGs document their infrastructure?

56 points by gtirloni 5y ago

Any methodologies? Public guides?

diskzero 5y ago
This is a very broad question and it would help if you could follow up with your definition of infrastructure. At Amazon, we would use a variety of solutions to document the build process, how to set up internal accounts, how to respond to pager calls, etc. Amazon has a custom documentation system, but it isn't a whole lot different than the various WIKIs that are available outside of Amazon. Some of what you see as documentation for AWS [1] are an evolution of internal documents from when the solution was used internally before being made public. My time at Apple was earlier, but the systems were similar. Various internal web sites, files checked into cvs/svn/etc and assorted custom issue tracking tools. I wish I could say there were amazing and magical tools that made documentation easy and painless, but there weren't during my time at Amazon and Apple. There might be some very nice tools available that can traverse and document network topology and services that I wasn't exposed to. Hopefully someone will come along and let us know. [1] https://docs.aws.amazon.com/index.html?nc2=h_ql_doc_do_v https://docs.aws.amazon.com/index.html?nc2=h_ql_doc_do_v
- iamevn 5y ago
  Same at Google, the most important thing is that the main documentation source internally is just markdown files checked in alongside code that is automatically served on the documentation site. Freshness is encouraged by owners periodically getting bugs to review pages and update a tag indicating the page was fresh as of X date. There's support for writing out diagrams in graphviz dot but people usually just draw them in google docs or some other external program and check a screenshot of that in to be embedded in the page. I'm sure some systems had more automation building documentation out of code directly but usually it's just the simple markdown written manually. There's also google docs which is nicer for quickly iterating and getting reviews but it's much less discoverable and falls out of sync with the system much more easily.
Sevii 5y ago
Its a smorgasbord of git commits, READMEs, wikis, Jira tickets, documentation 'websites', word documents, google doc like documents, etc.
- marto1 5y ago
  and copious amounts of https://xkcd.com/927/ https://xkcd.com/927/
oftenwrong 5y ago
There is a bunch of information on how Google does documentation in the book Software Engineering at Google: https://www.google.com/books/edition/_/WXTTDwAAQBAJ https://www.google.com/books/edition/_/WXTTDwAAQBAJ One notable practice mentioned in the book is their g3doc system. Documentation is kept in the (google3) repo alongside the things documented. It is encouraged to update the documentation and the code at the same time. This also means the full history of the documentation is available in repo rather than in a separate wiki history. There is also a system of URL-shortener-esque "go/ links" that allows people to easily link to or jump to documentation. For example, "look at go/foobar". The docs are also exposed in their code search system.
- cema 5y ago
  Athenahealth has borrowed the go system. I should say discoverability is not great.
  - oftenwrong 5y ago
    I'm interested to hear more about your experience with it if you're willing to share.
    - cema 5y ago
      When you know where to go (go/righthere) it's great. When you don't know where to go you need an external source (document or person) to tell you.

56 points by gtirloni 5y ago

Any methodologies? Public guides?

This is a very broad question and it would help if you could follow up with your definition of infrastructure. At Amazon, we would use a variety of solutions to document the build process, how to set up internal accounts, how to respond to pager calls, etc. Amazon has a custom documentation system, but it isn't a whole lot different than the various WIKIs that are available outside of Amazon. Some of what you see as documentation for AWS [1] are an evolution of internal documents from when the solution was used internally before being made public. My time at Apple was earlier, but the systems were similar. Various internal web sites, files checked into cvs/svn/etc and assorted custom issue tracking tools. I wish I could say there were amazing and magical tools that made documentation easy and painless, but there weren't during my time at Amazon and Apple. There might be some very nice tools available that can traverse and document network topology and services that I wasn't exposed to. Hopefully someone will come along and let us know. [1] https://docs.aws.amazon.com/index.html?nc2=h_ql_doc_do_v https://docs.aws.amazon.com/index.html?nc2=h_ql_doc_do_v

iamevn 5y ago
Same at Google, the most important thing is that the main documentation source internally is just markdown files checked in alongside code that is automatically served on the documentation site. Freshness is encouraged by owners periodically getting bugs to review pages and update a tag indicating the page was fresh as of X date. There's support for writing out diagrams in graphviz dot but people usually just draw them in google docs or some other external program and check a screenshot of that in to be embedded in the page. I'm sure some systems had more automation building documentation out of code directly but usually it's just the simple markdown written manually. There's also google docs which is nicer for quickly iterating and getting reviews but it's much less discoverable and falls out of sync with the system much more easily.

Sevii 5y ago

Its a smorgasbord of git commits, READMEs, wikis, Jira tickets, documentation 'websites', word documents, google doc like documents, etc.

marto1 5y ago
and copious amounts of https://xkcd.com/927/ https://xkcd.com/927/

oftenwrong 5y ago

There is a bunch of information on how Google does documentation in the book Software Engineering at Google: https://www.google.com/books/edition/_/WXTTDwAAQBAJ https://www.google.com/books/edition/_/WXTTDwAAQBAJ One notable practice mentioned in the book is their g3doc system. Documentation is kept in the (google3) repo alongside the things documented. It is encouraged to update the documentation and the code at the same time. This also means the full history of the documentation is available in repo rather than in a separate wiki history. There is also a system of URL-shortener-esque "go/ links" that allows people to easily link to or jump to documentation. For example, "look at go/foobar". The docs are also exposed in their code search system.

cema 5y ago
Athenahealth has borrowed the go system. I should say discoverability is not great.
- oftenwrong 5y ago
  I'm interested to hear more about your experience with it if you're willing to share.
  - cema 5y ago
    When you know where to go (go/righthere) it's great. When you don't know where to go you need an external source (document or person) to tell you.