10 ms·

Show HN: Zero-Config Documentation Websites for Python

233 points by timothycrosley 7y ago

noobiemcfoob 7y ago
I just used this on a random project I happen to be building documentation for. I'm pretty in love with the minimalism of 'portray'. I wish there was a chance to get my team to use it, but I'll definitely be using it in my personal projects going forward!
dragonsh 7y ago
My team at present use sphinx-doc[1] and generate both HTML and PDF. We write the documentation as part of the code using docstring and then other documents manually in rst. We use Sphinx plugin to automatically generate document from code. I will give portray a try and see. Is there any comparison between Sphinx-doc and portray? [1] http://www.sphinx-doc.org/en/master/ http://www.sphinx-doc.org/en/master/
- DanFeldman 7y ago
  Which plugin do you use to go docstring->docs? I found it difficult out of the box to determine w/ Sphinx how to get everything to play nice.
  - susam 7y ago
    I use sphinx-apidoc[1] command to extract docstrings from code and create Sphinx documentation files out of them. Enter this command to look for modules in a package, say, foopkg, and create .rst files to generate documentation for each module in the package and subpackages: sphinx-apidoc --module-first -o docs/api foopkg I am assuming that the Sphinx documentation project resides in a directory named docs. Each .rst file would contain automodule directives for Sphinx to automatically pull docstrings from foopkg and render them in the generated documentation. However, for Sphinx to understand these automodule directives, the autodoc[2] extension must be enabled in docs/conf.py. extensions = ['sphinx.ext.autodoc'] If the documentation is written using Google style or NumPy style docstrings, then the napolean[3] extension must also be enabled. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon'] The sphinx-apidoc command above would also write a file named docs/api/modules.rst which would be used to render the starting page of the automatically generated API documentation. So include this docs/api/modules.rst with the Sphinx toctree directive from the some other page that the user is likely to visit. For example, the start page of the project documentation would likely be rendered from `docs/index.rst`, so add this to `docs/index.rst`: API --- .. toctree:: api/modules Now render the documentation with Sphinx normally: cd docs && make html [1]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html [2]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html [3]: https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html https://www.sphinx-doc.org/en/master/usage/extensions/napole...
- weberc2 7y ago
  We've historically done this as well. One issue we perpetually run into is that the docstrings get out of sync with the actual signature / types of the methods/etc that they purport to document. I suspect there are plugins that generate documentation from types such that there is less to keep synchronized. It is also unfortunate that Python projects have to set up infrastructure to build and publish documentation packages. I really like the effortlessness that Go's ecosystem brings--https://godoc.org https://godoc.org just reads your source code from your git repository, so there is are no explicit steps for building and publishing documentation.
  - carapace 7y ago
    > the docstrings get out of sync with the actual signature / types of the methods/etc that they purport to document But that's on the devs, not Sphinx, eh?
    - weberc2 7y ago
      It's kind of on the whole dynamically typed model. If you can have static type annotations that are validated for correctness, then you can use those annotations to validate or generate your documentation. Python supports annotations and validation thereof via Mypy, and assuming the project takes advantage of these, then Sphinx could leverage them to validate/generate documentation.
  - baylessj 7y ago
    With Sphinx's Napoleon extension (https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html#type-annotations https://www.sphinx-doc.org/en/master/usage/extensions/napole...) Python 3 type annotations will be added to the generated docstrings so you don't have to write the annotations both in the function definition and in its docstring. I've found this to be very helpful in keeping docs in sync with the code.
- timothycrosley 7y ago
  The biggest differences between Sphinx-doc and portray: - Markdown - Zero Config required - Focus on Simplicity https://timothycrosley.com/project-2-portray https://timothycrosley.com/project-2-portray goes into why I created the project and how I viewed the current state of Python documentation tool.
contravariant 7y ago
It's always kind of interesting to look at a document created by the tool it's describing.
- ByThyGrace 7y ago
  I thought the GIF within a GIF was very clever. Did you notice? It's "playing" itself back.
  - jedberg 7y ago
    I think the number of playbacks is the number of times they had to re-record the video. :)
    - timothycrosley 7y ago
      Ha, I wish it was accidental! In reality, I created the gif many more times than needed just because I wanted to create the effect.
hultner 7y ago
Very neat, I especially like it's compability with mkdocs-themes. Will try this out on some project :)
amjith 7y ago
How is the search implemented?
- timothycrosley 7y ago
  portray uses MkDocs default search plugin which uses lunr.js behind the scenes.
- deleted 7y ago
  [deleted]
danking00 7y ago
Tried this on our project, GitHub.com/hail-is/hail. Sphinx is a persistent thorn in our side, but we've made it work. A couple things I had to do to get this working: - something thinks a filename with dots (e.g. example.8bits.bgen) means there is a python module that needs to be imported (e.g. example), and fails when that does not exist; we have files like this in a data directory - I could not quickly figure out how to ignore files, so I had to delete a Sphinx conf.py file. - Sphinx resolves the _templates folder relative to where it is called, not relative to the doc string's source file, so I had to add several symlinks (rather than manually edit every doc string). --- OK docs are generating. Quite slow, maybe a couple minutes. Looks like one core is at 100% during this process. We've got ~400 files. Maybe its parsing some data files? The whole directory is 318 MB. --- Now I'm looking at the docs. First thing I notice is that Python type hint forward references break everything. Consider: class GroupedMatrixTable( parent: 'MatrixTable', ... The generated doc page is missing the class name and the entire class definition is inlined as a preformatted block (with highlighting :shrug:). All my python function def's seem to be missing the function name? No arg functions look really weird def ( ) So all our docs are in ReST (thanks Sphinx :|). This means they're not valid Markdown. It seems that invalid markdown in a class doc string can break the formatting of all the methods (presenting them again as one reformatted block). --- On the bright side, the mobile version looks great and the search works better than my experience with Sphinx. EDIT: formatting
- pokeymcsnatch 7y ago
  A little off-topic, but as someone who's just started producing documentation in Sphinx, what are the problems/downsides that you've run in to? I've used it for two projects so far: documenting a relatively small python module, and for documenting a RESTful API. I do a lot of embedded work, so I'd like to use it as a place to document a piece of hardware in it's entirety, from schematics/layouts to firmware and build toolchains to actual use of the device. Do you (or anyone else reading this) have any comments on that use case? I think my main complaint at the moment is that it doesn't play nice with Markdown, so I had to re-format a bunch of pre-existing documentation for it to work in Sphinx. ReST seems to render alright in Gitlab though, so that's a plus.
fucking_tragedy 7y ago
How easy is it to theme for when Material starts looking dated?
- timothycrosley 7y ago
  https://timothycrosley.github.io/portray/docs/quick_start/4.-configuration/ https://timothycrosley.github.io/portray/docs/quick_start/4.... portray is compatible with all MkDocs themes and plugins out-of-the-box
  - fucking_tragedy 7y ago
    Awesome, looking forward to trying this out on a new project.
gillesjacobs 7y ago
This is great! You hit the need for configless docsites that many Pythonistas who are in data science and research have: we write a lot of experimental and prototype code. Docstring documentation in itself feels like a considerable investment in these contexts so anything that helps publication without any hassle is more than welcome. Are you planning to support Restructured Text next to Markdown? I find ResT is used more often (in the machine learning ecosystem at least).
- timothycrosley 7y ago
  Thanks! > Are you planning to support Restructured Text next to Markdown? I will look into supporting this, I agree that it definitely makes sense as a feature!
isbjorn16 7y ago
I was super pumped by this, and took some time to look into pdoc3 too. - https://github.com/pdoc3/pdoc/issues/87 https://github.com/pdoc3/pdoc/issues/87 - https://github.com/pdoc3/pdoc/issues/64 https://github.com/pdoc3/pdoc/issues/64 That's a hard pass from me. I'm not going to link my employer's name (much less my own name) to anything with a swastika on it.
- timothycrosley 7y ago
  I saw that as well, late into the project creation. The pdoc3 maintainer claims it's not that - but there are other reasons to want to want to avoid pdoc3. I'm going to be working on replace the pdoc3 dependency with pdoc in a way that is transparent to users.
  - isbjorn16 7y ago
    Fantastic! I look forward to this :)
    - timothycrosley 7y ago
      The change away from pdoc3 and to a new fork of the original pdoc (called pdocs) has landed: https://github.com/timothycrosley/portray/blob/master/CHANGELOG.md#120---september-3-2019 https://github.com/timothycrosley/portray/blob/master/CHANGE... https://timothycrosley.github.io/pdocs/ https://timothycrosley.github.io/pdocs/ Thanks! ~Timothy
  - dragonsh 7y ago
    Great to see you are receptive to this change. I didn't notice it earlier but then when I see on the right hand corner of the website I was astonished it's there.
    - timothycrosley 7y ago
      The change away from pdoc3 and to a new fork of the original pdoc (called pdocs) has landed: https://github.com/timothycrosley/portray/blob/master/CHANGELOG.md#120---september-3-2019 https://github.com/timothycrosley/portray/blob/master/CHANGE... https://timothycrosley.github.io/pdocs/ https://timothycrosley.github.io/pdocs/ Thanks! ~Timothy
MadWombat 7y ago
I am confused, all I get is "No Python project found in the given directory".
- timothycrosley 7y ago
  portray requires your project to be packaged for it's auto discovery - if you get this error and your project has a setup.py or pyproject.yaml in the root (and this is where you ran portray from) let me know and I'll create a ticket! Otherwise - I highly encourage looking at using poetry to setup your pyproject.yaml file: https://poetry.eustace.io/ https://poetry.eustace.io/
  - MadWombat 7y ago
    my project doesn't have a setup.py because it is not intended for packaging and it doesn't have a pyproject.yaml because to be honest, this is the first time I heard about such a file. But then again, the title says "zero-config".
    - timothycrosley 7y ago
      portray no longer requires a config to be present, instead you can just specify modules directly from the command line with `-m`: https://timothycrosley.github.io/portray/TROUBLESHOOTING/#noprojectfound https://timothycrosley.github.io/portray/TROUBLESHOOTING/#no...
  - brorfred 7y ago
    I have this problem when using the src folder tree conventions (poetry new --src new_package).
peteretep 7y ago
Excited to try this. From a Perl background, used to POD, Python's doc systems have been a sewer.
abakus 7y ago
Kudos to using toml for configuration.

I just used this on a random project I happen to be building documentation for. I'm pretty in love with the minimalism of 'portray'. I wish there was a chance to get my team to use it, but I'll definitely be using it in my personal projects going forward!

dragonsh 7y ago

My team at present use sphinx-doc[1] and generate both HTML and PDF. We write the documentation as part of the code using docstring and then other documents manually in rst. We use Sphinx plugin to automatically generate document from code. I will give portray a try and see. Is there any comparison between Sphinx-doc and portray? [1] http://www.sphinx-doc.org/en/master/ http://www.sphinx-doc.org/en/master/

DanFeldman 7y ago
Which plugin do you use to go docstring->docs? I found it difficult out of the box to determine w/ Sphinx how to get everything to play nice.
- susam 7y ago
  I use sphinx-apidoc[1] command to extract docstrings from code and create Sphinx documentation files out of them. Enter this command to look for modules in a package, say, foopkg, and create .rst files to generate documentation for each module in the package and subpackages: sphinx-apidoc --module-first -o docs/api foopkg I am assuming that the Sphinx documentation project resides in a directory named docs. Each .rst file would contain automodule directives for Sphinx to automatically pull docstrings from foopkg and render them in the generated documentation. However, for Sphinx to understand these automodule directives, the autodoc[2] extension must be enabled in docs/conf.py. extensions = ['sphinx.ext.autodoc'] If the documentation is written using Google style or NumPy style docstrings, then the napolean[3] extension must also be enabled. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon'] The sphinx-apidoc command above would also write a file named docs/api/modules.rst which would be used to render the starting page of the automatically generated API documentation. So include this docs/api/modules.rst with the Sphinx toctree directive from the some other page that the user is likely to visit. For example, the start page of the project documentation would likely be rendered from `docs/index.rst`, so add this to `docs/index.rst`: API --- .. toctree:: api/modules Now render the documentation with Sphinx normally: cd docs && make html [1]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html [2]: https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html [3]: https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html https://www.sphinx-doc.org/en/master/usage/extensions/napole...
weberc2 7y ago
We've historically done this as well. One issue we perpetually run into is that the docstrings get out of sync with the actual signature / types of the methods/etc that they purport to document. I suspect there are plugins that generate documentation from types such that there is less to keep synchronized. It is also unfortunate that Python projects have to set up infrastructure to build and publish documentation packages. I really like the effortlessness that Go's ecosystem brings--https://godoc.org https://godoc.org just reads your source code from your git repository, so there is are no explicit steps for building and publishing documentation.
- carapace 7y ago
  > the docstrings get out of sync with the actual signature / types of the methods/etc that they purport to document But that's on the devs, not Sphinx, eh?
  - weberc2 7y ago
    It's kind of on the whole dynamically typed model. If you can have static type annotations that are validated for correctness, then you can use those annotations to validate or generate your documentation. Python supports annotations and validation thereof via Mypy, and assuming the project takes advantage of these, then Sphinx could leverage them to validate/generate documentation.
- baylessj 7y ago
  With Sphinx's Napoleon extension (https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html#type-annotations https://www.sphinx-doc.org/en/master/usage/extensions/napole...) Python 3 type annotations will be added to the generated docstrings so you don't have to write the annotations both in the function definition and in its docstring. I've found this to be very helpful in keeping docs in sync with the code.
timothycrosley 7y ago
The biggest differences between Sphinx-doc and portray: - Markdown - Zero Config required - Focus on Simplicity https://timothycrosley.com/project-2-portray https://timothycrosley.com/project-2-portray goes into why I created the project and how I viewed the current state of Python documentation tool.

contravariant 7y ago

It's always kind of interesting to look at a document created by the tool it's describing.

ByThyGrace 7y ago
I thought the GIF within a GIF was very clever. Did you notice? It's "playing" itself back.
- jedberg 7y ago
  I think the number of playbacks is the number of times they had to re-record the video. :)
  - timothycrosley 7y ago
    Ha, I wish it was accidental! In reality, I created the gif many more times than needed just because I wanted to create the effect.

hultner 7y ago

Very neat, I especially like it's compability with mkdocs-themes. Will try this out on some project :)

amjith 7y ago

How is the search implemented?

timothycrosley 7y ago
portray uses MkDocs default search plugin which uses lunr.js behind the scenes.
deleted 7y ago
[deleted]

danking00 7y ago

Tried this on our project, GitHub.com/hail-is/hail. Sphinx is a persistent thorn in our side, but we've made it work. A couple things I had to do to get this working: - something thinks a filename with dots (e.g. example.8bits.bgen) means there is a python module that needs to be imported (e.g. example), and fails when that does not exist; we have files like this in a data directory - I could not quickly figure out how to ignore files, so I had to delete a Sphinx conf.py file. - Sphinx resolves the _templates folder relative to where it is called, not relative to the doc string's source file, so I had to add several symlinks (rather than manually edit every doc string). --- OK docs are generating. Quite slow, maybe a couple minutes. Looks like one core is at 100% during this process. We've got ~400 files. Maybe its parsing some data files? The whole directory is 318 MB. --- Now I'm looking at the docs. First thing I notice is that Python type hint forward references break everything. Consider: class GroupedMatrixTable( parent: 'MatrixTable', ... The generated doc page is missing the class name and the entire class definition is inlined as a preformatted block (with highlighting :shrug:). All my python function def's seem to be missing the function name? No arg functions look really weird def ( ) So all our docs are in ReST (thanks Sphinx :|). This means they're not valid Markdown. It seems that invalid markdown in a class doc string can break the formatting of all the methods (presenting them again as one reformatted block). --- On the bright side, the mobile version looks great and the search works better than my experience with Sphinx. EDIT: formatting

pokeymcsnatch 7y ago
A little off-topic, but as someone who's just started producing documentation in Sphinx, what are the problems/downsides that you've run in to? I've used it for two projects so far: documenting a relatively small python module, and for documenting a RESTful API. I do a lot of embedded work, so I'd like to use it as a place to document a piece of hardware in it's entirety, from schematics/layouts to firmware and build toolchains to actual use of the device. Do you (or anyone else reading this) have any comments on that use case? I think my main complaint at the moment is that it doesn't play nice with Markdown, so I had to re-format a bunch of pre-existing documentation for it to work in Sphinx. ReST seems to render alright in Gitlab though, so that's a plus.

fucking_tragedy 7y ago

How easy is it to theme for when Material starts looking dated?

timothycrosley 7y ago
https://timothycrosley.github.io/portray/docs/quick_start/4.-configuration/ https://timothycrosley.github.io/portray/docs/quick_start/4.... portray is compatible with all MkDocs themes and plugins out-of-the-box
- fucking_tragedy 7y ago
  Awesome, looking forward to trying this out on a new project.

gillesjacobs 7y ago

This is great! You hit the need for configless docsites that many Pythonistas who are in data science and research have: we write a lot of experimental and prototype code. Docstring documentation in itself feels like a considerable investment in these contexts so anything that helps publication without any hassle is more than welcome. Are you planning to support Restructured Text next to Markdown? I find ResT is used more often (in the machine learning ecosystem at least).

timothycrosley 7y ago
Thanks! > Are you planning to support Restructured Text next to Markdown? I will look into supporting this, I agree that it definitely makes sense as a feature!

isbjorn16 7y ago

I was super pumped by this, and took some time to look into pdoc3 too. - https://github.com/pdoc3/pdoc/issues/87 https://github.com/pdoc3/pdoc/issues/87 - https://github.com/pdoc3/pdoc/issues/64 https://github.com/pdoc3/pdoc/issues/64 That's a hard pass from me. I'm not going to link my employer's name (much less my own name) to anything with a swastika on it.

timothycrosley 7y ago
I saw that as well, late into the project creation. The pdoc3 maintainer claims it's not that - but there are other reasons to want to want to avoid pdoc3. I'm going to be working on replace the pdoc3 dependency with pdoc in a way that is transparent to users.
- isbjorn16 7y ago
  Fantastic! I look forward to this :)
  - timothycrosley 7y ago
    The change away from pdoc3 and to a new fork of the original pdoc (called pdocs) has landed: https://github.com/timothycrosley/portray/blob/master/CHANGELOG.md#120---september-3-2019 https://github.com/timothycrosley/portray/blob/master/CHANGE... https://timothycrosley.github.io/pdocs/ https://timothycrosley.github.io/pdocs/ Thanks! ~Timothy
- dragonsh 7y ago
  Great to see you are receptive to this change. I didn't notice it earlier but then when I see on the right hand corner of the website I was astonished it's there.
  - timothycrosley 7y ago
    The change away from pdoc3 and to a new fork of the original pdoc (called pdocs) has landed: https://github.com/timothycrosley/portray/blob/master/CHANGELOG.md#120---september-3-2019 https://github.com/timothycrosley/portray/blob/master/CHANGE... https://timothycrosley.github.io/pdocs/ https://timothycrosley.github.io/pdocs/ Thanks! ~Timothy

MadWombat 7y ago

I am confused, all I get is "No Python project found in the given directory".

timothycrosley 7y ago
portray requires your project to be packaged for it's auto discovery - if you get this error and your project has a setup.py or pyproject.yaml in the root (and this is where you ran portray from) let me know and I'll create a ticket! Otherwise - I highly encourage looking at using poetry to setup your pyproject.yaml file: https://poetry.eustace.io/ https://poetry.eustace.io/
- MadWombat 7y ago
  my project doesn't have a setup.py because it is not intended for packaging and it doesn't have a pyproject.yaml because to be honest, this is the first time I heard about such a file. But then again, the title says "zero-config".
  - timothycrosley 7y ago
    portray no longer requires a config to be present, instead you can just specify modules directly from the command line with `-m`: https://timothycrosley.github.io/portray/TROUBLESHOOTING/#noprojectfound https://timothycrosley.github.io/portray/TROUBLESHOOTING/#no...
- brorfred 7y ago
  I have this problem when using the src folder tree conventions (poetry new --src new_package).

peteretep 7y ago

Excited to try this. From a Perl background, used to POD, Python's doc systems have been a sewer.

abakus 7y ago

Kudos to using toml for configuration.