13 ms·
How to create a Python package in 2022
- d0mine 4y agoYou can use pyproject.toml for the tox configuration too: [tool.tox] legacy_tox_ini = """<tox.ini content here>"""
- RojerGS 4y agoYeah, but that looks a bit odd and you lose the syntax highlighting... I considered that but ended up going with a tox.ini file.
- tony-allan 4y ago"How do you create a Python package? How do you set up automated testing and code coverage? How do you publish the package? That's what this article teaches you." — delivered as promised!
- d0mine 4y agoEven more: "created a Python package; set up Poetry for dependency management; set up a GitHub repository to host the code; defined some pre-commit hooks to make sure we only make commits that meet certain criteria; added a (fairly permissive) license to the project; configured Poetry to allow uploading to PyPI and a test version of PyPI; tested uploading the package to a test version of PyPI; added Scriv to help us with changelog management and generation; tagged and published a release of our project; wrote a bunch of tests; automated testing and linting with tox; checked code coverage and got it to 100%; set up CI/CD with GitHub actions to run linting and testing; integrated with Codecov to get coverage reports in our pull requests; created a GitHub action to publish our package to PyPI automatically; and added some nifty badges to the README file."
- RojerGS 4y agoOriginal author here: don't make promises you won't keep, right? :P
- deleted 4y ago[deleted]
- la64710 4y agoVery well written , thanks!!
- RojerGS 4y agoThanks a lot! I'm the original author and I appreciate your nice words.
- aaur0 4y agoExcellent blog!
- RojerGS 4y agoThanks a lot! I'm the original author and I appreciate your nice words.
- woodruffw 4y agoThis is really nicely written; kudos to the author for compiling a great deal of information in a readable format. If I can be forgiven one nitpick: Poetry does not use a PEP 518-style[1] build configuration by default, which means that its use of `pyproject.toml` is slightly out of pace with the rest of the Python packaging ecosystem. That isn't to say that it isn't excellent, because it is! But you the standards have come a long way, and you can now use `pyproject.toml` with any build backend as long as you use the standard metadata. By way of example, here's a project that's completely PEP 517 and PEP 518 compatible without needing a setup.py or setup.cfg[2]. Everything goes through pyproject.toml. [1]: https://peps.python.org/pep-0518/ https://peps.python.org/pep-0518/ [2]: https://github.com/trailofbits/pip-audit/blob/main/pyproject.toml https://github.com/trailofbits/pip-audit/blob/main/pyproject...
- RojerGS 4y agoYour nitpick is forgiven! Thanks a lot for this information, I was not aware of this... However, I took a look at PEP 518 and failed to understand what was wrong with Poetry's default configuration. Can you help me out?
- pid-1 4y ago> By way of example, here's a project that's completely PEP 517 and PEP 518 compatible without needing a setup.py or setup.cfg[2]. Everything goes through pyproject.toml. Using pyproject.toml with pip / flit still has many rough edges such as pip being unable to install deps locally for development or not generating lock files. Poetry is way more mature IMO.
- davnn 4y agoI guess it's a great exercise to set up a repository yourself in this way, but once you have experience with the technologies involved, it's much easier to just use a cookiecutter template [1] to set up your package. Another aspect to consider is that there are often different tools to achieve the same goal, thus, it makes sense to experiment until you've found your perfect package setup. [1] https://github.com/search?q=python+package+cookiecutter https://github.com/search?q=python+package+cookiecutter
- RojerGS 4y agoExcellent points! I have seen several cookiecutter templates, but like you said, those aren't very useful when you are at the very start and everything looks weird and new.
- shadycuz 4y agoYou can take this a step further and completely automate the release of your package. That means the tagging, publishing and the GitHub release notes. I don't have a blog post but you can see the process on my personal project https://github.com/DontShaveTheYak/cf2tf https://github.com/DontShaveTheYak/cf2tf Check out the merged PR's and the GitHub actions. I even do alpha releases to test pypi.
- diarrhea 4y agoTo that end, I've had good success with `replease-please`: https://github.com/googleapis/release-please https://github.com/googleapis/release-please . It's available as a GitHub Action and works out of the box very easily. It does tagging, publishing, editing the CHANGELOG, creating a release and more. Whatever you want it to, really, using a bool flag in the CI pipeline that triggers after a release-please release was made aka merged into main.
- captn3m0 4y agoPyPi is adding support for GitHub OIDC for publishing packages soon, so there will be no need to generate API keys - you can just grant your GitHub Actions permissions to publish to PyPi. https://github.com/pypi/warehouse/issues/10619 https://github.com/pypi/warehouse/issues/10619
- woodruffw 4y agoHey, that’s my issue :-) Thank you for linking it! Yes, this will be a huge convenience and security win for the large number of packages that use GitHub to release new versions.
- RojerGS 4y agoOh, this would be neat! Looking forward to it!
- remram 4y agoSurely you mean OAuth2? I really hope you mean OAuth2.
- deleted 4y ago[deleted]
- fetzu 4y agoExcellent and very informative post, thank you very much !
- RojerGS 4y agoThanks for the nice words. Was there anything that was unclear or that you think was missing?
- lmeyerov 4y agoWe've been fine-ish with classic setup.py/setup.cfg + gha for publishing to pypi. But as we do OSS data science (gpu graph ai + viz), where conda is typical nowadays... ... Have to admit: We recently ended up contracting conda packaging out because it was nowhere near clear enough to make sense for our core team to untangle. Would love to see a similar tutorial on a github flow packaging & publishing to conda. Still no convinced we're doing it right for subtleties like optional dependencies: equivalent of `pip install graphistry` vs `pip install graphistry[ai]` vs `graphistry[umap-learn]`, etc.
- RojerGS 4y agoOh, I see... I personally barely use conga and I have no idea how that is done. I don't think I'll write any blog article like that any time soon :( Maybe you could do it!
- plonk 4y agoWe do a lot of deep learning and image processing and pip works much better for us. PyTorch makes wheels that contain all reauired DLLs on all systems. Maybe conda isn’t needed anymore.
- notemaker 4y agoI think of conda as jupyter notebooks. It's ok to experiment with and get something running quickly, but it's a bad idea to use for production code. Personally I never use either of these tools.
- quantumduck 4y agoConda let's you install a specific cuda version directly in a virtual environment with one click though. It's really useful when you have to switch between multiple PyTorch versions and convinient in general imo.
- plonk 4y agoThe PyTorch wheel comes with its own CUDA DLLs, they can’t be shared with other libraries but they won’t be your problem to install that way.
- zx14 4y agoSigh I don't see why I need to use a 3rd party tool for what should be a very straightforward process in Python out of the box. In fact, I think these days it actually is straightforward, of course once you work out what you need to do... Python is a mess.
- uranusjr 4y agoJust fyi, both npm and yarn are “third party” tools. Cargo is closer to the Rust core, and you can find tools in a similar position in Python as well (e.g. setuptools, hatch). Packaging tools being managed by a different group than people working on the “core language” is actually the norm, since those are very different topics and only very few brilliant people care about both of them at the same time. Python people tend to not hide some of these implementation details (probably self-selected by their choice to use Python in the first place), and it’s OK to not like it, but hopefully this helps clarify where your hate comes from so you don’t get burnt by wrong expectations elsewhere down the road.
- zx14 4y agoInstead of being insecure about criticisms to the Python ecosystem and calling my disappointment "hatred" I'd rather we focused on solutions. Just because this mess happens in some other languages doesn't mean it's the right thing. Having a very fragmented community is not a good thing for a beginner. Also, npm is far more of a de facto choice than poetry, which is still better than the state python finds itself in.
- doliveira 4y agoIt doesn't even happen in the example of Node. Npm is even bundled together with NodeJS itself, and yarn is fully compatible with package.json.
- throwaway892238 4y ago
- SNlMyhFioa 4y ago
- jfjqwqk 4y ago
- RojerGS 4y agoHey, original author here. Thanks a lot for sharing this! Also, can't believe everyone let me get away with not writing about documentation! I'll see to it that it gets done and added to the article.
- frumiousirc 4y agoA very nice post. Consider adding a prominent RSS/atom feed for your blog. My lack of finding one means I won't easily catch any future posts.
- slhck 4y agoThis is really nice. The only thing I'm missing here is a simple way to bump versions. Any ideas on how to do that? For Node, it's quite simple and even built into npm. Also the version is only part of the package.json file. For Python you probably have your version somewhere in __init__.py, and I always end up writing ugly bash scripts that modify multiple places with sed.
- udioron 4y agoTry commitizen.
- slhck 4y agoI know commitizen and it doesn't do that. It enforces particular forms of commit messages, but it does not bump versions in Python projects.
- remram 4y agoI never really understood this. Sure, changing the version in one place would be better and I would do that if it was possible with the current Python tools. But I have seen insane setups to achieve this, parsing __init__.py with regex from setup.py, using third-party dependencies that do only that, sometimes scripts with hundreds of lines included in the distribution to support it. Is changing the number 2 places really that big of a deal? You should have a release checklist anyway, with steps like sending an announcement email or tweet etc. How much time does this really save, at the cost of so much complexity?
- diekhans 4y agoLot of good info and saved away! However, it drinks the code coverage cool-aid that started like 30 years ago when code coverage tools emerged. Management types said "high test code coverage == high quality"; lets bean count that!! A great way to achieve high code coverage is to have less than robust code that does not check for crazy error cases that are really hard to reproduce in test cases. Code coverage is a tool to help engineers write good tests. One takes the time to look at the results and improve the test. It is a poor investment to be obsessed with code cover on paths where the cost to test them greatly exceeds the value. 10% coverage and 100% are both alarm bells. Don't assume naive, easy to produce metrics are the same as quality code. Otherwise, and excellent article.
- RojerGS 4y agoThanks for the words of caution! For such a small package, I think 100% code coverage isn't necessarily a bad thing yet :P But you raise valid points!
- zeotroph 4y agoPython is the language with one of the highest 100%-coverage-to-effort ratios. The included unittest.mock framework is making it quite easy to trigger obscure errors and ensure they are handled properly. Combined with thoughtful use of `# pragma: no cover` a 98% code coverage nowadays is an immediate warning that something was rushed. With this and type checking I feel RuntimeErrors much easier to avoid these days. And typing, not even a mention?! :) But otherwise a great article, thank you!
- HelloNurse 4y agoFrom the point of view of a programmer setting up a new project, type checking tools are yet another optional linting operation that can be integrated in pre-commit hooks or other automation.
- int_19h 4y agoThe problem with using mocks extensively for testing is that you then end up mostly testing the mocks. You'll know you've hit that point when you have 100% coverage, but things still break routinely due to interaction (sometimes very indirect) between components.
- dannyboland 4y agoThis is almost exactly how I set up python projects; it’s reassuring to see it set out in one place. I started using tox-poetry-installer[1] to make tox pick up pinned versions from the lock file and reuse the private package index credentials from poetry. [1] https://github.com/enpaul/tox-poetry-installer https://github.com/enpaul/tox-poetry-installer
- RojerGS 4y agoOh, this looks like an interesting tool! Thanks for the link.
- zmmmmm 4y agoI really dislike the Python convention of plonking source code effectively in the root directory. It means literally any random thing in there can get picked up if you put your package in the PYTHONPATH. Is there a reason the Python world did not standardise on putting source code in a "src" directory like every other language?
- pmontra 4y agoI don't have a single src directory in any of the Elixir, JavaScript, Ruby projects I'm currently working on. Python too, obviously. I got src directories in the Elixir dependencies written in Erlang, in about 12% of the node_modules used by a React project and in the few C extensions I'm using for Ruby. May I conclude that src is uncommon at least in scripting languages? (Elixir is compiled.) Maybe the reason is that there is only source code and there is no need for a separate directory for a build / dist.
- dannyboland 4y agoYeah, I adopted the “src” dir layout after reading this post some time back: https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-str... It describes some of the outdated motivations for the other layouts commonly seen with python, as well as the many benefits of the “src” layout.
- bvrmn 4y agoPost doesn't make sense. It clearly states that you only be sure that package is working by actually installing it into clean virtualenv and testing it there. `src` or any other layout doesn't matter.
- iggy_knights 4y agoThe difference is if you're running tests from your root project directory, the package is importable regardless of whether or not it is installed, as python picks up packages in the current working directory by name. src/ prevents this.
- anonymoushn 4y agoIt's impressive that this involves 3 human-readable configuration languages and 2 markup languages
- dr_kiszonka 4y agoThis excellent article references Textualize, which - as I have just found from their website - has a really great approach to job interviews: https://www.textualize.io/jobs https://www.textualize.io/jobs [I have no ties to this company and have never applied there.]
- TOMDM 4y agoThis is a fantastic resource. Are there any similar resources for setting up internal packages that you don't intend to publish publicly? I can think of a number of situations where I would have benifitted from it, but the process of configuring a package to publish, hosting it and then pulling it when necessary is a mystery to me.
- remram 4y agoYou can mostly just put wheels and source dists on any HTTP server with directory listing enabled and have it work (with --extra-index-url). Use a subdirectory per package. https://packaging.python.org/en/latest/guides/hosting-your-own-index/ https://packaging.python.org/en/latest/guides/hosting-your-o...
- softwareoiu 4y agoJust use https://pypi.org/project/zest.releaser/ https://pypi.org/project/zest.releaser/
- emacs28 4y agoMy only gripe with poetry is that when I tried it out last year there was no equivalent for `pip install -e` which can be used to install other python package dependancies that you are simultaneously developing. I found that feature useful enough to stick with setuptools and setup.py instead.
- brianskarda 4y agoPoetry has path dependencies and the develop flag to enable the same editable functionality https://python-poetry.org/docs/dependency-specification/#path-dependencies https://python-poetry.org/docs/dependency-specification/#pat...
- emacs28 4y agoThanks, I think that the thing this still lacks compared to `pip install -e` is that the develop dependency is then only visible inside other packages configured with poetry. So it is not useful for example if you want to locally install a package you make to be accessed through cli or any other python script not using poetry.
- remram 4y agoI'm not sure what you are talking about. It will be installed in the virtual environment just the same. There is no mechanism for it to "only be visible inside other packages configured with poetry".
- wyuenho 4y agoI wish people would just forget about pre-commit, this thing is especially useless in a setting where a CI/CD pipeline exists. It's not that hard to write a simple Makefile or shellscript to run linters on push. Pre-commit is one of the most annoying tools that have come into existence in recent years that everyone seems to be cargo-culting. It doesn't play well with editors since in order to find the actual binary path, you'd have to open up a sqlite database to fish out the virtualenv pre-commit created. Pre-commit also increases maintenance burden, since its configuration is completely separate from your usual requirements-dev.txt/pyproject.toml/setup.cfg etc. If you have dev dependencies in one of these files because making your editors to find the pre-commit created binaries are hard, now you have to keep both versions in sync. I really don't see the point of any pre-commit hooks unless you are the one guy that doesn't use a modern CI/CD platform.
- globular-toast 4y agoOur CI pipelines invoke pre-commit. That way it's trivial to run the exact same tools locally as would be run in CI. Running the tools locally is basically about tightening the development loop. Many of the commonly used tools (e.g. black, isort etc.) actually make the changes to the files so you'll never even commit failing versions. Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check? There's nothing at all stopping you from doing that. Pre-commit is completely optional for each developer. I would just recommend it for sanity reasons.
- wyuenho 4y ago> Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check? Yes, that's what CI/CD is for. > Pre-commit is completely optional for each developer. No it is not, it's installed at your git pre-commit hook, and it gets run every fucking time I commit, even if I've set up my editor correctly to auto-format and lint everything at I develop, meaning 99.9999% of the time sure the code I commit will pass all these linter checks. I can use git commit --no-verify to bypass pre-commit, but then again, what's the point of using pre-commit in the first place if you need to bypass it? There is absolutely no point of linting twice locally and thrice in total just to hit the first stage of your deployment.
- avidphantasm 4y agoFantastic article. Thank you so much for putting this together. Any thoughts of putting this in a Git repo somewhere so that it can be a living reference contributed to by others? Otherwise, I worry that in 6 months this will start to get confusing, and in 18 months it will be dangerously out of date (though I haven’t used poetry yet, so maybe they aim for more stability).
- bvrmn 4y agoHm, it's very strange argument regarding src directory. Article they refers to mentioned main issue with package in root is: it's possible to miss submodules in final released py-package. But src doesn't prevent this. You also can run your tests. But without proper packages option or find_packages in setup.py one gets a broken package.
- tpoacher 4y agoThe title should be: How to create a "Python DISTRIBUTION package". The term "python package" means something entirely different (or at the very least is ambiguous in a pypi/distribution context). To add to the confusion, creating a totally normal, runnable python package in a manner that makes it completely self-contained such that it can be "distributed" in a standalone manner, while still being a totally normal boring python package, is also totally possible (if not preferred, in my view). (shameless plug: https://github.com/tpapastylianou/self-contained-runnable-python-package-template https://github.com/tpapastylianou/self-contained-runnable-py... )
- radonek 4y agoWhat a great article. We start with learning that we absolutely need this Poetry thing because… it's what everyone else uses. It's refreshing to see author who can skip usual badly argued justifications and just plain admin that he does not know shit and is just following rest of the herd. Then we continue by "solving" depependencies by usual way of ignoring them and just freezing whatever happens to be present. Then there is inevitable firing up of virtualenv, because that's just what you have to do when dealing with messed up dependencies. Next one is new to me. Apparently, one does not just set up git hooks nowadays but use separate tool with declarative config. Because if you ever happen upon something not covered by the Tool, that would mean you are no longer part of the herd. Then we push our stuff straight to pypi, because of course our stuff can't possibly have any dependencies outside of python herd ecosystem. It's not like we knew our dependencies anyway. Then comes the fun part, pulling in tox, because when you have special tool to handle dependencies, what you just need is another tool with different environment and dependency model. Code quality section I will just skip over, seeing what pass for code quality these days makes me too sad. What follows is setup of several proprietary projects that modern opensource seemingly can't exist without. What is more interresting is "tyding up" by moving code from git root to subdir. Now, this is of course perfectly sensible thing to, but I wonder why is it called 'src'? Maybe some herd memeber saw compiled language somewhere and picked it up without understanding difference between compiled binary and source code? Now don't take this as if I have problem with the article content in itself. No, as a primer to modern python packaging it's great. It's not authors fault that his work is so comprehensive it lays out bare all the idiosyncrasies, herd mentality, cargocultism and general laziness of python ecosystem these days. Or is it?
- sigmonsays 4y agothe irony here is python packaging has sucked forever, and this is just another example of it. "Do more with less" has never entered the average python developers mind. you'd think herd mentality might help it but it only creates more packaging solutions. Now days, I've stopped using python outside of tiny scripts and I will never touch it for a large project.
- matheusmoreira 4y ago
- Bullfight2Cond 4y agoPoetry uses non-standard dependency specification formats. PDM is like Poetry but faster/more standards compliant. https://pdm.fming.dev/ https://pdm.fming.dev/
- iggy_knights 4y agoGood thing about finally converging on some sort of standard means tools become more interoperable: Another good tool (which was endorsed by the PyPA) is Hatch - https://hatch.pypa.io/latest/environment/ https://hatch.pypa.io/latest/environment/ I currently use PDM because it supports conda virtual environments for isolation, but am keeping an eye on Hatch.
- p33p 4y agoDidn't they also endorse PipEnv and that turned into a flop?
- lazrgatr 4y agocool - there's a lot of overlap with the "Hypermodern Python"[1] so if anyone enjoyed this and wants to read more, they might also enjoy that. [1]: https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu...
- nogbit 4y agoFantastic article. A clickable TOC at the top would be a great addition.
- linsomniac 4y agoI recently set up a new Python package and ran across the github instructions for automatically uploading the package to pypi. A month later I ran into a bug and needed to push out an update, man was that nice to have automated, and easy to set up.
- butz 4y agoRelated question: how to correctly package a graphical application, preferably for Linux?
- mixmastamyk 4y agoLook at debs/rpms, or flatpak etc for a cross-distro equivalent.