15 ms·
How to make a Python package in 2021
- mikepurvis 6y agoI hadn't heard of flit, it does seem like it's not brand new on the scene, however it is primarily a single author, so expect a tool which is opinionated and for which the opinions may not necessarily reflect a broad consensus: https://github.com/takluyver/flit/graphs/contributors https://github.com/takluyver/flit/graphs/contributors With a title like this, I'd be expecting to see an article describing the latest tools and recommendations from the PyPA, which are here: https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/ (In short, it's setup.cfg + pyproject.toml, `python3 -m build` to build, twine to upload.)
- neolog 6y agoThe PyPA recommendation is old and out of date. Using a system that doesn't manage constraints and hash-checked lockfiles is bad practice.
- carreau 6y agoThomas is well know as one of the maintainer of IPython and Jupyter, and developed flit while working on the pep for pyproject.toml and the pip backend allowing things like python -m build. Though `python -m build` only works _if_ you use something like flit or setup.py in the backend to do build the package and hence why you can set flit as a build-backend. So yes, flit is one of the latest tool, and yes it is one of the things that push for the ability to use pyproject.toml+python3 -m build, you just seem to miss some subtleties of the toolchain.
- mikepurvis 6y agoThe additional context is appreciated— it sounds like this tool is something which is likely to be supported long term, so that at least is good.
- erezsh 6y agoHow does flit compare to poetry? The seem to be both doing the same thing, and in a very similar way.
- takluyver 6y agoPoetry does much more than Flit, like resolving dependencies, creating a lock file, and managing an environment where you can run your code. In particular, Poetry is meant to support application development (where you want to have a fixed version of your dependencies) as well as library development. Flit is more aimed at being the simplest possible thing to put a package on PyPI, if that's all you want to do. It expects you to list dependencies in pyproject.toml manually.
- nerdponx 6y agoI feel like this is a good place to mention Pip-tools [0] which can generate a lockfile of sorts from a standard requirements.txt (or a setup.py). Specifically, it resolves all dependencies (including hashes), and writes them to a new "requirements" file that you can read with the usual `pip install -r`. The nice part about Pip-tools versus Flit or Poetry or Pipenv is that Pip-tools lets you keep using Setuptools if you want to, or if you're unable to switch to one of the others for some reason (and valid reasons do exist). [0]: https://pypi.org/project/pip-tools https://pypi.org/project/pip-tools
- takluyver 6y agoFlit is definitely opinionated, and not suitable for every use case. As Carreau hinted, I think its bigger impact will be from the specifications, especially PEP 517, which it helped to prompt, rather than people using Flit directly. The specifications mean it's practical to make new tools which interoperate nicely, without having to either wrap setuptools or carefully imitate its behaviour.
- neolog 6y agoJust use Poetry [1]. It's popular and works well. [1] https://python-poetry.org/ https://python-poetry.org/
- icy 6y agoYup, was surprised to see no mention of Poetry. Hands down, the best package manager for Python.
- santiagobasulto 6y ago+1 for poetry. It also includes deterministic dependency resolution with a lock file. I just published a repo today[0] using Poetry and it didn’t take me more than 5 minutes. Poetry build && poetry publish [0] https://github.com/santiagobasulto/hyper-inspector https://github.com/santiagobasulto/hyper-inspector
- rahimnathwani 6y agoThis looks cool. I had never come across rich[0] before. [0] https://github.com/willmcgugan/rich https://github.com/willmcgugan/rich
- void_mint 6y agoHoly crap, for some reason I never thought to consider another package manager than Pip, which I loathe.
- sergiomattei 6y agoHuge thumbs up to Poetry. It's drastically simplified package management for me and replaced Pipenv (which I simply dreaded working with due to performance and DX issues). I no longer start Python projects without Poetry. It's really good. EDIT: Also, it's being integrated to most PaaS as well. I deploy to Render.com with Poetry now.
- tcbasche 6y agoHow does it compare to pipenv?
- rtpg 6y agoThe recommendation to set up a Makefile on top of tox is a bit odd to be honest. Tox basically "just works", and you can do things like pass stuff to `pytest` by setting up `{posargs}` in the tox config (see [0]) I do feel like tox gets a bad rap despite having a complete feature set. I think a part of it is that the documentation is complete but not organized in the "Tox user"'s perspective, so for someone who shows up on a project using it, it's hard to figure out the quickstart( though the "general tips and tricks" page gets somewhere [1]) Anyways yeah, would not recommend Make over just leaning into tox more here. EDIT: also, this article reminded me of how much I really dislike Github Action's configuration syntax. Just balls of mud on top of the Docker "ball of mud" strategy. I will re-iterate my belief that a CI system where the configuration system isn't declarative but just like..... procedural Lua will be a billion dollar business. CI is about running commands one after another! If you want declarative DAGs use Bazel [0]: https://tox.readthedocs.io/en/latest/example/pytest.html?highlight=posargs https://tox.readthedocs.io/en/latest/example/pytest.html?hig... [1]: https://tox.readthedocs.io/en/latest/example/general.html https://tox.readthedocs.io/en/latest/example/general.html
- kristjansson 6y agoI’d argue the counterpoint actually: Writing Makefile targets for common commands significantly improves usability and ergonomics, especially when they follow common idioms (make test, make build, make install, ...). The recipes for each target describe not only how a project intends to run each tool, but which tools it intends to run. Instead of having to know that this project runs tests under tox, while that one runs only under pytest, we can run ‘make test’ in each, and count on the recipe doing the Right Thing. That consistency across projects makes it much easier for someone to get started on a new project (or to remember how the pieces fit together on your own project from a few months ago)
- rtpg 6y agoFor me it’s important for a testing command to be able to receive parameters at runtime (for example tox test —- —pdb) , is it possible to do that with make in general? I never knew how. I generally agree with your sentiment, though. I’m usually limiting myself to Python stuff so don’t have much exposure to make, it’s always felt like a less powerful task runner than other stuff
- zomglings 6y agoThe number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package. Want to publish to PyPI? Use twine. It's standard and it's simple. You don't need complicated tooling for simple projects.
- someoldguy 6y agoI would probably recommend just following https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/ instead.
- deleted 6y ago[deleted]
- zomglings 6y agoAh I'm stuck in the dark ages.
- neolog 6y agoHow do you handle version pinning? hash checking? CI? testing on multiple platforms? multiple python versions? deployment? credential management? package data? version bumps? Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.
- zomglings 6y agoIteratively. You don't need to solve all those problems at once. Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though. You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnable scripts. I typically bump version directly in a version file and move on with my life. This stuff usually builds up iteratively and at least for me has never been the starting point. Starting point should be a library worth sharing. It is not the end of the world of you release the first few versions manually.
- brandmeyer 6y agoLets see how many high-quality Python packaging guides we can cram into one discussion, shall we? I have no problems using setuptools directly, as outlined in https://packaging.python.org/guides/distributing-packages-using-setuptools/ https://packaging.python.org/guides/distributing-packages-us...
- perrygeo 6y agoWhy not to make a Python package in 2021. Even as a long time Python user, the packaging ecosystem feels fragmented and error-prone at best. Honestly, it sours the experience of writing Python code knowing you might eventually need to make it work on another computer.
- neolog 6y agoIt's fragmented, but it doesn't need to be error prone if people use the good tools instead of the old low-level tools.
- jimmaswell 6y agoThere would be no room for error if we just put the libraries in with the project as files instead of adding all these extra steps. Nobody seems to like this simple, bulletproof method anymore for some reason though.
- 1337shadow 6y agoThat's exactly what a package manager does
- jimmaswell 6y agoA package manager is a whole separate program with config files that adds an extra build step and works like a black box. I mean just having the libraries entirely present in the repository. If an update is needed then someone pastes it in and commits it. This also lets you organize how you want.
- neolog 6y agoThat's often called "vendoring" these days.
- saila 6y agoI think for the vast majority of at least pure-Python projects you could just use poetry and upload your packages to PyPI or a private index. You can go from an empty directory to publishing a package within minutes with poetry (although, of course, you probably shouldn't).
- modi123 6y agoTest
- tcbasche 6y agoI’m always glad to see Make being used. It’s such a powerful and simple tool that usually does the job just as well as more “bespoke” CLI’s for various frameworks and languages
- francislavoie 6y agoI rather just write a bash script. It's the lowest common denominator. Make may not be installed by default in many places and it has some weird syntax quirks that make it annoying to use IMO. Here's an example of how I like to do my "bash scripts that sorta work like make": https://github.com/francislavoie/laravel-websockets-example/blob/main/docker/utils https://github.com/francislavoie/laravel-websockets-example/... basically each function is a "command", so I do like "./utils start" or whatever.
- nikisweeting 6y agoI like this style as well, we keep all our scripts in a ./bin directory, e.g. ./bin/lint.sh, ./bin/test.sh, etc. just as discoverable as make commands (run ls ./bin) and much easier to maintain. If you really want make, you can also just call out to your bash scripts from make: test: ./bin/test.sh lint: ./bin/lint.sh ...
- fctorial 6y agoPlease don't write a bash script.
- francislavoie 6y agoPlease elaborate. This is a useless comment without more information.
- nikisweeting 6y agoI would strongly discourage using make on new projects, make syntax is full of footguns and quirks (not being able to pass multiple args to subcommands is an easy example). Bash, Python, or even Typescript are much easier, safer, and more widely standardized environments to maintain and grow your scripts once you get past a few lines.
- icy 6y agoHonestly, I don't even bother "packaging" Python tools anymore. Just put it in all in a git repo, and pip can install using pip install git+https://myg.it/repo.git
- bloaf 6y agoI do that where I work because we have an internal git that is wide open, but packages from the outside world need to get whitelisted. More controversially: With a few (<10) lines of code in your setup file, you can make an install-able module out of jupyter notebooks.
- anewhnaccount2 6y agoBut now packages on PyPI can't depend on it.
- nxpnsv 6y agoBut very often, that's ok...
- stinos 6y agoAlso automatcially handles the issue of public vs private etc. Note this [1] recommends using zip files for speed, especially for larger repositories: pip install https://github.com/django/django/archive/master.zip pip install https://github.com/django/django/archive/stable/1.7.x.zip [1] https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch https://stackoverflow.com/questions/20101834/pip-install-fro...
- amelius 6y agoHow does this work for dependencies? And what if some of the dependencies are incompatible with the versions already on your system?
- testtesting123 6y agoSlightly off topic but does any recommend any great guides for building browser javascript / node.js packages i.e listing recommended linters, profilers, testing strategy, documentation template, or a project structure?
- simonw 6y agoI have a cookiecutter template for this at https://github.com/simonw/python-lib https://github.com/simonw/python-lib (also click-app and datasette-plugin) It sets up GitHub actions for publishing the package to PyPI when you create a release on GitHub - which I find way to be a really productive way of working.
- nxpnsv 6y agocookiecutter is great. I recently moved from click to typer, which I so far really have enjoyed. I probably should make a cc template for that one day...
- anticodon 6y agoI prefer to put `.PHONY` before every phony target, not gather them all in one place. When phony targets are all written out in the beginning of the Makefile, it's easy to forget to alter this list when a phony target is added or removed later. This rarely causes an error, but still I've seen many Makefiles with outdated .PHONY lists.
- shadycuz 6y agoAfter reading this guide I would still recommend people to use this guide. https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu... It's great for beginners and experts. As a long time python veteran it completely changed how I work with python for the better. It is lengthy with lots of optional steps. Just skip the ones you don't find relevant.
- machineko 6y agoI see a lot of poetry and pyenv but i feel like anaconda/miniconda is a king :)
- 1337shadow 6y agoStill hardcoding version numbers in 2021? Why not use setupmeta
- amarshall 6y agoProbably because setupmeta seems to only supports setup.py, which this guide doesn’t use (and isn’t generally recommended for most use cases).
- 1337shadow 6y agoI maintain over 50 public packages with setup.py and most of them have setupmeta, it's great and I would recommend it for all use cases.
- amarshall 6y agoSure, and that’s fine. My statement was based off https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/ > dynamic metadata [(setup.py)] should be used only as an escape hatch when absolutely necessary.
- 1337shadow 6y agoDynamic version numbers based on git tags are absolutely necessary to me, to ease my continuous integration practice, that's also what openstack/pbr does amongst probably other things
- ThePhysicist 6y agoThe author mixes different things like linting and testing into the packaging process, which (IMHO) are not really part of making a package. The process is really much easier than this article makes it seem: - Write a simple setup.py file. - Generate a source or binary release by e.g. running "python setup.py sdist" - You're done! Adding a setup.py file is already enough to make your library pip-installable, so you could argue that this is a package already. The files generated by "setup.py" can also be pip-imported, so they are also packages. Now you might want to upload your package to a repository, for which there are different tools available. The simplest one being twine. Again, you just install it and run "twine upload -r dist/*" and your packages get uploaded to PyPi (it will ask for a username and password). So why complicate things?
- phodge 6y agoI don't see how your version is easier than the sequence of commands in the first few steps of the article, which is basically `pip install flit; flit init; flit publish`. Flit is just as easy to install as twine, but you save yourself the hassle of having to write a setup.py.
- ThePhysicist 6y agoMaybe I'm too old-fashioned then. But I like that you don't have any dependencies when using distutils/setuputils with a `setup.py` file, so if you don't distribute your code you're already done. I'm also not a fan of tools that are just wrappers around other tools.
- takluyver 6y agoFlit isn't (mostly) a wrapper around other tools - it has its own code to create and upload packages. This was one of the motivating cases for the PEPs (517, 518) defining a standard interface for build tools, so it's practical to make tools like this without wrapping setuptools. (flit install does wrap pip, however)
- cixinlui 6y ago
- Jukg7e3AFyskdj 6y agoJust use https://pypi.org/project/zest.releaser/ https://pypi.org/project/zest.releaser/
- 74d-fe6-2c6 6y agoCan somebody confirm that this is reasonable and state of the art? Or is it just a "look what I can do" type blog post?
- BerislavLopac 6y agoI would say it is quite up to date. First, it is using pyproject.toml which is now the standard way to define build requirements for Python packages. Second, its collection of additional (linting etc) tools is pretty solid; there are potential alternatives in few cases (e.g. I would personally use Poetry rather than flit, and wouldn't use make for development scripts) but that is pretty much it.
- lucb1e 6y agoI've never even heard of pyproject.toml and I open github projects behind packages to vet them (to a minimal degree, but still, making sure it's not a typosquat and still maintained and often also reading a bit of source code) on at least a weekly basis. It's not always python of course, but often enough. Maybe it's just some freak coincidence that I somehow never saw it or just don't recall while really it's mostly everywhere, but this broad statement for something I've never even heard of makes me think of JavaScript and the 'standard framework' that changes every six months.
- 74d-fe6-2c6 6y agoIt's actually not too uncommon - mostly due to the rising popularity of poetry.
- BerislavLopac 6y agoWith PEP-518 being 5 years old it's still relatively new, and most tools have implemented the support for it relatively recently. The key introduction article was written exactly one year ago: https://snarky.ca/what-the-heck-is-pyproject-toml/ https://snarky.ca/what-the-heck-is-pyproject-toml/
- edenhyacinth 6y ago
- say_it_as_it_is 6y agoWhat happened to setup.py and wheels?
- alknemeyer 6y agoSomewhat related: I have an example repository which I've been using to keep track of the tools I use, aimed at people in a research lab who are relatively new to Python. I made it because the existing example/template repositories I found don't gel nicely with the way I like to set up and think about things. Here it is -- hope you find it useful: https://github.com/alknemeyer/python-template/ https://github.com/alknemeyer/python-template/
- darkwater 6y agoI still don't understand why in 2021 `pip`, which is the standard package and dependencies manager in Python, cannot 1) build a package from some spec 2) generate the scaffolding needed to build such package `gem` from Ruby does that (well, it doesn't 2 AFAIK but at least it does 1)
- diimdeep 6y agoCan we have "How to make a X package" ? I would love to see also examples with C++(if it is possible at all) and Nim
- midrus 6y ago2021 and still there is no clear winner or official way to install dependencies and create packages. The worst part of Python.
- wdroz 6y agoWhen you are doing machine learning, conda is widely used. Why? Because you can install non-python things like cudatoolkit or ffmpeg (you can even install python, so you are sure that everybody are using the same version of python) Python is fantastic at gluing specialized tools/libraries, but a lot of these require non-python dependencies (most are written in more performant languages). IMO, this is a big differences when comparing with Cargo for Rust because most of the dependencies in Rust are written in Rust. The state of packaging in Python is kinda meh, the official documentation here [0] suggests to create 3 additional files in order to create a package: - pyproject.toml - setup.cfg - setup.py # optional, needed to make editable pip installs work if you add conda, you may need 2 additional files: - meta.yaml # to build you conda package - environment.yml With this much boilerplate, I understand why people are creating tools like flit. [0] -- https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/
- thinker5555 6y agoI work in a very small company where we are building the plane as we learn to fly, and I'm the only person there with any programming experience. I've been working on trying to improve my hobbyist-level (at best) knowledge of the Python ecosystem over the last year or so. I've gotten to the point where a number of smaller tools I've put together can now be used in larger projects. I learned the hard way that just copying files around makes it hard to know which version is in that project, and upgrading, especially once there is more than one file, becomes a lot tougher. I learned the harder way that trying to link the same file into multiple projects is a great way to really screw things up. About 4 or 5 months ago I made a real effort to try to learn how to use virtual environments (pipenv) and packaging to make it so that if I update one of the smaller tools, I don't clobber all the downstream projects that rely on it. I wanted to make it so that when I update something to add features or change things, I can go back and fix older projects it's used in at my leisure. I haven't even begun to touch on unit testing, and I have no clue what linting is. Things are kind of working so far, but it feels very hacky and fragile, and I know it can (and SHOULD) be better. All of this stuff around packaging and being able to install those packages very daunting, and trying to stumble on the right tutorials is extremely frustrating. The vast majority of them assume I want to share my stuff with the world on PyPI, or that I have servers available to me to create private PyPI indexes, but I don't. Yet I still want my packages to "resolve their own dependencies" when I install or upgrade them. And when it comes to learning things like testing, the few tutorials I've looked at either use different tools to do it, or their examples are so oversimplified that when I look at my own code, I don't know where to begin. I say all of this because looking at this tutorial, it's more of the same. I want to make my code better. I want to make it easier to use those smaller projects in larger projects. But then it says things like "Every solid open-source project runs cloud tests after each commit, so we will too," but it doesn't do anything to explain what that is or why it should be done, besides "everyone does it, so you should, too." I think what makes it even harder is that when something like this gets shared, there are so many conflicting opinions. Some people say to just use setuptools, other say that setuptools is on its way out and to use pyproject.toml (or some other tool) instead. It's all just so... hard! I'm sorry. This is coming off a bit ranty, and that's not what I intended. I'm just feeling frustrated and I'm not sure of a better way to express that I need help with finding help. There are even a lot of things that I'm sure I need help with, but I just don't know what they are. It makes it really hard to verbalize what I need to another person, let alone to get the right words into a search engine to take me there.
- jftuga 6y agoWhich is the more standard way to use venv for Python 3.8 or newer? 1) venv at the same level: create new project on GH git clone https://github.com/user/myproject https://github.com/user/myproject python -m venv myproject cd myproject (activate venv) ____ 2) clone first, venv in subdirectory: create new project on GH git clone https://github.com/user/myproject https://github.com/user/myproject cd myproject python -m venv venv (activate venv) ___ 3) something completely different? if so, what?