11 ms·

The number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify

The number of extra tools used in this article boggles my mind. Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package. Want to publish to PyPI? Use twine. It's standard and it's simple. You don't need complicated tooling for simple projects.

someoldguy 6y ago
I would probably recommend just following https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/ instead.
- deleted 6y ago
  [deleted]
- zomglings 6y ago
  Ah I'm stuck in the dark ages.
neolog 6y ago
How do you handle version pinning? hash checking? CI? testing on multiple platforms? multiple python versions? deployment? credential management? package data? version bumps? Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.
- zomglings 6y ago
  Iteratively. You don't need to solve all those problems at once. Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though. You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnable scripts. I typically bump version directly in a version file and move on with my life. This stuff usually builds up iteratively and at least for me has never been the starting point. Starting point should be a library worth sharing. It is not the end of the world of you release the first few versions manually.
  - neolog 6y ago
    > You don't need to solve all those problems at once. Why spend time tweaking the setup when you can just get it right in the first place with less work?
    - zomglings 6y ago
      Do you recommend poetry? I've been meaning to try it but haven't had the time to migrate an existing project to poetry.
      - neolog 6y ago
        Yeah it's pretty good. There are plenty of problems and holes of course, but for most pure-Python packages it does a nice job most of the time. If you're an expert, you can do it with setup.py or setup.cfg, but I don't think it's normally worth the trouble.
      - atoav 6y ago
        I used poetry for multiple projects already and I quite like it because it makes version management really straightforward and you can just go like poetry new foo And it will setup all the basic stuff for you right away. Having all the project dependecies and Metadata in pyproject.toml makes sense and reduces cognitive overload. Having poetry manage your venvs autmagically is a good extra. There is still room for improvement, e.g. I had some trouble with their install script at times (python vs python3)
  - ehsankia 6y ago
    To be fair, while this is a single article, if you only look at step 1, 2 and 3, you get a fully published package with only one tool used (flit) and not much extra. It's the succeeding sections (A, B, C, D, E) that get more advanced, but they're all optional. You should definitely do A, but the rest I'd say it's a lot more opinionated and definitely not needed.
    - zomglings 6y ago
- 1337shadow 6y ago
  I usually don't pin, I'd rather deal with upstream BC breaks as they are published instead of accumulating tech debt. I call this "continuously integrating upstream code", because Continuous Integration is a practice, not a tool.
- globular-toast 6y ago
  You don't pin anything for a package. I'm not aware of any "standard" CI that a package tool could set up. I guess you mean testing on multiple versions, in which case tox will help. Deployment for a package is handled by twine. Package data? What about it? Version bumps should always be manual but I recommend setuptools-scm. You seem to be confusing packages with "apps". It's very important to understand the clear distinction between these.
  - neolog 6y ago
    > You seem to be confusing packages with "apps". I'm not confusing libraries with applications. Pinning dependency versions enables a repeatable test environment.
- asdf123wtf 6y ago
  I generally reach for pip-tools, when I need to pin versions in a requirements for a deployable app, like an api. It's by far the simplest option I've found. If your project is a library, just use setup.py and express your deps abstractly (min, max or version range). Don't pin to specific versions at all, if you can help it.
da39a3ee 6y ago
You seem to have missed the point. The project in the article was simple because that is best for exposition. The article would hardly have been more helpful with the code for a realistic python package dumped into it, would it?
darthrupert 6y ago
I actually think one should do the exact opposite to what you're suggesting. I've experienced modern package management through Cargo and anything below that level now seems like returning to stone age. In the python ecosystem, poorly defined packages is a wide problem. You never know what you get in the next version upgrade. So my suggestion: burn all the "simple" python packaging tools in a big fire and move eveything to Poetry today. It will be painful at start but will increase the quality of the ecosystem by a huge margin if just 50% of the projects do this.
- hardwaresofton 6y ago
  Bit of a side note but remember cargo learned a lot from generations of package managers, across languages. It essentially represents some of the best that package managers have to offer. You only get that kind of result by starting from scratch every few years (as in, rust started from scratch when the Node/NPM ecosystem had been around for years to steal ideas from, Haskell had been around for years to steal language design from, etc.). Rust is incredibly lucky to have been created when it was because it benefits immensely from these things (I think it's the best new-to-middle-aged production-usable systems language out there today). I agree with the idea but languages like Python and their ecosystems are really hard to move (remember python 2->3? is that even over?) -- it's a herculean and often impossible task.
  - afarrell 6y ago
    Migrating python2 -> python3 is hard because it requires rewriting code and because you can't really run python3 code if you depend upon python2. If poetry create consume non-poetry packages and create packages which other package management systems can consume? If so, then: 1. Projects can move more effectively independently. 2. Projects can re-package as a single task rather than a massive rewrite effort.
  - lmm 6y ago
    Cargo is not hugely different from Maven which has been working fine for over a decade. Yes, it takes some polished ideas from other systems, but Python has had more packaging tools come and go than several other ecosystems put together.
geraneum 6y ago
Maybe the reason that the library itself is simple in the article is that that’s just an example and the author wants to show how to do it properly, end to end.
BerislavLopac 6y ago
Using setup.py does not mean "not using extra tools". It depends on setuptools, which is an "extra tools" just like flit (used in the article) or any other tool. In fact, with using only setuptools one will need a whole additional set of tools to manage things like: * virtual environments (granted, venv is now part of the stdlib, but it's still an "extra tool") * publishing to PyPI or another index (twine) * dependency management (both for development and actual dependencies) Plus the tools that are needed anyway, to manage common development actions like: * unit tests (pytests) * static typing (mypy) * linting (flake8, pylint) * styling (black) The article is correct in using pyproject.toml, which has become the standard way to specify build mechanism for your package [0]. Even setuptools supports it in the latest versions [1], meaning that setup.py is becoming obsolete, or at least unnecessary. Finally, tools like Poetry [2] offer a whole set of functionalities in one place (dependency management, virtual environments, publishing), which means that they need fewer "extra tools" than just setuptools. [0] https://www.python.org/dev/peps/pep-0518/ https://www.python.org/dev/peps/pep-0518/ [1] https://setuptools.readthedocs.io/en/latest/build_meta.html https://setuptools.readthedocs.io/en/latest/build_meta.html [2] https://python-poetry.org/ https://python-poetry.org/
- say_it_as_it_is 6y ago
  Where is there a comprehensive poetry packaging tutorial?
  - BerislavLopac 6y ago
    Right here: poetry build -f wheel poetry publish You asked for packaging, and that is pretty much it. Of course, setting up a project and its dependencies take a bit more work; the basic intro for that is here: https://python-poetry.org/docs/basic-usage/ https://python-poetry.org/docs/basic-usage/
    - say_it_as_it_is 6y ago
      That is fantastic.
cixinlui 6y ago
This is bad advice. Do not create a setup.py for a new package. (Keeping setup.py for an old package can be okay.) The author is correct that you want a tool such as flit or poetry which will work with pyproject.toml. Setting up a basic package will be no harder than using setuptools, and it is much more future-proof. You won't have to copy-paste some other crufty setup config either. It is fair that you don't need all the external tools in this tutorial. In particular, using make is very silly since you can configure different linting and testing workflows directly in pyproject.toml, rather than pull in a whole other system which only works decently on *nix. Poetry also removes the need for tox.
- 1337shadow 6y ago
  And why exactly do you think setup.py is deprecated?
  - cixinlui 6y ago
    Because of PEP 518.
- remram 6y ago
  Future-proof... Poetry 1.1 broke compatibility with 1.0. 1.1 lockfiles would crash Poetry 1.0, and 1.0 lockfiles would be thrown away by Poetry 1.1. It does not correctly verify hashes (if at all) [1]. You can't add packages without updating all your dependencies. Monorepos are not supported. PEP-508-compliant Git dependencies cause the tool to crash with TypeError [2]. I think Poetry is the right direction, I use it for everything, but it's not the silver bullet you're painting it to be (yet). It's definitely not on par with Cargo, or maybe even npm. [1]: https://github.com/python-poetry/poetry/issues/3765 https://github.com/python-poetry/poetry/issues/3765 [2]: https://github.com/python-poetry/poetry/issues/3425 https://github.com/python-poetry/poetry/issues/3425
  - cixinlui 6y ago
    I didn't say it was a silver bullet... I said "Don't Use setup.py".
fctorial 6y ago
For all the hate nodejs gets, it solved the software packaging problem. That is the sole reason npm ecosystem is so big. I have to say that it probably isn't a fare comparison because python is much older than nodejs. Python package management might very well have been state of the art in 1995.
- chriswarbo 6y ago
  My impression was that npm does little more than fetch code from the Web and stick it in a 'node_modules' directory. I've even seen npm used for things that aren't even JS, just bunch of files. This approach ends up with multiple, potentially-incompatible versions of the same package in a project. True that's less of a problem in JS, since it's interpreted (deferring imports to runtime) and un(i)typed (no need to check if interfaces match up). Yet even that has lead to replacements/complements like yarn.
  - fctorial 6y ago
    > This approach ends up with multiple, potentially-incompatible versions of the same package in a project. If data generated by one version of a library is being consumed by another version, that's a bug in the code that moves data between them.
  - oefrha 6y ago
    > My impression was that npm does little more than fetch code from the Web and stick it in a 'node_modules' directory. Yes. There's hardly even a standard directory structure, let alone a standard way to convert source code to published code. Every slightly non-trivial repo basically has an ad hoc build system of its own. Ever tried to fix a bug in a package, and realized that using git://github.com/user/repo#branch doesn't work, because npm only downloads the source code, which bears no resemblance to the built products? I fixed two bugs in two third party packages within the past week, had to deal with this twice. Ran into the Node 12+ and Gulp 3.x incompatibility issue twice in the past month (with totally modern, actively developed packages), too. npm has more sophisticated dependency resolution and locking than pip, sure. Python packaging is more consistent in basically every other regard.
kkirsche 6y ago
Copy from https://github.com/pypa/sampleproject https://github.com/pypa/sampleproject !
vonwoodson 6y ago
Comments on this reply seem to have forgotten two key things about python. 1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies. 2) “There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.py, pypi, and so on. Python is not Rust, or Node, or Clojure, or anything else. Do it Pythonicly and good things will happen. In the history of Python, it’s only because of the Zen’s guiding principles and the “come join us” attitude of the community that Python accidentally became “enterprise grade”. It succeeded in spite of the lack of Cargo or NPM, or (shudder) Mavin[, or CI/CD, or 3D game engines, or easy binary package productization, or everything else Python is obviously bad at]; almost precisely because it’s not those things. Python encourages me to get my task done, and all it asks is that I leave the flame wars at the door and embrace it for what it is. It saddens me to see so many people jump down zomglings throat for making the daring suggestion that Pythonistas use the language as designed. Because, he is absolutely right about this: “Complex is better than complicated.”
- patagonia 6y ago
  Unpopular opinion: Python is bait and switch. Naive, aspiring programmers are told to learn Python because it’s the perfect beginner language, only to quickly hit a brick wall of complication they are unable to foresee (because they’re completely new to all this) and ill-equipped navigate. Like it or not Python is now “enterprise grade”. The sooner it either grows up and figures out packages, ci/cd, binaries etc, or it looses its reputation at a great beginner and general programming language to something like Julia or Crystal, the better.
- cixinlui 6y ago
  The thing is that using setuptools is neither standard nor Pythonic. It isn't part of the standard library. It's a way of doing things that is broken, and has been specifically called out by the Python developer community as something that people should stop doing.

someoldguy 6y ago

I would probably recommend just following https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/ instead.

deleted 6y ago
[deleted]
zomglings 6y ago
Ah I'm stuck in the dark ages.

neolog 6y ago

How do you handle version pinning? hash checking? CI? testing on multiple platforms? multiple python versions? deployment? credential management? package data? version bumps? Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.

zomglings 6y ago
Iteratively. You don't need to solve all those problems at once. Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though. You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnable scripts. I typically bump version directly in a version file and move on with my life. This stuff usually builds up iteratively and at least for me has never been the starting point. Starting point should be a library worth sharing. It is not the end of the world of you release the first few versions manually.
- neolog 6y ago
  > You don't need to solve all those problems at once. Why spend time tweaking the setup when you can just get it right in the first place with less work?
  - zomglings 6y ago
    Do you recommend poetry? I've been meaning to try it but haven't had the time to migrate an existing project to poetry.
    - neolog 6y ago
      Yeah it's pretty good. There are plenty of problems and holes of course, but for most pure-Python packages it does a nice job most of the time. If you're an expert, you can do it with setup.py or setup.cfg, but I don't think it's normally worth the trouble.
    - atoav 6y ago
      I used poetry for multiple projects already and I quite like it because it makes version management really straightforward and you can just go like poetry new foo And it will setup all the basic stuff for you right away. Having all the project dependecies and Metadata in pyproject.toml makes sense and reduces cognitive overload. Having poetry manage your venvs autmagically is a good extra. There is still room for improvement, e.g. I had some trouble with their install script at times (python vs python3)
- ehsankia 6y ago
  To be fair, while this is a single article, if you only look at step 1, 2 and 3, you get a fully published package with only one tool used (flit) and not much extra. It's the succeeding sections (A, B, C, D, E) that get more advanced, but they're all optional. You should definitely do A, but the rest I'd say it's a lot more opinionated and definitely not needed.
  - zomglings 6y ago
1337shadow 6y ago
I usually don't pin, I'd rather deal with upstream BC breaks as they are published instead of accumulating tech debt. I call this "continuously integrating upstream code", because Continuous Integration is a practice, not a tool.
globular-toast 6y ago
You don't pin anything for a package. I'm not aware of any "standard" CI that a package tool could set up. I guess you mean testing on multiple versions, in which case tox will help. Deployment for a package is handled by twine. Package data? What about it? Version bumps should always be manual but I recommend setuptools-scm. You seem to be confusing packages with "apps". It's very important to understand the clear distinction between these.
- neolog 6y ago
  > You seem to be confusing packages with "apps". I'm not confusing libraries with applications. Pinning dependency versions enables a repeatable test environment.
asdf123wtf 6y ago
I generally reach for pip-tools, when I need to pin versions in a requirements for a deployable app, like an api. It's by far the simplest option I've found. If your project is a library, just use setup.py and express your deps abstractly (min, max or version range). Don't pin to specific versions at all, if you can help it.

da39a3ee 6y ago

You seem to have missed the point. The project in the article was simple because that is best for exposition. The article would hardly have been more helpful with the code for a realistic python package dumped into it, would it?

darthrupert 6y ago

I actually think one should do the exact opposite to what you're suggesting. I've experienced modern package management through Cargo and anything below that level now seems like returning to stone age. In the python ecosystem, poorly defined packages is a wide problem. You never know what you get in the next version upgrade. So my suggestion: burn all the "simple" python packaging tools in a big fire and move eveything to Poetry today. It will be painful at start but will increase the quality of the ecosystem by a huge margin if just 50% of the projects do this.

hardwaresofton 6y ago
Bit of a side note but remember cargo learned a lot from generations of package managers, across languages. It essentially represents some of the best that package managers have to offer. You only get that kind of result by starting from scratch every few years (as in, rust started from scratch when the Node/NPM ecosystem had been around for years to steal ideas from, Haskell had been around for years to steal language design from, etc.). Rust is incredibly lucky to have been created when it was because it benefits immensely from these things (I think it's the best new-to-middle-aged production-usable systems language out there today). I agree with the idea but languages like Python and their ecosystems are really hard to move (remember python 2->3? is that even over?) -- it's a herculean and often impossible task.
- afarrell 6y ago
  Migrating python2 -> python3 is hard because it requires rewriting code and because you can't really run python3 code if you depend upon python2. If poetry create consume non-poetry packages and create packages which other package management systems can consume? If so, then: 1. Projects can move more effectively independently. 2. Projects can re-package as a single task rather than a massive rewrite effort.
- lmm 6y ago
  Cargo is not hugely different from Maven which has been working fine for over a decade. Yes, it takes some polished ideas from other systems, but Python has had more packaging tools come and go than several other ecosystems put together.

geraneum 6y ago

Maybe the reason that the library itself is simple in the article is that that’s just an example and the author wants to show how to do it properly, end to end.

BerislavLopac 6y ago

Using setup.py does not mean "not using extra tools". It depends on setuptools, which is an "extra tools" just like flit (used in the article) or any other tool. In fact, with using only setuptools one will need a whole additional set of tools to manage things like: * virtual environments (granted, venv is now part of the stdlib, but it's still an "extra tool") * publishing to PyPI or another index (twine) * dependency management (both for development and actual dependencies) Plus the tools that are needed anyway, to manage common development actions like: * unit tests (pytests) * static typing (mypy) * linting (flake8, pylint) * styling (black) The article is correct in using pyproject.toml, which has become the standard way to specify build mechanism for your package [0]. Even setuptools supports it in the latest versions [1], meaning that setup.py is becoming obsolete, or at least unnecessary. Finally, tools like Poetry [2] offer a whole set of functionalities in one place (dependency management, virtual environments, publishing), which means that they need fewer "extra tools" than just setuptools. [0] https://www.python.org/dev/peps/pep-0518/ https://www.python.org/dev/peps/pep-0518/ [1] https://setuptools.readthedocs.io/en/latest/build_meta.html https://setuptools.readthedocs.io/en/latest/build_meta.html [2] https://python-poetry.org/ https://python-poetry.org/

say_it_as_it_is 6y ago
Where is there a comprehensive poetry packaging tutorial?
- BerislavLopac 6y ago
  Right here: poetry build -f wheel poetry publish You asked for packaging, and that is pretty much it. Of course, setting up a project and its dependencies take a bit more work; the basic intro for that is here: https://python-poetry.org/docs/basic-usage/ https://python-poetry.org/docs/basic-usage/
  - say_it_as_it_is 6y ago
    That is fantastic.

cixinlui 6y ago

This is bad advice. Do not create a setup.py for a new package. (Keeping setup.py for an old package can be okay.) The author is correct that you want a tool such as flit or poetry which will work with pyproject.toml. Setting up a basic package will be no harder than using setuptools, and it is much more future-proof. You won't have to copy-paste some other crufty setup config either. It is fair that you don't need all the external tools in this tutorial. In particular, using make is very silly since you can configure different linting and testing workflows directly in pyproject.toml, rather than pull in a whole other system which only works decently on *nix. Poetry also removes the need for tox.

1337shadow 6y ago
And why exactly do you think setup.py is deprecated?
- cixinlui 6y ago
  Because of PEP 518.
remram 6y ago
Future-proof... Poetry 1.1 broke compatibility with 1.0. 1.1 lockfiles would crash Poetry 1.0, and 1.0 lockfiles would be thrown away by Poetry 1.1. It does not correctly verify hashes (if at all) [1]. You can't add packages without updating all your dependencies. Monorepos are not supported. PEP-508-compliant Git dependencies cause the tool to crash with TypeError [2]. I think Poetry is the right direction, I use it for everything, but it's not the silver bullet you're painting it to be (yet). It's definitely not on par with Cargo, or maybe even npm. [1]: https://github.com/python-poetry/poetry/issues/3765 https://github.com/python-poetry/poetry/issues/3765 [2]: https://github.com/python-poetry/poetry/issues/3425 https://github.com/python-poetry/poetry/issues/3425
- cixinlui 6y ago
  I didn't say it was a silver bullet... I said "Don't Use setup.py".

fctorial 6y ago

For all the hate nodejs gets, it solved the software packaging problem. That is the sole reason npm ecosystem is so big. I have to say that it probably isn't a fare comparison because python is much older than nodejs. Python package management might very well have been state of the art in 1995.

chriswarbo 6y ago
My impression was that npm does little more than fetch code from the Web and stick it in a 'node_modules' directory. I've even seen npm used for things that aren't even JS, just bunch of files. This approach ends up with multiple, potentially-incompatible versions of the same package in a project. True that's less of a problem in JS, since it's interpreted (deferring imports to runtime) and un(i)typed (no need to check if interfaces match up). Yet even that has lead to replacements/complements like yarn.
- fctorial 6y ago
  > This approach ends up with multiple, potentially-incompatible versions of the same package in a project. If data generated by one version of a library is being consumed by another version, that's a bug in the code that moves data between them.
- oefrha 6y ago
  > My impression was that npm does little more than fetch code from the Web and stick it in a 'node_modules' directory. Yes. There's hardly even a standard directory structure, let alone a standard way to convert source code to published code. Every slightly non-trivial repo basically has an ad hoc build system of its own. Ever tried to fix a bug in a package, and realized that using git://github.com/user/repo#branch doesn't work, because npm only downloads the source code, which bears no resemblance to the built products? I fixed two bugs in two third party packages within the past week, had to deal with this twice. Ran into the Node 12+ and Gulp 3.x incompatibility issue twice in the past month (with totally modern, actively developed packages), too. npm has more sophisticated dependency resolution and locking than pip, sure. Python packaging is more consistent in basically every other regard.

kkirsche 6y ago

Copy from https://github.com/pypa/sampleproject https://github.com/pypa/sampleproject !

vonwoodson 6y ago

Comments on this reply seem to have forgotten two key things about python. 1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies. 2) “There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.py, pypi, and so on. Python is not Rust, or Node, or Clojure, or anything else. Do it Pythonicly and good things will happen. In the history of Python, it’s only because of the Zen’s guiding principles and the “come join us” attitude of the community that Python accidentally became “enterprise grade”. It succeeded in spite of the lack of Cargo or NPM, or (shudder) Mavin[, or CI/CD, or 3D game engines, or easy binary package productization, or everything else Python is obviously bad at]; almost precisely because it’s not those things. Python encourages me to get my task done, and all it asks is that I leave the flame wars at the door and embrace it for what it is. It saddens me to see so many people jump down zomglings throat for making the daring suggestion that Pythonistas use the language as designed. Because, he is absolutely right about this: “Complex is better than complicated.”

patagonia 6y ago
Unpopular opinion: Python is bait and switch. Naive, aspiring programmers are told to learn Python because it’s the perfect beginner language, only to quickly hit a brick wall of complication they are unable to foresee (because they’re completely new to all this) and ill-equipped navigate. Like it or not Python is now “enterprise grade”. The sooner it either grows up and figures out packages, ci/cd, binaries etc, or it looses its reputation at a great beginner and general programming language to something like Julia or Crystal, the better.
cixinlui 6y ago
The thing is that using setuptools is neither standard nor Pythonic. It isn't part of the standard library. It's a way of doing things that is broken, and has been specifically called out by the Python developer community as something that people should stop doing.