19 ms·

What the Heck Is Pyproject.toml?

133 points by BerislavLopac 7y ago

hprotagonist 7y ago
The thing i have yet to reliably figure out is this (i live in the science part of python): without a setup.py, how do I: from Cython.Build import cythonize import numpy as np import versioneer setup(..., version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), ext_modules=cythonize(extensions), include_dirs=[np.get_include()], ...) I'm 100% on board with not having executable configuration files, except that we've been happily abusing them for years now and there are a lot of edge cases that i rely on now.
- pdonis 7y ago
  All of those setup options are now setup.cfg options: https://docs.python.org/3/distutils/configfile.html https://docs.python.org/3/distutils/configfile.html At least, that's supposed to be the case; I am not 100% certain that it always is. However, even so, AFAIK setup.cfg does not directly support using third-party libraries to compute the setup options, as your setup.py does. The only workaround I'm aware of for that if you want to get rid of setup.py is to have a separate script that generates setup.cfg using whatever libraries you need. However, I don't think that's the vision of the Python developers.
  - hprotagonist 7y ago
    I get the idea -- but, how do I tell Cython "it's time to compile things now" from a file I can't execute?
    - pdonis 7y ago
      > how do I tell Cython "it's time to compile things now" from a file I can't execute? It looks like there are ways to specify which build tools your package requires; I would assume Cython would be one of the tools that can be specified since it's been around a while and gets a lot of usage, but I don't know for sure.
  - kevin_thibedeau 7y ago
    There are lot that of options that setup.cfg doesn't support. "use_2to3" is one.
    - takeda 7y ago
      I never needed it, but I see 2to3 options there [1], are they not working? [1] https://setuptools.readthedocs.io/en/latest/setuptools.html#options https://setuptools.readthedocs.io/en/latest/setuptools.html#...
- maxnoe 7y ago
  pyproject.toml and setup.cfg can in some cases make a setup.py unnecessary. That does not mean you should not use it anymore if you have a good reason to use it. The ones you listed are examples. How ever, as explained in the article, you should still add the pyproject.toml
fermienrico 7y ago
Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with great dependency management into one tool. In the spirit of Python, there should be only one way to package, manage dependencies and virtual environments. It should be super clean (see Go). Python needs a BDFL to enforce the hive mind. It is a fucking mess and it's unacceptable. Python is otherwise the most incredible language of our times. It is so far ahead of anything out there for general purpose use. Relevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/ [1] https://github.com/pypa/pipenv/issues/4058 https://github.com/pypa/pipenv/issues/4058
- nvrspyx 7y ago
  I haven't looked much into it myself, but I've heard good things about Poetry [0]. 0: https://python-poetry.org/ https://python-poetry.org/
  - rattray 7y ago
    This looks great. Can anyone who's used both comment on how poetry compares to yarn[0]? [0] https://yarnpkg.com/ https://yarnpkg.com/
    - franey 7y ago
      I think of Poetry as the Yarn of Python packaging for two main reasons (apart from package locking, which is the biggest similarity): the commands are more intuitive and the lockfile starts with a different letter from the dependency file. I don't know how many times I open package-lock.json instead of package.json because I tab-complete from my shell. Same with Pipfile and Pipfile.lock for anyone using Pipenv. I wrote this post[0] that compares Poetry and Pipenv if you're interested and talks about some of their differences. Poetry has both add and install commands, whereas Pipenv only has an install command, for instance. [0] https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchmarks-ergonomics/ https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchma...
goldenshale 7y ago
Our experience using pyenv, poetry and a Pyproject.toml for projects lately has been great. I highly recommend going this route compared to dealing with pip and pipenv.
deleted 7y ago
[deleted]
Waterluvian 7y ago
Is toml even supported in the standard library? This seems just bizarre. Python's really overtaking node for massive churn. I feel like an old guy when peers say, "don't use virtual environment use pipenv." then someone says no no don't even use that, use poetry. Etc. I think about 50% of why I enjoy, in my soul, about Rust when I'm hacking away and personal goofs is that the dependency and project management system is all first class and pretty decent and I sure hope they don't go changing it much.
- symmitchry 7y ago
  Python also doesn't support yaml in the standard library, interestingly.
  - mixmastamyk 7y ago
    Yaml is a big mess. I do really like strict-yaml, however (last time I checked) it is just a hack to simplify the large PyYaml compiled package.
    - pydry 7y ago
      Thanks for the vote of confidence. fwiw version 2.0 of strictyaml will feature its own specification and parser.
      - mixmastamyk 7y ago
        Great to hear. I've been manually deactivating almost all of the features of PyYaml for almost a decade now. Over and over. So quite happy someone is taking up the cause.
- benatkin 7y ago
  Here's their justification for using it. It makes sense to me. https://www.python.org/dev/peps/pep-0518/#overview-of-file-formats-considered https://www.python.org/dev/peps/pep-0518/#overview-of-file-f...
talentedcoin 7y ago
I don’t want to be too negative, and I realize this isn’t an option for everyone ... but seriously, just use conda if you can. Judging from comments here and in the past, we all know packaging is one of Python’s Achilles’ heels. I spent a couple years bouncing around with pyenv/pipenv, then poetry, then back. I lovingly followed all the flamewars here and on github. Ultimately, I realize now, I laboured under the conceit that it was desirable to have a “native” solution that did not depend on a suite like conda. I now think I was totally wrong. I’m so much happier and more productive now that I’ve switched. For any data science use case conda is hands-down better, and for everything else it’s still more useable than any of the alternatives. I hate to say this but it’s true. The only thing I can think of that was harder to work with a conda setup was the keychain in macOS, and I’m not even sure now that that’s an issue anymore. conda really does just work.
- pottertheotter 7y ago
  I've had the opposite experience. I used conda for several years, primarily for data science, but regularly ran into problems with packages not working, updates for packages bring severely delayed, and issues with conda itself. Switched to pyenv+virtualenv 18 months ago and haven't had a single issue. I believe both of our stories. I just always wonder how this happens that two people have such opposite experiences.
  - talentedcoin 7y ago
    Ha!! Likewise, I believe you 100%. It’s fascinating isn’t it? Just goes to show what a diverse (and frankly wonderful) platform Python is, despite the warts. If you don’t mind my asking, what kind of stuff we’re you working on? Just curious! I have definitely found it’s easy to bork a conda install with updates, enough to make it easier to just wipe it out and start over. I mostly work on financial software and only conda seemed to play nicely with pandas and brokerage APIs. I should probably mention that I do a lot of this is cross-platform and conda seemed to be the only thing that worked without a lot of tweaking on Windows, Mac and Linux consistently. But the Linux experience with pyenv and virtualenv (and pipenv too, for the most part) was great. If I was Linux-only I honestly might never have switched.
poooood 7y ago
I quite like the pyproject.toml; it + poetry + dephell is (relatively speaking) a breeze. Both poetry and dephell are quite good and they don’t not mesh with conda. Python packaging is awful but it’s gotten way better; it’s still maddening but it’s better.
carlosperate 7y ago
Apart from packaging I also love the idea of consolidating configurations from different dependencies into a single place. But when I first moved to pyproject.toml I didn't find its use as widespread as I hoped (flake8 being the dependency I hear about the most without support), so I started collecting a list of projects currently using it, and discussing its use, in an awesome list format. Hopefully it will be useful for other people as well: https://github.com/carlosperate/awesome-pyproject https://github.com/carlosperate/awesome-pyproject
yamrzou 7y ago
I have found this blog post[1] quite informative regarding Python projects packaging. It describes well how all the tools (pyenv, poetry, black, flake8, isort, pre-commit, pytest, coverage, tox) fit together. Poetry seems the go-to tool for dependencies management and it centralizes all your package configurations alongside its dependencies in pyproject.toml. [1] https://medium.com/georgian-impact-blog/python-tooling-makes-a-project-tick-181d567eea44 https://medium.com/georgian-impact-blog/python-tooling-makes...
- RMPR 7y ago
  Still have some critical bugs here and there[1], many about resolving dependencies. Don't get me wrong, I use it for my own current side project github.com/rmpr/atbswp, but imho a lot of things need improvement. [1]: github.com/python-poetry/poetry/issues
BerislavLopac 7y ago
Announcing a new Sponsorship Program for Python Packaging: http://pyfound.blogspot.com/2020/04/sponsoring-python-packaging.html http://pyfound.blogspot.com/2020/04/sponsoring-python-packag... Discussion here: https://news.ycombinator.com/item?id=22772410 https://news.ycombinator.com/item?id=22772410

133 points by BerislavLopac 7y ago

The thing i have yet to reliably figure out is this (i live in the science part of python): without a setup.py, how do I: from Cython.Build import cythonize import numpy as np import versioneer setup(..., version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), ext_modules=cythonize(extensions), include_dirs=[np.get_include()], ...) I'm 100% on board with not having executable configuration files, except that we've been happily abusing them for years now and there are a lot of edge cases that i rely on now.

pdonis 7y ago
All of those setup options are now setup.cfg options: https://docs.python.org/3/distutils/configfile.html https://docs.python.org/3/distutils/configfile.html At least, that's supposed to be the case; I am not 100% certain that it always is. However, even so, AFAIK setup.cfg does not directly support using third-party libraries to compute the setup options, as your setup.py does. The only workaround I'm aware of for that if you want to get rid of setup.py is to have a separate script that generates setup.cfg using whatever libraries you need. However, I don't think that's the vision of the Python developers.
- hprotagonist 7y ago
  I get the idea -- but, how do I tell Cython "it's time to compile things now" from a file I can't execute?
  - pdonis 7y ago
    > how do I tell Cython "it's time to compile things now" from a file I can't execute? It looks like there are ways to specify which build tools your package requires; I would assume Cython would be one of the tools that can be specified since it's been around a while and gets a lot of usage, but I don't know for sure.
- kevin_thibedeau 7y ago
  There are lot that of options that setup.cfg doesn't support. "use_2to3" is one.
  - takeda 7y ago
    I never needed it, but I see 2to3 options there [1], are they not working? [1] https://setuptools.readthedocs.io/en/latest/setuptools.html#options https://setuptools.readthedocs.io/en/latest/setuptools.html#...
maxnoe 7y ago
pyproject.toml and setup.cfg can in some cases make a setup.py unnecessary. That does not mean you should not use it anymore if you have a good reason to use it. The ones you listed are examples. How ever, as explained in the article, you should still add the pyproject.toml

fermienrico 7y ago

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with great dependency management into one tool. In the spirit of Python, there should be only one way to package, manage dependencies and virtual environments. It should be super clean (see Go). Python needs a BDFL to enforce the hive mind. It is a fucking mess and it's unacceptable. Python is otherwise the most incredible language of our times. It is so far ahead of anything out there for general purpose use. Relevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/ [1] https://github.com/pypa/pipenv/issues/4058 https://github.com/pypa/pipenv/issues/4058

nvrspyx 7y ago
I haven't looked much into it myself, but I've heard good things about Poetry [0]. 0: https://python-poetry.org/ https://python-poetry.org/
- rattray 7y ago
  This looks great. Can anyone who's used both comment on how poetry compares to yarn[0]? [0] https://yarnpkg.com/ https://yarnpkg.com/
  - franey 7y ago
    I think of Poetry as the Yarn of Python packaging for two main reasons (apart from package locking, which is the biggest similarity): the commands are more intuitive and the lockfile starts with a different letter from the dependency file. I don't know how many times I open package-lock.json instead of package.json because I tab-complete from my shell. Same with Pipfile and Pipfile.lock for anyone using Pipenv. I wrote this post[0] that compares Poetry and Pipenv if you're interested and talks about some of their differences. Poetry has both add and install commands, whereas Pipenv only has an install command, for instance. [0] https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchmarks-ergonomics/ https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchma...

goldenshale 7y ago

Our experience using pyenv, poetry and a Pyproject.toml for projects lately has been great. I highly recommend going this route compared to dealing with pip and pipenv.

deleted 7y ago

[deleted]

Waterluvian 7y ago

Is toml even supported in the standard library? This seems just bizarre. Python's really overtaking node for massive churn. I feel like an old guy when peers say, "don't use virtual environment use pipenv." then someone says no no don't even use that, use poetry. Etc. I think about 50% of why I enjoy, in my soul, about Rust when I'm hacking away and personal goofs is that the dependency and project management system is all first class and pretty decent and I sure hope they don't go changing it much.

symmitchry 7y ago
Python also doesn't support yaml in the standard library, interestingly.
- mixmastamyk 7y ago
  Yaml is a big mess. I do really like strict-yaml, however (last time I checked) it is just a hack to simplify the large PyYaml compiled package.
  - pydry 7y ago
    Thanks for the vote of confidence. fwiw version 2.0 of strictyaml will feature its own specification and parser.
    - mixmastamyk 7y ago
      Great to hear. I've been manually deactivating almost all of the features of PyYaml for almost a decade now. Over and over. So quite happy someone is taking up the cause.
benatkin 7y ago
Here's their justification for using it. It makes sense to me. https://www.python.org/dev/peps/pep-0518/#overview-of-file-formats-considered https://www.python.org/dev/peps/pep-0518/#overview-of-file-f...

talentedcoin 7y ago

I don’t want to be too negative, and I realize this isn’t an option for everyone ... but seriously, just use conda if you can. Judging from comments here and in the past, we all know packaging is one of Python’s Achilles’ heels. I spent a couple years bouncing around with pyenv/pipenv, then poetry, then back. I lovingly followed all the flamewars here and on github. Ultimately, I realize now, I laboured under the conceit that it was desirable to have a “native” solution that did not depend on a suite like conda. I now think I was totally wrong. I’m so much happier and more productive now that I’ve switched. For any data science use case conda is hands-down better, and for everything else it’s still more useable than any of the alternatives. I hate to say this but it’s true. The only thing I can think of that was harder to work with a conda setup was the keychain in macOS, and I’m not even sure now that that’s an issue anymore. conda really does just work.

pottertheotter 7y ago
I've had the opposite experience. I used conda for several years, primarily for data science, but regularly ran into problems with packages not working, updates for packages bring severely delayed, and issues with conda itself. Switched to pyenv+virtualenv 18 months ago and haven't had a single issue. I believe both of our stories. I just always wonder how this happens that two people have such opposite experiences.
- talentedcoin 7y ago
  Ha!! Likewise, I believe you 100%. It’s fascinating isn’t it? Just goes to show what a diverse (and frankly wonderful) platform Python is, despite the warts. If you don’t mind my asking, what kind of stuff we’re you working on? Just curious! I have definitely found it’s easy to bork a conda install with updates, enough to make it easier to just wipe it out and start over. I mostly work on financial software and only conda seemed to play nicely with pandas and brokerage APIs. I should probably mention that I do a lot of this is cross-platform and conda seemed to be the only thing that worked without a lot of tweaking on Windows, Mac and Linux consistently. But the Linux experience with pyenv and virtualenv (and pipenv too, for the most part) was great. If I was Linux-only I honestly might never have switched.

poooood 7y ago

I quite like the pyproject.toml; it + poetry + dephell is (relatively speaking) a breeze. Both poetry and dephell are quite good and they don’t not mesh with conda. Python packaging is awful but it’s gotten way better; it’s still maddening but it’s better.

carlosperate 7y ago

Apart from packaging I also love the idea of consolidating configurations from different dependencies into a single place. But when I first moved to pyproject.toml I didn't find its use as widespread as I hoped (flake8 being the dependency I hear about the most without support), so I started collecting a list of projects currently using it, and discussing its use, in an awesome list format. Hopefully it will be useful for other people as well: https://github.com/carlosperate/awesome-pyproject https://github.com/carlosperate/awesome-pyproject

yamrzou 7y ago

I have found this blog post[1] quite informative regarding Python projects packaging. It describes well how all the tools (pyenv, poetry, black, flake8, isort, pre-commit, pytest, coverage, tox) fit together. Poetry seems the go-to tool for dependencies management and it centralizes all your package configurations alongside its dependencies in pyproject.toml. [1] https://medium.com/georgian-impact-blog/python-tooling-makes-a-project-tick-181d567eea44 https://medium.com/georgian-impact-blog/python-tooling-makes...

RMPR 7y ago
Still have some critical bugs here and there[1], many about resolving dependencies. Don't get me wrong, I use it for my own current side project github.com/rmpr/atbswp, but imho a lot of things need improvement. [1]: github.com/python-poetry/poetry/issues

BerislavLopac 7y ago

Announcing a new Sponsorship Program for Python Packaging: http://pyfound.blogspot.com/2020/04/sponsoring-python-packaging.html http://pyfound.blogspot.com/2020/04/sponsoring-python-packag... Discussion here: https://news.ycombinator.com/item?id=22772410 https://news.ycombinator.com/item?id=22772410