5 ms·

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a si

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate python -m pip install .[dev] (We provide a `make dev` target that does that for you as well.) Similarly, to build distributions, all you need is `build`: python -m build [1]: https://github.com/pypa/pip-audit https://github.com/pypa/pip-audit

qbasic_forever 4y ago
Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or haven't been studying python packaging for decades).
- woodruffw 4y ago
  I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1]. Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particularly in that respect! [1]: https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... [2]: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/ https://packaging.python.org/en/latest/guides/distributing-p...
  - qbasic_forever 4y ago
    No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it. To be honest that entire pypa doc should be like two paragraphs long instead of 1000+ words. It should be basically, "ok to package your python app just run <official python tool to init a package>, that's it you're done!". Every decision should be made for me, like it is with npm, cargo, etc. I shouldn't have to think beyond running one command. That's what python end users need. We don't need a million tools and huge docs with seemingly no vision or goal. We need to get shit done and the less time we faff about with packaging and tooling the better.
    - KyeRussell 4y ago
      Yep. There being no documentation to really empower someone to work out what to use, to me very directly says “there’s some political spat going on that I’m being exposed to here”.
      - theacodes 4y ago
        that "political spat" was actually targeted harassment towards a former PyPA member the last time someone actively working on packaging.python.org tried to be remotely opinionated. Since this ended with them stepping away and no one else has volunteered, it remains unopinionated.
        qbasic_forever 4y ago
        Sounds like the steering council or whatever that runs python nowadays needs to step up. If people are being targeted for trying to improve packaging then the steering council should make a decision and direct all commentary, etc. to them, not individuals documenting best practices. The whole 'let the community figure it out' seems to have failed and is causing nothing but more confusion and now attacks on people. The council needs to step up and say, "this is how python packaging will work, period. End of story, end of debate. There is no more discussion on this, the decision is final. All other python packaging tools now are non-standard and not recommended for use anymore". That was the one good thing a BDFL model for leadership could achieve, making a hard decision in the face of many strong opinions.
- Izkata 4y ago
  AFAIK in the python 2 era it was the de-facto standard (just swap "python -m venv" with "virtualenv"). All these new tools have just made it more complicated and it's not clear to me what they gain, seems to me like they've simply succeeded in convincing people it's complicated by obscuring what's actually going on.
  - vonseel 4y ago
    Correct. And most people back then were just using `pip install -r requirements.txt`, including different files for dev/test/prod as needed. I worked at one company that used buildout back then, which seemed relatively rare, but buildout was more flexible and allowed you to do things like installing system packages or running your own scripts. We used it to pull down config files that weren't checked into the repo but still wanted to be shared, helping setup the local developer environment, and a few other things.
- fbdab103 4y ago
  A few weeks ago, I was attempting to help introduce someone to Python. I felt embarrassed at trying to paper-over how complicated and confusing the initial configuration is for a newbie. No officially endorsed best-practices to which I could point. Half a dozen tribes with their own competing solutions and guidance. A definitive statement from Python.org as to The Way would go so far.
Denvercoder9 4y ago
That repository doesn't seem to pin dependency versions. How do you integrate that in this workflow?
- woodruffw 4y ago
  You could use `pip-compile` if you want full pinning. That's what we do on another project -- we use GitHub Actions with `pip-compile` to provide a fully frozen copy of the dependency tree for users who'd like that[1]. In the context of `pip-audit`, that makes a little less sense: most of our dependencies are semantically versioned, and we'd rather users receive patches and fixes to our subdependencies automatically, rather than having to wait for us to release a corresponding fix version. Similarly, we expect users to install `pip-audit` into pre-existing virtual environments, meaning that excessive pinning will produce overly conservative dependency conflict errors. [1]: https://github.com/sigstore/sigstore-python/tree/main/install https://github.com/sigstore/sigstore-python/tree/main/instal...
  - Izkata 4y ago
    Or if you don't want to install something else and are willing to just use version numbers (instead of also hashes like pip-compile in that link), "pip freeze" is built in.
    - woodruffw 4y ago
      The tricky thing with `pip freeze` is that it dumps your environment, not your resolved set: your environment also contains things like your `pip` and `setuptools` versions, any developing tooling you have, and potentially your global environmental state (if the environment has global access and you forget to pass `--local` to `pip freeze`). In other words, it's generally a superset of the resolutions collected by `pip-compile`. This may or may not be what you want, or what your users expect!
      - Izkata 4y ago
        By default (at least in python3 nowadays) it also excludes pip, setuptools, distribute, and wheel; you need "--all" to include them in the output.
samwillis 4y ago
Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way". If you go to Python.org and follow through to the beginners guide [0] this is what's suggested: > There are several methods to install additional Python packages: > Packages can be installed via the standard Python distutils mode (python setup.py install). > Many packages can also be installed via the setuptools extension or pip wrapper, see https://pip.pypa.io/ https://pip.pypa.io/. That is so out of date and fundamentally confuses people coming to Python for the first time. How is pip secondary, and no mention of venv? The PyPA need to get buy in from Python Core to put one tool front and centre first. It needs to be like Rust with Cargo, literally the first thing you learn to use and core to all beginners guides. That's not to diminish the work of PyPA, you are all amazing I just want your work to be more obvious! 0: https://docs.python.org/3/using/mac.html#installing-additional-python-packages https://docs.python.org/3/using/mac.html#installing-addition...
- Godel_unicode 4y ago
  The future is here, it’s just inconsistently distributed. There’s also the problem of python.org not wanting to pick winners, i.e. not promoting activestate vs conda vs pypa. https://packaging.python.org/en/latest/tutorials/installing-packages/ https://packaging.python.org/en/latest/tutorials/installing-... https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...
  - KyeRussell 4y ago
    Which is just, utterly utterly ridiculous. The Python packaging story will continue to be a mess right up until this particular inconsistency is resolved.
agumonkey 4y ago
where is the install .[dev] format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools I found one mention in the docs but no more.
- woodruffw 4y ago
  It’s a pip-ism; as far as I know, it’s not defined in any PEP. It should be in their documentation, however.
  - agumonkey 4y ago
    gonna dig deeper there then, thanks
- baq 4y ago
  you may have done pip install jupyter[notebook] this is the same thing, just for . (current directory) and dev variant.
  - woodruffw 4y ago
    Yep. The key difference is that the former is specified in PEP 508, while `.` and its variants are a convenience feature that pip (and maybe some other tools) provide.
  - agumonkey 4y ago
    yeah but I'm missing data on how variant interacts with various requirements defined in setup.cfg (testing dependencies for instance)
- nprescott 4y ago
  PEP-508[0] explains the grammar for "extras": > Optional components of a distribution may be specified using the extras field: identifier_end = letterOrDigit | (('-' | '_' | '.' )* letterOrDigit) identifier = letterOrDigit identifier_end* name = identifier extras_list = identifier (wsp* ',' wsp* identifier)* extras = '[' wsp* extras_list? wsp* ']' as well as explaining their behavior, albeit briefly: > Extras union in the dependencies they define with the dependencies of the distribution they are attached to. The resolution on . is explained by the pip documentation[1]: > pip looks for packages in a number of places: on PyPI (if not disabled via --no-index), in the local filesystem, and in any additional repositories specified via --find-links or --index-url. There is no ordering in the locations that are searched. Rather they are all checked, and the “best” match for the requirements (in terms of version number - see PEP 440 for details) is selected. [0]: https://peps.python.org/pep-0508/#grammar https://peps.python.org/pep-0508/#grammar [1]: https://pip.pypa.io/en/stable/cli/pip_install/#finding-packages https://pip.pypa.io/en/stable/cli/pip_install/#finding-packa...
lmm 4y ago
That's still about 4 times as much command as I'd consider normal/reasonable, compared to say "mvn install".
Helmut10001 4y ago
First time I hear PyPA, and looking up their website [1]: It is not a good sign if a packaging authority has a broken logo link on their mainpage [2]. [1]: https://www.pypa.io/ https://www.pypa.io/ [2]: https://pypi.org/static/images/logo-large.svg/ https://pypi.org/static/images/logo-large.svg/
vonseel 4y ago
So, the project you linked has a pyproject.toml file uses flit: requires = ["flit_core >=3.2,<4"] build-backend = "flit_core.buildapi" Python.org defaults to using hatchling in the docs[1]. PyPA has a sample repository using setuptools[2]. This project is linked from Python.org on this page.[3] Python.org recommends using setuptools under packaging recommendations as well, with no mention of hatchling or flit even on that page.[4] No wonder so many devs are frustrated about this. I've been writing mostly Python for work and fun the past 11 years and I'm f*cking confused what tools to use. When I start a new side project I don't know whether a tool is going to exist and still work in a few years, because there's 10 tools trying to do the same things and it's not clear what the community is actually getting behind. [1]: https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml https://packaging.python.org/en/latest/tutorials/packaging-p... [2]: https://github.com/pypa/sampleproject/blob/main/pyproject.toml https://github.com/pypa/sampleproject/blob/main/pyproject.to... [3]: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/ https://packaging.python.org/en/latest/guides/distributing-p... [4]: https://packaging.python.org/en/latest/guides/tool-recommendations/ https://packaging.python.org/en/latest/guides/tool-recommend...

qbasic_forever 4y ago

Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or haven't been studying python packaging for decades).

woodruffw 4y ago
I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1]. Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particularly in that respect! [1]: https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... [2]: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/ https://packaging.python.org/en/latest/guides/distributing-p...
- qbasic_forever 4y ago
  No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it. To be honest that entire pypa doc should be like two paragraphs long instead of 1000+ words. It should be basically, "ok to package your python app just run <official python tool to init a package>, that's it you're done!". Every decision should be made for me, like it is with npm, cargo, etc. I shouldn't have to think beyond running one command. That's what python end users need. We don't need a million tools and huge docs with seemingly no vision or goal. We need to get shit done and the less time we faff about with packaging and tooling the better.
  - KyeRussell 4y ago
    Yep. There being no documentation to really empower someone to work out what to use, to me very directly says “there’s some political spat going on that I’m being exposed to here”.
    - theacodes 4y ago
      that "political spat" was actually targeted harassment towards a former PyPA member the last time someone actively working on packaging.python.org tried to be remotely opinionated. Since this ended with them stepping away and no one else has volunteered, it remains unopinionated.
      - qbasic_forever 4y ago
        Sounds like the steering council or whatever that runs python nowadays needs to step up. If people are being targeted for trying to improve packaging then the steering council should make a decision and direct all commentary, etc. to them, not individuals documenting best practices. The whole 'let the community figure it out' seems to have failed and is causing nothing but more confusion and now attacks on people. The council needs to step up and say, "this is how python packaging will work, period. End of story, end of debate. There is no more discussion on this, the decision is final. All other python packaging tools now are non-standard and not recommended for use anymore". That was the one good thing a BDFL model for leadership could achieve, making a hard decision in the face of many strong opinions.
Izkata 4y ago
AFAIK in the python 2 era it was the de-facto standard (just swap "python -m venv" with "virtualenv"). All these new tools have just made it more complicated and it's not clear to me what they gain, seems to me like they've simply succeeded in convincing people it's complicated by obscuring what's actually going on.
- vonseel 4y ago
  Correct. And most people back then were just using `pip install -r requirements.txt`, including different files for dev/test/prod as needed. I worked at one company that used buildout back then, which seemed relatively rare, but buildout was more flexible and allowed you to do things like installing system packages or running your own scripts. We used it to pull down config files that weren't checked into the repo but still wanted to be shared, helping setup the local developer environment, and a few other things.
fbdab103 4y ago
A few weeks ago, I was attempting to help introduce someone to Python. I felt embarrassed at trying to paper-over how complicated and confusing the initial configuration is for a newbie. No officially endorsed best-practices to which I could point. Half a dozen tribes with their own competing solutions and guidance. A definitive statement from Python.org as to The Way would go so far.

Denvercoder9 4y ago

That repository doesn't seem to pin dependency versions. How do you integrate that in this workflow?

woodruffw 4y ago
You could use `pip-compile` if you want full pinning. That's what we do on another project -- we use GitHub Actions with `pip-compile` to provide a fully frozen copy of the dependency tree for users who'd like that[1]. In the context of `pip-audit`, that makes a little less sense: most of our dependencies are semantically versioned, and we'd rather users receive patches and fixes to our subdependencies automatically, rather than having to wait for us to release a corresponding fix version. Similarly, we expect users to install `pip-audit` into pre-existing virtual environments, meaning that excessive pinning will produce overly conservative dependency conflict errors. [1]: https://github.com/sigstore/sigstore-python/tree/main/install https://github.com/sigstore/sigstore-python/tree/main/instal...
- Izkata 4y ago
  Or if you don't want to install something else and are willing to just use version numbers (instead of also hashes like pip-compile in that link), "pip freeze" is built in.
  - woodruffw 4y ago
    The tricky thing with `pip freeze` is that it dumps your environment, not your resolved set: your environment also contains things like your `pip` and `setuptools` versions, any developing tooling you have, and potentially your global environmental state (if the environment has global access and you forget to pass `--local` to `pip freeze`). In other words, it's generally a superset of the resolutions collected by `pip-compile`. This may or may not be what you want, or what your users expect!
    - Izkata 4y ago
      By default (at least in python3 nowadays) it also excludes pip, setuptools, distribute, and wheel; you need "--all" to include them in the output.

samwillis 4y ago

Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way". If you go to Python.org and follow through to the beginners guide [0] this is what's suggested: > There are several methods to install additional Python packages: > Packages can be installed via the standard Python distutils mode (python setup.py install). > Many packages can also be installed via the setuptools extension or pip wrapper, see https://pip.pypa.io/ https://pip.pypa.io/. That is so out of date and fundamentally confuses people coming to Python for the first time. How is pip secondary, and no mention of venv? The PyPA need to get buy in from Python Core to put one tool front and centre first. It needs to be like Rust with Cargo, literally the first thing you learn to use and core to all beginners guides. That's not to diminish the work of PyPA, you are all amazing I just want your work to be more obvious! 0: https://docs.python.org/3/using/mac.html#installing-additional-python-packages https://docs.python.org/3/using/mac.html#installing-addition...

Godel_unicode 4y ago
The future is here, it’s just inconsistently distributed. There’s also the problem of python.org not wanting to pick winners, i.e. not promoting activestate vs conda vs pypa. https://packaging.python.org/en/latest/tutorials/installing-packages/ https://packaging.python.org/en/latest/tutorials/installing-... https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...
- KyeRussell 4y ago
  Which is just, utterly utterly ridiculous. The Python packaging story will continue to be a mess right up until this particular inconsistency is resolved.

agumonkey 4y ago

where is the install .[dev] format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools I found one mention in the docs but no more.

woodruffw 4y ago
It’s a pip-ism; as far as I know, it’s not defined in any PEP. It should be in their documentation, however.
- agumonkey 4y ago
  gonna dig deeper there then, thanks
baq 4y ago
you may have done pip install jupyter[notebook] this is the same thing, just for . (current directory) and dev variant.
- woodruffw 4y ago
  Yep. The key difference is that the former is specified in PEP 508, while `.` and its variants are a convenience feature that pip (and maybe some other tools) provide.
- agumonkey 4y ago
  yeah but I'm missing data on how variant interacts with various requirements defined in setup.cfg (testing dependencies for instance)
nprescott 4y ago
PEP-508[0] explains the grammar for "extras": > Optional components of a distribution may be specified using the extras field: identifier_end = letterOrDigit | (('-' | '_' | '.' )* letterOrDigit) identifier = letterOrDigit identifier_end* name = identifier extras_list = identifier (wsp* ',' wsp* identifier)* extras = '[' wsp* extras_list? wsp* ']' as well as explaining their behavior, albeit briefly: > Extras union in the dependencies they define with the dependencies of the distribution they are attached to. The resolution on . is explained by the pip documentation[1]: > pip looks for packages in a number of places: on PyPI (if not disabled via --no-index), in the local filesystem, and in any additional repositories specified via --find-links or --index-url. There is no ordering in the locations that are searched. Rather they are all checked, and the “best” match for the requirements (in terms of version number - see PEP 440 for details) is selected. [0]: https://peps.python.org/pep-0508/#grammar https://peps.python.org/pep-0508/#grammar [1]: https://pip.pypa.io/en/stable/cli/pip_install/#finding-packages https://pip.pypa.io/en/stable/cli/pip_install/#finding-packa...

lmm 4y ago

That's still about 4 times as much command as I'd consider normal/reasonable, compared to say "mvn install".

Helmut10001 4y ago

First time I hear PyPA, and looking up their website [1]: It is not a good sign if a packaging authority has a broken logo link on their mainpage [2]. [1]: https://www.pypa.io/ https://www.pypa.io/ [2]: https://pypi.org/static/images/logo-large.svg/ https://pypi.org/static/images/logo-large.svg/

vonseel 4y ago

So, the project you linked has a pyproject.toml file uses flit: requires = ["flit_core >=3.2,<4"] build-backend = "flit_core.buildapi" Python.org defaults to using hatchling in the docs[1]. PyPA has a sample repository using setuptools[2]. This project is linked from Python.org on this page.[3] Python.org recommends using setuptools under packaging recommendations as well, with no mention of hatchling or flit even on that page.[4] No wonder so many devs are frustrated about this. I've been writing mostly Python for work and fun the past 11 years and I'm f*cking confused what tools to use. When I start a new side project I don't know whether a tool is going to exist and still work in a few years, because there's 10 tools trying to do the same things and it's not clear what the community is actually getting behind. [1]: https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml https://packaging.python.org/en/latest/tutorials/packaging-p... [2]: https://github.com/pypa/sampleproject/blob/main/pyproject.toml https://github.com/pypa/sampleproject/blob/main/pyproject.to... [3]: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/ https://packaging.python.org/en/latest/guides/distributing-p... [4]: https://packaging.python.org/en/latest/guides/tool-recommendations/ https://packaging.python.org/en/latest/guides/tool-recommend...