5 ms·

What’s New in Python 3.10

27 points by gilad 6y ago

dalke 6y ago
> The entire distutils package is deprecated, to be removed in Python 3.12. Wait, what? https://www.python.org/dev/peps/pep-0632/ https://www.python.org/dev/peps/pep-0632/ says: > Setuptools has recently integrated a complete copy of distutils and is no longer dependent on the standard library [3]. Pip has been silently replacing distutils with setuptools when installing packages for a long time already, and the distutils documentation has stated that it is being phased out since 2014 (or earlier). It is time to remove it from the standard library. Oh, okay. My package already requires setuptools for setup(), though it then uses distutils for the rest. https://setuptools.readthedocs.io/en/latest/userguide/quickstart.html#id12 https://setuptools.readthedocs.io/en/latest/userguide/quicks... says: > The landscape of Python packaging is shifting and Setuptools has evolved to only provide backend support, no longer being the de-facto packaging tool in the market. All python package must provide a pyproject.toml and specify the backend (build system) it wants to use. Wait, what? What's a "pyproject.toml"? My current setup.py's "build" step generates 10MB of auto-generated C extension code. Am I supposed to ship that too? Or, where do I put that hook? I have a custom build_ext subclass so I can pass per-file compilation flags to the C compiler. How's that handled? I can't even figure out the new way to specify C extension packages in the setuptools documentation?! And it appears that "build" in this modern era means "build a release package", and nothing like what "setup.py build", so keyword searches are failing me. (For example, "A simple, correct PEP 517 package builder" doesn't actually build C extensions.) https://setuptools.readthedocs.io/en/latest/userguide/quickstart.html https://setuptools.readthedocs.io/en/latest/userguide/quicks... goes on to say: > To learn more about Python packaging in general, navigate to the bottom of this page. but that link doesn't exist! Help?
- wrmsr 6y ago
  AIUI, as the official way setuptools is used is a setup.py calling setuptools.setup(), doing this will remain supported in perpetuity and will really still be powering everything under the hood, it's just no longer what people are told to do when they ask 'how do I make a new python library?' - newcomers and people with simple requirements will be directed to poetry or whatever and avoid that whole mess. For obvious historical reasons python's pretty averse to large breaking changes, and its support for all kinds of weird native interops and obscure platforms is one of its major assets (but also why setuptools is such a mess), so that's almost certainly not going anywhere, even if it's now kept even further back in the shadows. Regarding pyproject.toml if you already have a setup.py doing what you need then it's really only for specifying suported python versions and setup_requires (solving the problem of your setup.py itself needing Cython/wheel/etc).
  - dalke 6y ago
    Thanks for the reply! What throws me for a loop is that support for C extensions isn't documented in https://setuptools.readthedocs.io/en/latest/ https://setuptools.readthedocs.io/en/latest/ , nor is there any mention of how to do something equivalent to the current "python setup.py build", so I can't tell what is "official" regarding continued setuptools support for my current use of distutils. Here's how my setup.py starts: from setuptools import setup from distutils.core import Extension from distutils.command.build_ext import build_ext I use Extension() to define the C extension. I use build_ext to create a subclass which figures out the appropriate OpenMP compile and link flags, which depend on the compiler being used. I also use that build_ext subclass to handle code generation. This Python code replaced a horribly complex set of preprocessor #defines and made it much easier to debug and profile the code. Oh, and macOS's default C compiler doesn't support OpenMP so I added support for disabling the OpenMP flags, for example, with: CHEMFP_OPENMP=0 pip install chemfp-3.5.1.tar.gz I looked at poetry. It says "only pure python wheels are supported", so that's not (yet?) a replacement for distutils, yes? And it looks like pyproject.toml is supposed to replace MANIFEST/MANIFEST.in ? Lastly, I have a commercial product. I prefer to distribute source code to my customers. I grudgingly provide pre-compiled wheels for Linux, which people use for evaluation. My main source repository has the license check code ("src/chemfp_lm.h"), which is used to make the wheels. The MANIFEST.in excludes src/chemfp_lm.h during the sdist that goes to my paying customers. To handle source compilation for both cases, setup.py changes the Extension() configuration depending on if src/chemfp_lm.h is present. This was simple using distutils. I have no clue how to replace distutils in the first place, much less be able to handle my special case in this future Python world. Or as you say, "all kinds of weird native interops". FWIW, my code still supports 2.7 and 3.6+, and has no required dependencies on other Python packages, so I didn't realize there was a problem. ;) To be clear, I have optional dependencies on "zstandard" (in PyPI), on three computational chemistry packages with Python APIs but which aren't available through PyPI, and on one Java computational chemistry JAR file, connected via jpype (in PyPI) and itself with an optional dependency on a ZStandard JAR file. Which means pip-resolvable dependencies hasn't been something I've had to care about over the last 11 years of development.
    - wrmsr 6y ago
- BerislavLopac 6y ago
  > What's a "pyproject.toml"? https://snarky.ca/what-the-heck-is-pyproject-toml/ https://snarky.ca/what-the-heck-is-pyproject-toml/
  - dalke 6y ago
    Thanks! Quoting it: > but the purpose of PEP 518 was to come up with a way for projects to specify what build tools they required ... Now a tool like pip can read pyproject.toml, see what build tools are specified in it, and install those in a virtual environment to build your project Except, umm, I can't do that? Because I depend on a C compiler, and have optional dependencies on Python packages which aren't in PyPI. And I didn't see anything there about C extensions. I feel like the forgotten child. :( > editable installs Grr, now I have to figure out what an "editable install" means. EDIT: Oh, I see, I think. I work in the top-level directory with the project chemfp/ underneath and symlink to build/lib/_chemfp.so for the C extension, so I've not needed this.
    - BerislavLopac 6y ago
      You are right, building native extensions seems to have been left behind with the whole move to configuration-based builds (using pyproject.toml, as opposed to the "script-based" builds, using setup.py). That said, the configuration format itself is supposed to be universal -- but apparently none of the current tools support building extensions. That means that you are stuck with distutils, which should be available as part of setuptools -- PEP632 offers some migration advice [0]. I'd also recommend asking for advice on the packaging discussion board [1]. EDIT: And there is also the PyPA's official packaging problems tracker, FWIW [2]. [0] https://www.python.org/dev/peps/pep-0632/#migration-advice https://www.python.org/dev/peps/pep-0632/#migration-advice [1] https://discuss.python.org/c/packaging/14 https://discuss.python.org/c/packaging/14 [2] https://github.com/pypa/packaging-problems https://github.com/pypa/packaging-problems
    - BerislavLopac 6y ago
      > Because I depend on a C compiler But do you specify it in the Python build toolchain, e.g. setup.py? If I'm not mistaken, the binary compilation step happens independently, with the Python build packing everything up in a wheel. > and have optional dependencies on Python packages which aren't in PyPI. Such dependencies can still be declared in pyproject.toml, depending on the build tool - Poetry can add them by e.g. a local path, or git repo URL. I'm not sure what happens if you depend on them for the build itself, but it should be possible to add them to the build-system table in the TOML file. But pyproject.toml doesn't change anything compared to the current system; for backward compatibility, PEP 518 assumes this default file if a custom one is missing: [build-system] requires = ["setuptools", "wheel"] Which is enough to execute the standard setup.py. In any case, as per PEP 517, you can define your own custom packaging backend using the build-backend value in the [build-system] table.
pizza 6y ago
I like the new error messages and union types feature
killingtime74 6y ago
Nice improvements to typing, will be able to use these everyday

27 points by gilad 6y ago

> The entire distutils package is deprecated, to be removed in Python 3.12. Wait, what? https://www.python.org/dev/peps/pep-0632/ https://www.python.org/dev/peps/pep-0632/ says: > Setuptools has recently integrated a complete copy of distutils and is no longer dependent on the standard library [3]. Pip has been silently replacing distutils with setuptools when installing packages for a long time already, and the distutils documentation has stated that it is being phased out since 2014 (or earlier). It is time to remove it from the standard library. Oh, okay. My package already requires setuptools for setup(), though it then uses distutils for the rest. https://setuptools.readthedocs.io/en/latest/userguide/quickstart.html#id12 https://setuptools.readthedocs.io/en/latest/userguide/quicks... says: > The landscape of Python packaging is shifting and Setuptools has evolved to only provide backend support, no longer being the de-facto packaging tool in the market. All python package must provide a pyproject.toml and specify the backend (build system) it wants to use. Wait, what? What's a "pyproject.toml"? My current setup.py's "build" step generates 10MB of auto-generated C extension code. Am I supposed to ship that too? Or, where do I put that hook? I have a custom build_ext subclass so I can pass per-file compilation flags to the C compiler. How's that handled? I can't even figure out the new way to specify C extension packages in the setuptools documentation?! And it appears that "build" in this modern era means "build a release package", and nothing like what "setup.py build", so keyword searches are failing me. (For example, "A simple, correct PEP 517 package builder" doesn't actually build C extensions.) https://setuptools.readthedocs.io/en/latest/userguide/quickstart.html https://setuptools.readthedocs.io/en/latest/userguide/quicks... goes on to say: > To learn more about Python packaging in general, navigate to the bottom of this page. but that link doesn't exist! Help?

wrmsr 6y ago
AIUI, as the official way setuptools is used is a setup.py calling setuptools.setup(), doing this will remain supported in perpetuity and will really still be powering everything under the hood, it's just no longer what people are told to do when they ask 'how do I make a new python library?' - newcomers and people with simple requirements will be directed to poetry or whatever and avoid that whole mess. For obvious historical reasons python's pretty averse to large breaking changes, and its support for all kinds of weird native interops and obscure platforms is one of its major assets (but also why setuptools is such a mess), so that's almost certainly not going anywhere, even if it's now kept even further back in the shadows. Regarding pyproject.toml if you already have a setup.py doing what you need then it's really only for specifying suported python versions and setup_requires (solving the problem of your setup.py itself needing Cython/wheel/etc).
- dalke 6y ago
  Thanks for the reply! What throws me for a loop is that support for C extensions isn't documented in https://setuptools.readthedocs.io/en/latest/ https://setuptools.readthedocs.io/en/latest/ , nor is there any mention of how to do something equivalent to the current "python setup.py build", so I can't tell what is "official" regarding continued setuptools support for my current use of distutils. Here's how my setup.py starts: from setuptools import setup from distutils.core import Extension from distutils.command.build_ext import build_ext I use Extension() to define the C extension. I use build_ext to create a subclass which figures out the appropriate OpenMP compile and link flags, which depend on the compiler being used. I also use that build_ext subclass to handle code generation. This Python code replaced a horribly complex set of preprocessor #defines and made it much easier to debug and profile the code. Oh, and macOS's default C compiler doesn't support OpenMP so I added support for disabling the OpenMP flags, for example, with: CHEMFP_OPENMP=0 pip install chemfp-3.5.1.tar.gz I looked at poetry. It says "only pure python wheels are supported", so that's not (yet?) a replacement for distutils, yes? And it looks like pyproject.toml is supposed to replace MANIFEST/MANIFEST.in ? Lastly, I have a commercial product. I prefer to distribute source code to my customers. I grudgingly provide pre-compiled wheels for Linux, which people use for evaluation. My main source repository has the license check code ("src/chemfp_lm.h"), which is used to make the wheels. The MANIFEST.in excludes src/chemfp_lm.h during the sdist that goes to my paying customers. To handle source compilation for both cases, setup.py changes the Extension() configuration depending on if src/chemfp_lm.h is present. This was simple using distutils. I have no clue how to replace distutils in the first place, much less be able to handle my special case in this future Python world. Or as you say, "all kinds of weird native interops". FWIW, my code still supports 2.7 and 3.6+, and has no required dependencies on other Python packages, so I didn't realize there was a problem. ;) To be clear, I have optional dependencies on "zstandard" (in PyPI), on three computational chemistry packages with Python APIs but which aren't available through PyPI, and on one Java computational chemistry JAR file, connected via jpype (in PyPI) and itself with an optional dependency on a ZStandard JAR file. Which means pip-resolvable dependencies hasn't been something I've had to care about over the last 11 years of development.
  - wrmsr 6y ago
BerislavLopac 6y ago
> What's a "pyproject.toml"? https://snarky.ca/what-the-heck-is-pyproject-toml/ https://snarky.ca/what-the-heck-is-pyproject-toml/
- dalke 6y ago
  Thanks! Quoting it: > but the purpose of PEP 518 was to come up with a way for projects to specify what build tools they required ... Now a tool like pip can read pyproject.toml, see what build tools are specified in it, and install those in a virtual environment to build your project Except, umm, I can't do that? Because I depend on a C compiler, and have optional dependencies on Python packages which aren't in PyPI. And I didn't see anything there about C extensions. I feel like the forgotten child. :( > editable installs Grr, now I have to figure out what an "editable install" means. EDIT: Oh, I see, I think. I work in the top-level directory with the project chemfp/ underneath and symlink to build/lib/_chemfp.so for the C extension, so I've not needed this.
  - BerislavLopac 6y ago
    You are right, building native extensions seems to have been left behind with the whole move to configuration-based builds (using pyproject.toml, as opposed to the "script-based" builds, using setup.py). That said, the configuration format itself is supposed to be universal -- but apparently none of the current tools support building extensions. That means that you are stuck with distutils, which should be available as part of setuptools -- PEP632 offers some migration advice [0]. I'd also recommend asking for advice on the packaging discussion board [1]. EDIT: And there is also the PyPA's official packaging problems tracker, FWIW [2]. [0] https://www.python.org/dev/peps/pep-0632/#migration-advice https://www.python.org/dev/peps/pep-0632/#migration-advice [1] https://discuss.python.org/c/packaging/14 https://discuss.python.org/c/packaging/14 [2] https://github.com/pypa/packaging-problems https://github.com/pypa/packaging-problems
  - BerislavLopac 6y ago
    > Because I depend on a C compiler But do you specify it in the Python build toolchain, e.g. setup.py? If I'm not mistaken, the binary compilation step happens independently, with the Python build packing everything up in a wheel. > and have optional dependencies on Python packages which aren't in PyPI. Such dependencies can still be declared in pyproject.toml, depending on the build tool - Poetry can add them by e.g. a local path, or git repo URL. I'm not sure what happens if you depend on them for the build itself, but it should be possible to add them to the build-system table in the TOML file. But pyproject.toml doesn't change anything compared to the current system; for backward compatibility, PEP 518 assumes this default file if a custom one is missing: [build-system] requires = ["setuptools", "wheel"] Which is enough to execute the standard setup.py. In any case, as per PEP 517, you can define your own custom packaging backend using the build-backend value in the [build-system] table.

pizza 6y ago

I like the new error messages and union types feature

killingtime74 6y ago

Nice improvements to typing, will be able to use these everyday