4 ms·

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as

by dalke 4y ago

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for OpenMP, via environment variables so they can be passed through from pip. OpenMP is a compile-time option because the default C compiler on macOS doesn't include OpenMP. You need to install it, using one of various approaches. Which is why I only have a source distribution for macOS, along with a description of the approaches. I have not found a non-setup.py way to handle my configuration, nor to provide macOS wheels. Even for the Linux wheels, I have to patch the manylinux Docker container to whitelist libomp (the OpenMP library), using something like this: RUN perl -i -pe 's/"libresolv.so.2"/"libresolv.so.2", "libgomp.so.1"/' /opt/_internal/pipx/venvs/auditwheel/lib/python3.9/site-packages/ auditwheel/policy/manylinux-policy.json Oh, and if compiling where platform.machine() == "arm64" then I need to not add the AVX2 compiler flag. The non-setup.py packaging systems I've looked at are for Python-only code bases. Or, if I understand things correctly, I'm supposed to make a new specialized package which implements PEP 518, which I can then use to boot-strap my code. Except, that's still going to use effectively arbitrary code during the compilation step (to run the codegen) and still use setup.py to build the extension. So it's not like the evil disappears.

deleted 4y ago
[deleted]
orf 4y ago
To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or options in a structured way. Think of it like cargo.toml va build.rs.
- dalke 4y ago
  I agree it would be cool and useful. But it appears to be such a hard problem that modern packaging tools ignore it, preferring to take on other challenges instead. My own attempts at extracting Python configuration information to generate a Makefile for personal use (because Makefile understand dependencies better than setup.py) is a mess caused by my failure to understand what all the configuration options do. Given that's the case, when do you think we'll be able to "Remove setup.py files and mandate wheels"? I'm curious on what evils you're thinking of? I assume the need to run arbitrary Python code just to find metadata is one of them. But can't that be resolved with a pyproject.toml which uses setuptools only for the build backend? So you don't need to remove setup.py, only restrict when it's used, yes?
  - infogulch 4y ago
    The closest thing I've seen to a solution in this space is Riff, discussed yesterday [1], which solves the external dependency problem for rust projects. [1]: https://news.ycombinator.com/item?id=32739954 https://news.ycombinator.com/item?id=32739954
    - dalke 4y ago
      Agreed. In my answers to the survey, I mentioned "nix" was the technology most likely to affect the future of Python packaging, in part because of reading that same article on Riff. I think now I should have mentioned Riff too.
- dec0dedab0de 4y ago
  The ability to create a custom package that can run any custom code you want at install time is very powerful. I think a decent solution would be to have a way to mark a package as trusted, and only allow pre/post scripts if they are indeed trusted. Maybe even have specific permissions that can be granted, but that seems like a ton of work to get right across operating systems. My specific use cases are adding custom CA certs to certifi after it is installed, and modifying the maximum version of a requirement listed for an abandoned library that works fine with a newer version. I think the best solutions would be an official way to ignore dependencies for a specific package, and specify replacement packages in a project's dependencies. Something like this if it were a Pipfile: public-package = {version = "~=1.0",replace_with='path/to/local-package'} abandoned-package = {version = "~=*",ignore_dependencies=True} But the specific problem doesn't matter, what matters is that there will always be exceptions. This is Python, we're all adults here, and we should be able to easily modify things to get them to work the way we want them to. Any protections added should include a way to be dangerous. I know your point is more about requiring static metadata than using wheels per se. I just believe that all things Python should be flexible and hack-able. There are other more rigid languages if you're into that sort of thing. edit: before anyone starts getting angry I know there are other ways to solve the problems I mentioned. forking/vendoring is a bit of overkill for such a small change, and doesn't solve for when a dependency of a dependency needs to be modified. monkeypatching works fine, however it would need to be done at all the launch points of the project, and even then if I open a repl and import a specific module to try something it won't have my modifications. modifying an installed package at runtime works reasonably well, but it can cause a performance hit at launch, and while it only needs to be run once, it still needs to be run once. So if the first thing you do after recreating a virualenv is to try something with an existing module we have the same problem as monkey patching. 'just use docker' or maybe the more toned down version: 'create a real setup script for developers' are both valid solutions, and where I'll probably end up. It was just very useful to be able to modify things in a pinch.
korijn 4y ago
I emphatize with your situation and it's a great example. As crazy as this may sound, I think you would have to build every possible permutation of your library and make all of them available on pypi. You'd need a some new mechanism based on metadata to represent all the options and figure out how to resolve against available system libraries. Especially that last part seems very complicated. But I do think it's possible.
pabs3 4y ago
> not add the AVX2 compiler flag It is a better idea to do instruction selection at runtime in the code that currently uses AVX2. I recently wrote some docs for Debian contributors about the different ways to achieve this: https://wiki.debian.org/InstructionSelection https://wiki.debian.org/InstructionSelection
- dalke 4y ago
  I do that, using manual CPUID tests, along with allowing environment variables to override the default path choices. But if the compiler by default doesn't enable AVX2 then it will fail to compile the AVX2 intrinsics unless I add -mavx2. Even worse was ~10 years ago when I had an SSSE3 code path, with one file using SSSE3 intrinsics. I had to compile only that file for SSSE3, and not the rest of the package, as otherwise the compiler would issue SSSE3 instructions where it decided was appropriate. Including in code that wasn't behind a CPUID check. Thus crash on hardware without SSSE3. See https://stackoverflow.com/questions/15527611/how-do-i-specify-different-compiler-flags-for-just-one-python-c-extension-source https://stackoverflow.com/questions/15527611/how-do-i-specif... for more info about my solution. Someone last year contributed a solution for MS Windows.
  - pabs3 4y ago
    See the wiki page, the function multi-versioning stuff means you can use AVX2 in select functions without adding -mavx2. And using SIMD Everywhere you can automatically port that to ARM NEON, POWER AltiVec etc.
    - dalke 4y ago
      EDIT: after I wrote the below I realize I could use automatic multi-versioning solely to configure the individual functions, along with with a stub function indicating "was compiled for this arch?" I think that might be more effective should I need to revisit how I support multiple processor architecture dispatch. I will still need the code generation step. Automatic multi-versioning doesn't handled what I needed, at least not when I started. I needed a fast way to compute the popcount. 10 years ago, before most machines supported POPCNT, I implemented a variety of popcount algorithms (see https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0398-8/tables/1 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... ) and found that the fastest version depended on more that just the CPU instruction set. I ended up running some timings during startup to figure out the fastest version appropriate to the given hardware, with the option to override it (via environment variables) for things like benchmark comparisons. I used it to generate that table I linked to. Function multi-versioning - which I only learned about a few month ago - isn't meant to handle that flexibility. To my understanding. I still have one code path which uses __popcountll built-in intrinsics and another which has inline POPCNT assembly, so I can identify when it's no longer useful to have the inline assembly. (Though I used AVX2 if available, I've also read that some of the AMD processors have several POPCNT execution ports, so may be faster than using AVX2 for my 1024-bit popcount case. I have the run-time option to choose which to use, if I ever have access to those processors.) Furthermore, my code generation has one path for single-threaded use and one code path for OpenMP, because I found single-threaded-using-OpenMP was slower than single-threaded-without-OpenMP and it would crash on multithreaded macOS programs, due to conflicts between gcc's OpenMP implementation and Apple's POSIX threads implementation. The AVX2 popcount is from Muła, Kurz, and Lemire, https://academic.oup.com/comjnl/article-abstract/61/1/111/3852071?login=false https://academic.oup.com/comjnl/article-abstract/61/1/111/38... , with manually added prefetch instructions (implemented by Kurz). It does not appear that SIMD Everywhere is the right route for me.

by dalke 4y ago

deleted 4y ago

[deleted]

orf 4y ago

To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or options in a structured way. Think of it like cargo.toml va build.rs.

dalke 4y ago
I agree it would be cool and useful. But it appears to be such a hard problem that modern packaging tools ignore it, preferring to take on other challenges instead. My own attempts at extracting Python configuration information to generate a Makefile for personal use (because Makefile understand dependencies better than setup.py) is a mess caused by my failure to understand what all the configuration options do. Given that's the case, when do you think we'll be able to "Remove setup.py files and mandate wheels"? I'm curious on what evils you're thinking of? I assume the need to run arbitrary Python code just to find metadata is one of them. But can't that be resolved with a pyproject.toml which uses setuptools only for the build backend? So you don't need to remove setup.py, only restrict when it's used, yes?
- infogulch 4y ago
  The closest thing I've seen to a solution in this space is Riff, discussed yesterday [1], which solves the external dependency problem for rust projects. [1]: https://news.ycombinator.com/item?id=32739954 https://news.ycombinator.com/item?id=32739954
  - dalke 4y ago
    Agreed. In my answers to the survey, I mentioned "nix" was the technology most likely to affect the future of Python packaging, in part because of reading that same article on Riff. I think now I should have mentioned Riff too.
dec0dedab0de 4y ago
The ability to create a custom package that can run any custom code you want at install time is very powerful. I think a decent solution would be to have a way to mark a package as trusted, and only allow pre/post scripts if they are indeed trusted. Maybe even have specific permissions that can be granted, but that seems like a ton of work to get right across operating systems. My specific use cases are adding custom CA certs to certifi after it is installed, and modifying the maximum version of a requirement listed for an abandoned library that works fine with a newer version. I think the best solutions would be an official way to ignore dependencies for a specific package, and specify replacement packages in a project's dependencies. Something like this if it were a Pipfile: public-package = {version = "~=1.0",replace_with='path/to/local-package'} abandoned-package = {version = "~=*",ignore_dependencies=True} But the specific problem doesn't matter, what matters is that there will always be exceptions. This is Python, we're all adults here, and we should be able to easily modify things to get them to work the way we want them to. Any protections added should include a way to be dangerous. I know your point is more about requiring static metadata than using wheels per se. I just believe that all things Python should be flexible and hack-able. There are other more rigid languages if you're into that sort of thing. edit: before anyone starts getting angry I know there are other ways to solve the problems I mentioned. forking/vendoring is a bit of overkill for such a small change, and doesn't solve for when a dependency of a dependency needs to be modified. monkeypatching works fine, however it would need to be done at all the launch points of the project, and even then if I open a repl and import a specific module to try something it won't have my modifications. modifying an installed package at runtime works reasonably well, but it can cause a performance hit at launch, and while it only needs to be run once, it still needs to be run once. So if the first thing you do after recreating a virualenv is to try something with an existing module we have the same problem as monkey patching. 'just use docker' or maybe the more toned down version: 'create a real setup script for developers' are both valid solutions, and where I'll probably end up. It was just very useful to be able to modify things in a pinch.

korijn 4y ago

I emphatize with your situation and it's a great example. As crazy as this may sound, I think you would have to build every possible permutation of your library and make all of them available on pypi. You'd need a some new mechanism based on metadata to represent all the options and figure out how to resolve against available system libraries. Especially that last part seems very complicated. But I do think it's possible.

pabs3 4y ago

> not add the AVX2 compiler flag It is a better idea to do instruction selection at runtime in the code that currently uses AVX2. I recently wrote some docs for Debian contributors about the different ways to achieve this: https://wiki.debian.org/InstructionSelection https://wiki.debian.org/InstructionSelection

dalke 4y ago
I do that, using manual CPUID tests, along with allowing environment variables to override the default path choices. But if the compiler by default doesn't enable AVX2 then it will fail to compile the AVX2 intrinsics unless I add -mavx2. Even worse was ~10 years ago when I had an SSSE3 code path, with one file using SSSE3 intrinsics. I had to compile only that file for SSSE3, and not the rest of the package, as otherwise the compiler would issue SSSE3 instructions where it decided was appropriate. Including in code that wasn't behind a CPUID check. Thus crash on hardware without SSSE3. See https://stackoverflow.com/questions/15527611/how-do-i-specify-different-compiler-flags-for-just-one-python-c-extension-source https://stackoverflow.com/questions/15527611/how-do-i-specif... for more info about my solution. Someone last year contributed a solution for MS Windows.
- pabs3 4y ago
  See the wiki page, the function multi-versioning stuff means you can use AVX2 in select functions without adding -mavx2. And using SIMD Everywhere you can automatically port that to ARM NEON, POWER AltiVec etc.
  - dalke 4y ago
    EDIT: after I wrote the below I realize I could use automatic multi-versioning solely to configure the individual functions, along with with a stub function indicating "was compiled for this arch?" I think that might be more effective should I need to revisit how I support multiple processor architecture dispatch. I will still need the code generation step. Automatic multi-versioning doesn't handled what I needed, at least not when I started. I needed a fast way to compute the popcount. 10 years ago, before most machines supported POPCNT, I implemented a variety of popcount algorithms (see https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0398-8/tables/1 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... ) and found that the fastest version depended on more that just the CPU instruction set. I ended up running some timings during startup to figure out the fastest version appropriate to the given hardware, with the option to override it (via environment variables) for things like benchmark comparisons. I used it to generate that table I linked to. Function multi-versioning - which I only learned about a few month ago - isn't meant to handle that flexibility. To my understanding. I still have one code path which uses __popcountll built-in intrinsics and another which has inline POPCNT assembly, so I can identify when it's no longer useful to have the inline assembly. (Though I used AVX2 if available, I've also read that some of the AMD processors have several POPCNT execution ports, so may be faster than using AVX2 for my 1024-bit popcount case. I have the run-time option to choose which to use, if I ever have access to those processors.) Furthermore, my code generation has one path for single-threaded use and one code path for OpenMP, because I found single-threaded-using-OpenMP was slower than single-threaded-without-OpenMP and it would crash on multithreaded macOS programs, due to conflicts between gcc's OpenMP implementation and Apple's POSIX threads implementation. The AVX2 popcount is from Muła, Kurz, and Lemire, https://academic.oup.com/comjnl/article-abstract/61/1/111/3852071?login=false https://academic.oup.com/comjnl/article-abstract/61/1/111/38... , with manually added prefetch instructions (implemented by Kurz). It does not appear that SIMD Everywhere is the right route for me.