4 ms·
> What would in my opinion improve the situation? > Source only uploads and building of wheels for all platforms on PyPI infrastructure – this way we could hav
by dalke 5y ago
> What would in my opinion improve the situation?
> Source only uploads and building of wheels for all platforms on PyPI infrastructure – this way we could have wheels everywhere and remove the need to compile anything in pip install
I don't see how that can happen.
I distribute software through PyPI that doesn't support MS Windows.
It depends on OpenMP, and the stock macOS environment doesn't include OpenMP libraries. There are a few ways the end-user night support OpenMP on macOS, but I can't pre-compile a wheel for all those cases.
Neither can PyPI. Nor should they. It's my headache, not theirs.
Speaking more broadly, Python Packaging confuses me. I only vaguely understand the need for all this development in the last decade. What I do is so niche that it doesn't affect me.
1. My core distribution has no third-party dependencies so lock files and the like don't affect me.
2. I support third-party chemistry packages, but until a few weeks ago, none of them were distributed on PyPI. (RDKit just recently added PyPI downloads.) OTOH, all three are available through conda. And there's no way to express that in the existing Python packaging system.
3. The aforementioned OpenMP dependency.
4. I use per-file compilation flags, via a subclass of 'build_ext'. I have optimized implementations for certain instruction sets, which are chosen at run-time so the library run can still run on older hardware. To be clear, this is mostly historical - for machines which didn't support AVX2, and even machines which didn't support POPCNT - but AVX-512 might be an issue in the future.
5. To eek out more performance, the compilation set uses code generation for each of the hardware-specific implementations. This was done with #define macros in C, which became unmaintainable. I now have a Python program which does that just before compile time. C++ templates might work - I don't know C++.
That is, I use setup.py as a sort of Makefile, which it handles well-enough. This is because my needs fit the Unix-y cultural mindset that setup.py came out of. My target audience is academic and industrial researchers with training in programming.
This is not the way the winds of Python Packing seem to be going. In particular, #5 probably sent shivers down the spine of people who don't want potentially untrusted code to run on, say, their CI server during installation.
(One traditional Unix solution is to compile the code in an unprivileged account, build the files, and only install as root/sudo, after checking that the install only copies and chmods the files to install. This is hard to automate. Handwaving: compile in a container, make a local wheel, then install that wheel. But there's no portable container solution for pip etc. to use.)