2 ms·
Python packages are mostly fine if they're pure Python, the issue comes when you're installing native C/C++/Fortran dependencies, which most ML and scientific c
by physicsguy 3y ago
Python packages are mostly fine if they're pure Python, the issue comes when you're installing native C/C++/Fortran dependencies, which most ML and scientific codes have. People who complain about Python vs Go or Rust or Node in this area I think are not comparing like for like at all.
In academia, everyone pretty much uses Conda to get around these issues, because packages there if you stick to the main channel 'just work' but the licensing is prohibitive elsewhere. Many years ago, you couldn't generally expect to install a Python package on your machine if it had compiled dependencies without installing those libraries yourself with your system package manager and having a local compiler toolchain. That's changed because PyPi introduced wheels which allow you to bundled shared objects, and is why Python is now usable on Windows, but it's not perfect as the this post shows. Wheels generally are available for the most popular OS/architecture combinations - x86 processors, Windows, Mac, and 'manylinux' (an old Linux distribution with low version of glibc to maintain compatibility with newer distros. If you fall outside of that (ARM Mac, want to use a distro with MUSL rather than glibc, etc. etc.) then you basically are on your own and need to compile everything from scratch, and you're going to have a lot of... fun... doing so. In my previous job I worked for a University with a POWER9 GPU cluster, and so we used tools like EasyBuild and Spack to try and manage this, but many packages required manual patches in order to get them working.