4 ms·
The fundamental issue here is that Python users expect things to be install and go. That has actually never been the case, but it’s got more common over time -
by physicsguy 5y ago
The fundamental issue here is that Python users expect things to be install and go. That has actually never been the case, but it’s got more common over time - before wheels came along, you ended up getting compilation errors all the time from missing packages with pip. Enthought Canopy and later Conda came along and tried to fix that, and then later pip implemented binary wheels, but the wheels are often not a great solution because as someone in this thread said - every package ends up including duplicate stuff, so you can end up with like 5 versions of OpenBLAS and whatever other C/C++/FORTRAN dependencies your packages depend on. Intel released an MKL package onto PyPi which has reduced a lot of that somewhat but it is obviously a bit problematic, because lots of things now use this instead of OpenBLAS and fail to compile on other architectures (cough, PyTorch, cough).
The issue we now have is that scientific users almost always turn to Conda. Scientific authors also have increasingly turned to conda, because it solves a lot of distribution problems for them - it’s a lot easier to say to people ‘conda install -c conda-forge XYZ’ than it is to say ‘install the CUDA toolkit >v10.1 and then download dependency X and then install it from PyPi.’ Look for e.g. at Software Carpentry which aims to teach Python to researchers, or any undergrad course in Python outside of perhaps Comp Sci, and these days they almost always tell people to use Conda. In addition to that, it’s so much easier from a build perspective to add packages to Conda Forge than it is to build the various types of wheels that you need to do for PyPi. The manylinux instructions aren’t even particularly up to date.
To me, a lot of this stems back from Guido basically saying to the scientific community ‘it sounds like your problems are different to that of a lot of the Python ecosystem’ but without seeing that the whole compiled dependency thing affects everyone at some point. I’m not sure what the fix is - do we want people to be able to use PyPi to bundle compiled external libraries? It sounds like you’d just end up replicating Conda at the end of the day.
For me personally, this fragmentation causes a lot of issues because I help users install packages into our HPC cluster. I need to be able to install things from source and PyPi usually offers that, but increasingly, users bring packages with a chain of dependencies that have no source PyPi package and a long list of conda dependencies. I don’t blame package authors at all - they are doing what is easiest for them and most of their users - but I do think it needs a lot of thought and work by PyPa.
- fock 5y agomost commonly rdkit, where I've never managed to build all the dependencies
- navanchauhan 5y agoI agree. So many times I have had to reinstall rdkit using homebrew and link it again and again. Although, you can now install rdkit via pip: pip install rdkit-pypi GitHub Issue: https://github.com/rdkit/rdkit/issues/1812#issuecomment-808810574 https://github.com/rdkit/rdkit/issues/1812#issuecomment-8088...
- dalke 5y agoFor what it's worth, I have multiple RDKit instances on my Mac, so I can test chemfp against different toolkit releases. It's a headache! Here's my notes from installation - I have to keep notes as hints for newer releases: /Users/dalke/local/bin/python3.9 -m venv ~/venvs/py39-2021-4 source ~/venvs/py39-2021-4/bin/activate.csh pip install numpy scipy matplotlib pandas cd ~/ftps tar xf Release_2021_03_1.tar.gz cd rdkit-Release_2021_03_1 mkdir build_py39-2021-4 cd build_py39-2021-4/ cmake .. -DRDK_INSTALL_INTREE=OFF \ -DCMAKE_INSTALL_PREFIX=/Users/dalke/venvs/py39-2021-4 \ -DCMAKE_INSTALL_RPATH="/Users/dalke/venvs/py39-2021-4/lib/;/Users/dalke/local/lib/" \ -DRDK_BUILD_AVALON_SUPPORT=ON \ -DRDK_BUILD_CAIRO_SUPPORT=ON \ -DRDK_BUILD_INCHI_SUPPORT=ON \ -DRDK_BUILD_COMPRESSED_SUPPLIERS=ON make -j4 make install and then manual edit (!) activate.csh so the DYLD_LIBRARY_PATH is updated: alias deactivate 'test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH" && unset _OLD_VIRTUAL_PATH && setenv DYLD_LIBRARY_PATH "$_OLD_DYLD_LIBRARY_PATH" && unset _OLD_DYLD_LIBRARY_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; test "\!:*" != "nondestructive" && unalias deactivate' ... if (`printenv DYLD_LIBRARY_PATH` == '') then set _OLD_DYLD_LIBRARY_PATH="" setenv DYLD_LIBRARY_PATH "$VIRTUAL_ENV/lib" else set _OLD_DYLD_LIBRARY_PATH="$DYLD_LIBRARY_PATH" setenv DYLD_LIBRARY_PATH "$VIRTUAL_ENV/lib:$DYLD_LIBRARY_PATH" endif You'll note that I'm a tcsh user ('cause that was kewl in the 1990s when I started in Unix) and my directory structure uses "~/ftps" for things I've manually downloaded (ditto), and I'm probably using venv poorly. Oh, but am I done? No! See the "/Users/dalke/local/lib/" in the CMAKE_INSTALL_RPATH? That's because I installed my own Python, and Boost: cd ~/ftps curl -O 'https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tar.xz' tar xf Python-3.9.1.tar.xz cd Python-3.9.1 ./configure --prefix ~/local --with-openssl=/usr/local/opt/openssl make -j3 make install cd ~/ftps tar xf boost_1_72_0.tar.gz cd boost_1_72_0 ./bootstrap.sh --prefix=/Users/dalke/local --with-python=/Users/dalke/local/bin/python3.9 --with-toolset=clang --with-python-version=3.9 ./b2 -j 4 ./b2 install And even then it's a house of cards that will likely collapse when I install python 3.10. Oh, and I have multiple Open Babel installs from scratch, for the same reasons Pity me. :)