3 ms·

Hopefully virtualenv has fixed that issue, do you think it could be implemented after the fact in clusters?

by AUmrysh 14y ago

herge 14y ago
As great as virtualenv is, it won't help you with outdated versions of libc.
- ajdecon 14y ago
  The traditional answer to outdated libraries on clusters is to install them in a non-standard location and use Environment Modules[0] to select which libraries to use. Modules are a common tool since clusters have big user bases with lots of different needs, so you almost always have to have multiple versions of some libraries available. The classic example is keeping multiple MPI implementations around; but I know scientists who keep multiple versions of several libraries in their home directories depending on which apps they want to use, and just select the right module groups. That said, I've never tried this with anything as fundamental as libc... [0] http://modules.sourceforge.net/ http://modules.sourceforge.net/
- KMag 14y ago
  As I remember from my time at Google, we tried for a while statically linking all binaries against libc, but it got unwieldy patching libc, and I heard Ulrich Drepper (rightly) put zero effort into making it easier to make a statically linkable libc. Then, Google switched to having a redundant set of system libraries (including libc) installed, and periodic numbered releases of all of those libraries. I don't rightly recall if Google just had all of the distributed jobs, GWS, etc. run chroot or if some of them just had LD_LIBRARY_PATH not include the usual system directories. (I hope most things were run chroot.) The path to libc definitely contained a directory numbered for the release. That way, the OS of the servers could be updated independently of the libraries used to run GWS, the indexing system, etc. This helped reduce risk and testing effort when upgrading things. Some third parties compiled libraries for us, against our setup. When one of the versions of the system libraries was about to be removed from all of the servers, I remember editing the RPATH in the ELF headers of a third party library to make everything keep linking properly. (Edit: ... and without needing to ask them to recompile everything for us.) Anyway, it most certainly is possible (and often desirable) to have one set of libraries for the OS and all vendor-supplied binaries and a different set of libraries (including libc) for all of the code you write. Of course, Google has a lot more resources for this sort of thing, but it's not difficult to set up a chroot environment for your computations, or at least modify LD_LIBRARY_PATH.

Hopefully virtualenv has fixed that issue, do you think it could be implemented after the fact in clusters?

by AUmrysh 14y ago

Hopefully virtualenv has fixed that issue, do you think it could be implemented after the fact in clusters?

herge 14y ago

As great as virtualenv is, it won't help you with outdated versions of libc.

ajdecon 14y ago
The traditional answer to outdated libraries on clusters is to install them in a non-standard location and use Environment Modules[0] to select which libraries to use. Modules are a common tool since clusters have big user bases with lots of different needs, so you almost always have to have multiple versions of some libraries available. The classic example is keeping multiple MPI implementations around; but I know scientists who keep multiple versions of several libraries in their home directories depending on which apps they want to use, and just select the right module groups. That said, I've never tried this with anything as fundamental as libc... [0] http://modules.sourceforge.net/ http://modules.sourceforge.net/
KMag 14y ago
As I remember from my time at Google, we tried for a while statically linking all binaries against libc, but it got unwieldy patching libc, and I heard Ulrich Drepper (rightly) put zero effort into making it easier to make a statically linkable libc. Then, Google switched to having a redundant set of system libraries (including libc) installed, and periodic numbered releases of all of those libraries. I don't rightly recall if Google just had all of the distributed jobs, GWS, etc. run chroot or if some of them just had LD_LIBRARY_PATH not include the usual system directories. (I hope most things were run chroot.) The path to libc definitely contained a directory numbered for the release. That way, the OS of the servers could be updated independently of the libraries used to run GWS, the indexing system, etc. This helped reduce risk and testing effort when upgrading things. Some third parties compiled libraries for us, against our setup. When one of the versions of the system libraries was about to be removed from all of the servers, I remember editing the RPATH in the ELF headers of a third party library to make everything keep linking properly. (Edit: ... and without needing to ask them to recompile everything for us.) Anyway, it most certainly is possible (and often desirable) to have one set of libraries for the OS and all vendor-supplied binaries and a different set of libraries (including libc) for all of the code you write. Of course, Google has a lot more resources for this sort of thing, but it's not difficult to set up a chroot environment for your computations, or at least modify LD_LIBRARY_PATH.