3 ms·

You don't mix and match pieces of containers, just like you wouldn't mix and match binaries from different distributions of Linux. You can use a container as a

You don't mix and match pieces of containers, just like you wouldn't mix and match binaries from different distributions of Linux. You can use a container as a single environment in which to do development, and that works fine. But they are by definition an isolated environment with different dependencies than other containers. The result of compiling something in a container necessarily needs to end up in its own container. ...that said, you could use the exact same container base image, and make many different container images from it, and those files would be compatible (assuming you shipped all needed dependencies).

IshKebab 1y ago
> you wouldn't mix and match binaries from different distributions of Linux. You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. > and make many different container images from it Well yes, that's the problem. You end up either putting everything in one container (in which case why bother with a container?), or with a combinatorial explosion of every piece and version of software you might use. TCL modules are better. They don't let you cheat like containers do, but in return you get a better system.
- 0xbadcafebee 1y ago
  > You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. Doing this across different Linux distributions is inherently prone to failure. I don't know about your TCL modules specifically, but unless you have an identical and completely reproducible software toolchain across multiple linux distributions, it's going to end with problems. Honestly, it sounds like you just don't understand these systems and how they work. TCL modules aren't better than containers, this is like comparing apples and organgutans.
  - IshKebab 1y ago
    > Doing this across different Linux distributions is inherently prone to failure. Sure if you just take binaries from one distro that link with libraries on that distro and try and run it on a different one... But that's not what we're doing. All of our TCL modules are either portable binaries (e.g. commercial software) or compiled from source. > Honestly, it sounds like you just don't understand these systems and how they work. I do, but well done for being patronising. > TCL modules aren't better than containers, They are better for our use case. > this is like comparing apples and organgutans. If apples and orangutans were potential solutions to a single problem why couldn't you compare them?
  - elmolino89 1y ago
    The whole idea of maintaining module systems in a perfect sync on several systems as compared to i.e. just rsync-ing SIFs sounds strange to me. Often HPC systems (or rather their admins) are fairly (and for a good reason) conservative, keeping old system and libraries versions. Your mileage may vary, but in a small benchmark of a bioinformatics program called samtools depending on the version the fastest binaries were either run in a conda environment or inside singularity container using Clear Linux distro. Binaries compiled using either system's GCC or from a module were slower. One would have to repeat it throwing in at least Spack to see if this still holds water.
    - IshKebab 1y ago
      > maintaining module systems in a perfect sync on several systems We use NFS. It's a fairly common solution. I don't think it's perfect but for this sort of thing you can turn on aggressive caching and then it works pretty well. In a previous company I believe all the modules were rsync'd nightly.

IshKebab 1y ago

> you wouldn't mix and match binaries from different distributions of Linux. You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. > and make many different container images from it Well yes, that's the problem. You end up either putting everything in one container (in which case why bother with a container?), or with a combinatorial explosion of every piece and version of software you might use. TCL modules are better. They don't let you cheat like containers do, but in return you get a better system.

0xbadcafebee 1y ago
> You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. Doing this across different Linux distributions is inherently prone to failure. I don't know about your TCL modules specifically, but unless you have an identical and completely reproducible software toolchain across multiple linux distributions, it's going to end with problems. Honestly, it sounds like you just don't understand these systems and how they work. TCL modules aren't better than containers, this is like comparing apples and organgutans.
- IshKebab 1y ago
  > Doing this across different Linux distributions is inherently prone to failure. Sure if you just take binaries from one distro that link with libraries on that distro and try and run it on a different one... But that's not what we're doing. All of our TCL modules are either portable binaries (e.g. commercial software) or compiled from source. > Honestly, it sounds like you just don't understand these systems and how they work. I do, but well done for being patronising. > TCL modules aren't better than containers, They are better for our use case. > this is like comparing apples and organgutans. If apples and orangutans were potential solutions to a single problem why couldn't you compare them?
- elmolino89 1y ago
  The whole idea of maintaining module systems in a perfect sync on several systems as compared to i.e. just rsync-ing SIFs sounds strange to me. Often HPC systems (or rather their admins) are fairly (and for a good reason) conservative, keeping old system and libraries versions. Your mileage may vary, but in a small benchmark of a bioinformatics program called samtools depending on the version the fastest binaries were either run in a conda environment or inside singularity container using Clear Linux distro. Binaries compiled using either system's GCC or from a module were slower. One would have to repeat it throwing in at least Spack to see if this still holds water.
  - IshKebab 1y ago
    > maintaining module systems in a perfect sync on several systems We use NFS. It's a fairly common solution. I don't think it's perfect but for this sort of thing you can turn on aggressive caching and then it works pretty well. In a previous company I believe all the modules were rsync'd nightly.