3 ms·

We tried to use this on our compute cluster for silicon design/verification. We gave up in the end and just went with the traditional TCL (now Lua) modules. Th

We tried to use this on our compute cluster for silicon design/verification. We gave up in the end and just went with the traditional TCL (now Lua) modules. The problems are: 1. You can't have apptainers that use each other. The most common case was things like Make, GCC, Git etc. If Make is in a different apptainer to GCC then it won't work because as soon as you go into Make then it can't see GCC any more. 2. It doesn't work if any of your output artefacts depend on things inside the container. For example you use your GCC apptainer to compile a program. It appears to work, but when you run it you find it actually linked to something in the apptainer that isn't visible any more. This is also a problem for C headers. 3. We had constant issues with PATH getting messed up so you can't see things outside the apptainer that should have been available. All in all it was a nice idea but ended up causing way more hassle than it was worth. It was much easier just to use an old OS (RHEL8) and get everything to work directly on that.

mbreese 1y ago
I think of using Apptainer/Singularity as more like Docker than anything else (without the full networking configs). These are all issues with traditional Docker containers as well, so I’m not sure how you were using the containers or what you were expecting. For my workflows on HPC, I use apptainers as basically drop-in replacements for Docker, and for that, they work quite well. These biggest benefit is that the containers are unprivileged. This means you can’t do a lot of things (in particular complex networking), but it also makes it much more secure for multi-tenant systems (like HPC). (I know Docker and Apptainer are slightly different beasts, but I’m speaking in broad strokes in a general sense without extra permissions).
- d3Xt3r 1y ago
  You can also run Docker itself in rootless mode[1]. And if for some reason you don't want to run Docker, you can also use Podman or Incus instead, and they both support Docker images, as well as running unpriviliged. Finally, there's also Flox[2], which is a Nix-based application Sandbox, that I believe would align more towards your (and OP's) use case (unless you specifically require Docker image compatibility). So unfortunately your example doesn't illustrate why Apptainer is a better option. [1] https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/ [2] https://flox.dev/ https://flox.dev/
  - mbreese 1y ago
    For a long time you couldn’t run Docker in rootless mode. So Singularity/Apptainer was developed so that the benefit of containers could be used with shared HPC clusters. Podman also didn’t exist at the time. I don’t know why we should have waited for docker to get around to supporting a more secured use-case. With Docker, security has always seemed like an afterthought, which seemed perfectly reasonable considering their original use-case. But that’s not tenable on $$$ HPC clusters with thousands of users. So, the HPC world developed their own solution, better adapted to that environment.
0xbadcafebee 1y ago
You don't mix and match pieces of containers, just like you wouldn't mix and match binaries from different distributions of Linux. You can use a container as a single environment in which to do development, and that works fine. But they are by definition an isolated environment with different dependencies than other containers. The result of compiling something in a container necessarily needs to end up in its own container. ...that said, you could use the exact same container base image, and make many different container images from it, and those files would be compatible (assuming you shipped all needed dependencies).
- IshKebab 1y ago
  > you wouldn't mix and match binaries from different distributions of Linux. You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. > and make many different container images from it Well yes, that's the problem. You end up either putting everything in one container (in which case why bother with a container?), or with a combinatorial explosion of every piece and version of software you might use. TCL modules are better. They don't let you cheat like containers do, but in return you get a better system.
  - 0xbadcafebee 1y ago
    > You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. Doing this across different Linux distributions is inherently prone to failure. I don't know about your TCL modules specifically, but unless you have an identical and completely reproducible software toolchain across multiple linux distributions, it's going to end with problems. Honestly, it sounds like you just don't understand these systems and how they work. TCL modules aren't better than containers, this is like comparing apples and organgutans.
    - IshKebab 1y ago
      > Doing this across different Linux distributions is inherently prone to failure. Sure if you just take binaries from one distro that link with libraries on that distro and try and run it on a different one... But that's not what we're doing. All of our TCL modules are either portable binaries (e.g. commercial software) or compiled from source. > Honestly, it sounds like you just don't understand these systems and how they work. I do, but well done for being patronising. > TCL modules aren't better than containers, They are better for our use case. > this is like comparing apples and organgutans. If apples and orangutans were potential solutions to a single problem why couldn't you compare them?
Fokamul 1y ago
So you are using container app and your biggest problem with it is, that it's doing exactly as advertised -> containers :D
- IshKebab 1y ago
  If you want to be unnecessarily dismissive about the problems with containers, sure.
  - elmolino89 1y ago
    Frankly I have a problem with one of your problem as described. Who on the planet Earth creates a container with GCC, may I guess, to compile programs? and then complains about make being located in another container. If you miss some utility just convert the apptainer container to a sandbox, install utilities you need and convert the sandbox to .sif as needed. Also the whole point of building the program using compilers/libraries in the container is to use such container to run the aforementioned program in that very environment and not worry about libraryX or utility Y not installed in some box in a galaxy far, far away...
    - IshKebab 1y ago
      > Also the whole point of building the program using compilers/libraries in the container is to use such container to run the aforementioned program in that very environment Says who? That's only because that's the only way it works, not because it's desirable to only work like that. The only way Apptainers work well is if you put everything you need in one Apptainer, which makes them much less useful than you might imagine.
deleted 1y ago
[deleted]
je42 1y ago
How do you guys use Lua modules?
- IshKebab 1y ago
  We put all of our software in a module and then we have a script for our projects that loads the appropriate software. I'm not sure I understand the question...

mbreese 1y ago

I think of using Apptainer/Singularity as more like Docker than anything else (without the full networking configs). These are all issues with traditional Docker containers as well, so I’m not sure how you were using the containers or what you were expecting. For my workflows on HPC, I use apptainers as basically drop-in replacements for Docker, and for that, they work quite well. These biggest benefit is that the containers are unprivileged. This means you can’t do a lot of things (in particular complex networking), but it also makes it much more secure for multi-tenant systems (like HPC). (I know Docker and Apptainer are slightly different beasts, but I’m speaking in broad strokes in a general sense without extra permissions).

d3Xt3r 1y ago
You can also run Docker itself in rootless mode[1]. And if for some reason you don't want to run Docker, you can also use Podman or Incus instead, and they both support Docker images, as well as running unpriviliged. Finally, there's also Flox[2], which is a Nix-based application Sandbox, that I believe would align more towards your (and OP's) use case (unless you specifically require Docker image compatibility). So unfortunately your example doesn't illustrate why Apptainer is a better option. [1] https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/ [2] https://flox.dev/ https://flox.dev/
- mbreese 1y ago
  For a long time you couldn’t run Docker in rootless mode. So Singularity/Apptainer was developed so that the benefit of containers could be used with shared HPC clusters. Podman also didn’t exist at the time. I don’t know why we should have waited for docker to get around to supporting a more secured use-case. With Docker, security has always seemed like an afterthought, which seemed perfectly reasonable considering their original use-case. But that’s not tenable on $$$ HPC clusters with thousands of users. So, the HPC world developed their own solution, better adapted to that environment.

0xbadcafebee 1y ago

You don't mix and match pieces of containers, just like you wouldn't mix and match binaries from different distributions of Linux. You can use a container as a single environment in which to do development, and that works fine. But they are by definition an isolated environment with different dependencies than other containers. The result of compiling something in a container necessarily needs to end up in its own container. ...that said, you could use the exact same container base image, and make many different container images from it, and those files would be compatible (assuming you shipped all needed dependencies).

IshKebab 1y ago
> you wouldn't mix and match binaries from different distributions of Linux. You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. > and make many different container images from it Well yes, that's the problem. You end up either putting everything in one container (in which case why bother with a container?), or with a combinatorial explosion of every piece and version of software you might use. TCL modules are better. They don't let you cheat like containers do, but in return you get a better system.
- 0xbadcafebee 1y ago
  > You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules. Doing this across different Linux distributions is inherently prone to failure. I don't know about your TCL modules specifically, but unless you have an identical and completely reproducible software toolchain across multiple linux distributions, it's going to end with problems. Honestly, it sounds like you just don't understand these systems and how they work. TCL modules aren't better than containers, this is like comparing apples and organgutans.
  - IshKebab 1y ago
    > Doing this across different Linux distributions is inherently prone to failure. Sure if you just take binaries from one distro that link with libraries on that distro and try and run it on a different one... But that's not what we're doing. All of our TCL modules are either portable binaries (e.g. commercial software) or compiled from source. > Honestly, it sounds like you just don't understand these systems and how they work. I do, but well done for being patronising. > TCL modules aren't better than containers, They are better for our use case. > this is like comparing apples and organgutans. If apples and orangutans were potential solutions to a single problem why couldn't you compare them?

Fokamul 1y ago

So you are using container app and your biggest problem with it is, that it's doing exactly as advertised -> containers :D

IshKebab 1y ago
If you want to be unnecessarily dismissive about the problems with containers, sure.
- elmolino89 1y ago
  Frankly I have a problem with one of your problem as described. Who on the planet Earth creates a container with GCC, may I guess, to compile programs? and then complains about make being located in another container. If you miss some utility just convert the apptainer container to a sandbox, install utilities you need and convert the sandbox to .sif as needed. Also the whole point of building the program using compilers/libraries in the container is to use such container to run the aforementioned program in that very environment and not worry about libraryX or utility Y not installed in some box in a galaxy far, far away...
  - IshKebab 1y ago
    > Also the whole point of building the program using compilers/libraries in the container is to use such container to run the aforementioned program in that very environment Says who? That's only because that's the only way it works, not because it's desirable to only work like that. The only way Apptainers work well is if you put everything you need in one Apptainer, which makes them much less useful than you might imagine.

deleted 1y ago

[deleted]

je42 1y ago

How do you guys use Lua modules?

IshKebab 1y ago
We put all of our software in a module and then we have a script for our projects that loads the appropriate software. I'm not sure I understand the question...