5 ms·

How does this work? Fundamentally, containers are about namespace/isolation of a bunch of OS interfaces, so file system functions, network functions, memory ma

How does this work? Fundamentally, containers are about namespace/isolation of a bunch of OS interfaces, so file system functions, network functions, memory management, process functions, etc, can all pretend like they're the only game in town, but crucially without having to virtualize out the kernel. Does XNU have such namespacing functionality across all its interfaces? Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not. The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out: they actually run a Linux virtual machine (using Hypervisor.framework/hvf), and have that just provide the container environment. Is that what this project is doing? But then, how could it run a macOS container? [1] based on the foundation that Linux, unlike BSDs, has a stable syscall interface!

deleted 3y ago
[deleted]
robertlagrant 3y ago
The transitively linked docs have some answers: rund is an experimental containerd shim for running macOS containers on macOS. rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API. What rund provides: - Filesystem isolation via chroot(2) - Cleanup of container processes using process group - OCI Runtime Specification compatibility (to the extent it is possible on macOS) - Host-network mode only - bind mounts
- znpy 3y ago
  So essentially a chroot with a bit of make-up and a lot of marketing? Except for bind mounts (not even overlayfs...) there isn't much interesting. > - Host-network mode only Yeah expect a lot of things to break in subtle ways... most containers are developed kinda expecting you have your own network namespace (and that no one else is using ports)
  - lloeki 3y ago
    overlayfs and bind mounts are orthogonal: - bind mounting solves the exposition of filesystem within the root pivot. - overlayfs solves the persistence efficiency issue using a layered union fs. > most containers are developed Most Linux (and Windows) containers. Since these are macOS containers there are no containers developed yet so by definition there is nothing to break.
  - slonopotamus 3y ago
    Original author here. > So essentially a chroot with a bit of make-up Well. 1. It is not trivial to properly set up a chroot on macOS. If you try to find a working guide/tool that works with modern macOS, I doubt you'll find anything (at least, I failed, even though tried very hard) 2. I believe that ability to package stuff into a Docker image distributable via already existing infrastructure and compatible with already existing tools maybe "a bit of make-up", but it is an important makeup. 3. Kubernetes recently got HostProcesses for Windows: https://kubernetes.io/blog/2022/12/13/windows-host-process-containers-ga/ https://kubernetes.io/blog/2022/12/13/windows-host-process-c.... They are even less isolated from host than chroot and still, people find them useful for certain scenarios. > and a lot of marketing? Thanks for "a lot of marketing", that made me chuckle. My own submission got buried yesterday with humble 8 points: https://news.ycombinator.com/item?id=37640688 https://news.ycombinator.com/item?id=37640688
    - e12e 3y ago
      Great effort. I get why you call it container - but sounds more like jail or cheroot would give more appropriate expectations; like "tooling to build and run Darwin containers in a macOS chroot"?
      - slonopotamus 3y ago
        I didn't want to use "jail" term because it is mostly unheard of outside of FreeBSD. Container definition is very stretched nowadays. Look at Windows HostProcesses in Kubernetes [1]. They don't have neither process, network nor device isolation from the host. I also plan to try macOS sandbox-exec tool, which should offer additional isolation from the host. [1]: https://kubernetes.io/blog/2022/12/13/windows-host-process-containers-ga/ https://kubernetes.io/blog/2022/12/13/windows-host-process-c...
- lloeki 3y ago
  > bind mounts via MacFuse
lijok 3y ago
https://github.com/macOScontainers/rund/blob/main/README.adoc https://github.com/macOScontainers/rund/blob/main/README.ado...
plextoria 3y ago
> Does XNU have such namespacing functionality across all its interfaces? I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK > Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not. This project is about running macOS containers on macOS. It's not about running Linux containers.
- lloeki 3y ago
  > > Does XNU have such namespacing functionality across all its interfaces? > I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK Theoretically, probably, for coarse-grained yes/no things? I don't think it's able to go much further than "you can use the local network and/or internet" and "you can read/write to the filesystem location corresponding to you bundle identifier `com.foo.bar`" but not "hey let me present you with a namespaced view of loopback or process list". Also not sure if it can be dynamically set by a parent process for a child? Seems like it's very bundle oriented (except maybe for Apple processes) so not very practical.
  - slonopotamus 3y ago
    > Also not sure if it can be dynamically set by a parent process for a child? Yes, it can. See sandbox-exec tool. And I actually plan to use it: https://github.com/macOScontainers/rund/issues/15 https://github.com/macOScontainers/rund/issues/15
    - lloeki 3y ago
      Oh my I totally forgot about sandbox-exec! I played with it some time ago, can't recall the context but it was about build systems / packaging (maybe nix?), doing the configure/make/make install with reduced privileges.
highwaylights 3y ago
This introduces a long-standing problem that is really the main issue I have with Docker/Podman/containers on anything other than Linux - you have a dedicated VM that needs memory to be set aside for it exclusively. This is fine if you have a 32/64GB machine, but less so on an 8GB non-upgradeable laptop. I get it - memory is relatively cheap these days - and manufacturers that are building memory-limited devices are really only doing it to fleece you on obscene upgrade fees at the time of purchase - but it would be nice if there was a more elegant solution to this on Windows and macOS. WSL 1 had a solution to this that clearly took a lot of work to put together, wherein they'd have a Linux kernel running side-by-side as a Windows sub-process so that the memory pool was shared. Unfortunately it might have been too much work as they scrapped it entirely for WSL 2 and just used essentially the same VM route. If anyone knows of any projects trying to work around that problem I'd love to hear about it. If Apple really wanted to bring the development community back on board, focusing on these kind of use cases would be great, sadly it seems someone over there has taken the view that scrapping butterfly keys and the touchbar is "enough". Say what you will about Microsoft, but they've focused really hard on developer use cases for decades, and it shows.
- themoonisachees 3y ago
  VM dynamic allocation of ram has been a standard feature of KVM, hyper-v and VMware for the better part of 10 years now.
- pjmlp 3y ago
  However WSL for Docker works differently depending on the containers configuration. When Windows containers are being used, it is just there to keep the docker daemon happy.
- lloeki 3y ago
  > If anyone knows of any projects trying to work around that problem I'd love to hear about it. Containers are namespaced processes. These processes exec against the corresponding kernel they require. There is no workaround: if you have an ELF binary calling Linux syscalls it can only run on a Linux kernel†, so to run that you need a VM††. It's not as bad as it appears thanks to memory ballooning†††. Conversely if you want to exec a Windows binary in a container, the Windows kernel needs to provide process namespacing features (which it does). And if you want to exec a Darwin binary in a container, then the Darwin kernel needs to provide process namespacing features (which it doesn't). † WSL1 was implementing the Linux syscall API on the Windows kernel, which proved to be much more complex than it appears to be. †† Or colinux (https://en.wikipedia.org/wiki/Cooperative_Linux https://en.wikipedia.org/wiki/Cooperative_Linux), or user-mode Linux (https://en.wikipedia.org/wiki/User-mode_Linux https://en.wikipedia.org/wiki/User-mode_Linux). ††† https://en.wikipedia.org/wiki/Memory_ballooning https://en.wikipedia.org/wiki/Memory_ballooning
dwattttt 3y ago
> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not. There is more to the container ecosystem than Linux containers; Windows native containers function much the same way (well, in two ways, with VM-backing or the traditional kernel syscall interface, but with Windows syscalls).
lloeki 3y ago
> How does this work? Essentially, like this: https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/ https://earthly.dev/blog/chroot/ https://earthly.dev/blog/chroot/ > The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out It's not, it's a requirement for running Linux containers: https://news.ycombinator.com/item?id=37656401 https://news.ycombinator.com/item?id=37656401 DfM is more like running the CLI locally to a remote Linux machine, and all it does is conveniently expose /Users in the same place through the VM folder share so that you have the convenient illusion that it happens locally. If Darwin had process namespacing features it would not make it magically able to run Linux processes.
kevincox 3y ago
I am also interested in the API boundary. It seems that there are too options: 1. Rely on system call stability. This is like Linux containers but unlike Linux macOS doesn't provide a stable system call API. So this would break when the system updates with a change that modifies the system call API. 2. Install the host libraries into the container at runtime. This should provide as much stability as macOS apps usually have. It may also be beneficial as you wouldn't be embedding these into every container. It seems like 2 would be preferable. However it may be a bit weird when building as the libraries you build against would be updated without the container being aware, but this is unlikely to break anything unless they are copying them to new paths which seems unlikely.
- slonopotamus 3y ago
  > macOS doesn't provide a stable system call API I'm really wondering, do you have any links about macOS syscall stability over versions?
  - kevincox 3y ago
    I do not. I'm not really a macOS user but have heard this mentioned many times. I remember one recent example is Golang switched from doing raw syscalls to using the macOS libc due to this issue. That was probably ~5 years ago now so you may be able to dig up examples there.
  - lgg 3y ago
    Generally speaking macOS does not guarantee syscall stability, and does not generally guarantee compatibility for any binaries not linked to `libSystem.dylib` (that is the supported ABI boundary)[1]. This has a number of implications, including (but not limited to): * The most obvious is the commonly mentioned fact that syscalls may change. Here is an example where golang program broke because they were directly using the `gettimeofday()` syscalls[2]. * The interface between the kernel and the dynamic linker (which is required since ABI stability for statically linked executables is not guaranteed) is private and may change between versions. That means if your chroot contains a `dyld` from an OS version that is not the same as the host kernel it may not work. * The format of the dyld shared cache changes most releases, which means you can't just use the old dyld that matches the host kernel in your chroot because it may not work with the dyld shared cache for the OS you are trying to run in the chroot. * The system maintains a number of security policies around platform binaries, and those binaries are enumerated as part of the static trust cache[3]. Depending on what you are doing and what permissions it needs you may not be able to even run the system binaries from another release of macOS. In practice you can often get away with a slight skew (~1 year), but you can rarely get away with skews of more than 2-3 years. [1]: https://developer.apple.com/library/archive/qa/qa1118/_index.html https://developer.apple.com/library/archive/qa/qa1118/_index... [2]: https://github.com/golang/go/issues/16606 https://github.com/golang/go/issues/16606 [3]: https://support.apple.com/guide/security/trust-caches-sec7d38fbf97/web https://support.apple.com/guide/security/trust-caches-sec7d3...

deleted 3y ago

[deleted]

robertlagrant 3y ago

The transitively linked docs have some answers: rund is an experimental containerd shim for running macOS containers on macOS. rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API. What rund provides: - Filesystem isolation via chroot(2) - Cleanup of container processes using process group - OCI Runtime Specification compatibility (to the extent it is possible on macOS) - Host-network mode only - bind mounts

znpy 3y ago
So essentially a chroot with a bit of make-up and a lot of marketing? Except for bind mounts (not even overlayfs...) there isn't much interesting. > - Host-network mode only Yeah expect a lot of things to break in subtle ways... most containers are developed kinda expecting you have your own network namespace (and that no one else is using ports)
- lloeki 3y ago
  overlayfs and bind mounts are orthogonal: - bind mounting solves the exposition of filesystem within the root pivot. - overlayfs solves the persistence efficiency issue using a layered union fs. > most containers are developed Most Linux (and Windows) containers. Since these are macOS containers there are no containers developed yet so by definition there is nothing to break.
- slonopotamus 3y ago
  Original author here. > So essentially a chroot with a bit of make-up Well. 1. It is not trivial to properly set up a chroot on macOS. If you try to find a working guide/tool that works with modern macOS, I doubt you'll find anything (at least, I failed, even though tried very hard) 2. I believe that ability to package stuff into a Docker image distributable via already existing infrastructure and compatible with already existing tools maybe "a bit of make-up", but it is an important makeup. 3. Kubernetes recently got HostProcesses for Windows: https://kubernetes.io/blog/2022/12/13/windows-host-process-containers-ga/ https://kubernetes.io/blog/2022/12/13/windows-host-process-c.... They are even less isolated from host than chroot and still, people find them useful for certain scenarios. > and a lot of marketing? Thanks for "a lot of marketing", that made me chuckle. My own submission got buried yesterday with humble 8 points: https://news.ycombinator.com/item?id=37640688 https://news.ycombinator.com/item?id=37640688
  - e12e 3y ago
    Great effort. I get why you call it container - but sounds more like jail or cheroot would give more appropriate expectations; like "tooling to build and run Darwin containers in a macOS chroot"?
    - slonopotamus 3y ago
      I didn't want to use "jail" term because it is mostly unheard of outside of FreeBSD. Container definition is very stretched nowadays. Look at Windows HostProcesses in Kubernetes [1]. They don't have neither process, network nor device isolation from the host. I also plan to try macOS sandbox-exec tool, which should offer additional isolation from the host. [1]: https://kubernetes.io/blog/2022/12/13/windows-host-process-containers-ga/ https://kubernetes.io/blog/2022/12/13/windows-host-process-c...
lloeki 3y ago
> bind mounts via MacFuse

lijok 3y ago

https://github.com/macOScontainers/rund/blob/main/README.adoc https://github.com/macOScontainers/rund/blob/main/README.ado...

plextoria 3y ago

> Does XNU have such namespacing functionality across all its interfaces? I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK > Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not. This project is about running macOS containers on macOS. It's not about running Linux containers.

lloeki 3y ago
> > Does XNU have such namespacing functionality across all its interfaces? > I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK Theoretically, probably, for coarse-grained yes/no things? I don't think it's able to go much further than "you can use the local network and/or internet" and "you can read/write to the filesystem location corresponding to you bundle identifier `com.foo.bar`" but not "hey let me present you with a namespaced view of loopback or process list". Also not sure if it can be dynamically set by a parent process for a child? Seems like it's very bundle oriented (except maybe for Apple processes) so not very practical.
- slonopotamus 3y ago
  > Also not sure if it can be dynamically set by a parent process for a child? Yes, it can. See sandbox-exec tool. And I actually plan to use it: https://github.com/macOScontainers/rund/issues/15 https://github.com/macOScontainers/rund/issues/15
  - lloeki 3y ago
    Oh my I totally forgot about sandbox-exec! I played with it some time ago, can't recall the context but it was about build systems / packaging (maybe nix?), doing the configure/make/make install with reduced privileges.

highwaylights 3y ago

This introduces a long-standing problem that is really the main issue I have with Docker/Podman/containers on anything other than Linux - you have a dedicated VM that needs memory to be set aside for it exclusively. This is fine if you have a 32/64GB machine, but less so on an 8GB non-upgradeable laptop. I get it - memory is relatively cheap these days - and manufacturers that are building memory-limited devices are really only doing it to fleece you on obscene upgrade fees at the time of purchase - but it would be nice if there was a more elegant solution to this on Windows and macOS. WSL 1 had a solution to this that clearly took a lot of work to put together, wherein they'd have a Linux kernel running side-by-side as a Windows sub-process so that the memory pool was shared. Unfortunately it might have been too much work as they scrapped it entirely for WSL 2 and just used essentially the same VM route. If anyone knows of any projects trying to work around that problem I'd love to hear about it. If Apple really wanted to bring the development community back on board, focusing on these kind of use cases would be great, sadly it seems someone over there has taken the view that scrapping butterfly keys and the touchbar is "enough". Say what you will about Microsoft, but they've focused really hard on developer use cases for decades, and it shows.

themoonisachees 3y ago
VM dynamic allocation of ram has been a standard feature of KVM, hyper-v and VMware for the better part of 10 years now.
pjmlp 3y ago
However WSL for Docker works differently depending on the containers configuration. When Windows containers are being used, it is just there to keep the docker daemon happy.
lloeki 3y ago
> If anyone knows of any projects trying to work around that problem I'd love to hear about it. Containers are namespaced processes. These processes exec against the corresponding kernel they require. There is no workaround: if you have an ELF binary calling Linux syscalls it can only run on a Linux kernel†, so to run that you need a VM††. It's not as bad as it appears thanks to memory ballooning†††. Conversely if you want to exec a Windows binary in a container, the Windows kernel needs to provide process namespacing features (which it does). And if you want to exec a Darwin binary in a container, then the Darwin kernel needs to provide process namespacing features (which it doesn't). † WSL1 was implementing the Linux syscall API on the Windows kernel, which proved to be much more complex than it appears to be. †† Or colinux (https://en.wikipedia.org/wiki/Cooperative_Linux https://en.wikipedia.org/wiki/Cooperative_Linux), or user-mode Linux (https://en.wikipedia.org/wiki/User-mode_Linux https://en.wikipedia.org/wiki/User-mode_Linux). ††† https://en.wikipedia.org/wiki/Memory_ballooning https://en.wikipedia.org/wiki/Memory_ballooning

dwattttt 3y ago

> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not. There is more to the container ecosystem than Linux containers; Windows native containers function much the same way (well, in two ways, with VM-backing or the traditional kernel syscall interface, but with Windows syscalls).

lloeki 3y ago

> How does this work? Essentially, like this: https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/ https://earthly.dev/blog/chroot/ https://earthly.dev/blog/chroot/ > The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out It's not, it's a requirement for running Linux containers: https://news.ycombinator.com/item?id=37656401 https://news.ycombinator.com/item?id=37656401 DfM is more like running the CLI locally to a remote Linux machine, and all it does is conveniently expose /Users in the same place through the VM folder share so that you have the convenient illusion that it happens locally. If Darwin had process namespacing features it would not make it magically able to run Linux processes.

kevincox 3y ago

I am also interested in the API boundary. It seems that there are too options: 1. Rely on system call stability. This is like Linux containers but unlike Linux macOS doesn't provide a stable system call API. So this would break when the system updates with a change that modifies the system call API. 2. Install the host libraries into the container at runtime. This should provide as much stability as macOS apps usually have. It may also be beneficial as you wouldn't be embedding these into every container. It seems like 2 would be preferable. However it may be a bit weird when building as the libraries you build against would be updated without the container being aware, but this is unlikely to break anything unless they are copying them to new paths which seems unlikely.

slonopotamus 3y ago
> macOS doesn't provide a stable system call API I'm really wondering, do you have any links about macOS syscall stability over versions?
- kevincox 3y ago
  I do not. I'm not really a macOS user but have heard this mentioned many times. I remember one recent example is Golang switched from doing raw syscalls to using the macOS libc due to this issue. That was probably ~5 years ago now so you may be able to dig up examples there.
- lgg 3y ago
  Generally speaking macOS does not guarantee syscall stability, and does not generally guarantee compatibility for any binaries not linked to `libSystem.dylib` (that is the supported ABI boundary)[1]. This has a number of implications, including (but not limited to): * The most obvious is the commonly mentioned fact that syscalls may change. Here is an example where golang program broke because they were directly using the `gettimeofday()` syscalls[2]. * The interface between the kernel and the dynamic linker (which is required since ABI stability for statically linked executables is not guaranteed) is private and may change between versions. That means if your chroot contains a `dyld` from an OS version that is not the same as the host kernel it may not work. * The format of the dyld shared cache changes most releases, which means you can't just use the old dyld that matches the host kernel in your chroot because it may not work with the dyld shared cache for the OS you are trying to run in the chroot. * The system maintains a number of security policies around platform binaries, and those binaries are enumerated as part of the static trust cache[3]. Depending on what you are doing and what permissions it needs you may not be able to even run the system binaries from another release of macOS. In practice you can often get away with a slight skew (~1 year), but you can rarely get away with skews of more than 2-3 years. [1]: https://developer.apple.com/library/archive/qa/qa1118/_index.html https://developer.apple.com/library/archive/qa/qa1118/_index... [2]: https://github.com/golang/go/issues/16606 https://github.com/golang/go/issues/16606 [3]: https://support.apple.com/guide/security/trust-caches-sec7d38fbf97/web https://support.apple.com/guide/security/trust-caches-sec7d3...