7 ms·

I wish there was a usable Docker Engine API implementation that could run containers in jails using zfs for image layering. There is a Docker for FreeBSD port,

I wish there was a usable Docker Engine API implementation that could run containers in jails using zfs for image layering. There is a Docker for FreeBSD port, but last time I checked, it was super old. Having Docker as a first class citizen on FreeBSD would make it really incredibly as a server OS.

zokula 6y ago
> Having Docker as a first class citizen on FreeBSD would make it really incredibly as a server OS. Now if only if FreeBSD wasnt an slow, unstable, glued together example of an OS.
- jen20 6y ago
  > glued together example of an OS This is a strange assertion when the implication is that Linux is somehow not "glued together" despite requiring distributions to literally glue the pieces together.
- nickexyz 6y ago
  > Now if only if FreeBSD wasnt an slow, unstable, glued together example of an OS. I really don't see how this kind of comment adds to the conversation. As jen20 wrote, GNU/Linux really is glued together. Linux is great, but FreeBSD are in my opinion both more stable and less "glued together". Since you have a clear separation between the system and user installed packages, there usually are less things that breaks. In my experience of course. To be fair, I was kind of anti BSD as well before giving it a real chance. In any case, I really hope the BSD:s continue to grow. One OS is never a good thing, and Linux seems to eat up most of the market right now.
- centimeter 6y ago
  FreeBSD's point of distinction is that it's carefully designed and fitted together, compared to Linux's comparatively heavy use of glue. I also haven't heard credible reports of it being slow, except for things it's not really used for (like drivers for consumer wifi cards).
  - nix23 6y ago
    Well Linux is just a kernel, so it's a bit unfair to call it glued together...but the rest (a ditro)...lot of glue and clay...sometimes also pure mud.
    - a1369209993 6y ago
      Don't forget the duct tape, bailing wire, and gum.
    - yjftsjthsd-h 6y ago
      Nobody uses just the kernel alone, so I think it's fair to consider the other required components when looking at the system.
      - mulmen 6y ago
        But FreeBSD is an operating system. The userland and kernel are developed together, in the same source tree AFAIK. With Linux you have to glue a user land on to use the kernel, which is a "distribution" which is actually an operating system.
        yjftsjthsd-h 6y ago
        Agreed? I thought we were talking about Linux
        mulmen 6y ago
        This is an article and thread about FreeBSD. Linux was invoked and compared to FreeBSD in a completely nonsensical way.
        nix23 6y ago
        Linux is a Kernel...like you said, are you talking about gnu/linux musl/gnu/linux musl/busybox/linux libc++/uClibc/glibc/musl/busybox/matchbox/gnu/rpm/deb/tar.gz any other distro/composition of software that runs on linux? You see the problem? Linux is a kernel, FreeBSD/Open/Net/Dragonfly are real OS's...with pkg's.
justincormack 6y ago
Every couple of years people express interest but no one has ever done the work. It is very much doable as the original port showed, and the zfs driver already exists for image layering as that works on Linux.
- takeda 6y ago
  I think that the people who work on it have goal to run linux in it and that causes the project to be more complex than it should. For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there, so why it should be different with FreeBSD? Yes, it would be great to run the same docker images on FreeBSD, but that should be a later step, and stuff like described in this wiki, could be used to accomplish that.
  - danieldk 6y ago
    For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there, Actually, it would with qemu user-mode emulation. This can be done transparently with binfmt_misc, which is a one or two-liner on many distributions. BuildKit now can now also automatically without any additional setup: https://github.com/moby/buildkit/pull/1516 https://github.com/moby/buildkit/pull/1516 But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images.
    - whizzter 6y ago
      > But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images. This is really an important point, MS failed to get WSL1 good enough so WSL2 was just a plain Linux kernel being virtualized. Maybe running a some partially virtualized linux kernel(Linux already has modes for this?) via bhyve relying on a host FreeBSD filesystem (using the already developed layering) might be an feasible approach that requires less Linux API chasing while retaining the "native" FS advantage at the expense of some extra memory (bhyve) ?
ArchOversight 6y ago
Maybe not docker, but there is pot: https://potluck.honeyguide.net https://potluck.honeyguide.net Which aims to provide something similar as Docker for FreeBSD.
- pizzamig 6y ago
  I'm the one behind pot, a framework to manage jails as containers. Here, there is the presentation on how to orchestrate jail using nomad, from Hashicorp: https://archive.fosdem.org/2020/schedule/event/orchestrating_jails/ https://archive.fosdem.org/2020/schedule/event/orchestrating... pot is nothing more than a Proof of Concept, a model that uses FreeBSD technologies to implements a typical container workflow: jail, ZFS, pf, rctl and cpuset. There is no compatibility to OCI standards, but I believe it would be possible to implement them.
- m_sahaf 6y ago
  There's also Bastille, which looks very interesting: https://bastillebsd.org/ https://bastillebsd.org/
takeda 6y ago
Maybe I'm wrong, but isn't this a key component required to have native support (without having to use a VM?) Frankly, though docker failed to deliver most of its promises, but at least it became a standardized deployable unit.
- jdc 6y ago
  macOS seems to do alright
  - djxfade 6y ago
    Docker on macOS uses a VM for the kernel. Its much slower than on a Linux host
  - takeda 6y ago
    MacOS runs a VM with Linux installation on it. That's why it takes so long to start, it is slow and uses so much CPU power and memory to run.
    - big-malloc 6y ago
      Not to mention the unusable volume performance
q3k 6y ago
There used to be a thing like this called jetpack [1], but unfortunately the project has died. [1] - https://github.com/3ofcoins/jetpack https://github.com/3ofcoins/jetpack
accountofme 6y ago
Having played with SmartOS containers using linux compatibility (not docker). I will say this: Use native containers for that OS. The slight differences that you will find, will drive you up the wall. They will be esoteric and happen at the worst time. And i should say I love SmartOS, it is a really cool bit of tech... I just love simplicity more, and a compatibility layer is never simple.
- hedora 6y ago
  Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO. The ancient rhel something something doesn’t like the debian other thing, and I even saw a changelog about broken kernel compatibility and flipping a flag in proc or cmdline but then I forgot what it said. Whatever. Life is short. Docker is in no way independent of the host kernel. Spend time debugging something else. SmartOS is great though. I wish they’d take over the world.
  - yjftsjthsd-h 6y ago
    > Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO. I would love any details you can provide; that sounds interesting. (I'm an OS nerd)
  - danieldk 6y ago
    To give this argument some more meat with an example: some older glibc versions use vsyscalls and not vDSO. And kernels in some modern distributions disable vsyscall emulation. This is a problem because people use CentOS 5 or 6 images to build binaries that are compatible with many historical glibc versions. If the host system does not enable vsyscall emulation, running binaries on such old distributions will lead to segfaults. (Which can be resolved by booting the host kernel with vsyscall=emulate.)
- cchdmdldkdjfdk 6y ago
  https://en.wikipedia.org/wiki/SmartOS https://en.wikipedia.org/wiki/SmartOS If linux comes out with tools remotely approaching the quality of either freebsd or solaris planning everyone should drop what they're doing to go watch. Linux ain't gonna improve itself, and the pains of docker are an excellent entry to pressuring linux devs to do anything that can improve userspace experience, an otherwise deeply unlikely prospect. It took them more than a decade to implement basic jailing after BSDs and the tech is still, shall we say, extremely rough....

zokula 6y ago

> Having Docker as a first class citizen on FreeBSD would make it really incredibly as a server OS. Now if only if FreeBSD wasnt an slow, unstable, glued together example of an OS.

jen20 6y ago
> glued together example of an OS This is a strange assertion when the implication is that Linux is somehow not "glued together" despite requiring distributions to literally glue the pieces together.
nickexyz 6y ago
> Now if only if FreeBSD wasnt an slow, unstable, glued together example of an OS. I really don't see how this kind of comment adds to the conversation. As jen20 wrote, GNU/Linux really is glued together. Linux is great, but FreeBSD are in my opinion both more stable and less "glued together". Since you have a clear separation between the system and user installed packages, there usually are less things that breaks. In my experience of course. To be fair, I was kind of anti BSD as well before giving it a real chance. In any case, I really hope the BSD:s continue to grow. One OS is never a good thing, and Linux seems to eat up most of the market right now.
centimeter 6y ago
FreeBSD's point of distinction is that it's carefully designed and fitted together, compared to Linux's comparatively heavy use of glue. I also haven't heard credible reports of it being slow, except for things it's not really used for (like drivers for consumer wifi cards).
- nix23 6y ago
  Well Linux is just a kernel, so it's a bit unfair to call it glued together...but the rest (a ditro)...lot of glue and clay...sometimes also pure mud.
  - a1369209993 6y ago
    Don't forget the duct tape, bailing wire, and gum.
  - yjftsjthsd-h 6y ago
    Nobody uses just the kernel alone, so I think it's fair to consider the other required components when looking at the system.
    - mulmen 6y ago
      But FreeBSD is an operating system. The userland and kernel are developed together, in the same source tree AFAIK. With Linux you have to glue a user land on to use the kernel, which is a "distribution" which is actually an operating system.
      - yjftsjthsd-h 6y ago
        Agreed? I thought we were talking about Linux
        mulmen 6y ago
        This is an article and thread about FreeBSD. Linux was invoked and compared to FreeBSD in a completely nonsensical way.
        nix23 6y ago
        Linux is a Kernel...like you said, are you talking about gnu/linux musl/gnu/linux musl/busybox/linux libc++/uClibc/glibc/musl/busybox/matchbox/gnu/rpm/deb/tar.gz any other distro/composition of software that runs on linux? You see the problem? Linux is a kernel, FreeBSD/Open/Net/Dragonfly are real OS's...with pkg's.

justincormack 6y ago

Every couple of years people express interest but no one has ever done the work. It is very much doable as the original port showed, and the zfs driver already exists for image layering as that works on Linux.

takeda 6y ago
I think that the people who work on it have goal to run linux in it and that causes the project to be more complex than it should. For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there, so why it should be different with FreeBSD? Yes, it would be great to run the same docker images on FreeBSD, but that should be a later step, and stuff like described in this wiki, could be used to accomplish that.
- danieldk 6y ago
  For example if you use Linux on ARM you still need to build new containers, because the x86_64 won't work there, Actually, it would with qemu user-mode emulation. This can be done transparently with binfmt_misc, which is a one or two-liner on many distributions. BuildKit now can now also automatically without any additional setup: https://github.com/moby/buildkit/pull/1516 https://github.com/moby/buildkit/pull/1516 But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images.
  - whizzter 6y ago
    > But I do understand your point, Linux emulation on FreeBSD may be incomplete/incorrect in places, so you have to address to problems at the same time. Unfortunately, I think this is also what people would expect from Docker support - the ability to pull arbitrary images. This is really an important point, MS failed to get WSL1 good enough so WSL2 was just a plain Linux kernel being virtualized. Maybe running a some partially virtualized linux kernel(Linux already has modes for this?) via bhyve relying on a host FreeBSD filesystem (using the already developed layering) might be an feasible approach that requires less Linux API chasing while retaining the "native" FS advantage at the expense of some extra memory (bhyve) ?

ArchOversight 6y ago

Maybe not docker, but there is pot: https://potluck.honeyguide.net https://potluck.honeyguide.net Which aims to provide something similar as Docker for FreeBSD.

pizzamig 6y ago
I'm the one behind pot, a framework to manage jails as containers. Here, there is the presentation on how to orchestrate jail using nomad, from Hashicorp: https://archive.fosdem.org/2020/schedule/event/orchestrating_jails/ https://archive.fosdem.org/2020/schedule/event/orchestrating... pot is nothing more than a Proof of Concept, a model that uses FreeBSD technologies to implements a typical container workflow: jail, ZFS, pf, rctl and cpuset. There is no compatibility to OCI standards, but I believe it would be possible to implement them.
m_sahaf 6y ago
There's also Bastille, which looks very interesting: https://bastillebsd.org/ https://bastillebsd.org/

takeda 6y ago

Maybe I'm wrong, but isn't this a key component required to have native support (without having to use a VM?) Frankly, though docker failed to deliver most of its promises, but at least it became a standardized deployable unit.

jdc 6y ago
macOS seems to do alright
- djxfade 6y ago
  Docker on macOS uses a VM for the kernel. Its much slower than on a Linux host
- takeda 6y ago
  MacOS runs a VM with Linux installation on it. That's why it takes so long to start, it is slow and uses so much CPU power and memory to run.
  - big-malloc 6y ago
    Not to mention the unusable volume performance

q3k 6y ago

There used to be a thing like this called jetpack [1], but unfortunately the project has died. [1] - https://github.com/3ofcoins/jetpack https://github.com/3ofcoins/jetpack

accountofme 6y ago

Having played with SmartOS containers using linux compatibility (not docker). I will say this: Use native containers for that OS. The slight differences that you will find, will drive you up the wall. They will be esoteric and happen at the worst time. And i should say I love SmartOS, it is a really cool bit of tech... I just love simplicity more, and a compatibility layer is never simple.

hedora 6y ago
Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO. The ancient rhel something something doesn’t like the debian other thing, and I even saw a changelog about broken kernel compatibility and flipping a flag in proc or cmdline but then I forgot what it said. Whatever. Life is short. Docker is in no way independent of the host kernel. Spend time debugging something else. SmartOS is great though. I wish they’d take over the world.
- yjftsjthsd-h 6y ago
  > Yeah. I can’t even get some Linux containers to run under $ESCOTERIC_DEBIAN_BASED_DISTRO. I would love any details you can provide; that sounds interesting. (I'm an OS nerd)
- danieldk 6y ago
  To give this argument some more meat with an example: some older glibc versions use vsyscalls and not vDSO. And kernels in some modern distributions disable vsyscall emulation. This is a problem because people use CentOS 5 or 6 images to build binaries that are compatible with many historical glibc versions. If the host system does not enable vsyscall emulation, running binaries on such old distributions will lead to segfaults. (Which can be resolved by booting the host kernel with vsyscall=emulate.)
cchdmdldkdjfdk 6y ago
https://en.wikipedia.org/wiki/SmartOS https://en.wikipedia.org/wiki/SmartOS If linux comes out with tools remotely approaching the quality of either freebsd or solaris planning everyone should drop what they're doing to go watch. Linux ain't gonna improve itself, and the pains of docker are an excellent entry to pressuring linux devs to do anything that can improve userspace experience, an otherwise deeply unlikely prospect. It took them more than a decade to implement basic jailing after BSDs and the tech is still, shall we say, extremely rough....