9 ms·

With Linux having caught up with key Solaris features in recent years (DTrace -> eBPF, Zones -> Namespaces, ZFS -> ZFS on Linux), I always thought that the main

by wicket 5y ago

With Linux having caught up with key Solaris features in recent years (DTrace -> eBPF, Zones -> Namespaces, ZFS -> ZFS on Linux), I always thought that the main reason to use Illumos would be first-class SPARC support. With that now dropped, I'm concerned that Illumos soon become irrelevant. Are there any compelling reasons left to use Illumos, other than being something for those who just want a free Solaris alternative?

cyberpunk 5y ago
Not really. SmartOS was nice for a while but tbh you may as well go with openstack or proxmox these days. I used it for hosting a lot of java over the years but these days everyone wants a k8s endpoint and really the kind of hypervisor you are running doesn’t really make a difference. Shame, it was nice tech.
pjmlp 5y ago
Solaris SPARC is one of the few OSes in production taming C with hardware memory tagging (ADI). With Linux that will eventually happen in ARM, but currently only Android is adopting it, who know when it will ever come to upstream enabled by default like on Android.
jsiepkes 5y ago
In my opinion Linux hasn't caught up. * Namespaces don't come close to FreeBSD jails or Solaris / Illumos Zones. There is a reason Docker hosters put their Docker tenants in different hardware VM's. Because the isolation is too weak. * Due to CDDL and GPL problems ZFS on Linux will always be hard to use making every update cycle like playing Russian roulette. And there are other benefits. Like SMF offers nice service management while not providing half an operating system like systemd.
- tptacek 5y ago
  The problem with this jails/zones stuff is that I don't know anyone who seriously trusts jails and zones for real multitenant workloads anyways. The dealbreaker problem remains a shared kernel attack surface between tenants. It's one thing to propose that Zones are better than namespaces (they probably are), but another thing to cross the threshold where the distinction is meaningful in practice.
  - jchw 5y ago
    Also, tools for improving Docker for multi-tenant workloads exists, like gVisor. I don’t think equivalents exist for jails/zones really.
    - tptacek 5y ago
      gVisor isn't a shared-kernel multitenant system; it's essentially kernel emulation. It's a much stronger design.
      - jchw 5y ago
        I mostly mean that it is intended to be a solution to run containers from multiple tenants on the same host. Though I do agree, being essentially a kernel in itself, it is a bit in a different wheelhouse. It still is a huge value add that you can implement something like that on top of Docker, imo.
        tptacek 5y ago
        You can run container workloads in "real" VMs too; for instance, check out Kata Containers. Containers are a way of packaging applications; confusingly, they happen to also have a reference standard runtime associated with them. But you don't have to use it.
        jchw 5y ago
        Of course, and that's the value of the abstraction to me. Docker itself is obviously nothing to do with the Linux container technologies themselves that make up the equivalent functionality of FreeBSD jails, but I'm not aware of any equivalent abstraction that works around jails or zones even though it might be possible. So the way I see containers on Linux is not literally a composition of kernel features like cgroups or seccomp, but as an abstract thing that can be composed out of various primitives. And in practice, there's a number of different runtimes around it, including Docker clones like Podman, or tools that manage effectively chroots much closer to what you would do with jails. That said, I could just be completely wrong, and there could be similar things that can be done using jails and zones. But when I looked around for similar art with FreeBSD jails, either with regards to Docker's style of packaging and distribution, or with regards to additional layers like gVisor, it didn't seem like a thing well-suited to that kind of composition. In comparison, jails, at least, seem kind of like more powerful chroots. To me this is a pretty big difference versus Linux "containers".
- laumars 5y ago
  > In my opinion Linux hasn't caught up. I completely agree. I love Linux and it’s easily my preferred desktop OS but when it comes to stuff like ZFS, containerisation and other enterprise features, FreeBSD and Solaris are just more unified and consistent. A lot of it has to do with Linux being a hodgepodge of different contributors resulting in every feature effectively being a 3rd party feature. Which I think is the problem Pottering was trying to solve. And in many ways that’s quite a strength too. But ultimately it boils down to the old Perl mantra of “There's more than one way to do it” and how it’s fun for hackers but FreeBSD et al add the “but sometimes consistency is not a bad thing either” part of the mantra doesn’t too. https://en.m.wikipedia.org/wiki/There%27s_more_than_one_way_to_do_it https://en.m.wikipedia.org/wiki/There%27s_more_than_one_way_...
- wicket 5y ago
  > Namespaces don't come close to FreeBSD jails or Solaris / Illumos Zones. There is a reason Docker hosters put their Docker tenants in different hardware VM's. Because the isolation is too weak. This is largely a myth, please provide an namespace-related CVE that has gone unpatched to support your argument. The reason they run as VMs is that hypervisors run on ring 0 and require higher privileges than the kernel, therefore they are naturally more secure. Like Namespaces, Zones and Jails are also managed by their respective kernels. If there were any major hosters running managed services for Zones and Jails, you can bet they would implement them in a similar way. > Due to CDDL and GPL problems ZFS on Linux will always be hard to use making every update cycle like playing Russian roulette. You're right in that the CDDL causes complication but I don't consider this to be a compelling reason to use Illumos. Many who want to use ZFS on Linux will use it and get it to work despite the licensing issues and complications. > Like SMF offers nice service management while not providing half an operating system like systemd. SMF is relatively nice (apart from the use of XML) and like you, I would not touch systemd barge pole. Despite systemd making a lot noise in major distros, there are plenty alternative distros for those of us who don't want to use it. Don't get me wrong, I'm a Solaris guy, it made my career. I just fear that by dropping SPARC, Illumos have put the final nail in their own coffin.
  - jclulow 5y ago
    > I just fear that by dropping SPARC, Illumos have put the final nail in their own coffin. If that were true, most illumos users today would be SPARC users. There would be more than a couple of people working on SPARC support, and not merely as a part-time hobby. There would be software support for a SPARC machine that was sold some time after 2011. Instead, something like 99% of the people running illumos are doing so on 64-bit x86 machines. Dropping SPARC support will allow us to move forward much more easily with enhancements to the dramatically more relevant x86 bits. If anything I expect it will allow us to do interesting things that would garner new interest, like using Rust to implement bits of the operating system.
    - wicket 5y ago
      Thanks for the reply. That surprises me but I'm glad to hear that x86 support is strong. It does lead me to wonder, what sort of things are people using Illumos for? Maybe it's time I checked it out. :)
nimbius 5y ago
>ZFS on Linux its been said in the thread already but this was always a non-starter. Torvalds even said so himself. CDDL was the last poison pill of a dying giant who couldnt pull its foot from the well. What we, er, the linux community, chose instead, was BTRFS. It isnt ZFS, but its made incredible strides. for most use cases, it is a reasonable and working replacement for ZFS.
- _hyn3 5y ago
  I lost data and had to restore from 12 hour old backups one too many times with BTRFS. XFS + Ext4 for me from here on out, but that's one of the great things about Linux: lots of choices.
  - nix23 5y ago
    XFS all the way down...or ZFS (on FreeBSD)
  - acomjean 5y ago
    Lamentably, for those of us that just use linux, the lots of choices seem weird and frankly a little scary (Stories of data loss with BTRFS I have heard before.) I use the default Ext4? I think? But as a believer in open source, I really would love it if the choices we had were so much superior to the proprietary stuff that was out there that it made using an open source OS a no brainer.
- nix23 5y ago
  >Torvalds even said so himself. Torvald's comment about ZFS was as uninformed as it gets...and he calls himself an FS-Guy ;(
  - arid_eden 5y ago
    His comment was more on the wisdom (or otherwise) of running an out of tree filesystem. I think its hard to disagree with him. He went on to say you would never be able to merge the ZFS tree with Linux. Again he's the one who would know what code gets in Linux. His only actual comment against ZFS was that benchmarks didn't look great - which is unsurprising given all the extra work ZFS is doing wrt data integrity than other filesystems in production use. https://www.realworldtech.com/forum/?threadid=189711&curpostid=189841 https://www.realworldtech.com/forum/?threadid=189711&curpost...
shrubble 5y ago
As someone who has run both side by side on literally identical hardware... no, Linux has not caught up. Lxc containers vs Solaris Zones... zones clearly wins. SMF vs systemd (I know that you didn't include this, but it matters)... SMF is clearly superior as well.

by wicket 5y ago

cyberpunk 5y ago

Not really. SmartOS was nice for a while but tbh you may as well go with openstack or proxmox these days. I used it for hosting a lot of java over the years but these days everyone wants a k8s endpoint and really the kind of hypervisor you are running doesn’t really make a difference. Shame, it was nice tech.

pjmlp 5y ago

Solaris SPARC is one of the few OSes in production taming C with hardware memory tagging (ADI). With Linux that will eventually happen in ARM, but currently only Android is adopting it, who know when it will ever come to upstream enabled by default like on Android.

jsiepkes 5y ago

In my opinion Linux hasn't caught up. * Namespaces don't come close to FreeBSD jails or Solaris / Illumos Zones. There is a reason Docker hosters put their Docker tenants in different hardware VM's. Because the isolation is too weak. * Due to CDDL and GPL problems ZFS on Linux will always be hard to use making every update cycle like playing Russian roulette. And there are other benefits. Like SMF offers nice service management while not providing half an operating system like systemd.

tptacek 5y ago
The problem with this jails/zones stuff is that I don't know anyone who seriously trusts jails and zones for real multitenant workloads anyways. The dealbreaker problem remains a shared kernel attack surface between tenants. It's one thing to propose that Zones are better than namespaces (they probably are), but another thing to cross the threshold where the distinction is meaningful in practice.
- jchw 5y ago
  Also, tools for improving Docker for multi-tenant workloads exists, like gVisor. I don’t think equivalents exist for jails/zones really.
  - tptacek 5y ago
    gVisor isn't a shared-kernel multitenant system; it's essentially kernel emulation. It's a much stronger design.
    - jchw 5y ago
      I mostly mean that it is intended to be a solution to run containers from multiple tenants on the same host. Though I do agree, being essentially a kernel in itself, it is a bit in a different wheelhouse. It still is a huge value add that you can implement something like that on top of Docker, imo.
      - tptacek 5y ago
        You can run container workloads in "real" VMs too; for instance, check out Kata Containers. Containers are a way of packaging applications; confusingly, they happen to also have a reference standard runtime associated with them. But you don't have to use it.
        jchw 5y ago
        Of course, and that's the value of the abstraction to me. Docker itself is obviously nothing to do with the Linux container technologies themselves that make up the equivalent functionality of FreeBSD jails, but I'm not aware of any equivalent abstraction that works around jails or zones even though it might be possible. So the way I see containers on Linux is not literally a composition of kernel features like cgroups or seccomp, but as an abstract thing that can be composed out of various primitives. And in practice, there's a number of different runtimes around it, including Docker clones like Podman, or tools that manage effectively chroots much closer to what you would do with jails. That said, I could just be completely wrong, and there could be similar things that can be done using jails and zones. But when I looked around for similar art with FreeBSD jails, either with regards to Docker's style of packaging and distribution, or with regards to additional layers like gVisor, it didn't seem like a thing well-suited to that kind of composition. In comparison, jails, at least, seem kind of like more powerful chroots. To me this is a pretty big difference versus Linux "containers".
laumars 5y ago
> In my opinion Linux hasn't caught up. I completely agree. I love Linux and it’s easily my preferred desktop OS but when it comes to stuff like ZFS, containerisation and other enterprise features, FreeBSD and Solaris are just more unified and consistent. A lot of it has to do with Linux being a hodgepodge of different contributors resulting in every feature effectively being a 3rd party feature. Which I think is the problem Pottering was trying to solve. And in many ways that’s quite a strength too. But ultimately it boils down to the old Perl mantra of “There's more than one way to do it” and how it’s fun for hackers but FreeBSD et al add the “but sometimes consistency is not a bad thing either” part of the mantra doesn’t too. https://en.m.wikipedia.org/wiki/There%27s_more_than_one_way_to_do_it https://en.m.wikipedia.org/wiki/There%27s_more_than_one_way_...
wicket 5y ago
> Namespaces don't come close to FreeBSD jails or Solaris / Illumos Zones. There is a reason Docker hosters put their Docker tenants in different hardware VM's. Because the isolation is too weak. This is largely a myth, please provide an namespace-related CVE that has gone unpatched to support your argument. The reason they run as VMs is that hypervisors run on ring 0 and require higher privileges than the kernel, therefore they are naturally more secure. Like Namespaces, Zones and Jails are also managed by their respective kernels. If there were any major hosters running managed services for Zones and Jails, you can bet they would implement them in a similar way. > Due to CDDL and GPL problems ZFS on Linux will always be hard to use making every update cycle like playing Russian roulette. You're right in that the CDDL causes complication but I don't consider this to be a compelling reason to use Illumos. Many who want to use ZFS on Linux will use it and get it to work despite the licensing issues and complications. > Like SMF offers nice service management while not providing half an operating system like systemd. SMF is relatively nice (apart from the use of XML) and like you, I would not touch systemd barge pole. Despite systemd making a lot noise in major distros, there are plenty alternative distros for those of us who don't want to use it. Don't get me wrong, I'm a Solaris guy, it made my career. I just fear that by dropping SPARC, Illumos have put the final nail in their own coffin.
- jclulow 5y ago
  > I just fear that by dropping SPARC, Illumos have put the final nail in their own coffin. If that were true, most illumos users today would be SPARC users. There would be more than a couple of people working on SPARC support, and not merely as a part-time hobby. There would be software support for a SPARC machine that was sold some time after 2011. Instead, something like 99% of the people running illumos are doing so on 64-bit x86 machines. Dropping SPARC support will allow us to move forward much more easily with enhancements to the dramatically more relevant x86 bits. If anything I expect it will allow us to do interesting things that would garner new interest, like using Rust to implement bits of the operating system.
  - wicket 5y ago
    Thanks for the reply. That surprises me but I'm glad to hear that x86 support is strong. It does lead me to wonder, what sort of things are people using Illumos for? Maybe it's time I checked it out. :)

nimbius 5y ago

>ZFS on Linux its been said in the thread already but this was always a non-starter. Torvalds even said so himself. CDDL was the last poison pill of a dying giant who couldnt pull its foot from the well. What we, er, the linux community, chose instead, was BTRFS. It isnt ZFS, but its made incredible strides. for most use cases, it is a reasonable and working replacement for ZFS.

_hyn3 5y ago
I lost data and had to restore from 12 hour old backups one too many times with BTRFS. XFS + Ext4 for me from here on out, but that's one of the great things about Linux: lots of choices.
- nix23 5y ago
  XFS all the way down...or ZFS (on FreeBSD)
- acomjean 5y ago
  Lamentably, for those of us that just use linux, the lots of choices seem weird and frankly a little scary (Stories of data loss with BTRFS I have heard before.) I use the default Ext4? I think? But as a believer in open source, I really would love it if the choices we had were so much superior to the proprietary stuff that was out there that it made using an open source OS a no brainer.
nix23 5y ago
>Torvalds even said so himself. Torvald's comment about ZFS was as uninformed as it gets...and he calls himself an FS-Guy ;(
- arid_eden 5y ago
  His comment was more on the wisdom (or otherwise) of running an out of tree filesystem. I think its hard to disagree with him. He went on to say you would never be able to merge the ZFS tree with Linux. Again he's the one who would know what code gets in Linux. His only actual comment against ZFS was that benchmarks didn't look great - which is unsurprising given all the extra work ZFS is doing wrt data integrity than other filesystems in production use. https://www.realworldtech.com/forum/?threadid=189711&curpostid=189841 https://www.realworldtech.com/forum/?threadid=189711&curpost...

shrubble 5y ago

As someone who has run both side by side on literally identical hardware... no, Linux has not caught up. Lxc containers vs Solaris Zones... zones clearly wins. SMF vs systemd (I know that you didn't include this, but it matters)... SMF is clearly superior as well.