5 ms·

I don't understand why more people don't use Solaris Zones, they seem to me to be the superior solution by far, and with work done by Joyent you now have modern

I don't understand why more people don't use Solaris Zones, they seem to me to be the superior solution by far, and with work done by Joyent you now have modern LX-branded zones also. Is the lack of adoption mainly due to the fact that it's Solaris, and not Linux? (Solaris lives on in Illumos et al)

eikenberry 9y ago
Solaris is not free software and the free software forks never gained much traction. Plus it is now also associated with Oracle which brings along a lot of extra distrust, particularly given how litigious they are.
- willejs 9y ago
  FreeBSD jails it is then.
  - prewett 9y ago
    I don't know anything about Solaris Zones, but I'm guessing that the FreeBSD jails wouldn't help with the problem in the article. It's still the same kernel running in the jail, so the fadvise calls are going to bottleneck in the same place.
    - Skunkleton 9y ago
      Why would an entirely different piece of software have the same performance bottlenecks?
      - rbranson 9y ago
        Because the problem isn’t that there is one bug in a specific implementation, it’s that the whole model is fundamentally prone to this type of problem. cgroups, Jails, and Zones all suffer from having to cover an immense surface area. Contrast with VMs, which only require managing a few, significantly simpler interfaces. There are definitely differences in quality, but they all use a similar approach.
        Skunkleton 9y ago
        OP says that fadvise is going to bottleneck in the same places on BSD that it does under Linux, with no supporting evidence. I agree that containers in general are a tough problem.
      - jbn 9y ago
        Maybe because https://blog.regehr.org/archives/303 https://blog.regehr.org/archives/303 (N-version programming often exhibits the same defects...)
- travbrack 9y ago
  Illumos/SmartOS is FOSS
  - sigjuice 9y ago
    Not entirely FOSS. Looks like you need some binaries to actually build it. https://wiki.illumos.org/display/illumos/How+To+Build+illumos#HowToBuildillumos-Gettingclosedbinaries https://wiki.illumos.org/display/illumos/How+To+Build+illumo...
  - aaronchall 9y ago
    I remember being told Java is FOSS too... now tell that to Google...
    - asdbffg 9y ago
      Java is FOSS. Oracle used excuse, that it's current OpenJDK license (GPLv2) is incompatible with license, used by Google's runtime (Apache 2). If Google re-licensed it's Java implementation under GPL, some of arguments, used by Oracle lawyers (code reuse and patent (?) violations), would have been void, and arguing about reuse of APIs would have been a lot harder. Of course, this does not really matter, because the whole lawsuit is just excuse for power games between corporations. Oracle's goal wasn't about Java licensing, it was about gaining some degree of control over emerging Android ecosystem.
      - oneweekwonder 9y ago
        Can google pull a swift move?
        jordanrobinson 9y ago
        Many would say they already are with Kotlin.
        monocasa 9y ago
        No, the copyright trial is about the "structure, sequence, and organization" of the APIs, not any literal copying anymore. Switching languages, but keeping the same class library still leaves them open.
jbergstroem 9y ago
Last I used Illumos it had pretty poor hardware support (at least at entry to midlevel hardware which for obvious reasons must be a less important priority). I also ran into issues with their KVM port; mainly panics with various linux kernels (kvm clock, virt*, ..). Good experience in general though, especially the zones/zfs/kvm combo.
- mgerdts 9y ago
  At Joyent, we are working on bringing bhyve to SmartOS and illumos. Much attention is being paid to performance and stability. The plan of record is for this to eventually replace KVM. If you are itching for some slides, http://bhyvecon.org/bhyvecon2018-Gwydir.pdf http://bhyvecon.org/bhyvecon2018-Gwydir.pdf We have some work to do before we are ready for widespread use. As those pieces fall in place, we'll announce on smartos-discuss@lists.smartos.org.
sigjuice 9y ago
Because Linux userspace probably works best on the Linux kernel?
paxy 9y ago
And deploy it...where? AWS, Azure, GCP and virtually all other providers have first class Docker and Kubernetes support. You can barely even host a Solaris VM anywhere today.
- benjaminl 9y ago
  That would by Joyent [0] the makers of SmartOS (Basically, Open Source Solaris). Their Triton cloud product uses SmartOS under the hood. [0] - https://www.joyent.com/smartos https://www.joyent.com/smartos
  - oblio 9y ago
    So your solution is 1 vendor? Doesn’t seem too realistic.
  - amdavidson 9y ago
    They're not exactly competitive: https://www.joyent.com/pricing/cloud/compute https://www.joyent.com/pricing/cloud/compute https://www.digitalocean.com/pricing/ https://www.digitalocean.com/pricing/
    - benjaminl 9y ago
      They claim to be half the price of AWS [0]. Digital Ocean isn't relevant here as they don't offer container hosting. [0] - https://www.joyent.com/pricing/compare https://www.joyent.com/pricing/compare
      - amdavidson 9y ago
        In your link, they're comparing to EC2 VMs not Amazon's Container hosting. Their whole premise is that their containers are equivalent to/better than VMs[0]. 0: https://www.joyent.com/blog/understanding-triton-containers https://www.joyent.com/blog/understanding-triton-containers
rdtsc 9y ago
> Is the lack of adoption mainly due to the fact that it's Solaris, and not Linux? I am guessing it is to avoid to learn a whole new ecosystem, tools, environments, rules, package system etc. It's just simpler to stick to Linux. But it all depends, let's say if my application on Illumos shows a 60% performance improvement, well I can see spending time learning it and using it as a base. But it would really have to be large benefit to justify switching OSes. Of course how would I even bother benchmarking to start with? I'd probably have to hear other stories or mentions on HN and such...
mgerdts 9y ago
Zones are great and they are often the right solution. There are limits. If you have the choice of putting 100 zones on one 4 socket system or 25 zones on each of four 1 socket systems, I'd probably recommend going with the four systems. Hot locks and inefficient algorithms happen. For instance, the directory name lookup cache can greatly impact the performance of file operations. This is what makes it so that when you open /a/b/c/d/e/blah, the OS has a good chance of knowing how to open "blah" directly without first searching /a, /a/b, /a/b/c, /a/b/c/d, and /a/b/c/d/e. I don't have performance numbers handy (left at $job - 1 and $job - 2), but the default size is not great for a system that handles hundreds of thousands of files. If that sounds like a lot of files, count how many files are read as part of a reasonably large build. Then imagine there are dozens of them running concurrently. Or imagine that you are hosting git repos or a bunch of static web content. The obvious answer is to just increase the size of the DNLC. The problem is that there are other parts of the system that behave poorly with a large DNLC. For instance, whenever anyone tries to unmount a file system, dnlc_purge_vfsp() is called. This walks the cache, looking for entries that are associated with the file system. Who cares, right? We hardly ever unmount file systems. Well, if you use the automounter (by default Solaris uses it for at least /home/*), every few minutes it is trying to unmount all of the automounted file systems. The purge happens before EBUSY can be detected so hot cached entries may be purged while adding contention on dnlc-related locks. What's worse, the automounter doesn't reset its inactivity timer when it hits EBUSY, causing more frequent attempts to unmount than you would otherwise expect. There are other DNLC bottlenecks on a Solaris NFS server when a client removes a file. And then as you get to larger systems with more NUMA effects, these type of operations become even more expensive. Operating systems are hard. Making them scale to an infinite number of processes and processors is impossible. At a certain point, it becomes beneficial to use smaller hardware and/or add VMs into the mix.
ofrzeta 9y ago
It lives but it's not exactly thriving. OpenIndiana struggled, OmniOS gave up (and got picked up again) but still all that struggle makes me expect that Illumos will eventually fade out. At the latest when/if Joyent goes out of business. It's a shame because I like SmartOS a lot but it nowhere has the momentum Linux has. I don't see who will write all the device drivers for the next generation of hardware coming out.

eikenberry 9y ago

Solaris is not free software and the free software forks never gained much traction. Plus it is now also associated with Oracle which brings along a lot of extra distrust, particularly given how litigious they are.

willejs 9y ago
FreeBSD jails it is then.
- prewett 9y ago
  I don't know anything about Solaris Zones, but I'm guessing that the FreeBSD jails wouldn't help with the problem in the article. It's still the same kernel running in the jail, so the fadvise calls are going to bottleneck in the same place.
  - Skunkleton 9y ago
    Why would an entirely different piece of software have the same performance bottlenecks?
    - rbranson 9y ago
      Because the problem isn’t that there is one bug in a specific implementation, it’s that the whole model is fundamentally prone to this type of problem. cgroups, Jails, and Zones all suffer from having to cover an immense surface area. Contrast with VMs, which only require managing a few, significantly simpler interfaces. There are definitely differences in quality, but they all use a similar approach.
      - Skunkleton 9y ago
        OP says that fadvise is going to bottleneck in the same places on BSD that it does under Linux, with no supporting evidence. I agree that containers in general are a tough problem.
    - jbn 9y ago
      Maybe because https://blog.regehr.org/archives/303 https://blog.regehr.org/archives/303 (N-version programming often exhibits the same defects...)
travbrack 9y ago
Illumos/SmartOS is FOSS
- sigjuice 9y ago
  Not entirely FOSS. Looks like you need some binaries to actually build it. https://wiki.illumos.org/display/illumos/How+To+Build+illumos#HowToBuildillumos-Gettingclosedbinaries https://wiki.illumos.org/display/illumos/How+To+Build+illumo...
- aaronchall 9y ago
  I remember being told Java is FOSS too... now tell that to Google...
  - asdbffg 9y ago
    Java is FOSS. Oracle used excuse, that it's current OpenJDK license (GPLv2) is incompatible with license, used by Google's runtime (Apache 2). If Google re-licensed it's Java implementation under GPL, some of arguments, used by Oracle lawyers (code reuse and patent (?) violations), would have been void, and arguing about reuse of APIs would have been a lot harder. Of course, this does not really matter, because the whole lawsuit is just excuse for power games between corporations. Oracle's goal wasn't about Java licensing, it was about gaining some degree of control over emerging Android ecosystem.
    - oneweekwonder 9y ago
      Can google pull a swift move?
      - jordanrobinson 9y ago
        Many would say they already are with Kotlin.
      - monocasa 9y ago
        No, the copyright trial is about the "structure, sequence, and organization" of the APIs, not any literal copying anymore. Switching languages, but keeping the same class library still leaves them open.

jbergstroem 9y ago

Last I used Illumos it had pretty poor hardware support (at least at entry to midlevel hardware which for obvious reasons must be a less important priority). I also ran into issues with their KVM port; mainly panics with various linux kernels (kvm clock, virt*, ..). Good experience in general though, especially the zones/zfs/kvm combo.

mgerdts 9y ago
At Joyent, we are working on bringing bhyve to SmartOS and illumos. Much attention is being paid to performance and stability. The plan of record is for this to eventually replace KVM. If you are itching for some slides, http://bhyvecon.org/bhyvecon2018-Gwydir.pdf http://bhyvecon.org/bhyvecon2018-Gwydir.pdf We have some work to do before we are ready for widespread use. As those pieces fall in place, we'll announce on smartos-discuss@lists.smartos.org.

sigjuice 9y ago

Because Linux userspace probably works best on the Linux kernel?

paxy 9y ago

And deploy it...where? AWS, Azure, GCP and virtually all other providers have first class Docker and Kubernetes support. You can barely even host a Solaris VM anywhere today.

benjaminl 9y ago
That would by Joyent [0] the makers of SmartOS (Basically, Open Source Solaris). Their Triton cloud product uses SmartOS under the hood. [0] - https://www.joyent.com/smartos https://www.joyent.com/smartos
- oblio 9y ago
  So your solution is 1 vendor? Doesn’t seem too realistic.
- amdavidson 9y ago
  They're not exactly competitive: https://www.joyent.com/pricing/cloud/compute https://www.joyent.com/pricing/cloud/compute https://www.digitalocean.com/pricing/ https://www.digitalocean.com/pricing/
  - benjaminl 9y ago
    They claim to be half the price of AWS [0]. Digital Ocean isn't relevant here as they don't offer container hosting. [0] - https://www.joyent.com/pricing/compare https://www.joyent.com/pricing/compare
    - amdavidson 9y ago
      In your link, they're comparing to EC2 VMs not Amazon's Container hosting. Their whole premise is that their containers are equivalent to/better than VMs[0]. 0: https://www.joyent.com/blog/understanding-triton-containers https://www.joyent.com/blog/understanding-triton-containers

rdtsc 9y ago

> Is the lack of adoption mainly due to the fact that it's Solaris, and not Linux? I am guessing it is to avoid to learn a whole new ecosystem, tools, environments, rules, package system etc. It's just simpler to stick to Linux. But it all depends, let's say if my application on Illumos shows a 60% performance improvement, well I can see spending time learning it and using it as a base. But it would really have to be large benefit to justify switching OSes. Of course how would I even bother benchmarking to start with? I'd probably have to hear other stories or mentions on HN and such...

mgerdts 9y ago

Zones are great and they are often the right solution. There are limits. If you have the choice of putting 100 zones on one 4 socket system or 25 zones on each of four 1 socket systems, I'd probably recommend going with the four systems. Hot locks and inefficient algorithms happen. For instance, the directory name lookup cache can greatly impact the performance of file operations. This is what makes it so that when you open /a/b/c/d/e/blah, the OS has a good chance of knowing how to open "blah" directly without first searching /a, /a/b, /a/b/c, /a/b/c/d, and /a/b/c/d/e. I don't have performance numbers handy (left at $job - 1 and $job - 2), but the default size is not great for a system that handles hundreds of thousands of files. If that sounds like a lot of files, count how many files are read as part of a reasonably large build. Then imagine there are dozens of them running concurrently. Or imagine that you are hosting git repos or a bunch of static web content. The obvious answer is to just increase the size of the DNLC. The problem is that there are other parts of the system that behave poorly with a large DNLC. For instance, whenever anyone tries to unmount a file system, dnlc_purge_vfsp() is called. This walks the cache, looking for entries that are associated with the file system. Who cares, right? We hardly ever unmount file systems. Well, if you use the automounter (by default Solaris uses it for at least /home/*), every few minutes it is trying to unmount all of the automounted file systems. The purge happens before EBUSY can be detected so hot cached entries may be purged while adding contention on dnlc-related locks. What's worse, the automounter doesn't reset its inactivity timer when it hits EBUSY, causing more frequent attempts to unmount than you would otherwise expect. There are other DNLC bottlenecks on a Solaris NFS server when a client removes a file. And then as you get to larger systems with more NUMA effects, these type of operations become even more expensive. Operating systems are hard. Making them scale to an infinite number of processes and processors is impossible. At a certain point, it becomes beneficial to use smaller hardware and/or add VMs into the mix.

ofrzeta 9y ago

It lives but it's not exactly thriving. OpenIndiana struggled, OmniOS gave up (and got picked up again) but still all that struggle makes me expect that Illumos will eventually fade out. At the latest when/if Joyent goes out of business. It's a shame because I like SmartOS a lot but it nowhere has the momentum Linux has. I don't see who will write all the device drivers for the next generation of hardware coming out.