9 ms·

Joyent Open Sources SmartOS: Zones, ZFS, DTrace and KVM

279 points by timf 15y ago

wmf 15y ago
They're really going out of their way to not say "Solaris".
- po 15y ago
  They're really going out of their way to not use Solaris. SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level virtualization and next-generation KVM) with BSD package management and a GNU toolchain. I used Solaris for almost 5 years and when Sun was bought I thought for sure the cool parts of it were doomed. Illumos popped up and I thought that they didn't have a chance, it would wither and die. It's very nice to see that is not the case.
  - wmf 15y ago
    They're really going out of their way to not use Solaris. Considering that the Illumos kernel is basically the OpenSolaris kernel, I don't understand what you mean. They are using Solaris, but for some reason they're trying to hide it.
    - nknight 15y ago
      Illumos != Solaris, unless you'd like Oracle to beat you with a stick for trademark infringement. I suppose they could refer to it in technical documents as "the OpenSolaris kernel", but by that same logic, an OS using the DragonFly BSD kernel should say it's using "the FreeBSD kernel".
    - po 15y ago
      This project is certainly not using Solaris which is now an Oracle product (and trademark) and not open source. It is based on Illumos (which originally came from an OpenSolaris kernel as you said) but my understanding is that OpenSolaris as a project is dead. I don't think it's hiding something to name-check Illumos instead of OpenSolaris.
- killerswan 15y ago
  Trademark disputes with hyper-litigious competitors would not be fun...
- thirdhaf 15y ago
  Well since they have a lot of former Sun engineers working for them I don't really think they have to SAY Solaris. As others have mentioned there are trademarks involved here too. Sun/Oracle have been hemorrhaging talent, particularly the former Fishworks team, for a while now and a lot of those people are ending up in places like Joyent.
jjm 15y ago
Cool. I see this really helping me when testing my application prior to deployment. Would like to see some more movement with No.de. Have been waiting for a bit to get access.
nyellin 15y ago
To clarify, Joyent open sourced SmartOS. Zones, ZFS, DTrace, and KVM are maintained by other companies and have been open source for ages. Edit: That's not to imply Joyent is freeloading. There is another post on HN with an example of the awesome work they're doing: http://news.ycombinator.com/item?id=2887092 http://news.ycombinator.com/item?id=2887092
sciurus 15y ago
Can anyone either explain or link to material that explains the relationships between the different projects and distributions that have sprung up since Oracle discontinued OpenSolaris?
- piotrSikora 15y ago
  After OpenSolaris OS/Net kernel development was closed, 3 forks were made: Illumos, SchilliX-ON and Stormix (sadly, they all seem quite inactive). There are 3 distributions based on the first one: OpenIndiana, Nexenta and SmartOS. There is also SchilliX (based on SchilliX-ON) and StromOS (based on Stormix). This is just based on my existing knowledge, so there might be more forks and/or distributions out there.
  - binarycrusader 15y ago
    Both IllumOS and OpenIndiana are quite active (which shouldn't be surprising since the latter relies on the former).
    - piotrSikora 15y ago
      Is it, really? According to GitHub's commit history, there were 297 commits made since onnv_147, so ~1 commit/day. Prior to the fork, OpenSolaris was getting ~10 commits/day. To put this into perspective, there were over 8000 commits made to OpenBSD in the same time-frame. I won't even try to compare this with FreeBSD or Linux... Don't get me wrong, I love the effort, but at the same time I feel like there wasn't any real progress made since the project's inception.
      - binarycrusader 15y ago
        It depends on your definition of active. As an open source project, it's definitely active. Is it as active as the actual Solaris codebase? No. But it has also significantly less developers working on it.
mrb 15y ago
Wow this is huge. Having an OS be able to run both ZFS and KVM enables fantastic things. For example you can store each virtual machine disk template on a dedicated ZFS filesystem, and use "zfs clone" to rapidly deploy VMs (instead of using the KVM-level support for base disk images "qemu-img create -b ..."), as well as "zfs snapshot", "zfs revert", etc. The main advantage being that these clones and snapshots are possible while using the simple and fast "raw" KVM disk image format instead of the notoriously slower "qcow2" format that was, until today, the best format supporting base images and snapshots. I, for one, have been wanting to use ZFS specifically like that for a while. This does not compare at all to running ZFS on, say, an NFS server or iSCSI SAN serving data to a server running KVM (ZFS data integrity is only verified remotely on the storage server, it is slower, etc). Who ported KVM to Illumos? I know some old version of QEMU was running on Solaris at some point in the past, but I had no idea they had a full blown KVM port.
- amazingman 15y ago
  http://smartos.org/2011/08/15/kvm-on-illumos/ http://smartos.org/2011/08/15/kvm-on-illumos/
- bcantrill 15y ago
  KVM was ported to illumos by Max Bruning, Robert Mustacchi and me. Details are here: http://t.co/knW1UIJ http://t.co/knW1UIJ
  - rednaught 15y ago
    Can you provide more details of Crossbow? Who maintains it/status? I hadn't heard anything about it since OpenSolaris went dead.
    - bcantrill 15y ago
      Could you be more specific? It's there, it works, it's in shipping illumos-based products (from Joyent and others) and it's supported by those who ship it or stand it up. In that regard, it's like any other aspect of the system...
rednaught 15y ago
I'm having a difficult time wrapping my head around this. If Joyent wanted to stick with a UNIX then why not FreeBSD and help implement KVM there? Wasn't XEN already available for Dom0? I just don't see much mindshare in Illumos/SmartOS for keeping up with current server hardware drivers. I really would have liked to see one of the existing BSDs benefit from the time they've put into this project. Obviously Linux already has KVM, and there are new implmentations of DTrace and ZFS. Btrfs is not too many more kernel releases away from being considered stable(fsck being a glaring problem) and if you want container-based virtualization you've got LXC(some prefer OpenVZ). With that said, who is their target audience?
- hello_moto 15y ago
  They have a lot of investment around Solaris (let's not argue about Solaris, OpenSolaris, Illumos) since a few years ago. Brought a lot of ex-SUN engineers, hired open source activist around OpenSolaris, probably built their infrastructure around it as well.
  - rednaught 15y ago
    Thanks. But didn't they also have a lot of FreeBSD such as during the TextDrive days?
    - hello_moto 15y ago
      Probably customer + talent acquisition as opposed to technology/infrastructure.
- dap 15y ago
  You're suggesting Joyent should have finished porting DTrace, ZFS, and Zones to FreeBSD, and then port KVM to FreeBSD, instead of just porting KVM to Illumos?
  - rednaught 15y ago
    FreeBSD has already had ZFS and DTrace for years. They have Jails which would have benefited from Solaris' stronger Zones/Containers. There are already a lot of BSD users. Where are the Illumos users? Are you suggesting that all BSD users go to SmartOS? I used HP-UX and AIX for many years...the one off UNIX systems just end up recreating each other due to fragmentation. They are not the way forward. The smaller mind share/usage of OpenSolaris/Illumos isn't helping. Wouldn't contribution to a BSD have strengthened the community?
mkup 15y ago
Better they had ported KVM to FreeBSD. Who will add support for the new hardware to basically abandoned OpenSolaris kernel? Oracle? No. That small company? No. Leveraging FreeBSD community to grow the list of supported hardware would be much, much wiser.
nathanb 15y ago
"ZFS makes SANs and other expensive, redundant storage systems obsolete" Yes, because clearly what enterprise-level storage customers really want is to roll their own storage solutions using open source technology.
- daeken 15y ago
  Many, many companies do want this, yes. Back in 2005-2006, I worked at MP3tunes and among other things I helped design the storage system. After testing a bunch of storage solutions (many of them high-end systems), we ended up rolling our own using Linux + MogileFS. This was scaled up to a couple hundred terabytes before I left, and even higher afterwards. Sometimes the off-the-shelf solutions simply don't work, especially at scale.
  - mitchty 15y ago
    Pretty much, the amount of times we've had issues with new storage vendor arrays from a 3 letter company that starts with an E, and ends with a C, is a bit too much to count. Firmware bugs that affect anything on a fabric, that affect how it distributes its cache slot locks on writes, issues with the entire array acting funny, which are blamed on either the server hardware or os itself until proven to be an issue with the array (this is far too common to be honest, yay for "support" contracts), etc.... Yes you get "support", and I use the term lightly, with big vendors, but you also have to take a machete through their support organization to get to someone that can help you with a problem. At times rolling your own solution will end up being both cheaper and less problematic. The old adage of "Nobody ever got fired for choosing IBM" may soothe managers minds, but wait until you do end up buying those fancy pants high end arrays and find out how much snake oil turns out not to work on them.
    - acdha 15y ago
      We had great experiences with NetApp's products and support but it was priced accordingly.
- acdha 15y ago
strlen 15y ago
This is really great. Joyent has been running this internally for years. The combination is just great: ZFS, DTrace and KVM speak for themselves, but another great ingredient is their use of the NetBSD userland (in place of legacy Solaris one). Linux is great, but a monoculture benefits no one.
kraemate 15y ago
This is simply amazing. Tracing KVM using Dtrace is going to throw up some pretty useful and surprising results. And add ZFS on top that as well - with all its features (sp. dedup+COW) useful for virtualized hosting. Anyone know if linux has any variant/clone of dtrace yet? (and no, not systemtap)
- otoburb 15y ago
  Paul Fox seems to have been working on a Linux DTrace port since 2008 (based on the directory listing). He blogs about his progress on DTrace (and other things) at http://crtags.blogspot.com/ http://crtags.blogspot.com/
  - sogrady 15y ago
    Here's Bryan Cantrill, one of the original creators of DTrace, on the status of the port: https://plus.google.com/105843697186982227624/posts/5uk5SHrhPt6 https://plus.google.com/105843697186982227624/posts/5uk5SHrh...
darklajid 15y ago
The biggest yay for me is in the fine print: "SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level virtualization and next-generation KVM) with __BSD package management and a GNU toolchain__." Oh god, this makes me happy. I gave Solaris a couple of tries, but I've never felt at home there. These two facts mean that I'll try again.
- barrkel 15y ago
  Try out nexenta too. Debian userland.
- binarycrusader 15y ago
  Solaris 11 Express has the GNU toolchain and a modern package management system. For those that want something like SmartOS but free, and with a modern package management system, I would suggest OpenIndiana. OpenIndiana will be integrating many of the SmartOS changes.
sgt 15y ago
Fantastic news. I've been excited about Illumos for a long time, and now we're finally getting a decent distro with commercial backing. My servers will be very happy when they hear about this.
ristretto 15y ago
I 'm happy for all the involvement of Joyent in opensource, but how about also dropping the prices in their accelerators? I 've been using the same servers with the same prices for 3 years with zero upgrades.
foobarbazoo 15y ago
Would be sweet to run SmartOS on Backblaze's storage hardware.
jkahn 15y ago
I'm struggling to understand why ZFS makes enterprise storage obsolete. One of the benefits of the SAN is that it is shared amongst a number of servers. How does ZFS help with that? Assuming you install SmartOS as virtualisation host, you'd still need some kind of shared storage. [Edit:] The reason I point this out is that if you lose a host, you lose all VMs running on that host. And then you won't easily be able to get them back online on another host if you've lost your storage as well.
- wmf 15y ago
  You can use ZFS+Comstar to create a SAN controller.
  - organico 15y ago
    But you were able to do this before SmartOS... I'm curious about this too, and how Joyent have architected their cloud arrangement. I feel like perhaps hypervisors have local ZFS storage, as opposed to large SAN-backed virtual machines, but I can't really find much useful information on how Joyent/SmartOS implement/architect storage...
kermitthehermit 15y ago
This looks very good. How can packages be added? How can it be reconfigured? How is one supposed to add management tools (think libvirt, openstack) to it? I took it for a spin and noticed it had kvm, a whole lot of regular Illumos binaries and that it had a BSD-ish userland. I couldn't find anything like libvirtd, I didn't get to check if it has it, though.
RexRollman 15y ago
As an OS geek, I always enjoy the annoucement of a new OS, even if it is one I personally have no use for. Choice is good.

279 points by timf 15y ago

wmf 15y ago

They're really going out of their way to not say "Solaris".

po 15y ago
They're really going out of their way to not use Solaris. SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level virtualization and next-generation KVM) with BSD package management and a GNU toolchain. I used Solaris for almost 5 years and when Sun was bought I thought for sure the cool parts of it were doomed. Illumos popped up and I thought that they didn't have a chance, it would wither and die. It's very nice to see that is not the case.
- wmf 15y ago
  They're really going out of their way to not use Solaris. Considering that the Illumos kernel is basically the OpenSolaris kernel, I don't understand what you mean. They are using Solaris, but for some reason they're trying to hide it.
  - nknight 15y ago
    Illumos != Solaris, unless you'd like Oracle to beat you with a stick for trademark infringement. I suppose they could refer to it in technical documents as "the OpenSolaris kernel", but by that same logic, an OS using the DragonFly BSD kernel should say it's using "the FreeBSD kernel".
  - po 15y ago
    This project is certainly not using Solaris which is now an Oracle product (and trademark) and not open source. It is based on Illumos (which originally came from an OpenSolaris kernel as you said) but my understanding is that OpenSolaris as a project is dead. I don't think it's hiding something to name-check Illumos instead of OpenSolaris.
killerswan 15y ago
Trademark disputes with hyper-litigious competitors would not be fun...
thirdhaf 15y ago
Well since they have a lot of former Sun engineers working for them I don't really think they have to SAY Solaris. As others have mentioned there are trademarks involved here too. Sun/Oracle have been hemorrhaging talent, particularly the former Fishworks team, for a while now and a lot of those people are ending up in places like Joyent.

jjm 15y ago

Cool. I see this really helping me when testing my application prior to deployment. Would like to see some more movement with No.de. Have been waiting for a bit to get access.

nyellin 15y ago

To clarify, Joyent open sourced SmartOS. Zones, ZFS, DTrace, and KVM are maintained by other companies and have been open source for ages. Edit: That's not to imply Joyent is freeloading. There is another post on HN with an example of the awesome work they're doing: http://news.ycombinator.com/item?id=2887092 http://news.ycombinator.com/item?id=2887092

sciurus 15y ago

Can anyone either explain or link to material that explains the relationships between the different projects and distributions that have sprung up since Oracle discontinued OpenSolaris?

piotrSikora 15y ago
After OpenSolaris OS/Net kernel development was closed, 3 forks were made: Illumos, SchilliX-ON and Stormix (sadly, they all seem quite inactive). There are 3 distributions based on the first one: OpenIndiana, Nexenta and SmartOS. There is also SchilliX (based on SchilliX-ON) and StromOS (based on Stormix). This is just based on my existing knowledge, so there might be more forks and/or distributions out there.
- binarycrusader 15y ago
  Both IllumOS and OpenIndiana are quite active (which shouldn't be surprising since the latter relies on the former).
  - piotrSikora 15y ago
    Is it, really? According to GitHub's commit history, there were 297 commits made since onnv_147, so ~1 commit/day. Prior to the fork, OpenSolaris was getting ~10 commits/day. To put this into perspective, there were over 8000 commits made to OpenBSD in the same time-frame. I won't even try to compare this with FreeBSD or Linux... Don't get me wrong, I love the effort, but at the same time I feel like there wasn't any real progress made since the project's inception.
    - binarycrusader 15y ago
      It depends on your definition of active. As an open source project, it's definitely active. Is it as active as the actual Solaris codebase? No. But it has also significantly less developers working on it.

mrb 15y ago

Wow this is huge. Having an OS be able to run both ZFS and KVM enables fantastic things. For example you can store each virtual machine disk template on a dedicated ZFS filesystem, and use "zfs clone" to rapidly deploy VMs (instead of using the KVM-level support for base disk images "qemu-img create -b ..."), as well as "zfs snapshot", "zfs revert", etc. The main advantage being that these clones and snapshots are possible while using the simple and fast "raw" KVM disk image format instead of the notoriously slower "qcow2" format that was, until today, the best format supporting base images and snapshots. I, for one, have been wanting to use ZFS specifically like that for a while. This does not compare at all to running ZFS on, say, an NFS server or iSCSI SAN serving data to a server running KVM (ZFS data integrity is only verified remotely on the storage server, it is slower, etc). Who ported KVM to Illumos? I know some old version of QEMU was running on Solaris at some point in the past, but I had no idea they had a full blown KVM port.

amazingman 15y ago
http://smartos.org/2011/08/15/kvm-on-illumos/ http://smartos.org/2011/08/15/kvm-on-illumos/
bcantrill 15y ago
KVM was ported to illumos by Max Bruning, Robert Mustacchi and me. Details are here: http://t.co/knW1UIJ http://t.co/knW1UIJ
- rednaught 15y ago
  Can you provide more details of Crossbow? Who maintains it/status? I hadn't heard anything about it since OpenSolaris went dead.
  - bcantrill 15y ago
    Could you be more specific? It's there, it works, it's in shipping illumos-based products (from Joyent and others) and it's supported by those who ship it or stand it up. In that regard, it's like any other aspect of the system...

rednaught 15y ago

I'm having a difficult time wrapping my head around this. If Joyent wanted to stick with a UNIX then why not FreeBSD and help implement KVM there? Wasn't XEN already available for Dom0? I just don't see much mindshare in Illumos/SmartOS for keeping up with current server hardware drivers. I really would have liked to see one of the existing BSDs benefit from the time they've put into this project. Obviously Linux already has KVM, and there are new implmentations of DTrace and ZFS. Btrfs is not too many more kernel releases away from being considered stable(fsck being a glaring problem) and if you want container-based virtualization you've got LXC(some prefer OpenVZ). With that said, who is their target audience?

hello_moto 15y ago
They have a lot of investment around Solaris (let's not argue about Solaris, OpenSolaris, Illumos) since a few years ago. Brought a lot of ex-SUN engineers, hired open source activist around OpenSolaris, probably built their infrastructure around it as well.
- rednaught 15y ago
  Thanks. But didn't they also have a lot of FreeBSD such as during the TextDrive days?
  - hello_moto 15y ago
    Probably customer + talent acquisition as opposed to technology/infrastructure.
dap 15y ago
You're suggesting Joyent should have finished porting DTrace, ZFS, and Zones to FreeBSD, and then port KVM to FreeBSD, instead of just porting KVM to Illumos?
- rednaught 15y ago
  FreeBSD has already had ZFS and DTrace for years. They have Jails which would have benefited from Solaris' stronger Zones/Containers. There are already a lot of BSD users. Where are the Illumos users? Are you suggesting that all BSD users go to SmartOS? I used HP-UX and AIX for many years...the one off UNIX systems just end up recreating each other due to fragmentation. They are not the way forward. The smaller mind share/usage of OpenSolaris/Illumos isn't helping. Wouldn't contribution to a BSD have strengthened the community?

mkup 15y ago

Better they had ported KVM to FreeBSD. Who will add support for the new hardware to basically abandoned OpenSolaris kernel? Oracle? No. That small company? No. Leveraging FreeBSD community to grow the list of supported hardware would be much, much wiser.

nathanb 15y ago

"ZFS makes SANs and other expensive, redundant storage systems obsolete" Yes, because clearly what enterprise-level storage customers really want is to roll their own storage solutions using open source technology.

daeken 15y ago
Many, many companies do want this, yes. Back in 2005-2006, I worked at MP3tunes and among other things I helped design the storage system. After testing a bunch of storage solutions (many of them high-end systems), we ended up rolling our own using Linux + MogileFS. This was scaled up to a couple hundred terabytes before I left, and even higher afterwards. Sometimes the off-the-shelf solutions simply don't work, especially at scale.
- mitchty 15y ago
  Pretty much, the amount of times we've had issues with new storage vendor arrays from a 3 letter company that starts with an E, and ends with a C, is a bit too much to count. Firmware bugs that affect anything on a fabric, that affect how it distributes its cache slot locks on writes, issues with the entire array acting funny, which are blamed on either the server hardware or os itself until proven to be an issue with the array (this is far too common to be honest, yay for "support" contracts), etc.... Yes you get "support", and I use the term lightly, with big vendors, but you also have to take a machete through their support organization to get to someone that can help you with a problem. At times rolling your own solution will end up being both cheaper and less problematic. The old adage of "Nobody ever got fired for choosing IBM" may soothe managers minds, but wait until you do end up buying those fancy pants high end arrays and find out how much snake oil turns out not to work on them.
  - acdha 15y ago
    We had great experiences with NetApp's products and support but it was priced accordingly.
acdha 15y ago

strlen 15y ago

This is really great. Joyent has been running this internally for years. The combination is just great: ZFS, DTrace and KVM speak for themselves, but another great ingredient is their use of the NetBSD userland (in place of legacy Solaris one). Linux is great, but a monoculture benefits no one.

kraemate 15y ago

This is simply amazing. Tracing KVM using Dtrace is going to throw up some pretty useful and surprising results. And add ZFS on top that as well - with all its features (sp. dedup+COW) useful for virtualized hosting. Anyone know if linux has any variant/clone of dtrace yet? (and no, not systemtap)

otoburb 15y ago
Paul Fox seems to have been working on a Linux DTrace port since 2008 (based on the directory listing). He blogs about his progress on DTrace (and other things) at http://crtags.blogspot.com/ http://crtags.blogspot.com/
- sogrady 15y ago
  Here's Bryan Cantrill, one of the original creators of DTrace, on the status of the port: https://plus.google.com/105843697186982227624/posts/5uk5SHrhPt6 https://plus.google.com/105843697186982227624/posts/5uk5SHrh...

darklajid 15y ago

The biggest yay for me is in the fine print: "SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level virtualization and next-generation KVM) with __BSD package management and a GNU toolchain__." Oh god, this makes me happy. I gave Solaris a couple of tries, but I've never felt at home there. These two facts mean that I'll try again.

barrkel 15y ago
Try out nexenta too. Debian userland.
binarycrusader 15y ago
Solaris 11 Express has the GNU toolchain and a modern package management system. For those that want something like SmartOS but free, and with a modern package management system, I would suggest OpenIndiana. OpenIndiana will be integrating many of the SmartOS changes.

sgt 15y ago

Fantastic news. I've been excited about Illumos for a long time, and now we're finally getting a decent distro with commercial backing. My servers will be very happy when they hear about this.

ristretto 15y ago

I 'm happy for all the involvement of Joyent in opensource, but how about also dropping the prices in their accelerators? I 've been using the same servers with the same prices for 3 years with zero upgrades.

foobarbazoo 15y ago

Would be sweet to run SmartOS on Backblaze's storage hardware.

jkahn 15y ago

I'm struggling to understand why ZFS makes enterprise storage obsolete. One of the benefits of the SAN is that it is shared amongst a number of servers. How does ZFS help with that? Assuming you install SmartOS as virtualisation host, you'd still need some kind of shared storage. [Edit:] The reason I point this out is that if you lose a host, you lose all VMs running on that host. And then you won't easily be able to get them back online on another host if you've lost your storage as well.

wmf 15y ago
You can use ZFS+Comstar to create a SAN controller.
- organico 15y ago
  But you were able to do this before SmartOS... I'm curious about this too, and how Joyent have architected their cloud arrangement. I feel like perhaps hypervisors have local ZFS storage, as opposed to large SAN-backed virtual machines, but I can't really find much useful information on how Joyent/SmartOS implement/architect storage...

kermitthehermit 15y ago

This looks very good. How can packages be added? How can it be reconfigured? How is one supposed to add management tools (think libvirt, openstack) to it? I took it for a spin and noticed it had kvm, a whole lot of regular Illumos binaries and that it had a BSD-ish userland. I couldn't find anything like libvirtd, I didn't get to check if it has it, though.

RexRollman 15y ago

As an OS geek, I always enjoy the annoucement of a new OS, even if it is one I personally have no use for. Choice is good.