5 ms·

OpenZFS on illumos

48 points by bcantrill 12y ago

sgt 12y ago
Is illumos close to being production ready yet? (Or perhaps it has been for a while). I know there are several illumos-based distributions, so I'd like some clarification on the do's and don'ts.
- bcantrill 12y ago
  Yes, by inheriting the OpenSolaris legacy, illumos was essentially born in production, having been used in production very nearly since its inception. We at Joyent have our own illumos-derivative, SmartOS[1], that is the foundation of our own public cloud -- and of our (open source) orchestration software, SmartDataCenter[2] and of our object storage system, Manta[3]. OmniTI has OmniOS[4] which they (and others) have deployed extensively into production. Delphix (the origin of this post) has DelphixOS[5] which forms the basis of their data management virtual appliances, also widely deployed in production. Finally, if you're interested in the (complicated) history of the system, you might be interested in a talk that I gave a few years ago[6] that is somewhat infamous for its candor. ;) [1] http://smartos.org http://smartos.org [2] https://github.com/joyent/sdc/ https://github.com/joyent/sdc/ [3] https://github.com/joyent/manta https://github.com/joyent/manta [4] http://omnios.omniti.com/ http://omnios.omniti.com/ [5] https://github.com/delphix/delphix-os https://github.com/delphix/delphix-os [6] https://www.youtube.com/watch?v=-zRN7XLCRhc https://www.youtube.com/watch?v=-zRN7XLCRhc
  - ebiester 12y ago
    Is there a good link to the differences between SmartOS and illumos? Why does the fork exist?
    - bcantrill 12y ago
      SmartOS (like OmniOS, DelphixOS and the others) isn't a fork -- we're a downstream repo that merges from upstream on a daily basis, and regularly upstreams our own work. This model has been very successful for illumos (in my opinion) because it has allowed for a broad degree of experimentation in production-oriented systems while still having a conduit for getting successful experiments upstream. And by the time we go to upstream something, it's generally been running in production for us for a while (and ditto other downstream repos), giving reviewers confidence and allowing us to maintain very, very high quality in illumos itself. (Prakash alluded to this in the linked blog post, and it's absolutely essential to assure that downstream repos can frequently merge.) The net is that we have found the work going into illumos to be exactly what you want upon integration: innovative, pragmatic work that has already proven itself in production.
    - takeda 12y ago
      As I understand SmartOS was designated to be a host for VMs, for that reason Joyent ported KVM to Solaris and made it so it can be booted from USB drive. So for example updating system is as simple as loading new version on USB drive and rebooting the system.
- dmpk2k 12y ago
  illumos was already being used in production in 2010. Its development model is very much as the linked article implies -- every commit is expected to be stable, since the tree should always be stable. If you're looking for a general server OS, use OmniOS. If you're running armies of VMs, use SmartOS. For desktop, OpenIndiana Hipster, although illumos is more for servers. Personally, I'm fond of SmartOS, in large part because it comes with pkgsrc. The bad old Solaris days, with its anemic packages, blastwave and sunfreeware, are long gone.
- volkadav 12y ago
  It's 100% ready for production. Joyent runs a very large platform on SmartOS, and I can verify that at OmniTI we had the vast majority of our hosted customers running on OmniOS. I worked there for a bit over three years, starting just before the initial betas of OmniOS and watched as systems migrated from Solaris 10 to OmniOS without any major issues. (I say "watched" because I was in dev rather than ops, but I don't recall any inordinate grumbling.)
bsaul 12y ago
Could someone here make a description of what obvious gains one could have by moving a linux-based server (let's say ubuntu) hosting a regular web service & db to illumos ? Would i gain performance ? Would it let me backup my server more easily ? Make the stack easier to shard ? Be less prone to security issues ? Etc.
- alrs 12y ago
  Used to be that x86 Solaris was referred to as "Slowaris." edit: It wasn't?
- scarmig 12y ago
  It may or may not be worth your while, though I'd guess no, if you don't have the time to learn a new system and its capabilities. That said, Linux lags in many very useful features. It's only recently gotten a usable OS-virtualization solution (containers/Docker), while Solaris/illumos (and FreeBSD, for that matter) have had versions of it production ready for over a decade. Tracing on Linux is a pale shadow of DTrace--which bcantrill among others built a decade ago, and which has been available on the BSDs and Mac OSX for the better part of the decade since. ZFS, which many would argue is standing king of file systems, has been battle-tested on Solaris/illumos and FreeBSD for years, while licensing issues prevent it from being included as part of Linux. The file system that btrfs is meant to be better than is ZFS, a goal it's still seeking. Of course, as far as it goes, Linux is good enough to get by. But once you've worked with a more fully-featured OS, you find yourself thinking wistfully of them.
antonios 12y ago
Note for SSD owners: At this time, only FreeBSD's ZFS supports TRIM on SSD's.
- bsg75 12y ago
  Not Solaris / Illumios ?
  - _delirium 12y ago
    Not yet, if this bit on the wiki is up to date: http://open-zfs.org/wiki/Features#TRIM_Support http://open-zfs.org/wiki/Features#TRIM_Support Clicking through some of the links there, it sounds like the Illumos devs aren't entirely happy with the current FreeBSD code, but a new implementation looks stalled.
    - bsg75 12y ago
      As in not happy with the FreeBSD implementation of ZFS relative to the Illumos (OpenSolaris?) code?
      - _delirium 12y ago
        I think they're just not happy with the specific way TRIM support was added, and want to add that feature but architected in a somewhat different way. There was a thread on the illumos-zfs list in February 2013 where an alternate proposal w/ prototype patch was made (the thread also has some comments from the author of the FreeBSD version), but it doesn't seem to have been finished/merged.

48 points by bcantrill 12y ago

sgt 12y ago

Is illumos close to being production ready yet? (Or perhaps it has been for a while). I know there are several illumos-based distributions, so I'd like some clarification on the do's and don'ts.

bcantrill 12y ago
Yes, by inheriting the OpenSolaris legacy, illumos was essentially born in production, having been used in production very nearly since its inception. We at Joyent have our own illumos-derivative, SmartOS[1], that is the foundation of our own public cloud -- and of our (open source) orchestration software, SmartDataCenter[2] and of our object storage system, Manta[3]. OmniTI has OmniOS[4] which they (and others) have deployed extensively into production. Delphix (the origin of this post) has DelphixOS[5] which forms the basis of their data management virtual appliances, also widely deployed in production. Finally, if you're interested in the (complicated) history of the system, you might be interested in a talk that I gave a few years ago[6] that is somewhat infamous for its candor. ;) [1] http://smartos.org http://smartos.org [2] https://github.com/joyent/sdc/ https://github.com/joyent/sdc/ [3] https://github.com/joyent/manta https://github.com/joyent/manta [4] http://omnios.omniti.com/ http://omnios.omniti.com/ [5] https://github.com/delphix/delphix-os https://github.com/delphix/delphix-os [6] https://www.youtube.com/watch?v=-zRN7XLCRhc https://www.youtube.com/watch?v=-zRN7XLCRhc
- ebiester 12y ago
  Is there a good link to the differences between SmartOS and illumos? Why does the fork exist?
  - bcantrill 12y ago
    SmartOS (like OmniOS, DelphixOS and the others) isn't a fork -- we're a downstream repo that merges from upstream on a daily basis, and regularly upstreams our own work. This model has been very successful for illumos (in my opinion) because it has allowed for a broad degree of experimentation in production-oriented systems while still having a conduit for getting successful experiments upstream. And by the time we go to upstream something, it's generally been running in production for us for a while (and ditto other downstream repos), giving reviewers confidence and allowing us to maintain very, very high quality in illumos itself. (Prakash alluded to this in the linked blog post, and it's absolutely essential to assure that downstream repos can frequently merge.) The net is that we have found the work going into illumos to be exactly what you want upon integration: innovative, pragmatic work that has already proven itself in production.
  - takeda 12y ago
    As I understand SmartOS was designated to be a host for VMs, for that reason Joyent ported KVM to Solaris and made it so it can be booted from USB drive. So for example updating system is as simple as loading new version on USB drive and rebooting the system.
dmpk2k 12y ago
illumos was already being used in production in 2010. Its development model is very much as the linked article implies -- every commit is expected to be stable, since the tree should always be stable. If you're looking for a general server OS, use OmniOS. If you're running armies of VMs, use SmartOS. For desktop, OpenIndiana Hipster, although illumos is more for servers. Personally, I'm fond of SmartOS, in large part because it comes with pkgsrc. The bad old Solaris days, with its anemic packages, blastwave and sunfreeware, are long gone.
volkadav 12y ago
It's 100% ready for production. Joyent runs a very large platform on SmartOS, and I can verify that at OmniTI we had the vast majority of our hosted customers running on OmniOS. I worked there for a bit over three years, starting just before the initial betas of OmniOS and watched as systems migrated from Solaris 10 to OmniOS without any major issues. (I say "watched" because I was in dev rather than ops, but I don't recall any inordinate grumbling.)

bsaul 12y ago

Could someone here make a description of what obvious gains one could have by moving a linux-based server (let's say ubuntu) hosting a regular web service & db to illumos ? Would i gain performance ? Would it let me backup my server more easily ? Make the stack easier to shard ? Be less prone to security issues ? Etc.

alrs 12y ago
Used to be that x86 Solaris was referred to as "Slowaris." edit: It wasn't?
scarmig 12y ago
It may or may not be worth your while, though I'd guess no, if you don't have the time to learn a new system and its capabilities. That said, Linux lags in many very useful features. It's only recently gotten a usable OS-virtualization solution (containers/Docker), while Solaris/illumos (and FreeBSD, for that matter) have had versions of it production ready for over a decade. Tracing on Linux is a pale shadow of DTrace--which bcantrill among others built a decade ago, and which has been available on the BSDs and Mac OSX for the better part of the decade since. ZFS, which many would argue is standing king of file systems, has been battle-tested on Solaris/illumos and FreeBSD for years, while licensing issues prevent it from being included as part of Linux. The file system that btrfs is meant to be better than is ZFS, a goal it's still seeking. Of course, as far as it goes, Linux is good enough to get by. But once you've worked with a more fully-featured OS, you find yourself thinking wistfully of them.

antonios 12y ago

Note for SSD owners: At this time, only FreeBSD's ZFS supports TRIM on SSD's.

bsg75 12y ago
Not Solaris / Illumios ?
- _delirium 12y ago
  Not yet, if this bit on the wiki is up to date: http://open-zfs.org/wiki/Features#TRIM_Support http://open-zfs.org/wiki/Features#TRIM_Support Clicking through some of the links there, it sounds like the Illumos devs aren't entirely happy with the current FreeBSD code, but a new implementation looks stalled.
  - bsg75 12y ago
    As in not happy with the FreeBSD implementation of ZFS relative to the Illumos (OpenSolaris?) code?
    - _delirium 12y ago
      I think they're just not happy with the specific way TRIM support was added, and want to add that feature but architected in a somewhat different way. There was a thread on the illumos-zfs list in February 2013 where an alternate proposal w/ prototype patch was made (the thread also has some comments from the author of the FreeBSD version), but it doesn't seem to have been finished/merged.