5 ms·

At Joyent, we deployed public-facing multitenant workloads based on zones (and before that, jails) for many years. We seriously trusted it -- and had serious c

At Joyent, we deployed public-facing multitenant workloads based on zones (and before that, jails) for many years. We seriously trusted it -- and had serious customers who seriously depended on it. So, now you know someone!

wmf 5y ago
There were also tons of providers who trusted Linux containers for VPS hosting.
- tptacek 5y ago
  How'd that turn out?
  - wmf 5y ago
    I haven't heard any stories of people being hacked via container escape, but the whole VPS industry was so low-stakes that maybe customers didn't expect good isolation anyway.
mixmastamyk 5y ago
Security requirements (and awareness) have increased over the years, have they not?
- bcantrill 5y ago
  They definitely have! And we had a (zones-based) public cloud through it all. On that note, Alex Wilson's description of working with Robert Mustacchi on mitigating Meltdown by adding KPTI to illumos[0] definitely merits a read! [0] https://blog.cooperi.net/a-long-two-months https://blog.cooperi.net/a-long-two-months
unixhero 5y ago
And needless to say it became a billion dollar business, with a great product.
- psanford 5y ago
  They were acquired for $170m.
  - unixhero 5y ago
    I stand corrected. Still great product, business and team.
    - tptacek 5y ago
      I'm sure they're great. No part of what I have to say about this has anything to do with how competent they are.
psanford 5y ago
To be fair, y'all had some serious vulnerabilities, including zone escapes and arbitrary kernel memory reads, discovered by @benmmurphy.
- bcantrill 5y ago
  Yes, though I would like to believe that Ben's responsible disclosure coupled with our addressing those vulns (and auditing ourselves for similar) reflect exactly that seriousness around multitenant security. And for whatever it's worth, one of those vulnerabilities -- which was a bug in my code! -- very much informed by own thinking about the inherent unsafety of C, underscoring the appeal of Rust. So I am grateful in several dimensions!
  - tptacek 5y ago
    If you have a kernel implemented in Rust, (1) you should shout that from the rooftops and (2) use whatever isolation mechanism you like on it.
    - bsder 5y ago
      They're starting with the bootloader and management engine. That's a tough enough ocean to boil. Give them some time to get Rust above that.
      - pjmlp 5y ago
        Sadly Apple decided for a safe dialect of C for similar purposes e.g. iBoot, where they could have gone with Swift or Rust instead. Very big ocean indeed.
tptacek 5y ago
To this, all I can say is that I spent from 2005-2014, and then from 2016-2020, doing nothing but security evaluations of products, probably about 60% of which were serverside multitenant SAAS systems of one form or another, and I don't remember ever evaluating (or overseeing the evaluation of) a system that relied on Jails or Zones. Lots of Docker! And, until a few years ago, multitenant Docker isolation was an infamous joke! I'm not sticking up for it! You can look at the recent history of Linux kernel LPEs --- there has been sort of a renaissance because of mobile devices --- and count all the ways any shared-kernel multitenant system would have broken down. At the end of the day, it's not so much about predicting whether your system can get owned up (it can), so much as: "what do I need to do when there is a kernel LPE announced on my platform". If you're doing shared-kernel isolation, the right answer to that question is usually "fire drill". It's not a noodley thought-leadership kind of question; it's a simple, practical concern.

wmf 5y ago

There were also tons of providers who trusted Linux containers for VPS hosting.

tptacek 5y ago
How'd that turn out?
- wmf 5y ago
  I haven't heard any stories of people being hacked via container escape, but the whole VPS industry was so low-stakes that maybe customers didn't expect good isolation anyway.

mixmastamyk 5y ago

Security requirements (and awareness) have increased over the years, have they not?

bcantrill 5y ago
They definitely have! And we had a (zones-based) public cloud through it all. On that note, Alex Wilson's description of working with Robert Mustacchi on mitigating Meltdown by adding KPTI to illumos[0] definitely merits a read! [0] https://blog.cooperi.net/a-long-two-months https://blog.cooperi.net/a-long-two-months

unixhero 5y ago

And needless to say it became a billion dollar business, with a great product.

psanford 5y ago
They were acquired for $170m.
- unixhero 5y ago
  I stand corrected. Still great product, business and team.
  - tptacek 5y ago
    I'm sure they're great. No part of what I have to say about this has anything to do with how competent they are.

psanford 5y ago

To be fair, y'all had some serious vulnerabilities, including zone escapes and arbitrary kernel memory reads, discovered by @benmmurphy.

bcantrill 5y ago
Yes, though I would like to believe that Ben's responsible disclosure coupled with our addressing those vulns (and auditing ourselves for similar) reflect exactly that seriousness around multitenant security. And for whatever it's worth, one of those vulnerabilities -- which was a bug in my code! -- very much informed by own thinking about the inherent unsafety of C, underscoring the appeal of Rust. So I am grateful in several dimensions!
- tptacek 5y ago
  If you have a kernel implemented in Rust, (1) you should shout that from the rooftops and (2) use whatever isolation mechanism you like on it.
  - bsder 5y ago
    They're starting with the bootloader and management engine. That's a tough enough ocean to boil. Give them some time to get Rust above that.
    - pjmlp 5y ago
      Sadly Apple decided for a safe dialect of C for similar purposes e.g. iBoot, where they could have gone with Swift or Rust instead. Very big ocean indeed.

tptacek 5y ago

To this, all I can say is that I spent from 2005-2014, and then from 2016-2020, doing nothing but security evaluations of products, probably about 60% of which were serverside multitenant SAAS systems of one form or another, and I don't remember ever evaluating (or overseeing the evaluation of) a system that relied on Jails or Zones. Lots of Docker! And, until a few years ago, multitenant Docker isolation was an infamous joke! I'm not sticking up for it! You can look at the recent history of Linux kernel LPEs --- there has been sort of a renaissance because of mobile devices --- and count all the ways any shared-kernel multitenant system would have broken down. At the end of the day, it's not so much about predicting whether your system can get owned up (it can), so much as: "what do I need to do when there is a kernel LPE announced on my platform". If you're doing shared-kernel isolation, the right answer to that question is usually "fire drill". It's not a noodley thought-leadership kind of question; it's a simple, practical concern.