15 ms·

Sandboxing and workload isolation

158 points by sudhirj 6y ago

eyberg 6y ago
There's also unikernels which several work under firecracker. "The first, and really the big problem for the whole virtualization approach, is that you need bare metal servers to efficiently do lightweight virtualization;" <- this isn't really true anymore as unikernels can be deployed to {aws, gcloud, azure} as machine images and so can be as light-weight as your program needs.
- mrkurt 6y ago
  We've been playing a lot with OSv. But unikernels aren't strictly sandboxing or isolation. If you're deploying a unikernel to ec2, you're just using their underlying isolation. You can't safely deploy multiple unikernels on the same ec2 instances without figuring out your own, nested isolation.
  - eyberg 6y ago
    Sure - the hypervisor is the isolation vehicle - the same as firecracker. The only? reason I can see for wanting to deploy N unikernels to one AWS instance is if you want full control on the orchestration of things like multi-tenancy, however, it is very much possible and it is possible on GCloud as well. It should be noted that you are in the same boat as firecracker here. They both use virtualization.
    - yencabulator 6y ago
      The usual reason to want to do N things with 1 piece of hardware is cost savings.
hinkley 6y ago
I just cannot shake the feeling that everything Containers have promised us, I was promised back in 1991 with pre-emptive multitasking and protected memory. I also can't help but think we're going to end up with a microkernel with some sort of nested cgroups for user processes. Which is itself going to look a little bit like Erlang...
- jayd16 6y ago
  Is that a surprise? Containers can mostly be summed up by file system isolation on top of process isolation. I think this is why we're seeing some work done on capability based OSes like Google's Fuchsia.
  - kjeetgill 6y ago
    Sorta, but I think you're missing the value proposition forest for the feature trees. I'm going to speak to the docker-style containers because that's what I'm most familiar with. Filesystem, process, and even network isolation facilitate the illusion of an application having a fresh install of an OS to a degree. This includes details like /dev/, a package manager, things sitting in /etc/. Added to that there's decent tooling for sharing, composing/layering, shipping, baking images. They're declarative (enough-ish) too. This is nice because that's a pretty decent compatibility layer across a ton of software like databases and webservers that have always run expecting a full host, services written with bare metal in mind, and new things too. The kernel isolation features are powerful enablers, but don't undervalue how important it was to have things sorted out for you, like ldloader quirks with different glibc versions against the same kernel. Real issue I had. Source: Too much futzing around with chroots.
- pjmlp 6y ago
  For me we are kind of there already with hypervisors and managed language runtimes. The irony of having monoliths like Linux running type 1 hypervisors, on Intel CPUs running ME.
- watermelon0 6y ago
  Docker promised containers like we knew them from the past, and improved building and packaging procedures. In the past, we never had an easy way to package various applications (including all needed resources and executables). Each language ecosystem invented their own ways of packaging, distributing and installing the application, and this always relied on pre-existing environment. With Docker you don't care what versions of language runtime, or dependencies, each application runs, you can run them on the same machine. There are no incompatible dependencies between multiple Docker containers, everything almost always works. Same goes with building, image layers made building of images a lot easier, and doing incremental changes is fast. Of course there are downsides, image size can really grow if you are not careful, and you need to regularly rebuild images from the ground up, to ensure everything is patched.
jamestenglish 6y ago
Anyone else having a hard time even reading this article because of the lack of contrast from text to background?
- denysvitali 6y ago
  uhm, nope
- markbnj 6y ago
  No, but I am not sure why you got downvoted for asking.
  - tptacek 6y ago
    It's against the guidelines to do so.
- boring_twenties 6y ago
  Did you try Reader View[1]? Actually it's rather ugly for this specific page for some reason, with weird boxes around every paragraph. But at least the contrast isn't an issue. [1] https://support.mozilla.org/en-US/kb/firefox-reader-view-clutter-free-web-pages https://support.mozilla.org/en-US/kb/firefox-reader-view-clu...
louwrentius 6y ago
So light-weight virtual machines on a security-hardened code base. Have we come full circle?
- rhizome 6y ago
  It's not a circle! https://media0.giphy.com/media/F5rQlfTXqCJ8c/giphy.gif https://media0.giphy.com/media/F5rQlfTXqCJ8c/giphy.gif
- jrott 6y ago
  Probably since we're steadily reinventing all the techniques that people used to run stuff on mainframes.
hashamali 6y ago
Love what the folks at fly.io are doing with edge deployed containers. Interesting comparison with Cloudflare's isolate approach which they expanded on earlier this week: https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec... Seems like there is an edge compute option for any requirement these days!
- EE84M3i 6y ago
  What's the cutoff for something to be considered "Edge"? It looks like Fly has only about 20 POPs.
  - sudhirj 6y ago
    The ability to have an auto scaling auto distributive compute mesh, I’d say. Once a company builds that they can add POPs as fast as they can spend (or raise) money.
    - yencabulator 6y ago
      By that definition, AWS and GCP are both "edges" then. Doesn't sound like a useful definition...
iancarroll 6y ago
We've deployed nsjail for our file conversion pipeline (i.e. ImageMagick) and it's been great -- very nice configuration language and strong isolation properties, with a manageable performance hit. Definitely easy to write a configuration that would not securely sandbox you, though, which seems like a strong point towards Docker or other more high-level solutions.
- browsergap 6y ago
  I might look at nsjail. I run chrome headless inside of cgroups (for cpu and memory restrictions) and using a runtime generated no-login user (for privilege restrictions), and with a custom group id (for IP tables bandwidth and routing restrictions). Plus I monitor the pool of chromes with a shell script and if any exceed a resource threshold, I used cpulimit then pkill to take care of it. I could just use docker (and I have docker images of this app that some people like to run) but I think this way is more lightweight, gives me more explicit control, and leans into the OS level security features, like privileges and cgroups with a couple of Unix commands.
  - sitkack 6y ago
    I thought I was the only one running workloads in headless browsers! We should start a club for just us NaN.
    - browsergap 6y ago
      OK, come join my club! https://github.com/dosyago https://github.com/dosyago send me an email :)
0xdeadb00f 6y ago
> But you’re not running OpenBSD, so, moving on. But.. I am running OpenBSD ;(
fwsgonzo 6y ago
I've worked on unikernels for several years, and I've recently had some insight into low-overhead emulation. Depending on your use-case it's possible that a tiny emulator will be less overhead than everything else for the construction/destruction part. So, imagine that you have a programmable web server and each request needs to be served quickly. Then, reducing the overheading of forking the master machine is everything. The cost of doing a few emulated instructions into systemcalls (which largely model what you already have to check normally), you are going to have lots of room before you hit a point where something heavier like a JIT-enabled emulator or rv8 which has millisecond-overhead. I am working on such a thing now and it has a 6-microsecond overhead in a highly concurrent scenario (60k req/s). So, I think there are cases for everything.

158 points by sudhirj 6y ago

There's also unikernels which several work under firecracker. "The first, and really the big problem for the whole virtualization approach, is that you need bare metal servers to efficiently do lightweight virtualization;" <- this isn't really true anymore as unikernels can be deployed to {aws, gcloud, azure} as machine images and so can be as light-weight as your program needs.

mrkurt 6y ago
We've been playing a lot with OSv. But unikernels aren't strictly sandboxing or isolation. If you're deploying a unikernel to ec2, you're just using their underlying isolation. You can't safely deploy multiple unikernels on the same ec2 instances without figuring out your own, nested isolation.
- eyberg 6y ago
  Sure - the hypervisor is the isolation vehicle - the same as firecracker. The only? reason I can see for wanting to deploy N unikernels to one AWS instance is if you want full control on the orchestration of things like multi-tenancy, however, it is very much possible and it is possible on GCloud as well. It should be noted that you are in the same boat as firecracker here. They both use virtualization.
  - yencabulator 6y ago
    The usual reason to want to do N things with 1 piece of hardware is cost savings.

hinkley 6y ago

I just cannot shake the feeling that everything Containers have promised us, I was promised back in 1991 with pre-emptive multitasking and protected memory. I also can't help but think we're going to end up with a microkernel with some sort of nested cgroups for user processes. Which is itself going to look a little bit like Erlang...

jayd16 6y ago
Is that a surprise? Containers can mostly be summed up by file system isolation on top of process isolation. I think this is why we're seeing some work done on capability based OSes like Google's Fuchsia.
- kjeetgill 6y ago
  Sorta, but I think you're missing the value proposition forest for the feature trees. I'm going to speak to the docker-style containers because that's what I'm most familiar with. Filesystem, process, and even network isolation facilitate the illusion of an application having a fresh install of an OS to a degree. This includes details like /dev/, a package manager, things sitting in /etc/. Added to that there's decent tooling for sharing, composing/layering, shipping, baking images. They're declarative (enough-ish) too. This is nice because that's a pretty decent compatibility layer across a ton of software like databases and webservers that have always run expecting a full host, services written with bare metal in mind, and new things too. The kernel isolation features are powerful enablers, but don't undervalue how important it was to have things sorted out for you, like ldloader quirks with different glibc versions against the same kernel. Real issue I had. Source: Too much futzing around with chroots.
pjmlp 6y ago
For me we are kind of there already with hypervisors and managed language runtimes. The irony of having monoliths like Linux running type 1 hypervisors, on Intel CPUs running ME.
watermelon0 6y ago
Docker promised containers like we knew them from the past, and improved building and packaging procedures. In the past, we never had an easy way to package various applications (including all needed resources and executables). Each language ecosystem invented their own ways of packaging, distributing and installing the application, and this always relied on pre-existing environment. With Docker you don't care what versions of language runtime, or dependencies, each application runs, you can run them on the same machine. There are no incompatible dependencies between multiple Docker containers, everything almost always works. Same goes with building, image layers made building of images a lot easier, and doing incremental changes is fast. Of course there are downsides, image size can really grow if you are not careful, and you need to regularly rebuild images from the ground up, to ensure everything is patched.

jamestenglish 6y ago

Anyone else having a hard time even reading this article because of the lack of contrast from text to background?

denysvitali 6y ago
uhm, nope
markbnj 6y ago
No, but I am not sure why you got downvoted for asking.
- tptacek 6y ago
  It's against the guidelines to do so.
boring_twenties 6y ago
Did you try Reader View[1]? Actually it's rather ugly for this specific page for some reason, with weird boxes around every paragraph. But at least the contrast isn't an issue. [1] https://support.mozilla.org/en-US/kb/firefox-reader-view-clutter-free-web-pages https://support.mozilla.org/en-US/kb/firefox-reader-view-clu...

louwrentius 6y ago

So light-weight virtual machines on a security-hardened code base. Have we come full circle?

rhizome 6y ago
It's not a circle! https://media0.giphy.com/media/F5rQlfTXqCJ8c/giphy.gif https://media0.giphy.com/media/F5rQlfTXqCJ8c/giphy.gif
jrott 6y ago
Probably since we're steadily reinventing all the techniques that people used to run stuff on mainframes.

hashamali 6y ago

Love what the folks at fly.io are doing with edge deployed containers. Interesting comparison with Cloudflare's isolate approach which they expanded on earlier this week: https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec... Seems like there is an edge compute option for any requirement these days!

EE84M3i 6y ago
What's the cutoff for something to be considered "Edge"? It looks like Fly has only about 20 POPs.
- sudhirj 6y ago
  The ability to have an auto scaling auto distributive compute mesh, I’d say. Once a company builds that they can add POPs as fast as they can spend (or raise) money.
  - yencabulator 6y ago
    By that definition, AWS and GCP are both "edges" then. Doesn't sound like a useful definition...

iancarroll 6y ago

We've deployed nsjail for our file conversion pipeline (i.e. ImageMagick) and it's been great -- very nice configuration language and strong isolation properties, with a manageable performance hit. Definitely easy to write a configuration that would not securely sandbox you, though, which seems like a strong point towards Docker or other more high-level solutions.

browsergap 6y ago
I might look at nsjail. I run chrome headless inside of cgroups (for cpu and memory restrictions) and using a runtime generated no-login user (for privilege restrictions), and with a custom group id (for IP tables bandwidth and routing restrictions). Plus I monitor the pool of chromes with a shell script and if any exceed a resource threshold, I used cpulimit then pkill to take care of it. I could just use docker (and I have docker images of this app that some people like to run) but I think this way is more lightweight, gives me more explicit control, and leans into the OS level security features, like privileges and cgroups with a couple of Unix commands.
- sitkack 6y ago
  I thought I was the only one running workloads in headless browsers! We should start a club for just us NaN.
  - browsergap 6y ago
    OK, come join my club! https://github.com/dosyago https://github.com/dosyago send me an email :)

0xdeadb00f 6y ago

> But you’re not running OpenBSD, so, moving on. But.. I am running OpenBSD ;(

fwsgonzo 6y ago

I've worked on unikernels for several years, and I've recently had some insight into low-overhead emulation. Depending on your use-case it's possible that a tiny emulator will be less overhead than everything else for the construction/destruction part. So, imagine that you have a programmable web server and each request needs to be served quickly. Then, reducing the overheading of forking the master machine is everything. The cost of doing a few emulated instructions into systemcalls (which largely model what you already have to check normally), you are going to have lots of room before you hit a point where something heavier like a JIT-enabled emulator or rv8 which has millisecond-overhead. I am working on such a thing now and it has a 6-microsecond overhead in a highly concurrent scenario (60k req/s). So, I think there are cases for everything.