5 ms·

Hypervisor for multi-tenant computing, like it should be

16 points by ashishbijlani 5y ago

ashishbijlani 5y ago
Hello HN! We've been working on a new hypervisor https://kwarantine.xyz https://kwarantine.xyz that can run strongly isolated containers. This is still a WIP, but we wanted to give the community an idea about our approach, its benefits, and various use cases it unlocks. Today, VMs are used to host containers, and make up for the lack of strong security as well as kernel isolation in containers. This work adds this missing security piece in containers. We plan on launching a free private beta soon. Meanwhile, we'd deeply appreciate any feedback, and happy to answer any questions here or on our slack channel. Thanks!
ashishbijlani 5y ago
There was a question (now deleted) on difference between this and Firecracker. FC/gVisor/Kata use VMs to sandbox containers. However, VMs incur high runtime overhead (not to be confused with fast booting) and need to be provisioned. We use hardware virtualization to directly run system/app containers -- no I/O emulation, no expensive VM exits, scale as needed.
- rad_gruchalski 5y ago
  I was the one who posted that question and I deleted it because I understood I was asking to compare apple to oranges. Firecracker uses KVM under the hood. With regards to the overhead, sure, there's going to be resources needed but firecracker is pretty good at running VMs will low level overhead. We're talking thousands on a single host. I've done some fair share of evaluating firecracker for https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild and the need to provision is red herring. firebuild can run a VM directly from a Dockerfile and Docker image. Fly.io team does something similar. It's basically a fully functional app out of the container within a matter of milliseconds. What would be nice is to see a direct comparison between your solution and firecracker.
  - tptacek 5y ago
    (I'm a developer at Fly). I hesitated to chime in with specifics because I haven't done any real work on timing and profiling and because my dev environment is a NUC which, while zippy, is not the state of the art execution environment for Firecracker. But, long story short: I restart test Firecrackers constantly, from shell scripts, and from start to my Go application logging its first "I'm listening" message after its VM has fully initialized, it's essentially instantaneous. I'm sure it could be made faster, but making it faster would in no meaningful way change my life. Instances on Fly can take a second or two to come up, but that's orchestration overhead, not Firecracker (Fly is zippy, too, don't get me wrong.)
    - deleted 5y ago
      [deleted]
    - rad_gruchalski 5y ago
      That’s definitely true for no orchestration. If the kernel and rootfs is at the jailer location, things are instant. You guys have your owninit system which makes things pretty fast, indeed. I rely on whatever init system is the default for a given os so that adds an overhead because it implies running the app as an upstart / systemd / whatever service. Love your work by the way.
- new_realist 5y ago
  So, like gVisor's KVM support?
catern 5y ago
>Patent pending. I don't think there's much of a market for a new closed-source, patent-encumbered hypervisor for Linux... And, perhaps obviously, I personally would never use such a thing.
deleted 5y ago
[deleted]
smoldesu 5y ago
Not interested in using it, but the licensing here is going to be the real killer blow here. Word to the wise: the market of people who want to use your tool and pay for it is smaller than you think. It's the same mindset that kills most of these projects, thinking that the "average users" are going to represent a considerable portion of your income. It won't. Focus on milking enterprise users who want to pay hand over fist for stuff like this, but don't keep it closed source and slap your "patent pending" line on top. You're selling zero-marginal-utility here, and it's not super hard to see.
- ashishbijlani 5y ago
  Thanks for the feedback! We are still trying to figure out the business model.
tptacek 5y ago
I think people would take this more seriously and engage with it more deeply if you posted the technology white paper, rather than asking for people to sign up for it. There are I'm sure details I'm missing, but Firecracker startup (for instance) is very fast; so, for that matter, is vanilla gvisor.
- ashishbijlani 5y ago
  Thanks! We will post detailed technical docs on design/features soon. FC improved kernel boot up time (by stripping unnecessary functionality). However, runtime performance takes a hit due to I/O virtualization/emulation: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
  - tptacek 5y ago
    Have you measured the performance difference between your system and Firecracker? I'm interested in knowing more about how you're quantifying this. As I said, it'd be easier to engage with this if your technical overview was public. It's also a little tricky to get my head around how you're simultaneously winning on I/O performance by not virtualizing it, and also shielding me from host kernel vulnerabilities. Some of that I/O overhead you're talking about is spent in the service of putting a security boundary between my guests and my host.
    - ashishbijlani 5y ago
      Yes, we will post the numbers soon. However, what I was alluding to in my last comment was that FC runtime performance is far from native (i.e., no VMs). Therefore, there's definitely room for improvement. Depending on the use case, that may or may not be meaningful.
      - tptacek 5y ago
        Looking forward to seeing more about this. Thanks!
adgezaza 5y ago
I can see this being useful for something I'm doing with shared infrastructure. What are your timelines like for a stable version?
- ashishbijlani 5y ago
  It depends on the requirements. We can start as early as next month. Would it be possible for you to get in touch on our Slack?
efitz 5y ago
The site seems light on details. As a security guy, I'm particularly interested in (and a skeptic of) claims of "de-privileging" a Linux kernel. Firecracker[1] micro VMs are very fast, "feel" like containers, and don't depend on modifying the guest OS for security. Plus they're FOSS. This is what I'd measure Kwarantine against; you need to be in the ballpark on all these characteristics and knock it out of the park on manageability to interest me. [1] https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker
- ashishbijlani 5y ago
  Being security/systems researchers, that's exactly what our goals are as well.

16 points by ashishbijlani 5y ago

Hello HN! We've been working on a new hypervisor https://kwarantine.xyz https://kwarantine.xyz that can run strongly isolated containers. This is still a WIP, but we wanted to give the community an idea about our approach, its benefits, and various use cases it unlocks. Today, VMs are used to host containers, and make up for the lack of strong security as well as kernel isolation in containers. This work adds this missing security piece in containers. We plan on launching a free private beta soon. Meanwhile, we'd deeply appreciate any feedback, and happy to answer any questions here or on our slack channel. Thanks!

ashishbijlani 5y ago

There was a question (now deleted) on difference between this and Firecracker. FC/gVisor/Kata use VMs to sandbox containers. However, VMs incur high runtime overhead (not to be confused with fast booting) and need to be provisioned. We use hardware virtualization to directly run system/app containers -- no I/O emulation, no expensive VM exits, scale as needed.

rad_gruchalski 5y ago
I was the one who posted that question and I deleted it because I understood I was asking to compare apple to oranges. Firecracker uses KVM under the hood. With regards to the overhead, sure, there's going to be resources needed but firecracker is pretty good at running VMs will low level overhead. We're talking thousands on a single host. I've done some fair share of evaluating firecracker for https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild and the need to provision is red herring. firebuild can run a VM directly from a Dockerfile and Docker image. Fly.io team does something similar. It's basically a fully functional app out of the container within a matter of milliseconds. What would be nice is to see a direct comparison between your solution and firecracker.
- tptacek 5y ago
  (I'm a developer at Fly). I hesitated to chime in with specifics because I haven't done any real work on timing and profiling and because my dev environment is a NUC which, while zippy, is not the state of the art execution environment for Firecracker. But, long story short: I restart test Firecrackers constantly, from shell scripts, and from start to my Go application logging its first "I'm listening" message after its VM has fully initialized, it's essentially instantaneous. I'm sure it could be made faster, but making it faster would in no meaningful way change my life. Instances on Fly can take a second or two to come up, but that's orchestration overhead, not Firecracker (Fly is zippy, too, don't get me wrong.)
  - deleted 5y ago
    [deleted]
  - rad_gruchalski 5y ago
    That’s definitely true for no orchestration. If the kernel and rootfs is at the jailer location, things are instant. You guys have your owninit system which makes things pretty fast, indeed. I rely on whatever init system is the default for a given os so that adds an overhead because it implies running the app as an upstart / systemd / whatever service. Love your work by the way.
new_realist 5y ago
So, like gVisor's KVM support?

catern 5y ago

>Patent pending. I don't think there's much of a market for a new closed-source, patent-encumbered hypervisor for Linux... And, perhaps obviously, I personally would never use such a thing.

deleted 5y ago

[deleted]

smoldesu 5y ago

Not interested in using it, but the licensing here is going to be the real killer blow here. Word to the wise: the market of people who want to use your tool and pay for it is smaller than you think. It's the same mindset that kills most of these projects, thinking that the "average users" are going to represent a considerable portion of your income. It won't. Focus on milking enterprise users who want to pay hand over fist for stuff like this, but don't keep it closed source and slap your "patent pending" line on top. You're selling zero-marginal-utility here, and it's not super hard to see.

ashishbijlani 5y ago
Thanks for the feedback! We are still trying to figure out the business model.

tptacek 5y ago

I think people would take this more seriously and engage with it more deeply if you posted the technology white paper, rather than asking for people to sign up for it. There are I'm sure details I'm missing, but Firecracker startup (for instance) is very fast; so, for that matter, is vanilla gvisor.

ashishbijlani 5y ago
Thanks! We will post detailed technical docs on design/features soon. FC improved kernel boot up time (by stripping unnecessary functionality). However, runtime performance takes a hit due to I/O virtualization/emulation: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
- tptacek 5y ago
  Have you measured the performance difference between your system and Firecracker? I'm interested in knowing more about how you're quantifying this. As I said, it'd be easier to engage with this if your technical overview was public. It's also a little tricky to get my head around how you're simultaneously winning on I/O performance by not virtualizing it, and also shielding me from host kernel vulnerabilities. Some of that I/O overhead you're talking about is spent in the service of putting a security boundary between my guests and my host.
  - ashishbijlani 5y ago
    Yes, we will post the numbers soon. However, what I was alluding to in my last comment was that FC runtime performance is far from native (i.e., no VMs). Therefore, there's definitely room for improvement. Depending on the use case, that may or may not be meaningful.
    - tptacek 5y ago
      Looking forward to seeing more about this. Thanks!

adgezaza 5y ago

I can see this being useful for something I'm doing with shared infrastructure. What are your timelines like for a stable version?

ashishbijlani 5y ago
It depends on the requirements. We can start as early as next month. Would it be possible for you to get in touch on our Slack?

efitz 5y ago

The site seems light on details. As a security guy, I'm particularly interested in (and a skeptic of) claims of "de-privileging" a Linux kernel. Firecracker[1] micro VMs are very fast, "feel" like containers, and don't depend on modifying the guest OS for security. Plus they're FOSS. This is what I'd measure Kwarantine against; you need to be in the ballpark on all these characteristics and knock it out of the park on manageability to interest me. [1] https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker

ashishbijlani 5y ago
Being security/systems researchers, that's exactly what our goals are as well.