7 ms·

Atmanos: Build Go programs that run directly on the Xen hypervisor

193 points by loppers92 9y ago

StudentStuff 9y ago
This is quite interesting, are there major performance benefits to using something like AtmanOS? My main concern is if something like this will ever be able to run bare metal, and cut out the overhead of a hypervisor, if performance is critical, then a dedicated box is likely. Then again, this brings us to the same issues that the *BSD and Darwin kernels have, with really lackluster driver availability. While some big corps like Sony might get AMD to build them a performant GPU driver, the tragedy of the commons situation repeatedly occurs with BSD licensed projects, major improvements aren't contributed upstream reliably as there is no business reason.
- jsiepkes 9y ago
  Well performance aside there is also the benefit of not having to maintaining userland.
  - md_ 9y ago
    What are the overheads of maintaining userland? Not trying to be stupid, but I'm not really clear on what the advantage is of this system. Do I understand correctly that the dom0 OS is still providing drivers and hardware abstraction? So both that and some userland exists somewhere in the stack; it's just not duplicated in the virtualized OSes? Is the main goal simplicity, performance, or something else?
    - stonewhite 9y ago
      It also implies security due to reduced attack surface and non-standart interfaces.
    - jsiepkes 9y ago
      Userland isn't duplicated, it's 2 totally separated userlands. The userlands may be the same distro in case of Xen (for example RHEL 7) but they still lead separate lives; They are different installations. So that's also multiple userlands in which you have to for example deploy patches (RPM's, Deb's, etc.) for your SSH install.
      - md_ 9y ago
        Yes, that's what I meant by "duplicated." :) And yes, I see some value in the reduction of maintenance costs, but you're not doing this manually--you have a package manager of some sort and you're automatically tracking some standard image (either for your company or from some upstream maintainer like Canonical). So conceptually I get the simplicity argument, but practically speaking, it's not really more work to maintain two userlands vs one, right? I guess there's also an argument of resource (disk, memory footprint, etc) overhead of the second userland. It's not clear to me how significant that is, which was part of my question.
- jpgvm 9y ago
  That is why you use Xen as a base. Xen lets the dom0 (primary domain) handle the device drivers and then provides guests access to them via paravirtualised interfaces. I.e you get Linux compatibility and then you can run any OS that has paravirtualised drivers written for it. Generally if you are going to use something like this or MirageOS you will choose to pass through certain real hardware to the guest and write drivers for it. Say for instance pass through an Intel NIC and then have your application embed a TCP stack and DPDK like components so that you can run your application at line rate.
hamandcheese 9y ago
I’d be interested to see performance comaprisons between this and Go on a Linux variant. Aside from performance, what are the alleged benefits of a unikernel? Security?
- ComputerGuru 9y ago
  Much smaller attack surface.
  - zlynx 9y ago
    Not really. Now instead of having to break the application, then break the kernel it's running under and then attack the VM host, you only have to attack the app and can then go directly at the VM host. Unikernels just remove a whole security layer. May as well run the app as a user process on the host and forget the VM.
    - ComputerGuru 9y ago
      It depends on what you're securing and what the weakest link in your chain is. The assumption is that your app is doing something valuable (i.e. there is value in compromising the app, not just as a means of compromising the host). Perhaps I should have phrased that differently, but you've reduced the attack surface for compromising your _app_ via the environment (instead of the environment via your app). (big disclaimer: this is assuming Xen bugs are much more valuable than app bugs and someone with a Xen exploit won't be focusing on you.)
    - fiokoden 9y ago
      Wrong. Cracking the app does not give you access to the VM host. These apps are running in a VM. There's no relationship at all between cracking the app and cracking VM host.
      - viraptor 9y ago
        I believe what zlynx meant was that if you want to escalate from the VM to the host, you usually do that via the virtualised devices. On a normal system, your path (for a web service on Linux) would be: App exploit (-> LSM breakout?) -> local to root escalation -> VM to host escalation via device. For a unikernel deployment that's just: App exploit -> VM to host via device.
fiokoden 9y ago
How can any performance gain match the benefits of running on a tuned SMP kernel? What exactly happens with SMP with this sort of application?
- baruch 9y ago
  You either run it on a single CPU or the "os" layer needs to provide for thread migration. It also obviously needs to provide thread scheduling.
  - fiokoden 9y ago
    Which is to say, DIY symmetric multiprocessing, or BYO kernel code in-application. I'd have to read a compelling technical explanation before believing this could perform better than a Linux or BSD kernel. In most cases, the Go code is going to be single CPU, and that ain't the way the world works anymore. There's going to be a bunch of wasted computing power on that VM
    - farazbabar 9y ago
      You can bind processing for a consistent subset of requests to an individual cpu core - essentially sharding requests across cpu cores and benefitting from very high l1 and l2 cache utilization. The idea is to treat the system with multiple cores as bunch of single cpu nodes connected via bus instead of network and without the unnecessary overhead of thread and process related context swtiching.
      - fiokoden 9y ago
        And now you're implementing SMP in application. Zero chance of doing that as well as the Linux or bsd kernel.
        yazaddaruvala 9y ago
        Why? You're assuming there is no sharable code. Instead of sharing the code at runtime, i.e. what an OS does. You could easily share code at compile time, i.e. statically link a library. Because of sharable code, "implementing * in application" should always be at-least as performant as the best generic implementation (i.e. the implementation you find in a general purpose OS). However, when appropriate, customizing the implementation for the application would allow it to become even more performant.
rlonstein 9y ago
Nice. I immediately though of https://mirage.io/ https://mirage.io/ and http://erlangonxen.org/ http://erlangonxen.org/ but with a more "mainstream" language.
Timothycquinn 9y ago
Bryan Cantrill has a great post on unikernel models: https://www.joyent.com/blog/unikernels-are-unfit-for-production https://www.joyent.com/blog/unikernels-are-unfit-for-product... From my read, the benefits do not outweigh the costs. If you want light weight microservices, OS level virtualization is the way to go.
- zenlikethat 9y ago
  AtmanOS says in the README it's highly experimental and doesn't imply you should use it for production at all. So who cares? What else is HN for if not for posting cool hacks and projects and encouraging fellow builders?
  - xena 9y ago
    Every product shipped into production today started as a highly experimental prototype.
- im_down_w_otp 9y ago
  Brenden Gregg also has a great post on unikernel models: http://www.brendangregg.com/blog/2016-01-27/unikernel-profiling-from-dom0.html http://www.brendangregg.com/blog/2016-01-27/unikernel-profil... From my read, it counters Bryan Cantrill's claim that, "unikernels are undebuggable". From personal experience I'm also quite certain Bryan Cantrill's claim is spurious in that regard, as I've used both debugging and tracing facilities w/ LING unikernels to assess a number of runtime and clustering issues.
  - abrookewood 9y ago
    Are you still using LING? The project looks like it died.
  - dnautics 9y ago
    Ling is a special beast since erlang is basically begging to be it's own runtime os.
VirtualAirwaves 9y ago
We've been running Erlang directly on Xen with "no os" for a while. Works great and very efficient.
- pmarreck 9y ago
  I have always been fascinated by this combo. Can you discuss your use-cases and wins using this stack?
  - diwu1989 9y ago
    might be this or something similar, http://erlangonxen.org/ http://erlangonxen.org/
    - pmarreck 9y ago
      no, I know that's what he's referring to, but I wanted to know how s/he was actually using it in practice, for a real-world task
- abrookewood 9y ago
  Can you give any more details? I thought LING was dead?
liuw 9y ago
Somewhat related: for anyone who is interested in unikernel development, there is a new proposed project called Unicore under the Xen Project umbrella. That project aims to reduce the effort for porting applications to run in unikernels on different platforms (Xen, KVM and baremetal). https://lists.xen.org/archives/html/xen-devel/2017-09/msg03680.html https://lists.xen.org/archives/html/xen-devel/2017-09/msg036...
FrodeF 9y ago
is this something like includeos http://www.includeos.org/ http://www.includeos.org/
acidtrucks 9y ago
We're returning to the days of booting directly to you application, like we did with apple II

193 points by loppers92 9y ago

This is quite interesting, are there major performance benefits to using something like AtmanOS? My main concern is if something like this will ever be able to run bare metal, and cut out the overhead of a hypervisor, if performance is critical, then a dedicated box is likely. Then again, this brings us to the same issues that the *BSD and Darwin kernels have, with really lackluster driver availability. While some big corps like Sony might get AMD to build them a performant GPU driver, the tragedy of the commons situation repeatedly occurs with BSD licensed projects, major improvements aren't contributed upstream reliably as there is no business reason.

jsiepkes 9y ago
Well performance aside there is also the benefit of not having to maintaining userland.
- md_ 9y ago
  What are the overheads of maintaining userland? Not trying to be stupid, but I'm not really clear on what the advantage is of this system. Do I understand correctly that the dom0 OS is still providing drivers and hardware abstraction? So both that and some userland exists somewhere in the stack; it's just not duplicated in the virtualized OSes? Is the main goal simplicity, performance, or something else?
  - stonewhite 9y ago
    It also implies security due to reduced attack surface and non-standart interfaces.
  - jsiepkes 9y ago
    Userland isn't duplicated, it's 2 totally separated userlands. The userlands may be the same distro in case of Xen (for example RHEL 7) but they still lead separate lives; They are different installations. So that's also multiple userlands in which you have to for example deploy patches (RPM's, Deb's, etc.) for your SSH install.
    - md_ 9y ago
      Yes, that's what I meant by "duplicated." :) And yes, I see some value in the reduction of maintenance costs, but you're not doing this manually--you have a package manager of some sort and you're automatically tracking some standard image (either for your company or from some upstream maintainer like Canonical). So conceptually I get the simplicity argument, but practically speaking, it's not really more work to maintain two userlands vs one, right? I guess there's also an argument of resource (disk, memory footprint, etc) overhead of the second userland. It's not clear to me how significant that is, which was part of my question.
jpgvm 9y ago
That is why you use Xen as a base. Xen lets the dom0 (primary domain) handle the device drivers and then provides guests access to them via paravirtualised interfaces. I.e you get Linux compatibility and then you can run any OS that has paravirtualised drivers written for it. Generally if you are going to use something like this or MirageOS you will choose to pass through certain real hardware to the guest and write drivers for it. Say for instance pass through an Intel NIC and then have your application embed a TCP stack and DPDK like components so that you can run your application at line rate.

hamandcheese 9y ago

I’d be interested to see performance comaprisons between this and Go on a Linux variant. Aside from performance, what are the alleged benefits of a unikernel? Security?

ComputerGuru 9y ago
Much smaller attack surface.
- zlynx 9y ago
  Not really. Now instead of having to break the application, then break the kernel it's running under and then attack the VM host, you only have to attack the app and can then go directly at the VM host. Unikernels just remove a whole security layer. May as well run the app as a user process on the host and forget the VM.
  - ComputerGuru 9y ago
    It depends on what you're securing and what the weakest link in your chain is. The assumption is that your app is doing something valuable (i.e. there is value in compromising the app, not just as a means of compromising the host). Perhaps I should have phrased that differently, but you've reduced the attack surface for compromising your _app_ via the environment (instead of the environment via your app). (big disclaimer: this is assuming Xen bugs are much more valuable than app bugs and someone with a Xen exploit won't be focusing on you.)
  - fiokoden 9y ago
    Wrong. Cracking the app does not give you access to the VM host. These apps are running in a VM. There's no relationship at all between cracking the app and cracking VM host.
    - viraptor 9y ago
      I believe what zlynx meant was that if you want to escalate from the VM to the host, you usually do that via the virtualised devices. On a normal system, your path (for a web service on Linux) would be: App exploit (-> LSM breakout?) -> local to root escalation -> VM to host escalation via device. For a unikernel deployment that's just: App exploit -> VM to host via device.

fiokoden 9y ago

How can any performance gain match the benefits of running on a tuned SMP kernel? What exactly happens with SMP with this sort of application?

baruch 9y ago
You either run it on a single CPU or the "os" layer needs to provide for thread migration. It also obviously needs to provide thread scheduling.
- fiokoden 9y ago
  Which is to say, DIY symmetric multiprocessing, or BYO kernel code in-application. I'd have to read a compelling technical explanation before believing this could perform better than a Linux or BSD kernel. In most cases, the Go code is going to be single CPU, and that ain't the way the world works anymore. There's going to be a bunch of wasted computing power on that VM
  - farazbabar 9y ago
    You can bind processing for a consistent subset of requests to an individual cpu core - essentially sharding requests across cpu cores and benefitting from very high l1 and l2 cache utilization. The idea is to treat the system with multiple cores as bunch of single cpu nodes connected via bus instead of network and without the unnecessary overhead of thread and process related context swtiching.
    - fiokoden 9y ago
      And now you're implementing SMP in application. Zero chance of doing that as well as the Linux or bsd kernel.
      - yazaddaruvala 9y ago
        Why? You're assuming there is no sharable code. Instead of sharing the code at runtime, i.e. what an OS does. You could easily share code at compile time, i.e. statically link a library. Because of sharable code, "implementing * in application" should always be at-least as performant as the best generic implementation (i.e. the implementation you find in a general purpose OS). However, when appropriate, customizing the implementation for the application would allow it to become even more performant.

rlonstein 9y ago

Nice. I immediately though of https://mirage.io/ https://mirage.io/ and http://erlangonxen.org/ http://erlangonxen.org/ but with a more "mainstream" language.

Timothycquinn 9y ago

Bryan Cantrill has a great post on unikernel models: https://www.joyent.com/blog/unikernels-are-unfit-for-production https://www.joyent.com/blog/unikernels-are-unfit-for-product... From my read, the benefits do not outweigh the costs. If you want light weight microservices, OS level virtualization is the way to go.

zenlikethat 9y ago
AtmanOS says in the README it's highly experimental and doesn't imply you should use it for production at all. So who cares? What else is HN for if not for posting cool hacks and projects and encouraging fellow builders?
- xena 9y ago
  Every product shipped into production today started as a highly experimental prototype.
im_down_w_otp 9y ago
Brenden Gregg also has a great post on unikernel models: http://www.brendangregg.com/blog/2016-01-27/unikernel-profiling-from-dom0.html http://www.brendangregg.com/blog/2016-01-27/unikernel-profil... From my read, it counters Bryan Cantrill's claim that, "unikernels are undebuggable". From personal experience I'm also quite certain Bryan Cantrill's claim is spurious in that regard, as I've used both debugging and tracing facilities w/ LING unikernels to assess a number of runtime and clustering issues.
- abrookewood 9y ago
  Are you still using LING? The project looks like it died.
- dnautics 9y ago
  Ling is a special beast since erlang is basically begging to be it's own runtime os.

VirtualAirwaves 9y ago

We've been running Erlang directly on Xen with "no os" for a while. Works great and very efficient.

pmarreck 9y ago
I have always been fascinated by this combo. Can you discuss your use-cases and wins using this stack?
- diwu1989 9y ago
  might be this or something similar, http://erlangonxen.org/ http://erlangonxen.org/
  - pmarreck 9y ago
    no, I know that's what he's referring to, but I wanted to know how s/he was actually using it in practice, for a real-world task
abrookewood 9y ago
Can you give any more details? I thought LING was dead?

liuw 9y ago

Somewhat related: for anyone who is interested in unikernel development, there is a new proposed project called Unicore under the Xen Project umbrella. That project aims to reduce the effort for porting applications to run in unikernels on different platforms (Xen, KVM and baremetal). https://lists.xen.org/archives/html/xen-devel/2017-09/msg03680.html https://lists.xen.org/archives/html/xen-devel/2017-09/msg036...

FrodeF 9y ago

is this something like includeos http://www.includeos.org/ http://www.includeos.org/

acidtrucks 9y ago

We're returning to the days of booting directly to you application, like we did with apple II