5 ms·

Nanos: A kernel designed to run only one application in virtualized environment

125 points by yagizdegirmenci 5y ago

deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
sshine 5y ago
Somewhat related is MirageOS, a microkernel library for producing applications that run directly in a hypervisor, like Xen. So no virtual environment necessary. And thanks to the memory safety of OCaml (given certain assumptions about the style of programming, because any sufficiently versatile language comes with footguns) you don’t even really need the virtual memory system and other conveniences of a modern operating system.
eatonphil 5y ago
A couple of other interesting articles from the nanos team by way of the /r/unikernel subreddit: https://nanovms.com/dev/tutorials/debugging-nanos-unikernels-with-gdb-and-ops https://nanovms.com/dev/tutorials/debugging-nanos-unikernels... https://nanovms.com/dev/tutorials/finding-memory-management-errors-in-nanos https://nanovms.com/dev/tutorials/finding-memory-management-... https://nanovms.com/dev/tutorials/profiling-and-tracing-nanos-unikernel https://nanovms.com/dev/tutorials/profiling-and-tracing-nano...
brentcetinich 5y ago
See also https://github.com/direktiv/vorteil https://github.com/direktiv/vorteil The above also wrote a Uni kernel but it seems they abandoned that as was too large a problem and just link and package now
lmilcin 5y ago
I mean... if we go this way why even have a kernel? Why not just have a single application with everything that is needed to run it as VM?
- nomercy400 5y ago
  You mean, run an 'application' on a 'server'. Maybe we should call it.. an application server. And have it 'operate' it directly on hardware with only a minimal 'system' layer for common operations. I feel old.
- eyberg 5y ago
  The scope of "everything that is needed to run" is a lot higher than might appear and since it is common code that is applicable to every app that would be deployed it is packaged as a 'kernel'. Something has to talk to the network, to the disk, keep track of time, etc. You might be surprised at how much code is involved in merely writing a byte to disk, efficiently and in a manner that a random webapp might use. One very common misconception about unikernels is that they don't have kernels which every single implementation out there has one - it just might be smaller or less-featured than others. So, at least in our view, it's not about having a 'small' kernel it is more about the architecture.
  - lmilcin 5y ago
    You can have libraries that can implement device driver functionality and talk directly to devices. Actually there are some (DPDK - Data Plane Development Kit and SPDK - Storage Performance Development Kit, for example). > The scope of "everything that is needed to run" is a lot higher than might appear Having written one algotrading framework with full kernel bypass which required me to account for every single piece of kernel functionality in use by the application (mostly to eliminate its use) I actually think it is the opposite. Most applications do not need a lot from the kernel to function and what they are using could be supplied as library. Main reasons to have kernel -- protect shared resources and impose security constraints -- are not present when you intend to only have one application in the system.
    - eyberg 5y ago
      Whether code is packaged a a library or inside a base kernel is definitely open to interpretation/design. We, for instance have the concept of a 'klib' which is code we don't want packaged in the base kernel but is optional and can be included at build-time. For instance deploying to Azure requires a cloud-init call to check-in to the meta-data server to tell it the instance has booted - not something you want to put inside every image if you are only deploying to say Google Cloud. Likewise, we have another klib that provides APM functionality but checks-in to a proprietary server so clearly not everyone wants that inside their image either. However, there is a lot more than just networking/storage drivers and some of it is very common code. Page table management for instance. Do you have 2,3,4 page table levels? How do you allocate memory to the app? IOAPIC vs PIC? Do you support calls the way GLIBC wants? RTC vs TSC vs PIT? I'm not saying any of these can't be librarized but they are most definitely not choices I would expose to the vast majority of end-users to choose at build-time for.
tailspin2019 5y ago
From the FAQs: > Does this Work for My Mac M1 or M2? > No one has reported trying to run this on a Mac M1 or M2 yet but since these run ARM we don't feel that it is a good laptop to be using if you are deploying to X86 servers. At best you will experience slowness as the machines will need to emulate a different architecture. This isn't something we expect software updates to fix and both Docker and VMWare state the same thing. Even if you wish to deploy to ARM servers we don't feel that the M1s and M2s are going to be helpful as they are very different from most commodity ARM servers. Erm... they're not Mac people then... EDIT: I was probably being a bit grumpy - read the full replies below for further context...
- amelius 5y ago
  Is that a bad thing?
  - tailspin2019 5y ago
    I probably should have clarified: There's no such thing as a "Mac M2" for a start. (Yet) But I also found this paragraph a bit odd, because the machines I use locally never match the architecture of what I'm deploying to. In fact (in my view) it's largely irrelevant. I develop on Macs and deploy to Linux and Windows servers, with a mixture of ARM/Intel. I don't quite understand this sentiment. Not to mention the fact that they start off by saying that no one has even tried it yet (presumably to run Nanos in a VM on an M1 Mac). It just seems a bit uninformed and unnecessarily opinionated. As a Mac user it puts me off digging much deeper into the project. Maybe that's the wrong takeaway, but that was my first reaction.
- eyberg 5y ago
  We use macs. :) I'm typing on one right now. There are 2 large problems with macs for shipping to x86 servers in the cloud (our main target): * Different file format: elf vs mach-o - this is why many devs that use mac rely on things like docker or vagrant. * x86 vs arm: We do support ARM to a degree right now but the vast majority of our end users are deploying to x86 vms. The problem here of course is that ops produces machine images that are ran on a hypervisor so this works great on an x86 mac (for dev/test) but is very slow on apple silicon cause of the translation involved. Looks like one of our users has been playing around with it though so YMMV: https://github.com/imarsman/dockerops https://github.com/imarsman/dockerops
cyanydeez 5y ago
ah, the classic blind download this script and run, trust us guiz
- eyberg 5y ago
  You can also build from source here: https://github.com/nanovms/nanos https://github.com/nanovms/nanos && https://github.com/nanovms/ops https://github.com/nanovms/ops . There are also packages available through AUR/homebrew and the like: https://ops.city/downloads https://ops.city/downloads . The script is only there facilitate the 'install' such as ensuring you have qemu installed locally or assessing whether you have kvm/hvf rights/etc. Also, I don't think this is documented yet but you can target various PRs/builds with ops via this way: ops run /bin/ls --nanos-version d632de2
  - intpx 5y ago
    that's nice. It doesn't change the fact that its incredibly bad practice to ever run a piped curl, thus its incredibly irresponsible to make it the top of page delivery mechanism for your shiny new project. Its even more irresponsible to put that silly little shell script behind a url shortener. That's like, two major security, and just plain old fashioned hygiene violations and I haven't even moved past the first line. Things like this shouldn't be 'easy'. they should be well-documented, sane and safe. This is certainly an interesting and novel approach to tackle some of the absolute mess Docker has made of the world, but at the end of the day, if a system is compromised, its compromised, whether that be the docker user or the kvm/hvf group members.
ComputerGuru 5y ago
It would be great if the picture demo running the node application included timestamps. The landing page keeps using "fast" to mean "bandwidth" but without any mention of latency - my primary question is how long it takes to boot the kernel and start launching the userland process (i.e. cold start time) but there's no mention of that.
- eyberg 5y ago
  We've seen boot times in the 60s and 70s of ms but have put absolutely no work into optimizing that. We could drive that down substantially for payloads like virtual network functions. I should point out that boot time is highly dependent on two things: infrastructure of choice and application payload. For instance, your typical rails or JVM payload might take longer to init then actual boot time. Similarly booting on Azure can be different than booting under firecracker on your own hardware.
desktopninja 5y ago
Mmm. Fascinating ... am wondering already how well the JVM would run on it. Also maybe this could the solution I've been looking for for "Just enough OS for Virtualbox" and have it run on baremetal. VBox has one of the best VM management UIs in my opinion.
- eyberg 5y ago
  There are several pre-built JVM packages and VBox and OCI works.
GekkePrutser 5y ago
Cool, so this is a bit like MS-DOS? With the splitting of applications into microservices all running in separate containers this does make a lot of sense again. Why have a full multiuser os running for each process. Will look into this more.
nix23 5y ago
Whats the difference to OSv? https://github.com/cloudius-systems/osv https://github.com/cloudius-systems/osv
- dkjaudyeqooe 5y ago
  See https://nanos.org/faq https://nanos.org/faq where there is a comparison table but it's not complete or out of date, because OSv website says: "OSv supports many managed language runtimes including unmodified JVM, Python 2 and 3, Node.JS, Ruby, Erlang as well as languages compiling directly to native machine code like Golang and Rust" This seems like a big deal on the Nanos side: "Another big difference is that Nanos keeps the kernel/user boundary. In our testing removing the large process to process context switching that general purpose operating systems still removes quite a lot of the perceived cost in other systems. We keep the internal kernel <> user switch for security purposes. Without it page protections are basically useless as an attacker can adjust the permissions themselves with privileged instructions." This would seem to me that they are slower than OSv.
- convolvatron 5y ago
  they look pretty similar from the outside nanos is recently written for this particular use case and uses lwip for networking osv looks like FreeBSD with some machinery around it to package single applications and run them on boot
  - nix23 5y ago
    >osv looks like FreeBSD with some machinery around it to package single applications and run them on boot Just because ZFS? Because everything else is not from BSD's, it's a unikernel made to run on a hypervisor made to run linux-bin's, i really don't see a difference....or any plus to use nanos.
    - dkjaudyeqooe 5y ago
      OSv describes the project as BSD code with a Linux compatibility layer. ZFS is optional.
deleted 5y ago
[deleted]

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

sshine 5y ago

Somewhat related is MirageOS, a microkernel library for producing applications that run directly in a hypervisor, like Xen. So no virtual environment necessary. And thanks to the memory safety of OCaml (given certain assumptions about the style of programming, because any sufficiently versatile language comes with footguns) you don’t even really need the virtual memory system and other conveniences of a modern operating system.

eatonphil 5y ago

A couple of other interesting articles from the nanos team by way of the /r/unikernel subreddit: https://nanovms.com/dev/tutorials/debugging-nanos-unikernels-with-gdb-and-ops https://nanovms.com/dev/tutorials/debugging-nanos-unikernels... https://nanovms.com/dev/tutorials/finding-memory-management-errors-in-nanos https://nanovms.com/dev/tutorials/finding-memory-management-... https://nanovms.com/dev/tutorials/profiling-and-tracing-nanos-unikernel https://nanovms.com/dev/tutorials/profiling-and-tracing-nano...

brentcetinich 5y ago

See also https://github.com/direktiv/vorteil https://github.com/direktiv/vorteil The above also wrote a Uni kernel but it seems they abandoned that as was too large a problem and just link and package now

lmilcin 5y ago

I mean... if we go this way why even have a kernel? Why not just have a single application with everything that is needed to run it as VM?

nomercy400 5y ago
You mean, run an 'application' on a 'server'. Maybe we should call it.. an application server. And have it 'operate' it directly on hardware with only a minimal 'system' layer for common operations. I feel old.
eyberg 5y ago
The scope of "everything that is needed to run" is a lot higher than might appear and since it is common code that is applicable to every app that would be deployed it is packaged as a 'kernel'. Something has to talk to the network, to the disk, keep track of time, etc. You might be surprised at how much code is involved in merely writing a byte to disk, efficiently and in a manner that a random webapp might use. One very common misconception about unikernels is that they don't have kernels which every single implementation out there has one - it just might be smaller or less-featured than others. So, at least in our view, it's not about having a 'small' kernel it is more about the architecture.
- lmilcin 5y ago
  You can have libraries that can implement device driver functionality and talk directly to devices. Actually there are some (DPDK - Data Plane Development Kit and SPDK - Storage Performance Development Kit, for example). > The scope of "everything that is needed to run" is a lot higher than might appear Having written one algotrading framework with full kernel bypass which required me to account for every single piece of kernel functionality in use by the application (mostly to eliminate its use) I actually think it is the opposite. Most applications do not need a lot from the kernel to function and what they are using could be supplied as library. Main reasons to have kernel -- protect shared resources and impose security constraints -- are not present when you intend to only have one application in the system.
  - eyberg 5y ago
    Whether code is packaged a a library or inside a base kernel is definitely open to interpretation/design. We, for instance have the concept of a 'klib' which is code we don't want packaged in the base kernel but is optional and can be included at build-time. For instance deploying to Azure requires a cloud-init call to check-in to the meta-data server to tell it the instance has booted - not something you want to put inside every image if you are only deploying to say Google Cloud. Likewise, we have another klib that provides APM functionality but checks-in to a proprietary server so clearly not everyone wants that inside their image either. However, there is a lot more than just networking/storage drivers and some of it is very common code. Page table management for instance. Do you have 2,3,4 page table levels? How do you allocate memory to the app? IOAPIC vs PIC? Do you support calls the way GLIBC wants? RTC vs TSC vs PIT? I'm not saying any of these can't be librarized but they are most definitely not choices I would expose to the vast majority of end-users to choose at build-time for.

tailspin2019 5y ago

From the FAQs: > Does this Work for My Mac M1 or M2? > No one has reported trying to run this on a Mac M1 or M2 yet but since these run ARM we don't feel that it is a good laptop to be using if you are deploying to X86 servers. At best you will experience slowness as the machines will need to emulate a different architecture. This isn't something we expect software updates to fix and both Docker and VMWare state the same thing. Even if you wish to deploy to ARM servers we don't feel that the M1s and M2s are going to be helpful as they are very different from most commodity ARM servers. Erm... they're not Mac people then... EDIT: I was probably being a bit grumpy - read the full replies below for further context...

amelius 5y ago
Is that a bad thing?
- tailspin2019 5y ago
  I probably should have clarified: There's no such thing as a "Mac M2" for a start. (Yet) But I also found this paragraph a bit odd, because the machines I use locally never match the architecture of what I'm deploying to. In fact (in my view) it's largely irrelevant. I develop on Macs and deploy to Linux and Windows servers, with a mixture of ARM/Intel. I don't quite understand this sentiment. Not to mention the fact that they start off by saying that no one has even tried it yet (presumably to run Nanos in a VM on an M1 Mac). It just seems a bit uninformed and unnecessarily opinionated. As a Mac user it puts me off digging much deeper into the project. Maybe that's the wrong takeaway, but that was my first reaction.
eyberg 5y ago
We use macs. :) I'm typing on one right now. There are 2 large problems with macs for shipping to x86 servers in the cloud (our main target): * Different file format: elf vs mach-o - this is why many devs that use mac rely on things like docker or vagrant. * x86 vs arm: We do support ARM to a degree right now but the vast majority of our end users are deploying to x86 vms. The problem here of course is that ops produces machine images that are ran on a hypervisor so this works great on an x86 mac (for dev/test) but is very slow on apple silicon cause of the translation involved. Looks like one of our users has been playing around with it though so YMMV: https://github.com/imarsman/dockerops https://github.com/imarsman/dockerops

cyanydeez 5y ago

ah, the classic blind download this script and run, trust us guiz

eyberg 5y ago
You can also build from source here: https://github.com/nanovms/nanos https://github.com/nanovms/nanos && https://github.com/nanovms/ops https://github.com/nanovms/ops . There are also packages available through AUR/homebrew and the like: https://ops.city/downloads https://ops.city/downloads . The script is only there facilitate the 'install' such as ensuring you have qemu installed locally or assessing whether you have kvm/hvf rights/etc. Also, I don't think this is documented yet but you can target various PRs/builds with ops via this way: ops run /bin/ls --nanos-version d632de2
- intpx 5y ago
  that's nice. It doesn't change the fact that its incredibly bad practice to ever run a piped curl, thus its incredibly irresponsible to make it the top of page delivery mechanism for your shiny new project. Its even more irresponsible to put that silly little shell script behind a url shortener. That's like, two major security, and just plain old fashioned hygiene violations and I haven't even moved past the first line. Things like this shouldn't be 'easy'. they should be well-documented, sane and safe. This is certainly an interesting and novel approach to tackle some of the absolute mess Docker has made of the world, but at the end of the day, if a system is compromised, its compromised, whether that be the docker user or the kvm/hvf group members.

ComputerGuru 5y ago

It would be great if the picture demo running the node application included timestamps. The landing page keeps using "fast" to mean "bandwidth" but without any mention of latency - my primary question is how long it takes to boot the kernel and start launching the userland process (i.e. cold start time) but there's no mention of that.

eyberg 5y ago
We've seen boot times in the 60s and 70s of ms but have put absolutely no work into optimizing that. We could drive that down substantially for payloads like virtual network functions. I should point out that boot time is highly dependent on two things: infrastructure of choice and application payload. For instance, your typical rails or JVM payload might take longer to init then actual boot time. Similarly booting on Azure can be different than booting under firecracker on your own hardware.

desktopninja 5y ago

Mmm. Fascinating ... am wondering already how well the JVM would run on it. Also maybe this could the solution I've been looking for for "Just enough OS for Virtualbox" and have it run on baremetal. VBox has one of the best VM management UIs in my opinion.

eyberg 5y ago
There are several pre-built JVM packages and VBox and OCI works.

GekkePrutser 5y ago

Cool, so this is a bit like MS-DOS? With the splitting of applications into microservices all running in separate containers this does make a lot of sense again. Why have a full multiuser os running for each process. Will look into this more.

nix23 5y ago

Whats the difference to OSv? https://github.com/cloudius-systems/osv https://github.com/cloudius-systems/osv

dkjaudyeqooe 5y ago
See https://nanos.org/faq https://nanos.org/faq where there is a comparison table but it's not complete or out of date, because OSv website says: "OSv supports many managed language runtimes including unmodified JVM, Python 2 and 3, Node.JS, Ruby, Erlang as well as languages compiling directly to native machine code like Golang and Rust" This seems like a big deal on the Nanos side: "Another big difference is that Nanos keeps the kernel/user boundary. In our testing removing the large process to process context switching that general purpose operating systems still removes quite a lot of the perceived cost in other systems. We keep the internal kernel <> user switch for security purposes. Without it page protections are basically useless as an attacker can adjust the permissions themselves with privileged instructions." This would seem to me that they are slower than OSv.
convolvatron 5y ago
they look pretty similar from the outside nanos is recently written for this particular use case and uses lwip for networking osv looks like FreeBSD with some machinery around it to package single applications and run them on boot
- nix23 5y ago
  >osv looks like FreeBSD with some machinery around it to package single applications and run them on boot Just because ZFS? Because everything else is not from BSD's, it's a unikernel made to run on a hypervisor made to run linux-bin's, i really don't see a difference....or any plus to use nanos.
  - dkjaudyeqooe 5y ago
    OSv describes the project as BSD code with a Linux compatibility layer. ZFS is optional.

deleted 5y ago

[deleted]