10 ms·
The big idea around unikernels
- rackjack 5y agoMirageOS (unikernel) plug: https://mirage.io/ https://mirage.io/ and repo: https://github.com/mirage/mirage https://github.com/mirage/mirage
- yakubin 5y agoA bit OT: what kind of performance do you normally expect from OCaml? I looked at the Prime Sieve challenge[1] recently and with the current submissions the Standard ML solution is 3 times faster than the OCaml one. I've also spent some time optimizing this SML solution and got it several times faster still. With a bit more work I thinkg I could get it up to the speed of the fastest C solution. It's compiled with MLton. From that POV OCaml is a bit disappointing, since I've also viewed SML as a "hipster language" that hasn't had that much money put into it, while OCaml saw quite a bit more development, so I would expect the OCaml compiler to be able to produce better code. I've briefly looked at the OCaml code, but I don't know OCaml as much as SML, so can't really say what may be the issue in the OCaml case. [1]: <https://plummerssoftwarellc.github.io/PrimeView/report?id=rbergen-1638510661.json&hi=False&hf=False&hp=False&fi=&fp=mt&fa=&ff=uf&fb=&tp=False&sc=pp&sd=True https://plummerssoftwarellc.github.io/PrimeView/report?id=rb...> [2]: <https://github.com/PlummersSoftwareLLC/Primes https://github.com/PlummersSoftwareLLC/Primes>
- octachron 5y agoI would expect such a difference between OCaml and MLton (in either direction in fact) to be mostly the result of an apple-to-orange comparison. In this case, the OCaml code looks neither idiomatic nor optimized.
- naasking 5y agoMLTon is a whole program compiler, so it's not exactly comparable to OCaml. OCaml compiles insanely fast, MLTon not so much.
- tarjei_huse 5y agoI'm wondering if anyone has seen any attempts at creating unikernels that work with major scripting languages like PHP, Python or Ruby. I know this would seem a bit like a contradiction to the ethos of unikernels, but it would give the benefits of unikernels (i.e. smaller attack surface) to a much larger audience.
- BBC-vs-neolibs 5y agohttps://news.ycombinator.com/reply?id=29427884&goto=item%3Fid%3D29427449%2329427884 https://news.ycombinator.com/reply?id=29427884&goto=item%3Fi...
- tjansen 5y agoThat should be possible if you use a web server like Apache as main application and then link the module that your scripting language requires. Not sure whether NanoVMs already support it though.
- nderjung 5y agoHey, Unikraft has support for running the Python[0] and Ruby[1] runtimes as unikernels. We're about to release PHP too :) Watch this space! [1]: https://github.com/unikraft/app-python3 https://github.com/unikraft/app-python3 [1]: https://github.com/unikraft/app-ruby https://github.com/unikraft/app-ruby
- tarjei_huse 5y agoCool!
- kitd 5y agoNanosVM has a concept called Packages [1], ie prebuilt images that package a language runtime or application component eg Redis. It looks like they have packages for various scripting languages. [1] https://nanovms.gitbook.io/ops/packages https://nanovms.gitbook.io/ops/packages
- bfung 5y agoThe author of the article is CEO of NanosVM, who makes Nanos unikernel and tooling. It looks pretty interesting with Go, php, and nodejs support. I tried dabbling in MirageOS in the past, and actually love OCaml as a language, but the local setup required to code, build, and deploy is too much for after working a full day. The Ops tool and nanosvm looks much easier to use. I haven’t dug through the unikernel code to see how real it is, but it’s def. attractive efficiency after dealing w/k8 & docker stack all day. https://github.com/nanovms https://github.com/nanovms
- rwmj 5y agoHe's wrong about Linux, it's possible to link regular applications into Linux-based unikernels: https://github.com/unikernelLinux/linux https://github.com/unikernelLinux/linux
- NavinF 5y agoUmm what is this? “This branch is 71 commits ahead, 27784 commits behind torvalds:master” and the lack of any docs/readme isn’t very promising
- rwmj 5y agoIt's under development, and does indeed lack much in the way of docs at the moment, but it's a way to link regular applications into Linux to make unikernels that will run in VMs or on baremetal.
- tremon 5y agoGiven 15k commits per Linux release (see e.g. http://lkml.iu.edu/hypermail/linux/kernel/2105.1/00457.html http://lkml.iu.edu/hypermail/linux/kernel/2105.1/00457.html), that's actually trailing just two upstream kernel releases. That's pretty good for a small project.
- mikewarot 5y agoI didn't see in the documentation where the files are supplied to the run process... if it can be fed just the minimum files required, that's close to good enough to do capability based security.
- kitd 5y agoI believe that's one of the things the `ops` tool does, ie takes an image and bundles it with eg script or config files.
- eyberg 5y agohttps://nanovms.gitbook.io/ops/configuration#files https://nanovms.gitbook.io/ops/configuration#files || I most commonly use https://nanovms.gitbook.io/ops/configuration#dirs https://nanovms.gitbook.io/ops/configuration#dirs || https://nanovms.gitbook.io/ops/configuration#mapdirs https://nanovms.gitbook.io/ops/configuration#mapdirs to not do copies/keep same layout
- deleted 5y ago[deleted]
- hawk_ 5y ago> However, when it crashed, we could easily export the VM image and download it locally and run it. We could attach gdb to it and pinpoint directly where it was having an issue. How does this work, if running the VM again? How does one ensure the same issue is recreated?
- pjc50 5y agoImplicit in that is having a reproduction case, even if it's just "run the thing with a workload until it crashes".
- hawk_ 5y agoRight, that's worse than being able to connect to a VM with full O/S and investigate the problem there by analysing a core dump or something. An issue of ergonomics with unikernels which I was hoping they had a better solution for.
- eyberg 5y agoYou can easily get core dumps and there's a whole wealth of debugging tools available such as ftrace/strace/etc.
- pjc50 5y agoInteresting. Basically the argument is as follows: Conventional operating systems are multi-user: one user shares all their files with all their apps, but effort is put into protecting one user from another. This then gets breached all the time (escalation of privilege), and fails to capture the problem that attacks come from applications. Unikernel/cloud systems are "zero-user": we have something that might be called a "process" in a traditional operating system, but instead of lots of fallible fine grained security boundaries we have only one, the hardware virtualization. The "process" runs directly below the hypervisor and interacts with "virtual hardware", predominantly with other networked services. That raises the question - how is the hardware virtualization for these systems handled and secured? Do you get a virtual PCI bridge to a virtual network card?
- ksec 5y ago>zero-user Wouldn't the normal "single user" term be more appropriate? Or by Zero user you mean something else?
- convolvatron 5y agoI think 'zero user' is right, there just isn't any concept of users or permissions. its really moving the classic OS resource sharing interface to the hypervisor. that's where the storage and network comes from. given that, what machinery do you need to run just one application? maybe a little filesystem, a tcp stack, interfaces to manipulate threads and kernel locks. that's what gets packaged up with your application into a bootable VM
- m4rtink 5y agoSo whoever exploits that one process has complete control over everything, right ?
- pjc50 5y agoWell, they have control over the application, and whatever its VM has access to .. which is supposed to be limited to only the things that the application needs.
- mgaunard 5y agoOperating systems are there to help you share resources. You're going to have numerous applications share the same NIC, PCI bus and memory. As far as performance is concerned, running those applications bare metal on a traditional kernel makes a lot more sense than creating virtual resources that each application can pretend to have exclusive ownership of. The only way to get efficient networking out of a VM is to dedicate the hardware to that VM which defeats the point.
- rwmj 5y agoCertainly in 1970 that was the case. Nowadays things look different (in some ways): Hypervisors already mediate the hardware, presenting virtual NICs etc up to the guests. Partitioning hypervisors split up huge machines into smaller slices with dedicated (real) hardware per slice. And sometimes people really want to run a single application baremetal for performance reasons. And ask yourself how many CPUs you have even in your PC - it's a lot more than you think. Why not run single applications on one of the several CPUs that live in your hard disk for example?
- mgaunard 5y agoI guess you might think that if you never did high-performance networking. The whole setup adds a lot of latency and affects throughput too.
- YetAnotherNick 5y agoUnikernels don't want to replace systems where multiple competing process are running(heck, most unikernels don't even support process). The most obvious thing that you could run as unikernels are services and DBs.
- deleted 5y ago[deleted]
- eyberg 5y agoNanos has this basic assumption that it will never be deployed on bare metal. It assumes it is going into a virtualized environment such as AWS or GCP or Azure or whatever.
- DeathArrow 5y agoSo they've found out that unikernels are faster than running a kernel in a VM. But what about running your app in a Docker container on bare metal?
- MaxBarraclough 5y agoDocker is specific to the Linux kernel.
- DeathArrow 5y agoNot really. You can have Windows Docker containers.
- MaxBarraclough 5y agoThat's achieved by running a Linux kernel using virtualisation, if I understand correctly. (Wikipedia tells me WSL2 runs a full Linux kernel. [0]) Same goes for Docker on macOS. Docker containers are built atop the Linux syscall interface, after all. The goal of Docker is to allow you to bring your own userland without bringing your own kernel, so that many containers can run atop a single shared kernel instance. That's what distinguishes it from 'conventional' virtualisation. If you really want 'bare metal containers', that's pretty much what unikernels are going for. [0] https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux Also somewhat relevant: https://docs.docker.com/engine/security/seccomp/ https://docs.docker.com/engine/security/seccomp/
- monocasa 5y agoNo, there's docker windows images that run on an NT kernel. https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/run-your-first-container https://docs.microsoft.com/en-us/virtualization/windowsconta...
- MaxBarraclough 5y ago
- marcan_42 5y agoI'm not sure I get what the point of unikernels is given his explanation. GCP runs your VM as a Linux process with KVM. If you run a unikernel on that... that's roughly comparable to running your workload directly on the host - i.e. something more like App Engine. You still have Linux on top taking care of everything. You've just replaced the system call interface with a paravirtualized hardware interface. Which... is quite likely to be a bit slower if anything. You also have the overhead of two-stage paging (even for a unikernel, apps will want to use things like mmap(), which I assume means they can't run with guest paging off; and there's no paravirt API to mess with the VM memory layout to emulate this at the host level AFAIK) Of course you could in principle run a unikernel on bare metal, but I don't see that ever taking off. You need more software to manage a real machine, plus drivers. It's a lot easier to build your own OS when you're targeting fairly simple VM platforms with paravirtualized hardware. Bare metal isn't the same, and who wants to dedicate a whole machine to one process? Basically, this sounds to me like a less than ideal workaround to reduce VM overhead in existing cloud environments; not something that makes a whole lot of sense when you look at the big picture. The ASLR story is also silly. You can get the same effect with traditional fork/exec request handling (think CGI) on a normal OS, and of course that's going to be faster than booting the unikernel and the app for each request.
- johncolanduoni 5y agoI don’t think App Engine shares a Linux kernel between users. That would be a surprising level of confidence in lack of Linux system call vulnerabilities that no other cloud vendor seems to have (see AWS’s Firecracker VM or Google’s own gVisor). So at the end of the day, you’re still going to be dealing with a hypervisor. It’s also worth noting that at least AWS appears to hardware accelerate their paravirtualized interfaces so it’s not necessarily the case that IO operations are causing VM exits.
- marcan_42 5y agoApp Engine absolutely shares a Linux between users. They have a heavily audited sandbox with multiple layers of defence. Of course customers don't get to call arbitrary system calls, but it's also not a VM.
- madmax96 5y agoIf there is often no scheduler, how does multithreading work? Also, how do things like valgrind or compiler sanitizers work for unikernels?
- eyberg 5y agoWell, Nanos definitely does scheduling and it supports multiple threads. In this day/age you kind of have to. In fact we are at the tail end of removing our version of the BKL.
- jandrewrogers 5y agoMany high-throughput server architectures, like modern thread-per-core designs, already do all of their own thread scheduling in user space. Similarly, they tend to do their own storage I/O scheduling in user space as well, bypassing the kernel cache and sometimes the file system. Similar with networking. For this type of software, which treats Linux as little more than a device driver, a unikernel looks pretty similar to how they already interact with a normal OS, so porting to a unikernel should be relatively straightforward. I think this is more the case that unikernels target.
- jillesvangurp 5y agoBasically, this is not about operating system kernel but about the notion of running a single process with somebody else's PAAS, which runs an actual operating system with a kernel and virtualization layer. Doing this on bare metal makes a lot less sense because then suddenly you need to worry about managing all the hardware, initializing it, dealing with file systems, storage, memory and all the other stuff that operating systems tend to do (or at least orchestrate). Modern VMs on the other hand don't really need to do any of that and simply hand off that stuff to the underlying OS via kvm or some other virtulization API. You could do all of that in your unikernel on top of bare metal of course. That's what embedded software manufacturers of course do. They build firmware that goes into a ROM and the embedded hardware boots the firmware. That's a classic unikernel. I guess some network router manufacturers might do something similar though most of them seem to like running linux or bsd variants and not reinvent a lot of wheels instead.
- nickjj 5y agoIf anyone wants to learn more about Unikernels, Ian (the author of the blog post for this HN post) was on my podcast back in April. That episode is at: https://runninginproduction.com/podcast/79-nanovms-let-you-run-your-apps-faster-and-safer-with-unikernels https://runninginproduction.com/podcast/79-nanovms-let-you-r... It covers topics like what a Unikernel is, but also ways it can be practically applied to build web applications. We also talked about how he built a few apps using them. We also talked about a few of his open source projects related to Unikernels and how you can run them on popular cloud hosting providers today.
- bsenftner 5y agoDo you, or other readers, know of other/more companies operating in the Unikernel space? I'm curious about the landscape of opportunities to work in this area. Would it be wrong to think Unikernels and high CPU/GPU multiprocessing requirements might be a good combination?
- eyberg 5y agoNot something we've directly worked on yet but there is a lot of interest and other groups/companies doing ML at the edge utilizing things like GPU PCI passthrough with co-processors attached. That way you could isolate say ffmpeg in one unikernel, your inference app in another, maybe a stats app that shoots rollups to the cloud, etc. I think as costs keep coming down this will get ever more interesting over time. You can for instance already run multiple unikernels on a single rpi4 with ARMv8. The one that ships with v9 will be really nice.
- yjftsjthsd-h 5y agoThat sounds an awful lot like reinventing a multitasking operating system
- eyberg 5y agoAt the end of the day most software is not composed of a single application. To further complicate matters, many applications can't even run on a single server today. Many of the companies that users here work at have applications that span hundreds or thousands of servers. I think part of the argument here is that our software outgrew the one server/one operating system model a very long time ago.
- deleted 5y ago[deleted]
- ir193 5y agoI think we will be stuck with JVM on top of k8s container on top of linux VM on top of kvm on top of linux forever
- bsenftner 5y agoAt some point someone (will it require a combo developer/MBA?) is going to make a name for themselves by finally being heard when they say all this complexity is unnecessary. The raw compute at our finger tips is being wasted, replaced by pointless complexity for resume inflation and unnecessary cloud bills.
- deleted 5y ago[deleted]
- jasode 5y ago>What’s interesting about debugging this is that it actually is much easier to debug than a Linux system. Why? Because there is only one program in question. You aren’t whipping out lsof to figure out what process is spewing out crap connections or which process didn’t have a proper log rotation setup and prevented you from SSHing into the instance. [...] Unikernels are so much easier to debug than normal Linux systems. I have no strong opinion either way but just to add counterpoint from Bryan Cantrill's 2016 article[1]: >[...] the most profound reason that unikernels are unfit for production — and the reason that (to me, anyway) strikes unikernels through the heart when it comes to deploying anything real in production: Unikernels are entirely undebuggable. There are no processes, so of course there is no ps, no htop, no strace — but there is also no netstat, no tcpdump, no ping! And these are just the crude, decades-old tools. There is certainly nothing modern like DTrace or MDB. From a debugging perspective, to say this is primitive understates it: this isn’t paleolithic — it is precambrian. As one who has spent my career developing production systems and the tooling to debug them, I find the implicit denial of debugging production systems to be galling, and symptomatic of a deeper malaise among unikernel proponents: total lack of operational empathy. Production problems are simply hand-waved away — services are just to be restarted when they misbehave. And a previous 2018 HN subthread had differing opinions on debugging unikernels: https://news.ycombinator.com/item?id=17260604 https://news.ycombinator.com/item?id=17260604 [1] https://www.joyent.com/blog/unikernels-are-unfit-for-production https://www.joyent.com/blog/unikernels-are-unfit-for-product...
- throwaway894345 5y agoI think most of this can be significantly reduced with mature development processes and discipline. A mature org almost never needs to SSH into a production instance in the first place—extensive debug information is available via logging and monitoring, and when a problem arises in production you try to replicate it in lower environments if necessary. This doesn’t always work and sometimes it would be faster to just SSH and call it a day, but in exchange for that convenience you get a bunch of nice properties. Moreover, you can always compile in the debugging utilities that you absolutely need (or even better—create network interfaces for them).
- 5y ago
- hardwaresofton 5y agoGreat entrant in the space that is actually usable with ergonomic tooling and good docs: https://www.unikraft.org https://www.unikraft.org Promising project that's inactive but was one of the first ones I found with reasonable ergonomics and no lock-in to a specific language that I didn't use: https://github.com/rumpkernel/rumprun https://github.com/rumpkernel/rumprun Unfortunately it looks to be unmaintained as of now, but I expect the examples still work etc (https://github.com/rumpkernel/rumprun/issues/135 https://github.com/rumpkernel/rumprun/issues/135)
- 0xbadcafebee 5y agoSome points to consider: 1: This "big idea" will require a lot of differentiated feature work to make it be on-par with those dumb old slow Linux/Windows/Mac execution environments. You won't be able to develop or test or troubleshoot/debug apps for Unikernels the same way you do normal apps today. That might turn out to be a benefit... but any such huge change carries hurdles which need to be jumped over before we can become comfortable with it. 2: On "if the datacenter is the computer, then the cloud is its operating system". It's really better if you don't think about datacenters. It's fine to think about regions and availability zones. Those abstractions actually help you design a better, more resilient, more performant application. But there might be 5 different datacenters serving a single AZ. Datacenters are messy and have all kinds of weird considerations... best to stay in the logical abstraction world, rather than physical. 3: If we're talking about the future of computing, here's a few other big ideas for you: - Immutable SaaS. You've heard of immutable infrastructure? It's where, for example, you use Packer to create a VM image, and deploy a new EC2 instance with that VM image. If there's a bug in your app, you deploy the old VM image to an EC2 instance (or keep the old instance online and send traffic back to it, called blue/green deployment). No ssh'ing to a machine and 'git clone'ing and running puppet and restarting Nginx or any of that nonsense. Just spin up a VM with a pre-baked pre-tested validated set of software and configuration. It's the best, most reliable way to manage deployments, period. But that's just "the infrastructure". What about SaaS that you don't control like a virtual machine? We need SaaS to be immutable now, too. Literally the only reason Terraform exists is that 0% of the cloud's SaaS APIs are immutable. The majority of AWS APIs cannot be treated like giant blocks of versioned configuration that can be reverted at a moment's notice. Calling the APIs results in failure that has to be worked around by the client, rather than the SaaS taking care of it for you. This leads to complexity in management and lots of failures. Same goes for things like Kubernetes, which lie to you about "declarativeness", but meanwhile in the background the state is mutating like crazy and you have to pray that the change you deployed last week will work the same this week. - Versioned Immutable Data. A database should not be one giant constantly-mutating-pile-of-state that is impossible to revert without writing a bunch of scripts ahead of time. Modern databases need to start supporting not only transactions, but a dead-simple way to roll back to any change in time. This will result in a tectonic shift in how we can deploy and manage data in real time, enabling faster, more complex work. - First class support in every language for versioned function/subroutine DAGs. Essentially, every function that calls every other function in a program needs to have a version, and an explicit declaration of what versions of what functions it is compatible with. This functionality already exists in C today, but nobody uses it because they're unaware of its utility. If we all started using it, it would obsolete entire classes of software created to deal with the problems it would solve. Package management and backwards/forwards compatibility would just natively be taken care of, with no need to "manage" what versions of what software you have installed. We could support feature flags and upgrade-in-place code that could be reverted in a microsecond just by pointing at the previous DAG of software versions that an app used. "Deployment" would become flipping what version of a function is current, and it could automatically flip back to a previous one after a certain rate of errors. And because it's a DAG, it could work over entire distributed systems.
- solmag 5y agoVery good podcast (with transcript!) about kernels / lib kernels / unikernels: https://signalsandthreads.com/what-is-an-operating-system/ https://signalsandthreads.com/what-is-an-operating-system/