37 ms·

Ask HN: How are you using unikernels?

• mikepurvis 5y ago
  I'd be interested to know more about what the developer workflow is like on these. Like, do you always build the application as a unikernel, and run it locally on qemu or vmware? What is it like debugging something like that? What is the story for incremental builds? Or do you basically have to maintain a port of your software so that you can also run it on Linux with all the creature comforts of a normal system? If it's that, do you get weird bugs that only happen in prod and which are a gigantic pain to understand and work through?
  • rwmj 5y ago
    They differ very widely. It's instructive to look at unikraft which is probably the easiest to develop on (IMHO). eg: https://github.com/unikraft/app-helloworld-cpp https://github.com/unikraft/app-helloworld-cpp Our unikernel (UKL) lets you link your Linux program with the Linux kernel and produces a binary which run in a VM or on baremetal (essentially a custom vmlinuz). It's a bit laborious at the moment, one of many things to fix before release.
  • derefr 5y ago
    MirageOS compiles as either a unikernel (for prod) or a regular ELF image for running as a Linux process (for dev). The dev build still only sees the world through the same ABI the prod build does, though (it tries to have the semantics of being a VM plus an emulator, WPOed/Futamura-projected together) so any restriction prod has, you'll already run into in dev.
  • eyberg 5y ago
    As someone who works at NanoVMs you can run them locally but if you don't need to might as well run them natively (eg: not as a unikernel). As for "maintaining for linux" - most people are deploying their software to linux servers so if you can develop on linux that's great but if you develop on windows or mac you're going to have to do that regardless if you're using unikernels or not.
  • nderjung 5y ago
    [Unikraft maintainer] Your question breaks down into two I think: 1). whether someone working on a typical "userspace" application exclusively and then decides to package the application as a unikernel right before deployment and 2). whether you are working exclusively on a unikernel-centric application, that is, you are making use of non-POSIX, internal (kernel) APIs in order to exploit better performance or better interact with your platform (e.g. Xen, KVM, etc.) or hardware architecture. Unikernels can achieve excellent performance as they have a very tight coupling with their surrounding environment. In the case of 1), the developer workflow is the more or less the same in my opinion, you build your application in the comfortable way that you do before, without having to worry about the underlying kernel. There's two modes to this though. This is essentially whether your appplication is built using an interpreted language or a compiled one. In the former case, a python program is an apt example here, like using a framework like Flask where you build your HTTP APIs and work on application logic etc. In this case, you can use a pre-compiled unikernel which is tailored to run the python runtime. Unikraft makes these available actually[0]. :) When you are ready for deployment, you simply mount a filesystem with the source files of your application and point to the entry program as you would normally, like invoking the python3 binary with a path to your python program. When a developer is building some application which is compiled, e.g. with Golang, you can follow the same workflow as you would normally. However, when you are ready to deploy, you do something called binary rewriting. Here, you change invocations of kernel-space methods, mainly syscalls, to JMP instruction calls which are addressed inside of the unikernel binary. A final linking step puts the two together and you are then able to run your application as a unikernel. HermiTux does something like this[1] and Unikraft is soon to release some new tools to accomplish this too! Stay tuned! :) Unikraft aims to be POSIX compliant, and you simply select the relevant libraries and options you need to get the features your application needs to run. This way, you don't have to rely on maintaining two versions of your application. In the case of 2). where you are building a unikernel-centric application, it's no more different than working with a compiled language. At least in my workflow, when I am working on some application or a new internal library for Unikrat, I just re-compile the source files I'm working on and then perform the final linking step to create the unikernel binary. I mainly use kraft[2] to help me acquire the relevant source files and libraries I want to use and work on. The kraft repo ships with a suite of docker images, and this for me is my main way of creating a dev environment where I have the right version of gcc, qemu, and all relevant tools for debugging too. I typically invoke this environment like so: ``` cd to/my/project/path docker run -it --rm -e UK_KRAFT_GITHUB_TOKEN=<set token> v $(pwd):/usr/src/unikraft --device /dev/kvm --entrypoint bash unikraft/kraft:staging # you are now inside a container kraft list update kraft init # ..etc.. kraft configure # or kraft menuconfig kraft build kraft run # or use qemu-guest -k ./build/my_kernel ``` I hope this helps! [0]: https://releases.unikraft.org/unikraft/apps/python3/stable/ https://releases.unikraft.org/unikraft/apps/python3/stable/ [1]: https://dl.acm.org/doi/pdf/10.1145/3313808.3313817 https://dl.acm.org/doi/pdf/10.1145/3313808.3313817 [2]: https://github.com/unikraft/kraft https://github.com/unikraft/kraft
• eqvinox 5y ago
  Probably not the kind of answer you're looking for, but networking appliances (routers, switches, firewalls) essentially used to be "unikernels" in the early 90ies, particularly the original Cisco IOS. Which is also a great example that blurs the lines between embedded systems and unikernels… That said, no modern router is a "unikernel" anymore, and the low end switches fall more into the embedded systems category. (btw: out of curiosity, how would HN differentiate between an embedded system/RTOS and an unikernel?)
  • tenebrisalietum 5y ago
    Embedded system: Has an OS designed to be flashed to firmware with no dependency on external storage devices and survive for a long time without updates/reboots. RTOS: Any operating system that 100% absolutely guarantees IRQ response time or CPU scheduler time. An embedded system's OS may or may not be realtime. An RTOS may be possible on the standard PC platform. Unikernel: User space program linked with required kernel code and designed to run in kernel mode; also possibly designed with the expectation of a hypervisor. The things that come to mind with this are Xbox and Xbox 360 games. So all three of these to me are not mutually exclusive concepts. A unikernel program running in kernel mode probably doesn't need an OS, but there may be need for a hypervisor which the unikernel may use via some virtualized hardware type interface, and that may essentially be an embedded RTOS.
    • scrubs 5y ago
      Great answer. Thanks
    • djhworld 5y ago
      Forgive my ignorance, does this mean that when your user space program makes syscalls, it doesn't require the CPU to go into ring 0/"protected mode" as your program is already running in ring 0?
      • derefr 5y ago
        Yes; but furthermore, the "syscall" doesn't end up happening. You're really just writing a kernel driver calling kernel functions, with a libc-looking (or other-language's-runtime-looking) library abstraction (that can be optimized out at unikernel build time.) Usually this library abstraction also has a userspace actually-does-syscalls backend you can build against, giving you a regular userland process, rather than a unikernel, for low-overhead testing. (In this case, if there's anything that absolutely must load into kernelspace, it may compile to a separate DKMS module, which will then communicate through some form of IPC with your userspace process.) But all that complexity is just for testing; in prod, you just get a monolithic kernel with your code inside (that being either an existing kernel with your code being a driver; or your code being "the kernel", with the library being an exokernel framework.)
• sn 5y ago
  Let's define unikernels as running under a hypervisor. If it's not, it's more likely to be called an embedded operating system. I don't think unikernels are worth it unless you're running at massive scale. A realistic target audience is providers of serverless-style services, in which case you're taking a vanilla application someone else wrote and compiling it against a unikernel.
  • jlg23 5y ago
    > I don't think unikernels are worth it unless you're running at massive scale. "Worth" what? If deploying a microservice as a unikernel is just 'M-x deploy' (or a click, for mouse users) away, what's the cost?
    • mikepurvis 5y ago
      For me, the cost would be the loss of ecosystem— log collection, monitoring, cores + rr recordings, being able to connect with gdb and/or sniff around the live system from a familiar userland, etc. These concerns are reduced when you're at the thousands-of-machines scale since at that point you probably already have a somewhat hand-rolled solution for this kind of thing anyway.
      • rwmj 5y ago
        This isn't true of all unikernels. Ours lets you run regular programs like perf, gdb inside the unikernel. (Of course optionally, you can go "pure unikernel" if you want)
      • throwaway894345 5y ago
        We already solved those problems with containers (I know, half of HN doesn’t like containers and everyone should manage heavy VMs or bare metal machines just like our ancestors did). Notably logs and metrics are exfiltrated, detailed logging and monitoring largely obsolete debuggers for production (indeed, they aren’t even balle into the final image nor are they installable since the app user oughtn’t be root). These practices seem pretty portable to the unikernel model and they don’t require any hand-rolled workarounds or thousands of machines of scale.
• nickjj 5y ago
  If anyone is curious how Unikernels work and what the dev workflow is like, I had the creator of NanoVMs on my podcast last month at: https://runninginproduction.com/podcast/79-nanovms-let-you-run-your-apps-faster-and-safer-with-unikernels https://runninginproduction.com/podcast/79-nanovms-let-you-r... We talked about a mixture of Unikernels in general and how they run their infrastructure. What's interesting about Nanos is it's POSIX compliant. In other words you don't have to write your app differently to get it to run in their Unikernel.
  • vlovich123 5y ago
    There’s a large subtle gap in various APIs between Linux and POSIX that can matter. That might be a barrier for various applications since Linux is kind of the default target in the cloud with Windows following.
    • convolvatron 5y ago
      I agree completely, and Linux runtimes generally meaningfully address the difference. Nanos in particular though is written against Linux
  • pjmlp 5y ago
    I don't care 1 second about POSIX and don't have to rewrite the app differently, because I mostly use managed languages with rich libraries eco-systems.
• hardwaresofton 5y ago
  The definition of what a unikernel is needs to be narrowed down, a lot of these projects in the space (not all the ones listed above) have material differences that are not clear: - some run only one language - some require recompilation - some essentially swap out libraries, others do something closer to dropping your already mostly static binary in a minimal disk image - some build pid1 processes, others VMs images Anyway, here are some additional entries in the space: - https://ssrg-vt.github.io/hermitux/ https://ssrg-vt.github.io/hermitux/ - http://osv.io/ http://osv.io/ - https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit (more embedded/minimal VM than unikernel) - https://nabla-containers.github.io/ https://nabla-containers.github.io/ (runs on Solo5) I am going through using Linuxkit to build AMIs for cloud providers now from containers. I wouldn’t necessarily class linuxkit as a unikernel project because it doesn’t have the hallmark blurring of user and kernel space or kernel-as-a-library but you can customize the kernel so it’s an adjacent idea, and I think it’s the one most likely to be in actual use at non-hyperscalers. [EDIT] Added OSv since it's on one of the linked lists but is a pretty large active player in the field.
  • rwmj 5y ago
    https://github.com/unikernelLinux/ https://github.com/unikernelLinux/ once we get around to releasing it, which should happen in a few weeks. I agree with your point above that there are many different things called a unikernel, with widely differing scope and use cases.
• tinktank 5y ago
  I'm not. For my use-case the disadvantages (worse performance, hell debugging, no modularity) outweighed the advantages (theoretical increase in performance)
  • eyberg 5y ago
    We (Nanos) haven't done the work to put in a full suite of performance regression testing yet but in general we can run lots of workloads much faster. Rust webservers, go webservers, mysql, redis, nginx <-- all of this has been clocked running a lot faster. Go webservers in particular, since our websites and such are deployed as such, can run ~2X faster on GCP and 3X faster on AWS.
• TechBro8615 5y ago
  Many unikernel projects were ahead of their time. For example ClickOS [0] is ~7 years old but all its ideas still sound innovative. Someone could build an entire business on top of network function virtualization, using unikernels as an efficient sandboxing mechanism. I’m not sure why unikernels have not caught on widely. I suspect their time has yet to come for some applications, but at least for NFV and sandboxing, I would bet on solutions using eBPF or XDP with WASM for sandboxing. [0] https://github.com/kohler/click https://github.com/kohler/click
  • convolvatron 5y ago
    I think there isn't a compelling reason - For the sake of argument lets just say there is a difference in overall system performance - lets say 20% for a workload you care about. you get to change your tooling. I think the unikernel tooling for building deployable images is/could be superior, but its different. your service is now running as it did before, but you've also taken a whole lot of risk onboard that this experimental operating system used by a few tens of people around the world isn't going to cause problems for you. that there wont be some library you want to use that isn't supported. if your job is to prop up microservices or run websites, I dont see why you would do this unless you were really that bored and had no sense of responsibility.
    • TechBro8615 5y ago
      Tooling is really the chicken/egg problem of ecosystems. Good tools come from the best devs, who have a higher chance of existing in big, active ecosystems. But most devs won’t join an ecosystem if it doesn’t have good tooling, so how can it become big and active? An ecosystem needs its early adopters to be excellent devs, so the early community has a high concentration of them and therefore greater probability of developing good tools that will attract a wider audience. That’s why I wouldn’t give up quite yet on unikernels. There are a lot of really smart people working on them. What the unikernel ecosystem needs is a “killer app” to demonstrate a flagship use case. I used to think Fly.io could do that, but they seem to have settled on Firecracker (as has Amazon, tbf). Whatever happened to the team behind the unikernel company acquired by Docker?
• eyberg 5y ago
  Yes, the Solo team has been focused on service mesh/apis vs Unik. Unik is not a kernel implementation but a "orchestrator" of sorts. One of the cooler things that we've found though is that there is a very wide misconception that you'll need a k8s for unikernels - that is simply not the case. When we deploy them to the cloud we use the underlying storage/networking layers that already exist - so we don't have to manage all of that. Unikernels remove a lot of the complexity that comes with container infrastructure. When you deploy a Nanos unikernel we create a machine image, which if you are deploying to say AWS becomes an ec2 image and then the instance that spins up is that unikernel - there is no layer of Linux that you deploy something else to. I highly recommend that anyone that is remotely interested to just try it out - https://ops.city https://ops.city - it'll clear up any deployment/orchestration questions you have almost immediately. In fact, speaking of AWS we just reduced the deploy time there by 66% so now you can build/deploy your unikernel to AWS in < 20 seconds. It's actually faster than deploying to Google Cloud and that remains my favorite place to deploy them.