6 ms·

The article is Tl;Wbmloc;Dr (too long, way beyond my level of comprehension, didn't read), so forgive any stupid questions. Could the unikernel technology be us

The article is Tl;Wbmloc;Dr (too long, way beyond my level of comprehension, didn't read), so forgive any stupid questions. Could the unikernel technology be used to build extremely small Linux distros that contain just the bare minimum to satisfy all dependencies for running say a single software or a small group of them. I'm not interested in virtual machines for network related services but rather in small embedded systems where the user might need to use just one single application for the entire board lifetime. Security wouldn't be a concern there, assuming the user already has complete access to the hardware, that layer could be moved outside of the box by inspecting and firewalling the network traffic. If yes, does anyone know if there is/will be any automated way say to analyze a software, example: a .deb package containing an user software, then build a list of dependencies and create a bootable image for the requested architecture which will run just that software?

Maakuth 7y ago
This is how embedded linux systems are built already, AFAIK. There are vendors, such as MontaVista, that provide a toolchain to build images in this manner. There are free software alternatives too to do that, such as Embedded Gentoo.
vidarh 7y ago
It doesn't really make much difference in terms of size in itself, I suspect. You're still carrying along the whole kernel and the application code, they're just linked together into a single binary. You can certainly then trim things down a lot by stripping the kernel to its bones by removing drivers etc., but you can do that even without linking it together, and you can go a lot further than what the normal kernel configuration mechanism allows if you're prepared to strip out functionality you know you'll never need, but I'm hard pressed to think of approaches like that which you can apply to the kernel that will only work if you move your application code to kernel space. In the past I've brought up Linux on embedded boards with 4MB RAM and 4MB flash for both the kernel and the application, and we gained most by linking statically to a smaller libc on the user side and stripping out drivers on the kernel side, ditching bash for ash (currently dash) and using a stripped down barebones init. We could have ditches ash and init as well and booted straight into the app, but ash made debugging and testing slightly easier and we could fit it in. 4MB at the time was not even a challenge. I think the gap from something that you can fit a Linux kernel and an app mashed together into a unikernel up to being able to run them separately will be very small, so in most instances if you can make it work as a unikernel odds are you can produce a nearly a small solution without it. Going full on unikernel potentially costs you a lot of resilience against failure, it's not just about security. The main reason for wanting to do it is to reduce the amount of context switches and data copying to reduce latency.
- imglorp 7y ago
  It probably helps security a ton if you are the only binary: there is no shell, no other executables, and you can even disable fork() and exec() system calls.
- rwmj 7y ago
  The linking stage automatically removes all parts of Linux (eg. drivers) that are not needed. We can also apply various link-time and whole process profile-guided optimizations. Apparently most of the performance improvement comes from getting rid of system call overhead although we still have a student measuring these things more precisely.
  - vidarh 7y ago
    > The linking stage automatically removes all parts of Linux (eg. drivers) that are not needed. So does the normal kernel build process once you've specified which drivers you need compiled in. That's in no way unique to a unikernel approach. There'll be extremely little code that can be stripped from a unikernel that can not be stripped from a kernel built for the purpose of running a specific application. It's just not where the benefits of a unikernel are. As I said, and you reiterated, it's really cutting latency from syscalls that is the big benefit of a unikernel.
    - rwmj 7y ago
      That's not how most distro kernels are built, but sure you can build a custom kernel with everything statically linked in. What you can't do is link your application in as well and do whole-program optimizations on the whole binary, which is what Ali is trying to do here. We're also hoping that the eventual solution will be distro friendly enough that we can pack the "kernel library" into a future Fedora which you can use to link your own servers into unikernels.
      - vidarh 7y ago
        Of course it's not, but the comment I replied to was about whether using it to build binaries for extremely small embedded systems made sense. In that scenario a distro kernel from one of the desktop distros is a total non-starter. If a few MB of space matters, then you'll want a suitably configured kernel that disables a ton of stuff that is totally irrelevant for such a system, and you'll need to take that configuration into account whether or not you're building a normal kernel or a unikernel, as it involves not just excluding drivers and the like, but disabling functionality that if left enabled will not get removed by the linker because there will be calls into it from code paths that are reachable. Do that, and you'll find it is pretty straightforward to get a Linux kernel down under 1MB. Now, it's possible your tooling makes that easier, but my point is that it is equally possible to disable those parts of the kernel for a non-unikernel system as well, and you're not going to make significant additional savings. And if you want to, then ditching glibc for one of the smaller libcs would be a better starting point. This is not a criticism of this project - size is simply not what most people look to unikernels to address, especially not one built on a general purpose kernel.
throwaway_391 7y ago
Alpine Linux is a reasonably small (20mb) Linux distribution, I've seen it run on bare metal but it's more common to see it in Docker environments. Building minimal bootable to be very small is possible, but is difficult in comparison to an apt install - I imagine for the most part porting docker container configs to a bootable OS might be the best approach for small-medium projects.
- geggam 7y ago
  I miss Trinux ( ramdrive boot from floppy fits in 12MB ram OS ) http://trinux.sourceforge.net/legacy/ http://trinux.sourceforge.net/legacy/
  - yjftsjthsd-h 7y ago
    I suppose Tiny Core Linux is the modern equivalent.
- eyberg 7y ago
  This is brought up quite a lot and isn't answered correctly enough in my opinion. The problem with this approach is that you can make it as small as you want but it's still Linux. At a certain point are you going to start patching things out like support for users? Support for management of multiple processes? There's a non-trivial set of syscalls and data structures designed solely for these constructs. You can't just seccomp it and call it a day. For us it's not about the size (that's nice of course) but it's more about the performance and security.
  - throwaway_391 7y ago
    Size, Content. Pick one. I'm not suggesting it's a good idea, but it's there. I'm sure there's more minimal, and less minimal options available. I don't think there's any security impacts with using alpine Linux specifically, aside from default credentials in a bunch of containers a few months back.
syn0byte 7y ago
"Unikernel" is a overly generic term. Technically the Linux kernel is already a "Unikernel". It's just most of the time its discussed as a negative not a positive and the term used is "monolith". The project you want is "Yocoto", a complete toolchain to build customized embedded OS images. https://www.yoctoproject.org/ https://www.yoctoproject.org/ You would have to build your own analyzer. Grepping the dependencies from the makefile/build data or just parsing the output of dpkg and translating that into yocto build specs is not unreasonable.

Maakuth 7y ago

This is how embedded linux systems are built already, AFAIK. There are vendors, such as MontaVista, that provide a toolchain to build images in this manner. There are free software alternatives too to do that, such as Embedded Gentoo.

vidarh 7y ago

It doesn't really make much difference in terms of size in itself, I suspect. You're still carrying along the whole kernel and the application code, they're just linked together into a single binary. You can certainly then trim things down a lot by stripping the kernel to its bones by removing drivers etc., but you can do that even without linking it together, and you can go a lot further than what the normal kernel configuration mechanism allows if you're prepared to strip out functionality you know you'll never need, but I'm hard pressed to think of approaches like that which you can apply to the kernel that will only work if you move your application code to kernel space. In the past I've brought up Linux on embedded boards with 4MB RAM and 4MB flash for both the kernel and the application, and we gained most by linking statically to a smaller libc on the user side and stripping out drivers on the kernel side, ditching bash for ash (currently dash) and using a stripped down barebones init. We could have ditches ash and init as well and booted straight into the app, but ash made debugging and testing slightly easier and we could fit it in. 4MB at the time was not even a challenge. I think the gap from something that you can fit a Linux kernel and an app mashed together into a unikernel up to being able to run them separately will be very small, so in most instances if you can make it work as a unikernel odds are you can produce a nearly a small solution without it. Going full on unikernel potentially costs you a lot of resilience against failure, it's not just about security. The main reason for wanting to do it is to reduce the amount of context switches and data copying to reduce latency.

imglorp 7y ago
It probably helps security a ton if you are the only binary: there is no shell, no other executables, and you can even disable fork() and exec() system calls.
rwmj 7y ago
The linking stage automatically removes all parts of Linux (eg. drivers) that are not needed. We can also apply various link-time and whole process profile-guided optimizations. Apparently most of the performance improvement comes from getting rid of system call overhead although we still have a student measuring these things more precisely.
- vidarh 7y ago
  > The linking stage automatically removes all parts of Linux (eg. drivers) that are not needed. So does the normal kernel build process once you've specified which drivers you need compiled in. That's in no way unique to a unikernel approach. There'll be extremely little code that can be stripped from a unikernel that can not be stripped from a kernel built for the purpose of running a specific application. It's just not where the benefits of a unikernel are. As I said, and you reiterated, it's really cutting latency from syscalls that is the big benefit of a unikernel.
  - rwmj 7y ago
    That's not how most distro kernels are built, but sure you can build a custom kernel with everything statically linked in. What you can't do is link your application in as well and do whole-program optimizations on the whole binary, which is what Ali is trying to do here. We're also hoping that the eventual solution will be distro friendly enough that we can pack the "kernel library" into a future Fedora which you can use to link your own servers into unikernels.
    - vidarh 7y ago
      Of course it's not, but the comment I replied to was about whether using it to build binaries for extremely small embedded systems made sense. In that scenario a distro kernel from one of the desktop distros is a total non-starter. If a few MB of space matters, then you'll want a suitably configured kernel that disables a ton of stuff that is totally irrelevant for such a system, and you'll need to take that configuration into account whether or not you're building a normal kernel or a unikernel, as it involves not just excluding drivers and the like, but disabling functionality that if left enabled will not get removed by the linker because there will be calls into it from code paths that are reachable. Do that, and you'll find it is pretty straightforward to get a Linux kernel down under 1MB. Now, it's possible your tooling makes that easier, but my point is that it is equally possible to disable those parts of the kernel for a non-unikernel system as well, and you're not going to make significant additional savings. And if you want to, then ditching glibc for one of the smaller libcs would be a better starting point. This is not a criticism of this project - size is simply not what most people look to unikernels to address, especially not one built on a general purpose kernel.

throwaway_391 7y ago

Alpine Linux is a reasonably small (20mb) Linux distribution, I've seen it run on bare metal but it's more common to see it in Docker environments. Building minimal bootable to be very small is possible, but is difficult in comparison to an apt install - I imagine for the most part porting docker container configs to a bootable OS might be the best approach for small-medium projects.

geggam 7y ago
I miss Trinux ( ramdrive boot from floppy fits in 12MB ram OS ) http://trinux.sourceforge.net/legacy/ http://trinux.sourceforge.net/legacy/
- yjftsjthsd-h 7y ago
  I suppose Tiny Core Linux is the modern equivalent.
eyberg 7y ago
This is brought up quite a lot and isn't answered correctly enough in my opinion. The problem with this approach is that you can make it as small as you want but it's still Linux. At a certain point are you going to start patching things out like support for users? Support for management of multiple processes? There's a non-trivial set of syscalls and data structures designed solely for these constructs. You can't just seccomp it and call it a day. For us it's not about the size (that's nice of course) but it's more about the performance and security.
- throwaway_391 7y ago
  Size, Content. Pick one. I'm not suggesting it's a good idea, but it's there. I'm sure there's more minimal, and less minimal options available. I don't think there's any security impacts with using alpine Linux specifically, aside from default credentials in a bunch of containers a few months back.

syn0byte 7y ago

"Unikernel" is a overly generic term. Technically the Linux kernel is already a "Unikernel". It's just most of the time its discussed as a negative not a positive and the term used is "monolith". The project you want is "Yocoto", a complete toolchain to build customized embedded OS images. https://www.yoctoproject.org/ https://www.yoctoproject.org/ You would have to build your own analyzer. Grepping the dependencies from the makefile/build data or just parsing the output of dpkg and translating that into yocto build specs is not unreasonable.