4 ms·

I'm aware of those sorts of application level logs. I used that sort of thing when I worked at Uber, and I saw it built up over time as more classes of errors w

I'm aware of those sorts of application level logs. I used that sort of thing when I worked at Uber, and I saw it built up over time as more classes of errors were encountered. :) But there are a whole class of application failures that are handled out-of-band, which is what I was referring to: If you need to debug issues involving hardware then not having an interface to that hardware separate from your application can make it harder to inspect and determine what failure you're running into. And again I point to application failures within the application frameworks themselves having issues (such as logs not being written out at all) that having an out-of-band monitoring process with a mostly-uncorrelated probability of failure seems to me to be very necessary for handling these less common but potentially more catastrophic issues. I'll admit that this next statement is a bit of a Call to Authority fallacy, but you'll note that not even Uber or Google use unikernels in production, and I believe these two problem domains are why.

nostrademons 8y ago
Perhaps, though it seems that if you want to debug hardware, the place to do that is within the hypervisor itself. And if you want out-of-band monitoring, the place to do that is from a supervisor process running on a different box, a la Erlang supervisor trees. Very little of this infrastructure has been built yet, which is why we don't see this in production deployments. I'm talking decades out - I'll predict that the software industry will eventually tend towards this architecture because it's more efficient and better fits what we actually do with software now, but until it's a pressing pain point there's little urgency in getting there. Much like how all the folks who predicted in the 80s that CPUs would tend toward RISC were right, but it took 30 years and the ARM/mobile revolution to get there, and most desktops still have a hardware x86 emulation layer on top of RISC microcode.
- ISV_Damocles 8y ago
  I also agree that application -> cluster of machines is where we're going. I think it needs to be baked into an specialized programming language to really work well, though, and all of these things we rely on the OS for would have to be handled. But unikernels running on top of a hypervisor seems like a terrible solution to the problem? You're trading away the convenience of the OS tooling that's built up over the past 40+ years to get closer to the metal and squeeze more performance out, then injecting virtualization in between and throwing that performance gain back away? Why not just make a regular application at that point and not throw away all of the runtime debugging support? A unikernel application running on top of a hypervisor basically turns the unikernel part into just an incredibly large and inefficient libc, in my mind.
  - comex 8y ago
    > Why not just make a regular application at that point and not throw away all of the runtime debugging support? A few reasons: - There exist large public clouds willing to run any code you want under a hypervisor, but services that give you shell access to a user under a traditional kernel have mostly died out. This can partially be justified by security concerns: traditional syscall interfaces tend to be more complex and thus have more attack surface than the VM<->hypervisor interface. Hypervisors also tend to make it easier to divide system resources, e.g. by giving each VM a fixed RAM allocation. - Some clouds, like EC2 with the Elastic Network Adapter, give virtual machines direct access to (custom) networking hardware, rather than making them trap to the hypervisor for every send and receive. This should mitigate much of the performance overhead of using a VM, at least as far as networking is concerned. - Anyway, unikernels can put everything from filesystems and TCP to threading and even page table management "in-process"; this can reduce the number of syscalls that have to be performed and thus syscall overhead, even for operations that do ultimately delegate to "syscalls" in the hypervisor. In other words, they shouldn't be compared to just libc; they're also taking over many of the functions of a traditional kernel (just not all of them).
    - zaarn 8y ago
      >but services that give you shell access to a user under a traditional kernel have mostly died out. Shared hosting is alive and well, shell access is becoming more common to my knowledge. >Hypervisors also tend to make it easier to divide system resources, e.g. by giving each VM a fixed RAM allocation. CGroups. LXC and Docker are capable of RAM limits nowadays. >traditional syscall interfaces tend to be more complex and thus have more attack surface than the VM<->hypervisor interface A syscall interface is an assembly instruction and several registers that may point to some memory. The kernel is very thorough in checking the validity of such pointers (unless you use an ancient non-LTS kernel) > give virtual machines direct access to (custom) networking hardware, rather than making them trap to the hypervisor for every send and receive. Yes, VFIO and IOMMU have been around for a while. They do get pretty close to native performance (close enough for gaming atleast). It's not exactly new tech and full VMs with Linux images have been able to utilize full speed networking for a while now too. Also note that virtio adapters are close to baremetal even without passthrough of the adapter. >Anyway, unikernels can put everything from filesystems and TCP to threading and even page table management "in-process"; this can reduce the number of syscalls that have to be performed and thus syscall overhead For all the stuff you mentioned syscall overhead isn't the driving performance factor unless you're google or facebook scale. TCP, threading, filesystems do spend most of their time waiting for DMA or other interrupts. Page tables is usually not even a syscall and rather a interrupt from the CPU, the performance difference should be negligible. > In other words, they shouldn't be compared to just libc; they're also taking over many of the functions of a traditional kernel (just not all of them). It should be compared to libc because in the ideal deployment scenario there should be no difference.

nostrademons 8y ago

Perhaps, though it seems that if you want to debug hardware, the place to do that is within the hypervisor itself. And if you want out-of-band monitoring, the place to do that is from a supervisor process running on a different box, a la Erlang supervisor trees. Very little of this infrastructure has been built yet, which is why we don't see this in production deployments. I'm talking decades out - I'll predict that the software industry will eventually tend towards this architecture because it's more efficient and better fits what we actually do with software now, but until it's a pressing pain point there's little urgency in getting there. Much like how all the folks who predicted in the 80s that CPUs would tend toward RISC were right, but it took 30 years and the ARM/mobile revolution to get there, and most desktops still have a hardware x86 emulation layer on top of RISC microcode.

ISV_Damocles 8y ago
I also agree that application -> cluster of machines is where we're going. I think it needs to be baked into an specialized programming language to really work well, though, and all of these things we rely on the OS for would have to be handled. But unikernels running on top of a hypervisor seems like a terrible solution to the problem? You're trading away the convenience of the OS tooling that's built up over the past 40+ years to get closer to the metal and squeeze more performance out, then injecting virtualization in between and throwing that performance gain back away? Why not just make a regular application at that point and not throw away all of the runtime debugging support? A unikernel application running on top of a hypervisor basically turns the unikernel part into just an incredibly large and inefficient libc, in my mind.
- comex 8y ago
  > Why not just make a regular application at that point and not throw away all of the runtime debugging support? A few reasons: - There exist large public clouds willing to run any code you want under a hypervisor, but services that give you shell access to a user under a traditional kernel have mostly died out. This can partially be justified by security concerns: traditional syscall interfaces tend to be more complex and thus have more attack surface than the VM<->hypervisor interface. Hypervisors also tend to make it easier to divide system resources, e.g. by giving each VM a fixed RAM allocation. - Some clouds, like EC2 with the Elastic Network Adapter, give virtual machines direct access to (custom) networking hardware, rather than making them trap to the hypervisor for every send and receive. This should mitigate much of the performance overhead of using a VM, at least as far as networking is concerned. - Anyway, unikernels can put everything from filesystems and TCP to threading and even page table management "in-process"; this can reduce the number of syscalls that have to be performed and thus syscall overhead, even for operations that do ultimately delegate to "syscalls" in the hypervisor. In other words, they shouldn't be compared to just libc; they're also taking over many of the functions of a traditional kernel (just not all of them).
  - zaarn 8y ago
    >but services that give you shell access to a user under a traditional kernel have mostly died out. Shared hosting is alive and well, shell access is becoming more common to my knowledge. >Hypervisors also tend to make it easier to divide system resources, e.g. by giving each VM a fixed RAM allocation. CGroups. LXC and Docker are capable of RAM limits nowadays. >traditional syscall interfaces tend to be more complex and thus have more attack surface than the VM<->hypervisor interface A syscall interface is an assembly instruction and several registers that may point to some memory. The kernel is very thorough in checking the validity of such pointers (unless you use an ancient non-LTS kernel) > give virtual machines direct access to (custom) networking hardware, rather than making them trap to the hypervisor for every send and receive. Yes, VFIO and IOMMU have been around for a while. They do get pretty close to native performance (close enough for gaming atleast). It's not exactly new tech and full VMs with Linux images have been able to utilize full speed networking for a while now too. Also note that virtio adapters are close to baremetal even without passthrough of the adapter. >Anyway, unikernels can put everything from filesystems and TCP to threading and even page table management "in-process"; this can reduce the number of syscalls that have to be performed and thus syscall overhead For all the stuff you mentioned syscall overhead isn't the driving performance factor unless you're google or facebook scale. TCP, threading, filesystems do spend most of their time waiting for DMA or other interrupts. Page tables is usually not even a syscall and rather a interrupt from the CPU, the performance difference should be negligible. > In other words, they shouldn't be compared to just libc; they're also taking over many of the functions of a traditional kernel (just not all of them). It should be compared to libc because in the ideal deployment scenario there should be no difference.