6 ms·

Instead of straw-manning the GP, why not address the very specific concerns raised? If there's an error you're seeing in a cluster of, say, 30 machines that onl

Instead of straw-manning the GP, why not address the very specific concerns raised? If there's an error you're seeing in a cluster of, say, 30 machines that only seems to happen once every few days, how are we supposed to debug this? Since the application and kernel are joined together so closely, there aren't any parallel processes that could be monitoring the behavior of the unikernel app. You can't log in on the affected machine and pull logs really, because the process may not be able to even write a log about what's failing when its in the process of failing. If you have a situation where a single server is having performance issues in a unikernel approach, there is no UNIX-style environment to log in and poke the hardware configuration and health, whereas on a more normal stack you could log in and see that NetworkManager has degraded the connection down from 10Gbps to 100Mbps because it can't negotiate a higher speed successfully, anymore, and then you can configure your alerting system to watch that stat, remove any boxes in that state from the load balancer, and inform DevOps about the bad hardware. All of that to then say that I really do like the idea of Unikernels, and I believe they have their place, but that place tends to be very well-defined and infrequently changing problem domains that need very high performance, such as in supercomputers. It doesn't work as well elsewhere because the application, by being joined so closely to the kernel, is also responsible for these things operating systems have traditionally taken care of for us, and problems in the application code can have an even more catastrophic effect than normal. Most applications are neither well-defined enough, time-invariant enough, or latency-constrained enough for the unikernel tradeoff to make sense.

nostrademons 8y ago
One of the things that struck me when I first joined Google was the extent to which logging, debugging, and tracing information was built into the application itself, as well as the extent to which this information was routinely useful in a way that basic UNIX performance tools are not. They have, for example, a distributed tracing system [1] integrated into their RPC layer that lets you instantly view the complete set of RPCs kicked off by a request, the services touched, and statistics for latency either for a single request or across many requests. This means that if you're a developer trying to integrate a new service into the search stack, you can get an answer within a couple of minutes as to "Your service will be on the critical path, so you need to count microseconds in implementation" vs. "Your service is off the critical path, go wild" vs. "You have a budget of 25ms, as long as you come in under that you should be fine." It lets you tell whether your application requests were slow because you triggered an edge-case in your code vs. because that BigTable your dependency relies upon was undergoing a compaction. Combined with some other application-level logging, you could tell if your ranking algorithm was slow vs. it happened to share a machine with a process that hogged the machine resources. It's probably one of the features I miss most now that I'm no longer at Google and doing my own startup, and I've tried to apply the same development philosophy to my own code. It's been invaluable, for instance, to tell whether I'm not getting results because I can't parse a page but forgot to write the error-handler to mark it as done & unparseable vs. because a complicated algorithm I use is accidentally exponential-time in some cases vs. because an HTTP connection is taking too long to complete vs. because CloudFlare blocks robots and the site really doesn't want to be crawled so I should give up on it vs. because an algorithm I wrote to avoid overloading hosts has a bug that makes it get stuck performing requests once every month. Some of these could perhaps be caught with lsof or top, but they'd give me nowhere near as precise info when trying to track down the problem. All this is to say that perhaps the application (or a library used by it) really is the right place for those tools to go, because it knows far more about its operation and the likely questions you'll ask than a generic system-wide tool. This'd require a change in how we write applications and what we consider to be the responsibilities of an application developer, which probably explains why we don't have mainstream adoption yet. But in a world that's moving from multiple-apps-on-one-computer to multiple-computers-for-one-app, it makes sense, and so I wonder if the long-term trend will go in that direction. [1] https://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf https://static.googleusercontent.com/media/research.google.c...
- ISV_Damocles 8y ago
  I'm aware of those sorts of application level logs. I used that sort of thing when I worked at Uber, and I saw it built up over time as more classes of errors were encountered. :) But there are a whole class of application failures that are handled out-of-band, which is what I was referring to: If you need to debug issues involving hardware then not having an interface to that hardware separate from your application can make it harder to inspect and determine what failure you're running into. And again I point to application failures within the application frameworks themselves having issues (such as logs not being written out at all) that having an out-of-band monitoring process with a mostly-uncorrelated probability of failure seems to me to be very necessary for handling these less common but potentially more catastrophic issues. I'll admit that this next statement is a bit of a Call to Authority fallacy, but you'll note that not even Uber or Google use unikernels in production, and I believe these two problem domains are why.
  - nostrademons 8y ago
    Perhaps, though it seems that if you want to debug hardware, the place to do that is within the hypervisor itself. And if you want out-of-band monitoring, the place to do that is from a supervisor process running on a different box, a la Erlang supervisor trees. Very little of this infrastructure has been built yet, which is why we don't see this in production deployments. I'm talking decades out - I'll predict that the software industry will eventually tend towards this architecture because it's more efficient and better fits what we actually do with software now, but until it's a pressing pain point there's little urgency in getting there. Much like how all the folks who predicted in the 80s that CPUs would tend toward RISC were right, but it took 30 years and the ARM/mobile revolution to get there, and most desktops still have a hardware x86 emulation layer on top of RISC microcode.
    - ISV_Damocles 8y ago
      I also agree that application -> cluster of machines is where we're going. I think it needs to be baked into an specialized programming language to really work well, though, and all of these things we rely on the OS for would have to be handled. But unikernels running on top of a hypervisor seems like a terrible solution to the problem? You're trading away the convenience of the OS tooling that's built up over the past 40+ years to get closer to the metal and squeeze more performance out, then injecting virtualization in between and throwing that performance gain back away? Why not just make a regular application at that point and not throw away all of the runtime debugging support? A unikernel application running on top of a hypervisor basically turns the unikernel part into just an incredibly large and inefficient libc, in my mind.
int0x80 8y ago
What you say is not wrong but people have very good debuggin tools for kernels. You can surely make a small debuggin monitoring layer in the unikernel and instrumentation. If it is not already done, Im sure it will be soon.

nostrademons 8y ago

One of the things that struck me when I first joined Google was the extent to which logging, debugging, and tracing information was built into the application itself, as well as the extent to which this information was routinely useful in a way that basic UNIX performance tools are not. They have, for example, a distributed tracing system [1] integrated into their RPC layer that lets you instantly view the complete set of RPCs kicked off by a request, the services touched, and statistics for latency either for a single request or across many requests. This means that if you're a developer trying to integrate a new service into the search stack, you can get an answer within a couple of minutes as to "Your service will be on the critical path, so you need to count microseconds in implementation" vs. "Your service is off the critical path, go wild" vs. "You have a budget of 25ms, as long as you come in under that you should be fine." It lets you tell whether your application requests were slow because you triggered an edge-case in your code vs. because that BigTable your dependency relies upon was undergoing a compaction. Combined with some other application-level logging, you could tell if your ranking algorithm was slow vs. it happened to share a machine with a process that hogged the machine resources. It's probably one of the features I miss most now that I'm no longer at Google and doing my own startup, and I've tried to apply the same development philosophy to my own code. It's been invaluable, for instance, to tell whether I'm not getting results because I can't parse a page but forgot to write the error-handler to mark it as done & unparseable vs. because a complicated algorithm I use is accidentally exponential-time in some cases vs. because an HTTP connection is taking too long to complete vs. because CloudFlare blocks robots and the site really doesn't want to be crawled so I should give up on it vs. because an algorithm I wrote to avoid overloading hosts has a bug that makes it get stuck performing requests once every month. Some of these could perhaps be caught with lsof or top, but they'd give me nowhere near as precise info when trying to track down the problem. All this is to say that perhaps the application (or a library used by it) really is the right place for those tools to go, because it knows far more about its operation and the likely questions you'll ask than a generic system-wide tool. This'd require a change in how we write applications and what we consider to be the responsibilities of an application developer, which probably explains why we don't have mainstream adoption yet. But in a world that's moving from multiple-apps-on-one-computer to multiple-computers-for-one-app, it makes sense, and so I wonder if the long-term trend will go in that direction. [1] https://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf https://static.googleusercontent.com/media/research.google.c...

ISV_Damocles 8y ago
I'm aware of those sorts of application level logs. I used that sort of thing when I worked at Uber, and I saw it built up over time as more classes of errors were encountered. :) But there are a whole class of application failures that are handled out-of-band, which is what I was referring to: If you need to debug issues involving hardware then not having an interface to that hardware separate from your application can make it harder to inspect and determine what failure you're running into. And again I point to application failures within the application frameworks themselves having issues (such as logs not being written out at all) that having an out-of-band monitoring process with a mostly-uncorrelated probability of failure seems to me to be very necessary for handling these less common but potentially more catastrophic issues. I'll admit that this next statement is a bit of a Call to Authority fallacy, but you'll note that not even Uber or Google use unikernels in production, and I believe these two problem domains are why.
- nostrademons 8y ago
  Perhaps, though it seems that if you want to debug hardware, the place to do that is within the hypervisor itself. And if you want out-of-band monitoring, the place to do that is from a supervisor process running on a different box, a la Erlang supervisor trees. Very little of this infrastructure has been built yet, which is why we don't see this in production deployments. I'm talking decades out - I'll predict that the software industry will eventually tend towards this architecture because it's more efficient and better fits what we actually do with software now, but until it's a pressing pain point there's little urgency in getting there. Much like how all the folks who predicted in the 80s that CPUs would tend toward RISC were right, but it took 30 years and the ARM/mobile revolution to get there, and most desktops still have a hardware x86 emulation layer on top of RISC microcode.
  - ISV_Damocles 8y ago
    I also agree that application -> cluster of machines is where we're going. I think it needs to be baked into an specialized programming language to really work well, though, and all of these things we rely on the OS for would have to be handled. But unikernels running on top of a hypervisor seems like a terrible solution to the problem? You're trading away the convenience of the OS tooling that's built up over the past 40+ years to get closer to the metal and squeeze more performance out, then injecting virtualization in between and throwing that performance gain back away? Why not just make a regular application at that point and not throw away all of the runtime debugging support? A unikernel application running on top of a hypervisor basically turns the unikernel part into just an incredibly large and inefficient libc, in my mind.

int0x80 8y ago

What you say is not wrong but people have very good debuggin tools for kernels. You can surely make a small debuggin monitoring layer in the unikernel and instrumentation. If it is not already done, Im sure it will be soon.