8 ms·

Unikernels can be difficult to debug. All the tools that come with your operating system are missing. A big chunk of managing distributed systems at scale is t

Unikernels can be difficult to debug. All the tools that come with your operating system are missing. A big chunk of managing distributed systems at scale is trying to figure out what's wrong with them when they don't behave. Most operators will gladly trade some performance overhead for access to a suite of tools they're accustomed too.

andrewstuart 8y ago
>>Unikernels can be difficult to debug. This is an irrelevant objection that seems to get massive traction. Just like some politicians have scathing criticisms of other politicians, which despite being not true, are easily and widely believed. It's not even a relevant point. Nothing I ever did with Unikernels hit this as even an issue on the horizon. If you want to knock down the technology, this is the most effective way to do it because it very effectively frightens off people considering the technology. The origin of the "Unikernels can be difficult to debug." is from someone who works at a company that bet everything on containers. Nuff said.
- kjeetgill 8y ago
  Honest question: Do you still have the ability to ls, find,grep, netstat, lsof stuff? I don't know if unikernels can be difficult to debug but if you told me I didn't have those I would be nervous at a glance at least. Maybe we've been mislead.
  - lkurusa 8y ago
    Most unikernels are a kernel specifically packaged for a certain application, so the answer would be no.
  - pjmlp 8y ago
    No, but they again you don't really need them. Remote debugging and cluster monitoring tools like JEE servers provide would already be quite good. Debugging UNIX way is not the only path.
- noselasd 8y ago
  People need to debug their application. If you need to convince people that this is a myth, you need to tell people how to debug and inspect their application and system. If anything your own response here is like the typical politician- avoiding the issue, providing anecdotes that promotes your view, going on the offense and changing the subject.
  - andrewstuart 8y ago
    >> People need to debug their application. I'm not saying applications don't need to be debugged, that would be silly. What I am saying is that this "Unikernels are hard to debug" is a hand wavey way of knocking down the technology, and it has really worked - witness this subthread. The unikernel is the final build, the package, the output of application development and debugging which happens upstream, before you put it into the package. There's always ways you can expose the inner workings of a unikernel to gain insight into whatever might be going wrong for whatever reason - if you really need to do it at the stage that it is a running unikernel - but in most cases its just not necessary because application development and debugging happens, as I say, upstream of the final building of the thing into a unikernel.
    - ehsanu1 8y ago
      and debugging which happens upstream I believe the objection is clearly about debugging in production, not during development. This is absolutely essential if you want to be able to fix things in production that broke unexpectedly long after development/deployment.
      - weberc2 8y ago
        Yeah, it's really strange that he's not addressing this, since the unikernel answer is relatively straight forward--you just compile in the debugging abilities (e.g., a protected endpoint that lets you inspect the application dynamically). That said, I don't know of any unikernels that support this.
    - noselasd 8y ago
- ISV_Damocles 8y ago
  Instead of straw-manning the GP, why not address the very specific concerns raised? If there's an error you're seeing in a cluster of, say, 30 machines that only seems to happen once every few days, how are we supposed to debug this? Since the application and kernel are joined together so closely, there aren't any parallel processes that could be monitoring the behavior of the unikernel app. You can't log in on the affected machine and pull logs really, because the process may not be able to even write a log about what's failing when its in the process of failing. If you have a situation where a single server is having performance issues in a unikernel approach, there is no UNIX-style environment to log in and poke the hardware configuration and health, whereas on a more normal stack you could log in and see that NetworkManager has degraded the connection down from 10Gbps to 100Mbps because it can't negotiate a higher speed successfully, anymore, and then you can configure your alerting system to watch that stat, remove any boxes in that state from the load balancer, and inform DevOps about the bad hardware. All of that to then say that I really do like the idea of Unikernels, and I believe they have their place, but that place tends to be very well-defined and infrequently changing problem domains that need very high performance, such as in supercomputers. It doesn't work as well elsewhere because the application, by being joined so closely to the kernel, is also responsible for these things operating systems have traditionally taken care of for us, and problems in the application code can have an even more catastrophic effect than normal. Most applications are neither well-defined enough, time-invariant enough, or latency-constrained enough for the unikernel tradeoff to make sense.
  - nostrademons 8y ago
    One of the things that struck me when I first joined Google was the extent to which logging, debugging, and tracing information was built into the application itself, as well as the extent to which this information was routinely useful in a way that basic UNIX performance tools are not. They have, for example, a distributed tracing system [1] integrated into their RPC layer that lets you instantly view the complete set of RPCs kicked off by a request, the services touched, and statistics for latency either for a single request or across many requests. This means that if you're a developer trying to integrate a new service into the search stack, you can get an answer within a couple of minutes as to "Your service will be on the critical path, so you need to count microseconds in implementation" vs. "Your service is off the critical path, go wild" vs. "You have a budget of 25ms, as long as you come in under that you should be fine." It lets you tell whether your application requests were slow because you triggered an edge-case in your code vs. because that BigTable your dependency relies upon was undergoing a compaction. Combined with some other application-level logging, you could tell if your ranking algorithm was slow vs. it happened to share a machine with a process that hogged the machine resources. It's probably one of the features I miss most now that I'm no longer at Google and doing my own startup, and I've tried to apply the same development philosophy to my own code. It's been invaluable, for instance, to tell whether I'm not getting results because I can't parse a page but forgot to write the error-handler to mark it as done & unparseable vs. because a complicated algorithm I use is accidentally exponential-time in some cases vs. because an HTTP connection is taking too long to complete vs. because CloudFlare blocks robots and the site really doesn't want to be crawled so I should give up on it vs. because an algorithm I wrote to avoid overloading hosts has a bug that makes it get stuck performing requests once every month. Some of these could perhaps be caught with lsof or top, but they'd give me nowhere near as precise info when trying to track down the problem. All this is to say that perhaps the application (or a library used by it) really is the right place for those tools to go, because it knows far more about its operation and the likely questions you'll ask than a generic system-wide tool. This'd require a change in how we write applications and what we consider to be the responsibilities of an application developer, which probably explains why we don't have mainstream adoption yet. But in a world that's moving from multiple-apps-on-one-computer to multiple-computers-for-one-app, it makes sense, and so I wonder if the long-term trend will go in that direction. [1] https://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf https://static.googleusercontent.com/media/research.google.c...
wmf 8y ago
Yeah, production-quality unikernels would need to have built-in observability, tracing, and debugging support (probably in the libOS) but AFAIK no one has started building such tooling. I suspect that production-quality unikernels would end up re-introducing a lot of the "bloat" that unikernels were supposed to eliminate.
- convolvatron 8y ago
  yes. idk how bloaty it would be. an x86-64 gdb stub weighs in at < 1000 lines. if one were starting from scratch, a uniform measurement/management interface along the lines of osquery, sysctl, or /proc would be alot more straightforward on both the back and front end than the current weird mix of command line tools that linux presents. I think the most interesting question, as others have mentioned in the thread, is providing decent debugging and introspection in the distributed context. the development of those kinds of tools seems pretty orthogonal to the size of your kernel and whether or not it runs systemd.
- himom 8y ago
  Yup. GNU/Linux has just been reinvented. Another flavor of OS to support equals $$$$.
  - jacobush 8y ago
    IBM OS 360 has just been reinvented. It's turtles all the way down to the 1960s. This is just like nature. We are witnessing something like the evolution of one celled organisms. Also, multicelled organisms have started to appear. That would be cloud services (mats of algae and bacteria) and machine learning (true multicelled organisms).
- perbu 8y ago
  I've heard this quite a few times. There is really nothing fundamental in keeping you from having the same or better observability in a Unikernel as opposed to a traditional POSIX kernel. A couple of examples. The Gnu Debugger (gdb) supports connecting to qemu-backed virtual machines, so you have the ability to everything that gdb can on a unikernel as well, given the unikernel is implemented in something gdb groks. In addition we've created a trace layer, similar to strace that allows you to run strace inside the vm itself. Since the POSIX libs are just libs wrapping them inside something like strace is rather trivial. Also the unikernels have a huge advantage of having everything in the same memory space, meaning that you can observe the application and operating system at the same time. High level languages like Python or Node.js can typically have debuggers loaded dynamically and as such would be able to offer full debuggability without having to do tricks with gdb. All this doesn't have to induce bloat. The socalled bloat in POSIX systems is mostly caused by the security barriers raised by a design that forces strict separation between unprivileged and privileged access to the system. In addition a POSIX kernel will typically be built to support "everything" that a user might need whereas a unikernel can take a build-time decision on what to include into the OS.
baybal2 8y ago
>Unikernels can be difficult to debug. All the tools that come with your operating system are missing. Do you even need to? If something so simple breaks, why not to use, say, JTAG?
- wmf 8y ago
  Did Elastic Cloud JTAG come out while I wasn't looking?
  - baybal2 8y ago
    i think giving JTAG access on cloud machines may not be that a bad idea. Connect as low-end sbb to server's JTAG output, and have it connected to the network. JTAG is beautiful tool for debugging because it requires no modification of software being debugged.
    - comex 8y ago
      That would give the user full control over the physical CPU, though, whereas most "cloud" services only give you access to a virtual machine. On the other hand, it should be possible for the hypervisor to provide roughly the same capabilities in software, without having to modify the software inside the virtual machine. Indeed, most hypervisors already provide a GDB stub for this purpose, including, e.g., Xen [1]; it's just that public clouds don't typically give customers access to it. It would be nice if that changed. [1] https://github.com/xen-project/xen/tree/master/tools/debugger/gdbsx https://github.com/xen-project/xen/tree/master/tools/debugge...
      - baybal2 8y ago
        Somebody's cloud is engineered the wrong way if somebody can wreck the entirety of it when only having access to a single server
        Nullabillity 8y ago
        Usually the same physical server has multiple tenants, which do not trust each other.

andrewstuart 8y ago

>>Unikernels can be difficult to debug. This is an irrelevant objection that seems to get massive traction. Just like some politicians have scathing criticisms of other politicians, which despite being not true, are easily and widely believed. It's not even a relevant point. Nothing I ever did with Unikernels hit this as even an issue on the horizon. If you want to knock down the technology, this is the most effective way to do it because it very effectively frightens off people considering the technology. The origin of the "Unikernels can be difficult to debug." is from someone who works at a company that bet everything on containers. Nuff said.

kjeetgill 8y ago
Honest question: Do you still have the ability to ls, find,grep, netstat, lsof stuff? I don't know if unikernels can be difficult to debug but if you told me I didn't have those I would be nervous at a glance at least. Maybe we've been mislead.
- lkurusa 8y ago
  Most unikernels are a kernel specifically packaged for a certain application, so the answer would be no.
- pjmlp 8y ago
  No, but they again you don't really need them. Remote debugging and cluster monitoring tools like JEE servers provide would already be quite good. Debugging UNIX way is not the only path.
noselasd 8y ago
People need to debug their application. If you need to convince people that this is a myth, you need to tell people how to debug and inspect their application and system. If anything your own response here is like the typical politician- avoiding the issue, providing anecdotes that promotes your view, going on the offense and changing the subject.
- andrewstuart 8y ago
  >> People need to debug their application. I'm not saying applications don't need to be debugged, that would be silly. What I am saying is that this "Unikernels are hard to debug" is a hand wavey way of knocking down the technology, and it has really worked - witness this subthread. The unikernel is the final build, the package, the output of application development and debugging which happens upstream, before you put it into the package. There's always ways you can expose the inner workings of a unikernel to gain insight into whatever might be going wrong for whatever reason - if you really need to do it at the stage that it is a running unikernel - but in most cases its just not necessary because application development and debugging happens, as I say, upstream of the final building of the thing into a unikernel.
  - ehsanu1 8y ago
    and debugging which happens upstream I believe the objection is clearly about debugging in production, not during development. This is absolutely essential if you want to be able to fix things in production that broke unexpectedly long after development/deployment.
    - weberc2 8y ago
      Yeah, it's really strange that he's not addressing this, since the unikernel answer is relatively straight forward--you just compile in the debugging abilities (e.g., a protected endpoint that lets you inspect the application dynamically). That said, I don't know of any unikernels that support this.
  - noselasd 8y ago
ISV_Damocles 8y ago
Instead of straw-manning the GP, why not address the very specific concerns raised? If there's an error you're seeing in a cluster of, say, 30 machines that only seems to happen once every few days, how are we supposed to debug this? Since the application and kernel are joined together so closely, there aren't any parallel processes that could be monitoring the behavior of the unikernel app. You can't log in on the affected machine and pull logs really, because the process may not be able to even write a log about what's failing when its in the process of failing. If you have a situation where a single server is having performance issues in a unikernel approach, there is no UNIX-style environment to log in and poke the hardware configuration and health, whereas on a more normal stack you could log in and see that NetworkManager has degraded the connection down from 10Gbps to 100Mbps because it can't negotiate a higher speed successfully, anymore, and then you can configure your alerting system to watch that stat, remove any boxes in that state from the load balancer, and inform DevOps about the bad hardware. All of that to then say that I really do like the idea of Unikernels, and I believe they have their place, but that place tends to be very well-defined and infrequently changing problem domains that need very high performance, such as in supercomputers. It doesn't work as well elsewhere because the application, by being joined so closely to the kernel, is also responsible for these things operating systems have traditionally taken care of for us, and problems in the application code can have an even more catastrophic effect than normal. Most applications are neither well-defined enough, time-invariant enough, or latency-constrained enough for the unikernel tradeoff to make sense.
- nostrademons 8y ago
  One of the things that struck me when I first joined Google was the extent to which logging, debugging, and tracing information was built into the application itself, as well as the extent to which this information was routinely useful in a way that basic UNIX performance tools are not. They have, for example, a distributed tracing system [1] integrated into their RPC layer that lets you instantly view the complete set of RPCs kicked off by a request, the services touched, and statistics for latency either for a single request or across many requests. This means that if you're a developer trying to integrate a new service into the search stack, you can get an answer within a couple of minutes as to "Your service will be on the critical path, so you need to count microseconds in implementation" vs. "Your service is off the critical path, go wild" vs. "You have a budget of 25ms, as long as you come in under that you should be fine." It lets you tell whether your application requests were slow because you triggered an edge-case in your code vs. because that BigTable your dependency relies upon was undergoing a compaction. Combined with some other application-level logging, you could tell if your ranking algorithm was slow vs. it happened to share a machine with a process that hogged the machine resources. It's probably one of the features I miss most now that I'm no longer at Google and doing my own startup, and I've tried to apply the same development philosophy to my own code. It's been invaluable, for instance, to tell whether I'm not getting results because I can't parse a page but forgot to write the error-handler to mark it as done & unparseable vs. because a complicated algorithm I use is accidentally exponential-time in some cases vs. because an HTTP connection is taking too long to complete vs. because CloudFlare blocks robots and the site really doesn't want to be crawled so I should give up on it vs. because an algorithm I wrote to avoid overloading hosts has a bug that makes it get stuck performing requests once every month. Some of these could perhaps be caught with lsof or top, but they'd give me nowhere near as precise info when trying to track down the problem. All this is to say that perhaps the application (or a library used by it) really is the right place for those tools to go, because it knows far more about its operation and the likely questions you'll ask than a generic system-wide tool. This'd require a change in how we write applications and what we consider to be the responsibilities of an application developer, which probably explains why we don't have mainstream adoption yet. But in a world that's moving from multiple-apps-on-one-computer to multiple-computers-for-one-app, it makes sense, and so I wonder if the long-term trend will go in that direction. [1] https://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf https://static.googleusercontent.com/media/research.google.c...

wmf 8y ago

Yeah, production-quality unikernels would need to have built-in observability, tracing, and debugging support (probably in the libOS) but AFAIK no one has started building such tooling. I suspect that production-quality unikernels would end up re-introducing a lot of the "bloat" that unikernels were supposed to eliminate.

convolvatron 8y ago
yes. idk how bloaty it would be. an x86-64 gdb stub weighs in at < 1000 lines. if one were starting from scratch, a uniform measurement/management interface along the lines of osquery, sysctl, or /proc would be alot more straightforward on both the back and front end than the current weird mix of command line tools that linux presents. I think the most interesting question, as others have mentioned in the thread, is providing decent debugging and introspection in the distributed context. the development of those kinds of tools seems pretty orthogonal to the size of your kernel and whether or not it runs systemd.
himom 8y ago
Yup. GNU/Linux has just been reinvented. Another flavor of OS to support equals $$$$.
- jacobush 8y ago
  IBM OS 360 has just been reinvented. It's turtles all the way down to the 1960s. This is just like nature. We are witnessing something like the evolution of one celled organisms. Also, multicelled organisms have started to appear. That would be cloud services (mats of algae and bacteria) and machine learning (true multicelled organisms).
perbu 8y ago
I've heard this quite a few times. There is really nothing fundamental in keeping you from having the same or better observability in a Unikernel as opposed to a traditional POSIX kernel. A couple of examples. The Gnu Debugger (gdb) supports connecting to qemu-backed virtual machines, so you have the ability to everything that gdb can on a unikernel as well, given the unikernel is implemented in something gdb groks. In addition we've created a trace layer, similar to strace that allows you to run strace inside the vm itself. Since the POSIX libs are just libs wrapping them inside something like strace is rather trivial. Also the unikernels have a huge advantage of having everything in the same memory space, meaning that you can observe the application and operating system at the same time. High level languages like Python or Node.js can typically have debuggers loaded dynamically and as such would be able to offer full debuggability without having to do tricks with gdb. All this doesn't have to induce bloat. The socalled bloat in POSIX systems is mostly caused by the security barriers raised by a design that forces strict separation between unprivileged and privileged access to the system. In addition a POSIX kernel will typically be built to support "everything" that a user might need whereas a unikernel can take a build-time decision on what to include into the OS.

baybal2 8y ago

>Unikernels can be difficult to debug. All the tools that come with your operating system are missing. Do you even need to? If something so simple breaks, why not to use, say, JTAG?

wmf 8y ago
Did Elastic Cloud JTAG come out while I wasn't looking?
- baybal2 8y ago
  i think giving JTAG access on cloud machines may not be that a bad idea. Connect as low-end sbb to server's JTAG output, and have it connected to the network. JTAG is beautiful tool for debugging because it requires no modification of software being debugged.
  - comex 8y ago
    That would give the user full control over the physical CPU, though, whereas most "cloud" services only give you access to a virtual machine. On the other hand, it should be possible for the hypervisor to provide roughly the same capabilities in software, without having to modify the software inside the virtual machine. Indeed, most hypervisors already provide a GDB stub for this purpose, including, e.g., Xen [1]; it's just that public clouds don't typically give customers access to it. It would be nice if that changed. [1] https://github.com/xen-project/xen/tree/master/tools/debugger/gdbsx https://github.com/xen-project/xen/tree/master/tools/debugge...
    - baybal2 8y ago
      Somebody's cloud is engineered the wrong way if somebody can wreck the entirety of it when only having access to a single server
      - Nullabillity 8y ago
        Usually the same physical server has multiple tenants, which do not trust each other.