4 ms·

>What’s interesting about debugging this is that it actually is much easier to debug than a Linux system. Why? Because there is only one program in question. Yo

by jasode 5y ago

>What’s interesting about debugging this is that it actually is much easier to debug than a Linux system. Why? Because there is only one program in question. You aren’t whipping out lsof to figure out what process is spewing out crap connections or which process didn’t have a proper log rotation setup and prevented you from SSHing into the instance. [...] Unikernels are so much easier to debug than normal Linux systems. I have no strong opinion either way but just to add counterpoint from Bryan Cantrill's 2016 article[1]: >[...] the most profound reason that unikernels are unfit for production — and the reason that (to me, anyway) strikes unikernels through the heart when it comes to deploying anything real in production: Unikernels are entirely undebuggable. There are no processes, so of course there is no ps, no htop, no strace — but there is also no netstat, no tcpdump, no ping! And these are just the crude, decades-old tools. There is certainly nothing modern like DTrace or MDB. From a debugging perspective, to say this is primitive understates it: this isn’t paleolithic — it is precambrian. As one who has spent my career developing production systems and the tooling to debug them, I find the implicit denial of debugging production systems to be galling, and symptomatic of a deeper malaise among unikernel proponents: total lack of operational empathy. Production problems are simply hand-waved away — services are just to be restarted when they misbehave. And a previous 2018 HN subthread had differing opinions on debugging unikernels: https://news.ycombinator.com/item?id=17260604 https://news.ycombinator.com/item?id=17260604 [1] https://www.joyent.com/blog/unikernels-are-unfit-for-production https://www.joyent.com/blog/unikernels-are-unfit-for-product...

throwaway894345 5y ago
I think most of this can be significantly reduced with mature development processes and discipline. A mature org almost never needs to SSH into a production instance in the first place—extensive debug information is available via logging and monitoring, and when a problem arises in production you try to replicate it in lower environments if necessary. This doesn’t always work and sometimes it would be faster to just SSH and call it a day, but in exchange for that convenience you get a bunch of nice properties. Moreover, you can always compile in the debugging utilities that you absolutely need (or even better—create network interfaces for them).
- lostcolony 5y ago
  Not to mention he even mentions an obvious fix - "Production problems are simply hand-waved away — services are just to be restarted when they misbehave. " Even if my telemetry doesn't tell me all I need, if it's easy to reproduce, I can reproduce it in an environment that has all the debug stuff I want (and I can futz with it to my heart's content safely, too). If it's hard to reproduce...then I don't know that I really want to spend time on it if a reboot will just fix it. While there are probably some in between cases (hard to reproduce, unable to be caught by solid telemetry, we have an instance that is in a bad state that I could remove from the LB and safely futz with, and actually gain useful information from an after-the-fact debugging rather than inspection as it runs under real load), I'm not sure I can recollect one that fits those criteria across 6 or so different companies and domains. That said, that's hardly exhaustive, and I'll admit at only two of those were we using anything akin to a unikernel.
  - throwaway894345 5y ago
    Very well said. This is basically what I picture when I think of "mature organizations". You strive to never touch things in production except to restart them or deploy a different version. When you find yourself wishing you had some debug utility, that's certainly a sign that you need better observability. Rolling out a new release with your improved instrumentation isn't a good fit for an urgent problem, but these cases become increasingly rare over time as your observability improves and moreover you can always deploy a "debug" version of your application with various debug utilities compiled in).
  - naasking 5y ago
    From what I recall, a lot of unikernel hype started around MirageOS and OCaml. When your language encourages immutability, debugging on the live system is simply not that useful. Take a snapshot of the live system and replay it on a local machine like you describe where you have considerably more sophisticated tools. I assume unikernels are also big on reproducible builds, which means if there are issues you can't debug like above, then you can easily and quickly deploy a new version with more tracing code inserted exactly where you want. "Traditional debugging" tools can't get this precise, so I can't say I'm convinced by that objection, although I agree developing in this paradigm would require some domain knowledge/experience to set things up right.
nine_k 5y ago
I don't see why a unikernel may not have a dedicated debug interface, much like MCUs that lack a console but have JTAG.
- marcosdumay 5y ago
  It seems to be a tooling problem. But it's a large deficit, and there's a catch 22 in that the people most concerned with tooling try to use architectures with mature tooling, and don't write it for the immature ones. There is also the problem of a lot extra code volume without visible interfaces separating them. Once it's a networking problem, is it on your code, on the driver, on some other part of the code that is interfering with ours? But that also is a matter of architecture details and tooling.
- gnulinux 5y ago
  Because someone needs to go write code for them, or make linux tools work in the said unikernel.
  - deleted 5y ago
    [deleted]
- deleted 5y ago
  [deleted]
- marcan_42 5y ago
  That interface would have to exist at the host level (just like JTAG exists at the hardware level), but you won't see cloud vendors exposing a GDB stub interface to your VM instances any time soon.
cryptonector 5y ago
That was 2016. I don't know the state of unikernel debugging today, but I can imagine a unikernel having all the observability goodies that Solaris/Illumos have, or Linux for that matter. mdb? maybe via a console just like on Solaris/Illumos, or from the hypervisor. gdb? maybe via a gdb server in the unikernel, or from the hypervisor. DTrace, eBPF? maybe via a server like a gdb server, but for those, in the unikernel. I can't imagine why those things wouldn't be possible.

by jasode 5y ago

throwaway894345 5y ago

I think most of this can be significantly reduced with mature development processes and discipline. A mature org almost never needs to SSH into a production instance in the first place—extensive debug information is available via logging and monitoring, and when a problem arises in production you try to replicate it in lower environments if necessary. This doesn’t always work and sometimes it would be faster to just SSH and call it a day, but in exchange for that convenience you get a bunch of nice properties. Moreover, you can always compile in the debugging utilities that you absolutely need (or even better—create network interfaces for them).

lostcolony 5y ago
Not to mention he even mentions an obvious fix - "Production problems are simply hand-waved away — services are just to be restarted when they misbehave. " Even if my telemetry doesn't tell me all I need, if it's easy to reproduce, I can reproduce it in an environment that has all the debug stuff I want (and I can futz with it to my heart's content safely, too). If it's hard to reproduce...then I don't know that I really want to spend time on it if a reboot will just fix it. While there are probably some in between cases (hard to reproduce, unable to be caught by solid telemetry, we have an instance that is in a bad state that I could remove from the LB and safely futz with, and actually gain useful information from an after-the-fact debugging rather than inspection as it runs under real load), I'm not sure I can recollect one that fits those criteria across 6 or so different companies and domains. That said, that's hardly exhaustive, and I'll admit at only two of those were we using anything akin to a unikernel.
- throwaway894345 5y ago
  Very well said. This is basically what I picture when I think of "mature organizations". You strive to never touch things in production except to restart them or deploy a different version. When you find yourself wishing you had some debug utility, that's certainly a sign that you need better observability. Rolling out a new release with your improved instrumentation isn't a good fit for an urgent problem, but these cases become increasingly rare over time as your observability improves and moreover you can always deploy a "debug" version of your application with various debug utilities compiled in).
- naasking 5y ago
  From what I recall, a lot of unikernel hype started around MirageOS and OCaml. When your language encourages immutability, debugging on the live system is simply not that useful. Take a snapshot of the live system and replay it on a local machine like you describe where you have considerably more sophisticated tools. I assume unikernels are also big on reproducible builds, which means if there are issues you can't debug like above, then you can easily and quickly deploy a new version with more tracing code inserted exactly where you want. "Traditional debugging" tools can't get this precise, so I can't say I'm convinced by that objection, although I agree developing in this paradigm would require some domain knowledge/experience to set things up right.

nine_k 5y ago

I don't see why a unikernel may not have a dedicated debug interface, much like MCUs that lack a console but have JTAG.

marcosdumay 5y ago
It seems to be a tooling problem. But it's a large deficit, and there's a catch 22 in that the people most concerned with tooling try to use architectures with mature tooling, and don't write it for the immature ones. There is also the problem of a lot extra code volume without visible interfaces separating them. Once it's a networking problem, is it on your code, on the driver, on some other part of the code that is interfering with ours? But that also is a matter of architecture details and tooling.
gnulinux 5y ago
Because someone needs to go write code for them, or make linux tools work in the said unikernel.
- deleted 5y ago
  [deleted]
deleted 5y ago
[deleted]
marcan_42 5y ago
That interface would have to exist at the host level (just like JTAG exists at the hardware level), but you won't see cloud vendors exposing a GDB stub interface to your VM instances any time soon.

cryptonector 5y ago

That was 2016. I don't know the state of unikernel debugging today, but I can imagine a unikernel having all the observability goodies that Solaris/Illumos have, or Linux for that matter. mdb? maybe via a console just like on Solaris/Illumos, or from the hypervisor. gdb? maybe via a gdb server in the unikernel, or from the hypervisor. DTrace, eBPF? maybe via a server like a gdb server, but for those, in the unikernel. I can't imagine why those things wouldn't be possible.