3 ms·

I think most of this can be significantly reduced with mature development processes and discipline. A mature org almost never needs to SSH into a production ins

I think most of this can be significantly reduced with mature development processes and discipline. A mature org almost never needs to SSH into a production instance in the first place—extensive debug information is available via logging and monitoring, and when a problem arises in production you try to replicate it in lower environments if necessary. This doesn’t always work and sometimes it would be faster to just SSH and call it a day, but in exchange for that convenience you get a bunch of nice properties. Moreover, you can always compile in the debugging utilities that you absolutely need (or even better—create network interfaces for them).

lostcolony 5y ago
Not to mention he even mentions an obvious fix - "Production problems are simply hand-waved away — services are just to be restarted when they misbehave. " Even if my telemetry doesn't tell me all I need, if it's easy to reproduce, I can reproduce it in an environment that has all the debug stuff I want (and I can futz with it to my heart's content safely, too). If it's hard to reproduce...then I don't know that I really want to spend time on it if a reboot will just fix it. While there are probably some in between cases (hard to reproduce, unable to be caught by solid telemetry, we have an instance that is in a bad state that I could remove from the LB and safely futz with, and actually gain useful information from an after-the-fact debugging rather than inspection as it runs under real load), I'm not sure I can recollect one that fits those criteria across 6 or so different companies and domains. That said, that's hardly exhaustive, and I'll admit at only two of those were we using anything akin to a unikernel.
- throwaway894345 5y ago
  Very well said. This is basically what I picture when I think of "mature organizations". You strive to never touch things in production except to restart them or deploy a different version. When you find yourself wishing you had some debug utility, that's certainly a sign that you need better observability. Rolling out a new release with your improved instrumentation isn't a good fit for an urgent problem, but these cases become increasingly rare over time as your observability improves and moreover you can always deploy a "debug" version of your application with various debug utilities compiled in).
- naasking 5y ago
  From what I recall, a lot of unikernel hype started around MirageOS and OCaml. When your language encourages immutability, debugging on the live system is simply not that useful. Take a snapshot of the live system and replay it on a local machine like you describe where you have considerably more sophisticated tools. I assume unikernels are also big on reproducible builds, which means if there are issues you can't debug like above, then you can easily and quickly deploy a new version with more tracing code inserted exactly where you want. "Traditional debugging" tools can't get this precise, so I can't say I'm convinced by that objection, although I agree developing in this paradigm would require some domain knowledge/experience to set things up right.
  - deleted 5y ago
    [deleted]
toast0 5y ago
Having worked mostly in cowboy organizations, I just don't see how you can fix tricky network issues without access to the machines. And once you need access sometimes, you might as well use it always, because it's certainly handy. Application issues usually aren't so hard to replication, but even there, sometimes it's easiest to figure out how to replicate the issue in a test environment by debugging on production. Once the issue is understood, then you can validate the changes in a test environment and push to production. All that to say, if you don't have access to an equivalent of tcpdump, netstat, top, etc, it's going to be a lot harder to debug your system. And if you have a system that needs the performance of a Unikernel, you're going to experience tricky problems. If you just want a simple system, it's better IMHO to just run a minimum daemon system. sshd, getty(if you have a console), ntpd, syslogd, crond, a local caching dns server (I like unbound) a monitoring program, and your application. A servicable unikernel is going to need an equivalent to most of those and the debugging tool, so maybe not as simple as one might hope.
- throwaway894345 5y ago
  > Having worked mostly in cowboy organizations, I just don't see how you can fix tricky network issues without access to the machines. And once you need access sometimes, you might as well use it always, because it's certainly handy. The big, abstract idea is that you don't worry about individual instances (be they VMs or container instances or unikernel apps or etc), but you worry about the process for stamping out those instances. If you have a process for reproducibly creating machines, then you can test that process in lower environments (stamp out some instances, validate them, and destroy them) before promoting that process to production. When you have production issues, your first line of defense should be comprehensive logging and monitoring, but failing that you try to reproduce issues in lower environments and failing that you can deploy a version of your app with monitoring tools baked in (e.g., instead of a scratch base image, you deploy a version with a full ubuntu base image). If you're running into issues so often that you feel the need to deploy your debug tools to production all the time, then you're not appropriately investing in your logging/monitoring (ideally every time you need a debug tool in production, you should go back and add equivalent instrumentation so you don't need that debug tool any more). This is the theory, anyway. Practice involves a lot more nuance and judgement.

lostcolony 5y ago

Not to mention he even mentions an obvious fix - "Production problems are simply hand-waved away — services are just to be restarted when they misbehave. " Even if my telemetry doesn't tell me all I need, if it's easy to reproduce, I can reproduce it in an environment that has all the debug stuff I want (and I can futz with it to my heart's content safely, too). If it's hard to reproduce...then I don't know that I really want to spend time on it if a reboot will just fix it. While there are probably some in between cases (hard to reproduce, unable to be caught by solid telemetry, we have an instance that is in a bad state that I could remove from the LB and safely futz with, and actually gain useful information from an after-the-fact debugging rather than inspection as it runs under real load), I'm not sure I can recollect one that fits those criteria across 6 or so different companies and domains. That said, that's hardly exhaustive, and I'll admit at only two of those were we using anything akin to a unikernel.

throwaway894345 5y ago
Very well said. This is basically what I picture when I think of "mature organizations". You strive to never touch things in production except to restart them or deploy a different version. When you find yourself wishing you had some debug utility, that's certainly a sign that you need better observability. Rolling out a new release with your improved instrumentation isn't a good fit for an urgent problem, but these cases become increasingly rare over time as your observability improves and moreover you can always deploy a "debug" version of your application with various debug utilities compiled in).
naasking 5y ago
From what I recall, a lot of unikernel hype started around MirageOS and OCaml. When your language encourages immutability, debugging on the live system is simply not that useful. Take a snapshot of the live system and replay it on a local machine like you describe where you have considerably more sophisticated tools. I assume unikernels are also big on reproducible builds, which means if there are issues you can't debug like above, then you can easily and quickly deploy a new version with more tracing code inserted exactly where you want. "Traditional debugging" tools can't get this precise, so I can't say I'm convinced by that objection, although I agree developing in this paradigm would require some domain knowledge/experience to set things up right.
- deleted 5y ago
  [deleted]

toast0 5y ago

Having worked mostly in cowboy organizations, I just don't see how you can fix tricky network issues without access to the machines. And once you need access sometimes, you might as well use it always, because it's certainly handy. Application issues usually aren't so hard to replication, but even there, sometimes it's easiest to figure out how to replicate the issue in a test environment by debugging on production. Once the issue is understood, then you can validate the changes in a test environment and push to production. All that to say, if you don't have access to an equivalent of tcpdump, netstat, top, etc, it's going to be a lot harder to debug your system. And if you have a system that needs the performance of a Unikernel, you're going to experience tricky problems. If you just want a simple system, it's better IMHO to just run a minimum daemon system. sshd, getty(if you have a console), ntpd, syslogd, crond, a local caching dns server (I like unbound) a monitoring program, and your application. A servicable unikernel is going to need an equivalent to most of those and the debugging tool, so maybe not as simple as one might hope.

throwaway894345 5y ago
> Having worked mostly in cowboy organizations, I just don't see how you can fix tricky network issues without access to the machines. And once you need access sometimes, you might as well use it always, because it's certainly handy. The big, abstract idea is that you don't worry about individual instances (be they VMs or container instances or unikernel apps or etc), but you worry about the process for stamping out those instances. If you have a process for reproducibly creating machines, then you can test that process in lower environments (stamp out some instances, validate them, and destroy them) before promoting that process to production. When you have production issues, your first line of defense should be comprehensive logging and monitoring, but failing that you try to reproduce issues in lower environments and failing that you can deploy a version of your app with monitoring tools baked in (e.g., instead of a scratch base image, you deploy a version with a full ubuntu base image). If you're running into issues so often that you feel the need to deploy your debug tools to production all the time, then you're not appropriately investing in your logging/monitoring (ideally every time you need a debug tool in production, you should go back and add equivalent instrumentation so you don't need that debug tool any more). This is the theory, anyway. Practice involves a lot more nuance and judgement.