4 ms·
The problem is how to debug sporadic production problems. It doesn't matter how good your tooling is to debug a problem in a dev environment if you have no ide
by btilly 11y ago
The problem is how to debug sporadic production problems. It doesn't matter how good your tooling is to debug a problem in a dev environment if you have no idea how to reproduce it there.
You need some level of decent logging in the production environment (with optional extra logging that can be turned on) to capture WTF went wrong. THEN you can try to reproduce it. When a logged system goes boom, you need it to come out of production and remain around until those logs are saved somewhere.
It is old, but http://lkml.iu.edu/hypermail/linux/kernel/0009.1/1307.html http://lkml.iu.edu/hypermail/linux/kernel/0009.1/1307.html is an interesting data point about IBM's experience. They implemented exactly the kind of logging that I advocate in all of their systems until OS/2. And the fact that they didn't have it in OS/2 was a big problem.
- derefr 11y agoYou're right. Coming from the Erlang world, I almost implicitly assume good logging (and extremely useful task-granular crashdumps.) If you're implementing your app on a custom unikernel "platform" that isn't basically-an-OS, the first and most important step is logging the hell out of everything.
- jupp0r 11y agoBut that's the case for every system of sufficient complexity (that I've seen).
- derefr 11y agoThere's an alternate approach that I was mentally contrasting with. You tend to see it with e.g. Ruby apps on Heroku: you can't "log into" a live app, but you can launch an IRB REPL—with all the app's dependencies loaded—in the context of the live app. Having this ability to "futz around in prod" frequently obviates the need for prescient levels of logging. You can poke at the problem until it crashes for you.
- eropple 11y agoRuby folks, don't let not being in Heroku stop you from taking this approach, because it's the best. pry-remote (are you using irb instead of pry? please stop) will give you the same fantastic behavior. It's part of every persistent Ruby service I deploy (guarded for internal access only, of course, don't be crazy). https://github.com/Mon-Ouie/pry-remote https://github.com/Mon-Ouie/pry-remote
- edwintorok 11y agoI haven't tried deploying a unikernel in production yet - I've been mostly using/debugging only OCaml code on Linux in production - but it should be possible to implement the kind of logging you describe. For example I've seen a project that would collect&dump its log ring-buffer when an exception was encountered in a unikernel, or one that collects very detailed profiling information from a unikernel. It would be nice to have some kind of a "shell" to inspect details about the application when something goes wrong, but that applies equally to non-unikernel applications. The difference is that with unikernels the debugging tool would be provided as a library, whereas with traditional applications debugging is provided as a language-independent OS tool.
- nulltype 11y agoI use Go on App Engine and have never been able to SSH or GDB those machines. Nevertheless, I am still able to debug issues in my app. I admit that debugging is easier on platforms where you get more control.
- talex5 11y agoJust to add some links for those (I assume these are the ones you mean): Dumping the debug-level log ring-buffer on exception: http://lists.xenproject.org/archives/html/mirageos-devel/2016-01/msg00027.html http://lists.xenproject.org/archives/html/mirageos-devel/201... Detailed profiling (interactive JavaScript viewer for thread interactions): https://mirage.io/wiki/profiling https://mirage.io/wiki/profiling