4 ms·

I listened to your DockerCon talk on debugging. So in a server that can handle multiple concurrent requests (e.g. a web application server), should each unhandl

I listened to your DockerCon talk on debugging. So in a server that can handle multiple concurrent requests (e.g. a web application server), should each unhandled exception abort the whole process (after delivering something like an HTTP 500 to the client) so the developer can do postmortem debugging on a core dump? That would cause all other in-progress requests to be aborted too, unlike simply logging the exception and moving on. But then again, maybe that would increase the motivation to root-cause every failure.

dap 11y ago
I don't mean to speak for bcantrill, but yes, this is exactly how we (at Joyent) build programs. We wrote in detail about why we do it this way: https://www.joyent.com/developers/node/design/errors https://www.joyent.com/developers/node/design/errors It's not that this approach motivates root-causing failures (though that's true). It's that uncaught exceptions are programmer errors, and it's tautologically impossible to correctly deal with them. Attempting to do so can make things much, much worse. To make this concrete: I've dealt more than one production outage caused by the mere presence of an uncaughtException handler. If the program had merely crashed, a hundred requests may have been interrupted, but the program would have restarted and resumed servicing requests. Instead, the exception was thrown from a code path that had a database transaction open with database locks held. Because the uncaughtException handler just logged the exception and otherwise ignored it, that transaction stayed open (and the locks remained held) until a human got into the loop -- interfering with tens of thousands of subsequent requests. That's much, much worse. If the process had just exited, the db connection would have been closed, the transaction aborted, and the locks released. An unexpected error handler can't know the complete state that the program was in because by definition this wasn't a state that the programmer thought about.
- mwcampbell 11y ago
  Thanks. Now I gotta figure out the best way to do that in Python. Most Python web frameworks and WSGI servers catch all exceptions and keep the process going.
Licenser 11y ago
The solution to this problem is simply process isolate, handle each request in a own process and crashing one won't affect the others. Of cause you won't get away with OS level processes. If it interests you look at Erlang's OTP, it solved this problem years ago and works very well with that.

dap 11y ago

I don't mean to speak for bcantrill, but yes, this is exactly how we (at Joyent) build programs. We wrote in detail about why we do it this way: https://www.joyent.com/developers/node/design/errors https://www.joyent.com/developers/node/design/errors It's not that this approach motivates root-causing failures (though that's true). It's that uncaught exceptions are programmer errors, and it's tautologically impossible to correctly deal with them. Attempting to do so can make things much, much worse. To make this concrete: I've dealt more than one production outage caused by the mere presence of an uncaughtException handler. If the program had merely crashed, a hundred requests may have been interrupted, but the program would have restarted and resumed servicing requests. Instead, the exception was thrown from a code path that had a database transaction open with database locks held. Because the uncaughtException handler just logged the exception and otherwise ignored it, that transaction stayed open (and the locks remained held) until a human got into the loop -- interfering with tens of thousands of subsequent requests. That's much, much worse. If the process had just exited, the db connection would have been closed, the transaction aborted, and the locks released. An unexpected error handler can't know the complete state that the program was in because by definition this wasn't a state that the programmer thought about.

mwcampbell 11y ago
Thanks. Now I gotta figure out the best way to do that in Python. Most Python web frameworks and WSGI servers catch all exceptions and keep the process going.

Licenser 11y ago

The solution to this problem is simply process isolate, handle each request in a own process and crashing one won't affect the others. Of cause you won't get away with OS level processes. If it interests you look at Erlang's OTP, it solved this problem years ago and works very well with that.