* Netty has a boss thread that accepts incoming connections and assigns each successfully-opened socket to a single particular I/O thread. * Each I/O thread ru

* Netty has a boss thread that accepts incoming connections and assigns each successfully-opened socket to a single particular I/O thread. * Each I/O thread runs an infinite loop that repeatedly waits for activity on its assigned sockets (using epoll/kqueue/select) and runs each received TCP segment through our Netty pipeline on that thread. * Usually when an I/O thread writes to one of its own sockets, the write takes place synchronously. However, crucially, it may defer at least part of the write until later, for example if the kernel’s buffer is full. The write would then be performed on a later loop when the selector (epoll/kqueue/select) reports that the socket is ready for writing. * We were writing to a channel and then blocking on the result Future to see if it succeeded. Since we were writing from the I/O thread, the write would usually be performed synchronously so the returned future would already be complete and the application would continue. However, sometimes the write wouldn’t fully complete and the future could not be completed until the next time around the selector loop. But the I/O thread was blocked, so the loop couldn’t proceed, so the future would never complete. * Since the I/O thread was stuck, it couldn’t respond to any further messages from any of its managed sockets. Eventually the client would give up and send a FIN segment, and the kernel’s TCP/IP stack would put the socket in the CLOSE_WAIT state. Usually the I/O thread handles this in its worker loop by calling close on the socket, but it was stuck so this code never ran and the socket would never close. * The boss thread was still running, so the system continued to accept new connections and assign some of them to the stuck I/O thread. To add insult to injury, Netty actually has a deadlock detector to prevent this scenario from occurring. Unfortunately, one of the libraries that we were using at the time is also based on Netty, and they explicitly disabled the deadlock checker, globally! https://github.com/AsyncHttpClient/async-http-client/commit/b4a6dbd103ec611f70879cfc78fc5cce5c63a4e6#diff-f86cd91228f3cde51e071f609d997d5bR205 https://github.com/AsyncHttpClient/async-http-client/commit/... So there you have it. Don't block in your Netty worker threads. This should have been obvious at the time - I had worked with Netty before - but we were too busy investigating to sit back and just think. If I had stood in front of a whiteboard for an hour, I suspect I could have worked it out, but instead we spent anxious days tearing our hair out attacking the problem by stepping through multiple threads of code in a debugger (racing timeouts is always fun) and taking endless stack traces and thread dumps. The short-term mitigation ended up being trivial: we could still block the worker thread; we just had to check to see if the channel was already closed first! This is not a perfect fix, but it let us preserve the illusion that the application code had a real bona-fide blocking java.io.OutputStream. The long-term solution is to switch the API handlers over to using real async I/O. To this day, my fellow engineer investigating this issue refers to this saga as the Christmas Miracle bug. :)