4 ms·
Not handling the return of sendmsg (or any send operation on a socket) is a common mistake (I hate to say "rookie",but the shoe does fit here). The proposed so
by muststopmyths 8y ago
Not handling the return of sendmsg (or any send operation on a socket) is a common mistake (I hate to say "rookie",but the shoe does fit here).
The proposed solution of setting the TCP_USER_TIMEOUT is worth internalizing.
In my experience, the one thing anyone writing an RPC transport over TCP should always do is set a short TCP keepalive timer to detect network connectivity breakdowns. I am talking about a per-socket basis, of course. On Linux, I guess you'd use the above option to get the same effect.
On Windows, you can set SO_KEEPALIVE_VALS, which allows you to set the TCP keepalive intervals. I couldn't find an equivalent Linux socket option by a cursory google.
The TCP keepalive defaults were designed for an age when sending packets too frequently was considered impolite. As a result, a network hole (where packets are being silently dropped) can take several minutes to detect.
Have been bitten a few times by this, so now I just make this socket option part of my standard template code for connection protocols.
- eeeeffdsx 8y agoIf I understand it correctly, the issue was’t caused by not checking the result of sendmsg, but sendmsg blocking the program when the send buffer was full.
- muststopmyths 8y agono, if you look at the github issue, sendmsg fails with EAGAIN. It's probably an upper layer that hangs.
- noselasd 8y agoNote that in this case EAGAIN is just an indication the socket buffer is full - which can easily also happen on normal operation when you're blasting off calls at full speed.
- iforgotpassword 8y agoNot that I know anything about the grpc codebase, but in my experience it is often easier to implement timeouts at a higher level instead of tweaking several socket options. Apart from being more portable, it now doesn't matter whether the connection dropped, stalled, or the handshake hangs. When your timeout fires you mark the endpoint as bad and tear down the connection, or optionally retry once if the last failure wasn't too recent.