2 ms·
Last month I had a deployment go wrong on one box, and the part of deployments outside of my control is all or nothing. No partial credit for 96% success. Some
by hinkley 4y ago
Last month I had a deployment go wrong on one box, and the part of deployments outside of my control is all or nothing. No partial credit for 96% success. Some random consul call consumed the port we listen on, a shutdown timeout expired and the process was killed, and so that socket was left in CLOSE_WAIT (like seriously, Hashicorp, SO_LINGER has been around for at least 30 years).
This led to an existential crisis because given the number of ports we open and the number of machines we run and the number of processes per machine, there must be over a 0.1% chance of any deployment blowing up this way. We do hundreds a year in prod and probably hundreds a month in preprod. We've been winning the lottery this whole time.
Throw enough events around and a one in a million corner case will happen every week, every day, twice a day, three times in a row. That gets old really really quickly.
- metadat 4y ago> SO_LINGER. Lingers on close if data is present. If this option is enabled and there is unsent data present when close() is called, the calling application program is blocked during the close() call, until the data is transmitted or the connection has timed out. I had to look this one up for a refresher, but 100% violently agree - Such behavior certainly warrants a bug submission.
- hinkley 4y agoThe obnoxious thing about CLOSE_WAIT is that it's supposed to time out after 2 minutes or 10 minutes, but I gave up and kicked the box out of the cluster after a half hour of trying to ask it nicely. Which is probably what everyone else does.