Random example from today - we have a new product being deployed from a prominent vendor, and they started running into a ton of "out of socket" errors. This

Random example from today - we have a new product being deployed from a prominent vendor, and they started running into a ton of "out of socket" errors. This product was important to one of our telemetry organizations Q4 OKRs, and they weren't getting traction with the vendor, so they started reaching out internally. I got tagged as someone with a background in networking (though this was clearly not a networking issue) - and started poking around. The first thing that comes to mind of course, is file descriptions, but ulimit -n returned a massive number and "ss -a" showed there were a tiny number of sockets opened (like, 1300 - we have lots of nodes with 500K+ sockets opened - so this really didn't feel like a node or docker container issue). So - I just reached out to Chat, and we started going back and forth, starting with "In linux, how many open sockets can there be at a time?". What's nice, about that wide open question, is you don't get a single answer, instead you get a briefing on Linux Sockets. File Descriptors, sure, but also memory, port range, TCP/IP stack limits, etc... It starts to lay out a roadmap towards solving the issue, answering the question you were interested in, rather than the one you asked. I do a bit of back and forth on some scenarios, asking about /proc, and ss, etc.. seeing if I can track anything else out. And then, after spending about 5 minutes, and building context, I ask it "Is there anything else that can cause an error regarding too many sockets with low socket use" - at which point it lays out a number of scenarios, one of which is FD_SETSIZE. So - we dig into FD_SETSIZE, and immediately that looks interesting - it's the limit that you can use with a select() call, and, even better, I get recommendations to use POLL or EPOLL (which anyone who has ever straced has seen a ton of). I ask it how to determine FD_SETSIZE, discover it's 1024 on the client, which matches our low socket count, confirm that we should never increase the size of FD_SETSIZE, check the vendor code, see they've got it hard configured to talk with select() instead of poll - we recommend they give us a new build with poll() defined - and voila - problem goes away. On to the next issue. Where Chat excels is not in solving or completing anything, or, in fact, even being correct 100% of the time (it clearly isn't) - but it's an endlessly enthusiastic savant intern - frequently wrong in hilarious ways, but always willing to dig in and pull on technical threads and come up with things that I can try with a bunch of rapid iterations, and close off an issue. It's willingness to write code that is 90% correct just reduces the time, and cognitive load of constantly having to do it all yourself.