13 ms·
It's always TCP_NODELAY
- theamk 3y agoI don't by the reasoning for never needing Nagle anymore. Sure, telnet isn't a thing today, but I bet there are still plenty of apps which do equivalent of: write(fd, "Host: ") write(fd, hostname) write(fd, "\r\n") write(fd, "Content-type: ") etc... this may not be 40x overhead, but it'd still 5x or so.
- Arnt 3y agoThose aren't the ones you debug, so they won't be seen by OP. Those are the ones you don't need to debug because Nagle saves you.
- otterley 3y agoMarc addresses that: “That’s going to make some “write every byte” code slower than it would otherwise be, but those applications should be fixed anyway if we care about efficiency.”
- rwmj 3y agoThe comment about telnet had me wondering what openssh does, and it sets TCP_NODELAY on every connection, even for interactive sessions. (Confirmed by both reading the code and observing behaviour in 'strace').
- c0l0 3y agoEspecially for interactive sessions, it absolutely should! :)
- syncsynchalt 3y agoIronic since Nagle's Algorithm (which TCP_NODELAY disables) was invented for interactive sessions. It's hard to imagine interactive sessions making more than the tiniest of blips on a modern network.
- eru 3y agoIsn't video calling an interactive session?
- semi 3y agoI think that's more two independent byte streams. You want low latency but what is transfered doesnt really impact the other side, you just constantly want to push the next frame
- eru 3y agoThanks, that makes sense! It's interesting that it's very much an interactive experience for the end-user. But for the logic of the computer, it's not interactive at all. You can make the contrast even stronger: if both video streams are transmitted over UDP, you don't even need to sent ACKs etc. To be truly one-directional from a technical point of view. Then compare that to transferring a file via TCP. For the user this is as one-directional and non-interactive as it gets, but the computers constantly talk back and forth.
- saagarjha 3y agoYou don't want ACKs on every frame but video streaming can be made more efficient if you do checkpoints (which informs the encoder that the other side has some data, which means it can compress more efficiently).
- TorKlingberg 3y agoVideo calls indeed almost always use UDP. TCP retransmission isn't really useful since by the time a retransmitted packet arrives it's too old to display. Worse, a single lost packet will block a TCP stream. Sometimes TCP is the only way to get through a firewall, but the experience is bad if there's any packet loss at all. VC systems do constantly send back packet loss statistics and adjust the video quality to avoid saturating a link. Any buffering in routers along the way will add delay, so you want to keep the bitrate low enough to keep buffers empty.
- 3y ago
- temac 3y agoFix the apps. Nobody expect magical perf if you do that when writing to files, even though the OS also has its own buffers. There is no reason to expect otherwise when writing to a socket and actually nagle already doesn't save you from syscall overhead.
- toast0 3y agoNagle doesn't save the derpy side from syscall overhead, but it would save the other side. It's not just apps doing this stuff, it also lives in system libraries. I'm still mad at the Android HTTPS library for sending chunked uploads as so many tinygrams. I don't remember exactly, but I think it's reasonable packetization for the data chunk (if it picked a reasonable size anyway), then one packet for \r\n, one for the size, and another for another \r\n. There's no reason for that, but it doesn't hurt the client enough that I can convince them to avoid the system library so they can fix it and the server can manage more throughput. Ugh. (It might be that it's just the TLS packetization that was this bogus and the TCP packetization was fine, it's been a while) If you take a pcap for some specific issue, there's always so many of these other terrible things in there. </rant>
- meinersbur 3y agoThose are the apps are quickly written and do not care if they unnecessarily congest the network. The ones that do get properly maintained can set TCP_NODELAY. Seems like a reasonable default to me.
- ale42 3y agoApps can always misbehave, you never know what people implement, and you don't always have source code to patch. I don't think the role of the OS is to let the apps do whatever they wish, but it should give the possibility of doing it if it's needed. So I'd rather say, if you know you're properly doing things and you're latency sensitive, just TCP_NODELAY on all your sockets and you're fine, and nobody will blame you about doing it.
- bjourne 3y ago> Fix the apps. Nobody expect magical perf if you do that when writing to files, We write to files line-by-line or even character-by-character and expect the library or OS to "magically" buffer it into fast file writes. Same with memory. We expect multiple small mallocs to be smartly coalesced by the platform.
- jrockway 3y agoDoes this matter? Yes, there's a lot of waste. But you also have a 1Gbps link. Every second that you don't use the full 1Gbps is also waste, right?
- tedunangst 3y agoThis is why I always pad out the end of my html files with a megabyte of . A half empty pipe is a half wasted pipe.
- dessimus 3y agoJust be sure HTTP Compression is off though, or you're still half-wasting the pipe. Better to just dump randomized uncompressible data into html comments.
- arp242 3y agoI am finally starting to understand some of these OpenOffice/LibreOffice commit messages like https://github.com/LibreOffice/core/commit/a0b6744d3d77 https://github.com/LibreOffice/core/commit/a0b6744d3d77
- jrockway 3y agoI think that's an unfair comparison. By using Nagle's algorithm for interactive work, you save bytes, but the software you're interacting with is that much less responsive. (If the client was responsible for echoing typed characters, then it wouldn't matter. But ssh and telnet don't work like that, unfortunately.) So by saving bytes and leaving your pipe empty, you just suffer in user experience. Why not use something you're already paying for to make your life better? (In the end, it seems like SSH agrees with me, and just wastes the bytes by enabling TCP_NODELAY.)
- grishka 3y agoAnd they really shouldn't do this. Even disregarding the network aspect of it, this is still bad for performance because syscalls are kinda expensive.
- eatonphil 3y agoI imagine the write calls show up pretty easily as a bottleneck in a flamegraph.
- wbl 3y agoThey don't. Maybe if you're really good you notice the higher overhead but you expect to be spending time writing to the network. The actual impact shows up when the bandwidth consumption is way up on packet and TCP headers which won't show on a flamegraph that easily.
- silisili 3y agoWe shouldn't penalize the internet at large because some developers write terrible code.
- littlestymaar 3y agoIsn't it how SMTP is working though?
- leni536 3y agoNo?
- loopdoend 3y agoAh yeah I fixed this exact bug in net-http in Ruby core a decade ago.
- tptacek 3y agoThe discussion here mostly seems to miss the point. The argument is to change the default, not to eliminate the behavior altogether.
- the8472 3y agoshouldn't autocorking help with even without nagle?
- asveikau 3y agoI don't think that's actually super common anymore when you consider that doing asynchronous I/O, the only sane way to do that is put it into a buffer rather than blocking at every small write(2). Then you consider that asynchronous I/O is usually necessary both on server (otherwise you don't scale well) and client (because blocking on network calls is terrible experience, especially in today's world of frequent network changes, falling out of network range, etc.)
- sophacles 3y agoTCP_CORK handles this better than nagle tho.
- jabl 3y agoEven if you do nothing 'fancy' like Nagle, corking, or userspace building up the complete buffer before writing etc., at the very least the above should be using a vectored write (writev() ).
- Too 3y agoShouldn’t that go through some buffer? Unless you fflush() between each write?
- mannyv 3y agoWe used to call them "packlets." His "tinygrams" is pretty good too, but that sort of implies UDP (D -> datagrams)
- chuckadams 3y ago> We used to call them "packlets." setsockopt(fd, IPPROTO_TCP, TCP_MAKE_IT_GO, &go, sizeof(go));
- obelos 3y agoNot every time. Sometimes it's DNS.
- marcosdumay 3y agoWhen it fails, it's DNS. When it just stops moving, it's either TCP_NODELAY or stream buffering. Really complex systems (the Web) also fail because of caching.
- jeffrallen 3y agoOnce every 50 years and 2 billion kilometers, it's a failing memory chip. But you can usually just patch around them, so no big deal.
- skunkworker 3y agoDon’t forget BGP or running out of disk space without an alert.
- p_l 3y agoOnce it was a failing line card in router zeroing last bit in IPv4 addresses, resulting in ticket about "only even IPv4 addresses are accessible" ...
- jcgrillo 3y agoFor some reason this reminded me of the "500mi email" bug [1], maybe a similar level of initial apparent absurdity? [1] https://www.ibiblio.org/harris/500milemail.html https://www.ibiblio.org/harris/500milemail.html
- chuckadams 3y agoThe most absurd thing to me about the 500 mile email situation is that sendmail just happily started up and soldiered on after being given a completely alien config file. Could be read as another example of "be liberal in what you accept" going awry, but sendmail's wretched config format is really a volume of war stories all its own...
- batmanthehorse 3y agoDoes anyone know of a good way to enable TCP_NODELAY on sockets when you don't have access to the source for that application? I can't find any kernel settings to make it permanent, or commands to change it after the fact. I've been able to disable delayed acks using `quickack 1` in the routing table, but it seems particularly hard to enable TCP_NODELAY from outside the application. I've been having exactly the problem described here lately, when communicating between an application I own and a closed source application it interacts with.
- coldpie 3y agoWould some kind of LD_PRELOAD interception for socket(2) work? Call the real function, then do setsockopt or whatever, and return the modified socket.
- cesarb 3y ago> Would some kind of LD_PRELOAD interception for socket(2) work? That would only work if the call goes through libc, and it's not statically linked. However, it's becoming more and more common to do system calls directly, bypassing libc; the Go language is infamous for doing that, but there's also things like the rustix crate for Rust (https://crates.io/crates/rustix https://crates.io/crates/rustix), which does direct system calls by default.
- zbowling 3y agoAnd go is wrong for doing that, at least on Linux. It bypasses optimizations in the vDSO in some cases. On Fuchsia, we made direct syscalls not through the vDSO illegal and it was funny the hacks to go that required. The system ABI of Linux really isn't the syscall interface, its the system libc. That's because the C ABI (and the behaviors of the triple it was compiled for) and its isms for that platform are the linga franca of that system. Going around that to call syscalls directly, at least for the 90% of useful syscalls on the system that are wrapped by libc, is asinine and creates odd bugs, makes crash reporters heuristical unwinders, debuggers, etc all more painful to write. It also prevents the system vendor from implementing user mode optimizations that avoid mode and context switches when necessary. We tried to solve these issues in Fuchsia, but for Linux, Darwin, and hell, even Windows, if you are making direct syscalls and it's not for something really special and bespoke, you are just flat-out wrong.
- mirekrusin 3y agoCan't it have "if payload is 1 byte (or less than X) then wait, otherwise don't" condition?
- chuckadams 3y agoSome network stacks like those in Solaris and HP/UX let you tune the "Nagle limit" in just such a fashion, up to disabling it entirely by setting it to 1. I'm not aware of it being tunable on Linux, though you can manually control the buffering using TCP_CORK. https://baus.net/on-tcp_cork/ https://baus.net/on-tcp_cork/ has some nice details.
- fweimer 3y agoThere is a socket option, SO_SNDLOWAT. It's not implement Linux according to the manual page. The description in UNIX Network Programming and TCP Illustrated conflict, too. So it's probably not useful.
- the8472 3y agoYou can buffer in userspace. Don't do small writes to the socket and no bytes will be sent. Don't do two consecutive small writes and nagle won't kick in.
- astrange 3y agoFreeBSD has accept filters, which let you do something like wait for a complete HTTP header (inaccurate from memory summary.) Not sure about the sending side.
- deathanatos 3y agoHow is what you're describing not just Nagle's algorithm? If you mean TCP_NODELAY, you should use it with TCP_CORK, which prevents partial frames. TCP_CORK the socket, do your writes to the kernel via send, and then once you have an application level "message" ready to send out — i.e., once you're at the point where you're going to go to sleep and wait for the other end to respond, unset TCP_CORK & then go back to your event loop & sleep. The "uncork" at the end + nodelay sends the final partial frame, if there is one.
- elhosots 3y agoThis sounds like the root of my vncviewer / server interaction bugs i experience with some vnc viewer/server combo’s between ubuntu linux and freebsd… (tight/tiger)
- evanelias 3y agoJohn Nagle has posted insightful comments about the historical background for this many times, for example https://news.ycombinator.com/item?id=9048947 https://news.ycombinator.com/item?id=9048947 referenced in the article. He's a prolific HN commenter (#11 on the leaderboard) so it can be hard to find everything, but some more comments searchable via https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=%22delayed%20ack%22%20%22animats%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... or https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=%22tcp_nodelay%22%20%22animats%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- Animats 3y agoThe sending pattern matters. Send/Receive/Send/Receive won't trigger the problem, because the request will go out immediately and the reply will provide an ACK and allow another request. Bulk transfers won't cause the problem, because if you fill the outgoing block size, there's no delay. But Send/Send/Receive will. This comes up a lot in game systems, where most of the traffic is small events going one way.
- EvanAnderson 3y agoI love it when Nagle's algorithm comes up on HN. Inevitably someone, not knowing "Animats" is John Nagle, responds a comment from Animats with a "knowing better" tone. >smile< (I also really like Animats' comments, too.)
- pclmulqdq 3y agoIn a world where bandwidth was limited, and the packet size minimum was 64 bytes plus an inter-frame gap (it still is for most Ethernet networks), sending a TCP packet for literally every byte wasted a huge amount of bandwidth. The same goes for sending empty acks. On the other hand, my general position is: it's not TCP_NODELAY, it's TCP.
- metadaemon 3y agoI'd just love a protocol that has a built in mechanism for realizing the other side of the pipe disconnected for any reason.
- koverstreet 3y agoLike TCP keepalives?
- mort96 3y agoIf the feature already technically exists in TCP, it's either broken or disabled by default, which is pretty much the same as not having it.
- voxic11 3y agokeepalives are an optional TCP feature so they are not necessarily supported by all TCP implementations and therefor default to off even when supported.
- dilyevsky 3y agoWhere is it off? Most linux distros have it on it’s just the default kickoff timer is ridiculously long (like 2 hours iirc). Besides, TCP keepalives won't help with the issue at hand and were put in for totally different purpose (gc'ing idle connections). Most of the time you don't even need them because the other side will send RST packet if it already closed the socket.
- zengid 3y agoRelevant Oxide and Friends podcast episode https://www.youtube.com/watch?v=mqvVmYhclAg https://www.youtube.com/watch?v=mqvVmYhclAg
- matthavener 3y agoThis was a great episode and the really drove home the importance of visualization.
- rsc 3y agoNot if you use a modern language that enables TCP_NODELAY by default, like Go. :-)
- andrewfromx 3y agohttps://news.ycombinator.com/item?id=34179426 https://news.ycombinator.com/item?id=34179426 https://github.com/golang/go/issues/57530 https://github.com/golang/go/issues/57530 huh, TIL.
- silverwind 3y agoNode.js also does this since at least 2020.
- Sammi 3y agoSince 2022 v.18. PR: https://github.com/nodejs/node/pull/42163 https://github.com/nodejs/node/pull/42163 Changelog entry: https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V18.md#other-notable-changes-7 https://github.com/nodejs/node/blob/main/doc/changelogs/CHAN...
- eru 3y agoWhy do you need a whole language for that? Couldn't you just use a 'modern' networking library?
- rsc 3y agoSure, like the one in https://9fans.github.io/plan9port/ https://9fans.github.io/plan9port/. :-)
- ironman1478 3y agoI've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a setting that people might not know about that manipulates how data is sent over the wire so aggressively.
- hinkley 3y agoWhat you really want is for the delay to be n microseconds, but there’s no good way to do that except putting your own user space buffering in front of the system calls (user space works better, unless you have something like io_uring amortizing system call times)
- bobmcnamara 3y agoI'd rather have portable TCP_CORK
- hinkley 3y agoCork is probably how you’d implement this in userspace so why not both?
- mjevans 3y agoIt'd probably be amazing how many poorly coded games would work better if something like... TCP_60FPSBUFFER Would wait for ~16mS after the first packet is queued and batch the data stream up.
- JoshTriplett 3y agoI do wish that TCP_NODELAY was the default, and there was a TCP_DELAY option instead. That'd be a world in which people who want the batch-style behavior (optimizing for throughput and fewer packets at the expense of latency) could still opt into it.
- mzs 3y agoSo do I, but I with there was a new one TCP_RTTDELAY. It would take a byte that would be what 128th of RTT you want to use for Nagle instead of one RTT or full* buffer. 0 would be the default, behaving as you and I prefer. * "Given the vast amount of work a modern server can do in even a few hundred microseconds, delaying sending data for even one RTT isn’t clearly a win." I don't think that's such an issue anymore either, given that the server produces so much data it fills the output buffer quickly anyway, the data is then immediately sent before the delay runs its course.
- stonemetal12 3y ago>To make a clearer case, let’s turn back to the justification behind Nagle’s algorithm: amortizing the cost of headers and avoiding that 40x overhead on single-byte packets. But does anybody send single byte packets anymore? That is a bit of a strawman there. While he uses single byte packets as the worst case example, the issue as stated is any not full packet.
- somat 3y agoWhat about the opposite, disable delayed acks. The problem is the pathological behavior when tinygram prevention interacts with delayed acks. There is an exposed option to turn off tinygram prevention(TCP_NODELAY), how would you tun off delayed acks instead? Say if you wanted to benchmark all four combinations and see what works best. doing a little research I found: linux has the TCP_QUICKACK socket option but you have to set it every time you receive. there is also /proc/sys/net/ipv4/tcp_delack_min and /proc/sys/net/ipv4/tcp_ato_min freebsd has net.inet.tcp.delayed_ack and net.inet.tcp.delacktime
- Animats 3y ago> linux has the TCP_QUICKACK socket option but you have to set it every time you receive Right. What were they thinking? Why would you want it off only some of the time?
- batmanthehorse 3y agoIn CentOS/RedHat you can add `quickack 1` to the end of a route to tell it to disable delayed acks for that route.
- rbjorklin 3y agoAnd with systemd >= 253 you can set it as part of the network config to have it be applied automatically. https://github.com/systemd/systemd/issues/25906 https://github.com/systemd/systemd/issues/25906
- mjb 3y agoTCP_QUICKACK does fix the worst version of the problem, but doesn't fix the entire problem. Nagles algorithm will still wait for up to one round-trip time before sending data (at least as specified in the RFC), which is extra latency with nearly no added value.
- Culonavirus 3y agoApparently you have time to "do a little research" but not to read the entire article you're reacting to? It specifically mentions TCP_QUICKACK.
- benreesman 3y agoNagle and no delay are like like 90+% of the latency bugs I’ve dealt with. Two reasonable ideas that mix terribly in practice.
- tempaskhn 3y agoWow, never would have thought of that.
- jedberg 3y agoThis is an interesting thing that points out why abstraction layers can be bad without proper message passing mechanisms. This could be fixed if there was a way for the application at L7 to tell the TCP stack at L4 "hey, I'm an interactive shell so I expect to have a lot of tiny packets, you should leave TCP_NODELAY on for these packets" so that it can be off by default but on for that application to reduce overhead. Of course nowadays it's probably an unnecessary optimization anyway, but back in '84 it would have been super handy.
- Dylan16807 3y ago"I'm an interactive shell so I expect to have a lot of tiny packets" is what the delay is for. If you want to turn it off for those, you should turn it off for everything. (If you're worried about programs that buffer badly, then you could compensate with a 1ms delay. But not this round trip stuff.)
- deleted 3y ago[deleted]
- eru 3y agoThe take-away I get is that abstraction layers (in the kernel) can be bad. Operating system kernels should enable secure multiplexing of resources. Abstraction and portability should be done via libraries. See https://en.wikipedia.org/wiki/Exokernel https://en.wikipedia.org/wiki/Exokernel
- landswipe 3y agoUse UDP ;)
- gafferongames 3y agoBingo
- hi-v-rocknroll 3y agoToo many applications end up reinventing TCP or SCTP in user-space. Also, network-level QoS applied to unrecognized UDP protocols typically means it gets throttled before TCP. Use UDP when nothing else will work, when the use-case doesn't need a persistent connection, and when no other messaging or transport library is suitable.
- epolanski 3y agoI was curious whether I had to change anything in my applications after reading that so did a bit of research. Both Node.js and Curl use TCP_NODELAY by default from a long time.
- Sammi 3y agoNodejs enabled TCP_NODELAY by default in 2022 v.18. PR: https://github.com/nodejs/node/pull/42163 https://github.com/nodejs/node/pull/42163 Changelog entry: https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V18.md#other-notable-changes-7 https://github.com/nodejs/node/blob/main/doc/changelogs/CHAN...
- epolanski 3y agoThat's the HTTP module, but it was already moved to NODELAY for the 'net' module in 2015.
- gafferongames 3y agoIt's always TCP_NODELAY. Except when it's head of line blocking, then it's not.
- kaoD 3y agoAs a counterpoint, here's the story of how for me it wasn't TCP_NODELAY: for some reason my Nodejs TCP service was talking a few seconds to reply to my requests in localhost (Windows machine). After the connection was established everything was pretty normal but it consistently took a few seconds to establish the connection. I even downloaded netcat for Windows to go as bare ones as possible... and the exact same thing happened. I rewrote a POC service in Rust and... oh wow, the same thing happens. It took me a very long time of not finding anything on the internet (and getting yelled at in Stack Overflow, or rather one of its sister sites) and painstakingly debugging (including writing my own tiny client with tons of debug statements) until I realized "localhost" was resolving first to IPv6 loopback in Windows and, only after quietly timing out there (because I was only listening on IPv4 loopback), it did try and instantly connect through IPv4.
- littlestymaar 3y agoI've seen this too, but luckily someone one the internet gave me a pointer to the exact problem so I didn't have to go deep to figure out.
- pandemicsyn 3y agoI was gonna say its always lro offload but my experience is dated.
- 0xbadcafebee 3y agoThe takeaway is odd. Clearly Nagle's Algorithm was an attempt at batched writes. It doesn't matter what your hardware or network or application or use-case or anything is; in some cases, batched writes are better. Lots of computing today uses batched writes. Network applications benefit from it too. Newer higher-level protocols like QUIC do batching of writes, effectively moving all of TCP's independent connection and error handling into userspace, so the protocol can move as much data into the application as fast as it can, and let the application (rather than a host tcp/ip stack, router, etc) worry about the connection and error handling of individual streams. Once our networks become saturated the way they were in the old days, Nagle's algorithm will return in the form of a QUIC modification, probably deeper in the application code, to wait to send a QUIC packet until some criteria is reached. Everything in technology is re-invented once either hardware or software reaches a bottleneck (and they always will as their capabilities don't grow at the same rate). (the other case besides bandwidth where Nagle's algorithm is useful is if you're saturating Packets Per Second (PPS) from tiny packets)
- Spivak 3y agoYes but it seems this particular implementation is using a heuristic for how to batch that made some assumptions that didn't pan out.
- p_l 3y agoThe difference between QUIC and TCP is the original sin of TCP (and its predecessor) - that of emulating an async serial port connection, with no visible messaging layer. It meant that you could use a physical teletypewriter to connect to services (simplified description - slap a modem on a serial port, dial into a TIP, write host address and port number, voila), but it also means that TCP has no idea of message boundaries, and while you can push some of that knowledge now the early software didn't. In comparison, QUIC and many other non-TCP protocols (SCTP, TP4) explicitly provide for messaging boundaries - your interface to the system isn't based on emulated serial ports but on messages that might at most get reassembled.
- utensil4778 3y ago
- ryjo 3y agoI just ran into this this week implementing a socket library in CLIPS. I used Berkley sockets, and before that I had only worked with higher-level languages/frameworks that abstracts a lot of these concerns away. I was quite confused when Firefox would show a "connection reset by peer." It didn't occur to me it could be an issue "lower" in the stack. `tcpdump` helped me to observe the port and I saw that the server never sent anything before my application closed the connection.
- AtNightWeCode 3y agoAgreed. Another thing along the same path is expect. Needs to be disabled in many cloud services.
- meisel 3y agoIs this something I should also adjust on my personal Ubuntu machine for better network performance?
- projectileboy 3y agoThe real issue in modern data centers is TCP. Of course at present, we need to know about these little annoyances at the application layer, but what we really need is innovation in the data center at level 4. And yes I know that many people are looking into this and have been for years, but the economic motivation clearly has not yet been strong enough. But that may change if the public's appetite for LLM-based tooling causes data centers to increase 10x (which seems likely).
- maple3142 3y agoNot sure if this is a bit off topic or not, but I recently encountered a problem where my program are continuously calling write to a socket in a loop that loops N times, with each of them sending about a few hundred bytes of data representing an application-level message. The loop can be understanded as some "batched messages" to server. After that, the program will try to receive data from server and do some processing. The problem is that if N is above certain limit (e.g. 4), the server will resulting in some error saying that the data is truncated somehow. I want to make N larger because the round-trip latency is already high enough, so being blocked by this is pretty annoying. Eventually, I found an answer on stackoverflow saying that setting TCP_NODELAY can fix this, and it actually magically enable be to increase N to a larger number like 64 or 128 without causing issues. Still not sure why TCP_NODELAY can fix this issue and why this problem happens in the first place.
- blahgeek 3y ago> The problem is that if N is above certain limit (e.g. 4), the server will resulting in some error saying that the data is truncated somehow. Maybe your server expects full application-level messages from single "recv" call? This is is not correct. A message may be spited across multiple recv buffers.
- gizmo686 3y agoMy guess would be that the server assumes that every call to recv() terminates on a message boundary. With TCP_NODELAY and small messages, this works out fine. Every message is contained in a single packet, and the userspace buffer being read into is large enough to contain it. As such, whenever the kernel has any data to give to userspace, it has an integer number of messages to give. Nothing requires the kernel to respect that, but it will not go out of its way to break it. In contrast, without TCP_NODELAY, messages get concatenated and then fragmented based on where packet boundaries occur. Now, the natural end point for a call to recv() is not the message boundary, but the packet boundary. The server is supposed to see that it is in the middle of a message, and make another call to recv() to get the rest of it; but clearly it does not do that.
- 3y ago
- deleted 3y ago[deleted]
- ramblemonkey 3y agoWhat if we changed the kernel or tcp stack to hold on to the packet for only a short time before sending it out. This could allow you to balance the latency against the network cost of many small packets. The tcp stack could even do it dynamically if needed.
- tucnak 3y agoGenius
- Ono-Sendai 3y agoFrom my blog > 10 years ago but sadly still relevant: "Sockets should have a flushHint() API call.": https://forwardscattering.org/post/3 https://forwardscattering.org/post/3
- hi-v-rocknroll 3y agoApropos repost from 2015: > That still irks me. The real problem is not tinygram prevention. It's ACK delays, and that stupid fixed timer. They both went into TCP around the same time, but independently. I did tinygram prevention (the Nagle algorithm) and Berkeley did delayed ACKs, both in the early 1980s. The combination of the two is awful. Unfortunately by the time I found about delayed ACKs, I had changed jobs, was out of networking, and doing a product for Autodesk on non-networked PCs. > Delayed ACKs are a win only in certain circumstances - mostly character echo for Telnet. (When Berkeley installed delayed ACKs, they were doing a lot of Telnet from terminal concentrators in student terminal rooms to host VAX machines doing the work. For that particular situation, it made sense.) The delayed ACK timer is scaled to expected human response time. A delayed ACK is a bet that the other end will reply to what you just sent almost immediately. Except for some RPC protocols, this is unlikely. So the ACK delay mechanism loses the bet, over and over, delaying the ACK, waiting for a packet on which the ACK can be piggybacked, not getting it, and then sending the ACK, delayed. There's nothing in TCP to automatically turn this off. However, Linux (and I think Windows) now have a TCP_QUICKACK socket option. Turn that on unless you have a very unusual application. > Turning on TCP_NODELAY has similar effects, but can make throughput worse for small writes. If you write a loop which sends just a few bytes (worst case, one byte) to a socket with "write()", and the Nagle algorithm is disabled with TCP_NODELAY, each write becomes one IP packet. This increases traffic by a factor of 40, with IP and TCP headers for each payload. Tinygram prevention won't let you send a second packet if you have one in flight, unless you have enough data to fill the maximum sized packet. It accumulates bytes for one round trip time, then sends everything in the queue. That's almost always what you > want. If you have TCP_NODELAY set, you need to be much more aware of buffering and flushing issues. > None of this matters for bulk one-way transfers, which is most HTTP today. (I've never looked at the impact of this on the SSL handshake, where it might matter.) > Short version: set TCP_QUICKACK. If you find a case where that makes things worse, let me know. > John Nagle (2015) https://news.ycombinator.com/item?id=10608356 https://news.ycombinator.com/item?id=10608356 --- Support platform survey: TCP_QUICKACK: Linux (must set again every every recv()) TCP_NODELAY: Linux, Apple, Windows, Solaris, FreeBSD, OpenBSD, and NetBSD References: https://www.man7.org/linux/man-pages/man7/tcp.7.html https://www.man7.org/linux/man-pages/man7/tcp.7.html https://opensource.apple.com/source/xnu/xnu-1504.9.17/bsd/netinet/tcp.h.auto.html https://opensource.apple.com/source/xnu/xnu-1504.9.17/bsd/ne... https://learn.microsoft.com/en-us/windows/win32/winsock/ipproto-tcp-socket-options https://learn.microsoft.com/en-us/windows/win32/winsock/ippr... https://docs.oracle.com/cd/E88353_01/html/E37851/esc-tcp-4p.html https://docs.oracle.com/cd/E88353_01/html/E37851/esc-tcp-4p.... https://man.freebsd.org/cgi/man.cgi?query=tcp https://man.freebsd.org/cgi/man.cgi?query=tcp https://man.openbsd.org/tcp https://man.openbsd.org/tcp https://man.netbsd.org/NetBSD-8.0/tcp.4 https://man.netbsd.org/NetBSD-8.0/tcp.4
- resonious 3y ago~15 years ago I played an MMO that was very real-time, and yet all of the communication was TCP. Literally you'd click a button, and you would not even see your action play out until a response packet came back. All of the kids playing this game (me included) eventually figured out you could turn on TCP_NODELAY to make the game buttery smooth - especially for those in California close to the game servers.
- jonathanlydall 3y agoNot sure if you're talking about WoW, but around that time ago an update to the game did exactly this change (and possibly more). An interesting side-effect of this was that before the change if something stalled the TCP stream, the game would hang for a while then very quickly replay all the missed incoming events (which was very often you being killed). After the change you'd instead just be disconnected.
- mst 3y agoI think I have a very vague memory of the "hang, hang, hang, SURPRISE! You're dead" thing happening in Diablo II but it's been so long I wouldn't bet on having remembered correctly.
- spintin 3y ago[dead]
- trollied 3y agoBrings back memories. This was a big Sybase performance win back in the day.
- f1shy 3y agoIf you are sooo worried about latency, maybe TCP is a bad choice to start with… I hate to see people using TCP for all, without the minimal understanding of what problems TCP wants to solve, and specially which dont.
- pcai 3y agoTCP solves for “when i send a message i want the other side to actually receive it” which is…fairly common
- adgjlsfhk1 3y agotcp enforces a much stricter ordering than desirable (head of line blocking). quic does a much better job of emulating a stream of independent tasks.
- kazinator 3y agoHere is the thing. Nagle's and the delayed ACK may suck for individual app performance, but fewer packets on the network is better for the entire network.
- hartvile02 3y ago[flagged]
- hartvile02 3y ago[flagged]
- sciencesama 3y agoFor folks looking to disable in windows this is how it is done fyi !! https://support.microsoft.com/en-us/topic/fix-tcp-ip-nagle-algorithm-for-microsoft-message-queue-server-can-be-disabled-74ba2f6a-e558-d1df-1c60-57b0fab68ccc https://support.microsoft.com/en-us/topic/fix-tcp-ip-nagle-a...