11 ms·
40 Milliseconds of latency that just would not go away
- rsclient 6y agoThis is exactly why the Socket API in WinRT has Nagle off by default. The old way of dealing with sockets was to treat them like buffered files, or to drive them from a keyboard (so that Nagle is useful). But newer socket programs seem to just make a full chunk of information, and send it at once. Those newer programs either turn off Nagle, or would be improved if they did. So we bit the bullet, and decided to make Nagle off by default.
- ohazi 6y agoYou don't need to do it this way, though... the general rule is that you shouldn't enable both Nagle's Algorithm and TCP delayed ACK at the same time.
- nly 6y agoThat's easy to say but the bad interaction happens when the algorithms operate at opposite ends of the connection, and you often don't control both ends. The bottom line is, you need to understand the semantics of your application protocol to best know how to apply them.
- deafcalculus 6y agoWas delayed ACKs the problem? Disabling delayed ACKs seems like a better bet than using TCP_NODELAY which turns off Nagle's algorithm.
- ganzuul 6y agoThe article is refreshingly short and to the point. Unusually, it does not aim the waste the maximum amount of your time so why don't you read it.
- deafcalculus 6y agoI liked the article, but it's not entirely clear to me what the cause of the problem was. Linux delays sending out data only if the data size is less than the packet size AND the previous packet in not yet ACKed (Nagle's algorithm). My guess was that this app is doing a write, write, read and hitting the delayed ACK problem.
- Animats 6y agoI generally say yes. The fixed timer is for the delayed ACK. That was a terrible idea. Both Linux and Windows now have a way to turn delayed ACKs off, but they're still on by default. TCP_QUICKACK, which turns off delayed ACKs, is in Linux, but the manual page is very confused about what it actually does. Apparently it turns itself off after a while. I wish someone would get that right. I'd disable delayed ACKs by default. It's hard to think of a case today where they're a significant win. As I've written in the past, delayed ACKs were a hack to make remote Telnet character echo work better. A key point is asymmetry. If you're the one who's doing lots of little writes, you can either turn set TCP_NODELAY at your end, or turn off delayed ACKs at the other end. If you can. Things doing lots of little writes but not filling up the pipe, typically game clients, can't change the settings at the other end. So it became a standard practice to do what you could do at your end.
- deadmutex 6y agoYou should sign your post with your name (in this topic), in case a lot of people are not familiar with who you are :).
- twsted 6y agoOn macOS add the line net.inet.tcp.delayed_ack=0 to the file /etc/sysctl.conf (which doesn't initially exist).
- JoeAltmaier 6y agoSome linux settings apply to every TCP connection which can be a very surprising result and hard to debug. I can see someone wanting to time out a feature, so it doesn't upset the rest of the system forever.
- lilyball 6y agoThe problem with delayed ACKs is that requires controlling the server. If you control the client, you can’t remotely turn off delayed ACKs, so instead you have to disable Nagle’s algorithm.
- deleted 6y ago[deleted]
- rgossiaux 6y agoAs soon as I saw the title I remembered this Julia Evans post about the same issue: https://jvns.ca/blog/2015/11/21/why-you-should-understand-a-little-about-tcp/ https://jvns.ca/blog/2015/11/21/why-you-should-understand-a-...
- punnerud 6y agoFrom John Nagle: "(..) Unfortunately, delayed ACKs went in after I got out of networking in 1986, and this was never fixed. Now it's too late." https://stackoverflow.com/a/16663206/2326672 https://stackoverflow.com/a/16663206/2326672
- oefrha 6y agoSee also this thread straight from the horse’s mouth, with some additional technical considerations: https://news.ycombinator.com/item?id=9048947 https://news.ycombinator.com/item?id=9048947
- punnerud 6y agoNot topic related, but John’s transfer from networking to Autodesk got me interested. I don’t immediately see the good connection between the two, when they where a startup. Would love to see a blog post from John Nagel on this, now that we know the outcome of Autodesk. (I see he is active on HN almost every day)
- ignoramous 6y agoSee also another amusing exchange: https://news.ycombinator.com/item?id=21088736 https://news.ycombinator.com/item?id=21088736
- mercer 6y agoHa, I've been here for years and never knew that Animats was John Nagle. cool!
- JdeBP 6y agoThe Stack Overflow answer was "straight from the horse's mouth" too, doubters here at https://news.ycombinator.com/item?id=24799256 https://news.ycombinator.com/item?id=24799256 notwithstanding. (-:
- pedrocr 6y agoSo the kernel is using a static decision that's really bad sometimes? Would it be too expensive to treat this like a branch predictor and keep some state to have the kernel enable/disable the delayed ACK dynamically depending on how it has won/lost the bet recently?
- jorangreef 6y agoI was dealing with "Nagle's delay" only yesterday, adding "setsockopt(TCP_NODELAY)" for the alpha version of a new high-performance payments database called TigerBeetle: https://github.com/coilhq/tiger-beetle https://github.com/coilhq/tiger-beetle
- coddle-hark 6y agoHey, cool, I was actually just using TCP myself working on a new high-performance beetle database called TigerPayments: https://github.com/coddlehark/tigerpayments https://github.com/coddlehark/tigerpayments Comments like this are just spam and don’t contribute anything to the discussion. If you have some interesting insights, share them, but don’t shift the discussion to your own completely unrelated project.
- dang 6y agoI appreciate your positive intention of watching out for the quality of the threads. But please don't do it this way. Instead, please follow the site guidelines and be kind. HN threads are conversations, and it's natural in a conversation for someone to connect a topic to something they were working on recently. That's not shifting discussion to something unrelated. I can see how one could interpret it just as an excuse for spamming one's project, but such an interpretation is just a guess, and if you apply the site guideline "Assume good faith", you'd guess the opposite. It's clear from https://news.ycombinator.com/item?id=24801075 https://news.ycombinator.com/item?id=24801075 that the good-faith guess is the true one. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html, we'd be grateful.
- coddle-hark 6y agoYou’re right, I’m sorry.
- dang 6y agoAppreciated!
- acvny 6y agoNot clear what is this about. What delay?
- doubletgl 6y agoThe network kind.
- 2rsf 6y agoI was challenged with similar problems when doing performance testing trying to determine our new wireless router limits only to find out they are far too low. > you can’t fix TCP problems without understanding TCP True, but many problems are not TCP problems and it is not always easy to determine where does your delay is coming from
- euph0ria 6y agoWhy not just use tcpdump or wireshark when troubleshooting network latencies? Usually only takes a minute or two to pinpoint the issue. Then you would need to spend time understanding why the pinpointed behavior is what it is and sometimes it is in the application, sometimes not.. I've solved so many issues over the years with tcpdump that it has become one of the most valuable tools I know.
- LeonM 6y agoOut of curiosity, how would you debug this issue with Wireshark? Just look for multiple messages in a single TCP packet? Or is there a better way?
- deleted 6y ago[deleted]
- pwinnski 6y agoRight? If seeing multiple messages in a single TCP packet immediately clues you in to the issue, the truth is that you already knew about the issue.
- marcosdumay 6y agoNo kidding. I imagined Google or DDG would turn "multiple messages in a single TCP package" into something useful, since it's basically the description of the algorithm. But no luck. The best I got was somebody with an IO buffering problem in Stack Overflow that commented that he turned the algorithm off.
- bostik 6y agoIf you're debugging weird latency problems, I'd recommend to start from relative timestamps. Then filter to just one or two TCP streams that exhibit the problem and go from there. Inspect packet contents. The packet dissectors are doing a lot of heavy lifting, so look at the deconstructed data. I realise this may not be a welcome suggestion, but a sharp pencil, an A3 (or larger) scratch pad and a good ruler go a long way. I've dug to the bottom of quite a few network and traffic problems in my life by drawing the observed traffic patterns into sequence diagrams. Once you have the diagram visualised, it's easier to spot the places where something funky either happens or looks to be missing.
- Ono-Sendai 6y agoRelated: an old blog post of mine: http://forwardscattering.org/post/3 http://forwardscattering.org/post/3 'Sockets should have a flushHint() API call.'
- fanf2 6y agoLinux has a MSG_MORE flag for sendmsg() which sort of does the opposite. https://linux.die.net/man/2/sendmsg https://linux.die.net/man/2/sendmsg
- lmilcin 6y agoTo be fair, this can be fixed with well designed libraries that don't rely on TCP doing job for them merging buffers and preventing small writes. The issue is vast majority of libraries treat the problem as if it did not exist and prefer to not get their hands dirty and just conveniently write a stream of data to the socket leaving to the user to correctly configure options on the socket. But yes, in general, performance is at least in significant part about remembering a huge amount of trivia.
- derefr 6y agoWell, yes. The point of TCP is that it's an opaque reliable linear-stream abstraction. If you're not treating it like an opaque reliable linear-stream abstraction, you shouldn't be using TCP. If you want to manage your own datagrams, use a datagram transport. (Not necessarily UDP. I'd suggest SCTP, personally. Or maybe QUIC.)
- lmilcin 6y agoThat's naive at best. You say, if I wanted to write good HTTP client I should decide not to use TCP because TCP is meant to relieve the user from managing datagrams? The reality is, when you design a client library you work with what you have and in case when you are writing HTTP client what you have is TCP on one side and a user application on the other side that expects maximum performance and minimum latency possible (just look at the benchmarks on the net that compare tens of frameworks in competition for who can produces largest TPS of hello world with lowest latency).
- unilynx 6y agoHTTP is specifically the case where Nagle will probably hit most often. If you treat TCP just like file stream (it's a stream after all) and you're implementing, say, a webserver, the most straightforward way to implement a response is to: - figure out what was requested - build headers - send headers - send the file body especially if you intend to send the file body using sendfile. but this pattern is broken by Nagle - because there's no return traffic between the two sends in HTTP and the headers often won't fill out a full TCP packet, you'll trigger the wait.
- nt2h9uh238h 6y agoI have the same issue with my in-memory nodeJS app on Amazon AWS. Nice story, but how to FIX it?
- tus88 6y ago> Yep. Have you ever looked at TCP code and noticed a couple of calls to setsockopt() and one of them is TCP_NODELAY? That's why. When that algorithm is enabled on Linux, TCP tries to collapse a bunch of tiny sends into fewer bigger ones to not blow a lot of network bandwidth with the overhead. Unfortunately, in order to actually gather this up, it involves a certain amount of delay and a timeout before flushing smaller quantities of data to the network. IT'S CALLED BUFFERING.
- errantspark 6y agoI remember first learning of Nagle's algorithm back in the early WoW days in my endless quest to get lower latency for PvP on my neighbor's cracked WEP. I don't really know if it matters much in 2020, but I still habitually run the *.reg file to disable it on every new windows install.
- blibble 6y agoBlizzard finally figured out how to call setsockopt with TCP_NODELAY
- bborud 6y agoW Richard Stevens wrote some good books on TCP/IP. May I suggest reading them?
- sdoering 6y ago> Down that path also lies madness. Ohhhhh so true. I sadly have no such story to tell regarding performance optimization, but figuring out the intricacies of any complex system (for me at least) inevitably leads to you knowing arcane stuff that might come in handy some time. But on the other hand, it also - in my humble experience - leads to you knowing a lot of arcane stuff that might have an impact on a problem, but is absolutely not related in the specific case one is dealing with. Knowing when to discard arcane knowledge and when to jump onto that train of thought imho is crucial. But on the other hand debugging arcane stuff in complex systems is just so much fun. One learns so much.
- ganafagol 6y agoPeople should know about that more so that they can learn lessons. Any "little tweak" to an otherwise simple and elegant spec increases complexity which generations will have to deal with. This complexity often times compounds exponentially. Just see the interaction between Nagle and delayed acks. Each on their own they sound like a cool idea, but the compounding complexity is what kills understanding. Unfortunately, new generations don't learn the lesson. Modern web dev for example has so many layers of complexity and bloat, and they all interact and you need to know all layers intimitly for any real understanding as the complexity explodes. It does not have to be that way, if only every layer has a clean, small abstraction. Then you don't need to know the details, only the small spec. But that does not work if everybody just adds a hack here and there which breaks some separation but oh well. And that's why we can't have nice things.
- slx26 6y agoThere's nothing that makes me sadder in the world. How a lack of understanding can poison what could otherwise be in our reach, and moves it again further away from our understanding. It's almost as if we love "simple" because it's the perfect platform to start complicating. I mean, it's everywhere, not just IT systems. And part of it is unavoidable, but... when you start looking at it, it becomes impossible to unsee. So many absurd roadblocks, so much wasted potential.
- lawrjone 6y agoIt’s one of those trapdoors that people continually fall down: https://gocardless.com/blog/in-search-of-performance-how-we-shaved-200ms-off-every-post-request/ https://gocardless.com/blog/in-search-of-performance-how-we-...
- throwdbaaway 6y agoSomething is not right with this. The blog post has a link to https://bugs.ruby-lang.org/issues/8681 https://bugs.ruby-lang.org/issues/8681, which made the change to disable Nagle's algorithm since ruby 2.1.0 released in 2013.
- jdblair 6y agoThis isn't explicitly related, but interesting, so I offer it up here. When I read 40ms, it triggered a memory from tracking down a different 40ms latency bug a few years ago. I work on the Netflix app for set top boxes, and a particular pay TV company had a box based on AOSP L. Testing discovered that after enough times switching the app between foreground and background, playback would start to stutter. The vendor doing the integration blamed Netflix - they showed that in the stutter case, the Netflix app was not feeding video data quickly enough for playback. They stopped their analysis at this point, since as far as they were concerned, they had found the issue and we had to fix the Netflix app. I doubted the app was the issue, as it ran on millions of other devices without showing this behavior. I instrumented the code and measured 40ms of extra delay from the thread scheduler. The 40ms was there, and was outside of our app's context. Literally, I measured it between the return of the thread handler and the next time the handler was called. So I responded, to paraphrase, its not us, its you. Your Android scheduler is broken. But the onus was on me to prove it by finding the bug. I read the Android code, and learned Android threads are a userspace construct - the Android scheduler uses epoll() as a timer and calls your thread handler based on priority level. I thought, epoll() performance isn't guaranteed, maybe something obscure changed, and this change is adding an additional 40ms in this particular case. So I dove into the kernel, thinking the issue must be somewhere inside epoll(). Lucky for me, another engineer, working for a different vendor on the project, found the smoking gun in this patch in Android M (the next version). It was right there, an extra 40ms explicitly (and mistakenly) added when a thread is created while the app is in the background. https://android.googlesource.com/platform/system/core/+/4cdce42%5E%21/#F0 https://android.googlesource.com/platform/system/core/+/4cdc... Fix janky navbar ripples -- incorrect timerslack values If a thread is created while the parent thread is "Background", then the default timerslack value gets set to the current timerslack value of the parent (40ms). The default value is used when transitioning to "Foreground" -- so the effect is that the timerslack value becomes 40ms regardless of foreground/background. This does occur intermittently for systemui when creating its render thread (pretty often on hammerhead and has been seen on shamu). If this occurs, then some systemui animations like navbar ripples can wait for up to 40ms to draw a frame when they intended to wait 3ms -- jank. This fix is to explicitly set the foreground timerslack to 50us. A consequence of setting timerslack behind the process' back is that any custom values for timerslack get lost whenever the thread has transition between fg/bg. --- a/libcutils/sched_policy.c +++ b/libcutils/sched_policy.c @@ -50,6 +50,7 @@ // timer slack value in nS enforced when the thread moves to background #define TIMER_SLACK_BG 40000000 +#define TIMER_SLACK_FG 50000 static pthread_once_t the_once = PTHREAD_ONCE_INIT; @@ -356,7 +357,8 @@ ¶m); } - prctl(PR_SET_TIMERSLACK_PID, policy == SP_BACKGROUND ? TIMER_SLACK_BG : 0, tid); + prctl(PR_SET_TIMERSLACK_PID, + policy == SP_BACKGROUND ? TIMER_SLACK_BG : TIMER_SLACK_FG, tid); return 0;
- nialv7 6y agoMy first reaction to the 40ms number is "TCP_NODELAY?". That number is probably craved into my brain now.
- jeffbee 6y agoI have the same traumatic association with 5 second delays. 5 second tail latency? Look for SYN retransmits.
- taneq 6y agoWorld of Warcraft had Nagle's algorithm enabled for YEARS. That's one reason that VPN services were so popular and could cut 50-100ms off your ping time, especially if you were playing from Oceania.
- unilynx 6y ago> Stuff like this just proves that a large part of this job is just remembering a bunch of weird data points and knowing when to match this story to that problem. I've hit Nagle far in the past, and reading the title I thought 'well that can't be about Nagle because that was a 200ms delay' Looks like someone tuned it down to 40ms but didn't dare removing it. It would be interesting to know how they came to that choice
- JoeAltmaier 6y agoI thought the delay was set in the 'delayed ack' setting?
- nickdothutton 6y agoEveryone should spend a day trying to optimise iSCSI traffic.
- bullen 6y agoCan you disable Neagles (enable nodelay) on .js XMLHttpRequest or the newer fetch stuff? Maybe chrome/firefox disables it by default? Edit: This community is stupid/toxic.
- jorangreef 6y agoNo, the TCP_NODELAY setsockopt is not exposed by the browser. If you only control the client then you could make your http/fetch requests as big as possible, i.e. do the Nagle algorithm yourself but at a higher level to avoid the delays, but I don't think there's not much else you can do. Then again, if you control the server you could just run HTTP/3 to be on QUIC and then you're not on TCP any more, you would also save alot of connection handshake latency and benefit from advances in bufferbloat-sensitive congestion control algorithms much quicker. To be fair, I would be more concerned about bufferbloat delays on small shared networks than delays due to Nagle's algorithm. Bufferbloat delays can be up to tens of seconds, and you can trigger them disturbingly easily, just have someone else in the office upload a large attachment to an email in Gmail's web client to saturate the router's send buffer, while you watch your ping times.
- kevsim 6y agoSecond Nagle-related post I've seen this week! https://www.kdab.com/there-and-back-again/ https://www.kdab.com/there-and-back-again/
- nly 6y agoIf TCP had a header flag to indicate that the next segment was <MSS in size, then the receiver could be a lot smarter about whether it delayed the ACK.
- drosan 6y agoWhole page dedicated to something extremely simple that usually is being found in 2 mins by even amateur programmers. I love 2020 internet that presents something basic over and over while putting more and more air about it.
- pronoiac 6y agoI wish the fixes from the first fork were offered back to the main project.
- JoeAltmaier 6y agoSimilar to Nagle, there are reasons to combine packets on a session. Network equipment that fools with every packet can get backed up if the traffic packet count exceeds a limit. By Nagling (or doing something similar in your transmit code) you can increase your message rate through such bottlenecks. Used to have a server cluster that used some 'hologram' style router on the receiving end, to spread load. It had a hard limit on # packets per second it could handle. I changed our app to combine sends (2ms timer, not 40ms!) and halved our total traffic packet count. Put off the day they had to buy more server-side hardware to handle the load. Btw if the clients are on wifi networks, then there's no point in aggregating sends past a pretty small size (512 bytes?) because wifi fragments (used to fragment?) packets to that smaller size over the air, and never reassembles them, leaving that to the target server.
- oasisbob 6y ago> wifi fragments (used to fragment?) packets 802.11n and 802.11ac both don't do this. In contrast, they have several layers of packet aggregation (AMPDU/AMSDU). [1] Learned about this while troubleshooting latency problems on a noisy 10km point-to-point link. [1] https://arxiv.org/pdf/1704.07015.pdf https://arxiv.org/pdf/1704.07015.pdf
- dekhn 6y agoI once had to debug the scaling performance of a MPI-based simulation algorithm on cheap linux machines with TCP. I finally collected a TCP trace and showed it to the local expert who said: "hmm, 250ms delay right there.. that's the TCP retransmit timer... you're flooding the ethernet switch with too many packets and the switch is dropping them. Enable <such and such a feature>." Since then I've always kept various constants in human RAM because it helps root cause.
- whoisburbansky 6y agoWhat are some examples of other such constants you have hanging around?
- dekhn 6y agoa good one (although it's really a functional constant that has to be determined per-system) is the amount of time it takes to look up data in local RAM vs RAM that is attached to the other CPU socket (IE, over the system bus); it's 50% longer. I typically remember all the TCP timing constants, the seek time of a hard drive, everything in "Programmer's rule of thumb" (how long it takes for data to travel by light from CA to NL, etc)
- jgalt212 6y ago> Stuff like this just proves that a large part of this job is just remembering a bunch of weird data points and knowing when to match this story to that problem. Sounds a like interns doing rounds with the chief resident.
- nh2 6y agoFor those wondering "so, so how do I do it right?": I was in that situation 4 years ago and did a short write up on it: https://gist.github.com/nh2/9def4bcf32891a336485 https://gist.github.com/nh2/9def4bcf32891a336485 It explains how to avoid the 40ms delay and still batch data where possible for maximum efficiency. The key part is that you can toggle the TCP options during the lifetime of the connection to force flushes. Review appreciated.
- jlokier 6y agoSeems like generally a good approach. However you also have MSG_MORE instead of TCP_CORK to save a few system calls.
- nh2 6y agoHow does MSG_MORE save syscalls, given that it's a flag you pass to syscalls? Do you mean "save packets"?
- jlokier 6y agoSystem calls not packets. By using sendmsg() with MSG_MORE instead of write(), you can avoid the setsockopt() with TCP_CORK to cork before the write, and the later setsockopt() with TCP_NODELAY to push despite the cork. You can't give MSG_MORE to sendfile(), though with HTTP you don't need to. But if you need an equivalent of MSG_MORE with sendfile() you can in theory use SPLICE_F_MORE with splice() instead.
- nh2 6y agoAh, I see, you were referring to saving `setsockopt()`, not data-carrying syscalls. Yes that makes sense. But I guess that in most cases where you have control over the `sendmsg()`'s calls flags, you'd also have control over its buffer, so you may be able to build the buffer in userspace in many situations, thus even saving multiple data-carrying syscalls. The `setsockopt()` approach has the benefit that it works even when you have no control over the sending syscalls, e.g. when some library does it for you that you cannot modify or configure.
- scott_s 6y agoJohn Nagle is a commenter here on HN, and has commented on this very thing: https://news.ycombinator.com/item?id=10608356 https://news.ycombinator.com/item?id=10608356 I have also ran into this, but for me it was a periodic latency spike with steady but periodic messages. That latency spike went away when the messages were sent as-fast-as-possible.
- Kiro 6y agoAnimats is John Nagle?! I've seen good comments from him to the level where I think "oh, a comment from Animats" but never realized.
- jlokier 6y agoThis is one ancient problem. I remember dealing with it in 2003. Writeup from 1997 here (P-HTTP basically means HTTP version 1.1): https://www.isi.edu/~johnh/PAPERS/Heidemann97a.html https://www.isi.edu/~johnh/PAPERS/Heidemann97a.html > John Heidemann. Performance Interactions Between P-HTTP and TCP Implementations. ACM Computer Communication Review. 27, 2 (Apr. 1997), 65–73. > This document describes several performance problems resulting from interactions between implementations of persistent-HTTP (P-HTTP) and TCP. Two of these problems tie P-HTTP performance to TCP delayed-acknowledgments, thus adding up to 200ms to each P-HTTP transaction. A third results in multiple slow-starts per TCP connection. Unresolved, these problems result in P-HTTP transactions which are 14 times slower than standard HTTP and 20 times slower than potential P-HTTP over a 10 Mb/s Ethernet. We describe each problem and potential solutions. After implementing our solutions to two of the problems, we observe that P-HTTP performs better than HTTP on a local Ethernet. Although we observed these problems in specific implementations of HTTP and TCP (Apache-1.1b4 and SunOS 4.1.3, respectively), we believe that these problems occur more widely. Solutions for efficient batching of HTTP headers + data without delays involve TCP_NODELAY, and MSG_MORE / SPLICE_F_MORE / TCP_CORK / TCP_NOPUSH. Possibly TCP_QUICKACK may come in handy. Same for any protocol really, but HTTP is the one where there tends to be a separate sendmsg() and sendfile() on Linux.
- brundolf 6y ago> Stuff like this just proves that a large part of this job is just remembering a bunch of weird data points and knowing when to match this story to that problem.