7 ms·
I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it
by ironman1478 2y ago
I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a setting that people might not know about that manipulates how data is sent over the wire so aggressively.
- hinkley 2y agoWhat you really want is for the delay to be n microseconds, but there’s no good way to do that except putting your own user space buffering in front of the system calls (user space works better, unless you have something like io_uring amortizing system call times)
- bobmcnamara 2y agoI'd rather have portable TCP_CORK
- hinkley 2y agoCork is probably how you’d implement this in userspace so why not both?
- mjevans 2y agoIt'd probably be amazing how many poorly coded games would work better if something like... TCP_60FPSBUFFER Would wait for ~16mS after the first packet is queued and batch the data stream up.
- jnordwick 2y agolinux has auto-corking (and I know of no way to disable it) that will do these short delays on small packets even if the dev doesn't want it
- Bluecobra 2y agoI agree, it has been fairly well known to disable Nagle's Algorithm in HFT/low latency trading circles for quite some time now (like > 15 years). It's one of the first things I look for.
- Scubabear68 2y agoI was setting TCP_NODELAY at Bear Stearns for custom networking code circa 1994 or so.
- kristjansson 2y agoThis is why I love this place
- mcoliver 2y agoSame in M&E / vfx
- Reason077 2y agoSurely serious HFT systems bypass TCP altogether now days. In that world, every millisecond of latency can potentially cost a lot of money. These are the guys that use microwave links to connect to exchanges because fibre-optics have too much latency.
- jm__87 2y agoThey still need to send their orders to an exchange, which is often done with FIX protocol over TCP (some exchanges have binary protocols which are faster than FIX, but the ones I'm aware of still use TCP)
- nsguy 2y agoThe logic is really for things like Telnet sessions. IIRC that was the whole motivation.
- bobmcnamara 2y agoAnd for block writes! The Nagler turns a series of 4KB pages over TCP into a stream of MTU sized packets, rather than a short packet aligned to the end of each page.
- nailer 2y agoYou’re right re: making delay explicit, but also crappy use the space networking tools don’t show whether no_delay is enabled on sockets. Last time I had to do some Linux stuff, maybe 10 years ago you had to write a systemtap program. I guess it’s EBNF now. But I bet the userspace tools still suck.
- nailer 2y ago> use the space Userspace. Sorry, was using voice dictation.
- Sebb767 2y ago> It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. If the intention is mostly to fix applications with bad `write`-behavior, this would make setting TCP_DELAY a pretty exotic option - you would need a software engineer to be both smart enough to know to set this option, but not smart enough to distribute their write-calls well and/or not go for writing their own (probably better fitted) application-specific version of Nagles.
- nh2 2y agoSame here. I have a hobby that on any RPC framework I encounter, I file a Github issue "did you think of TCP_NODELAY or can this framework do only 20 calls per second?". So far, it's found a bug every single time. Some examples: https://cloud-haskell.atlassian.net/browse/DP-108 https://cloud-haskell.atlassian.net/browse/DP-108 or https://github.com/agentm/curryer/issues/3 https://github.com/agentm/curryer/issues/3 I disagree on the "not a good / bad option" though. It's a kernel-side heuristic for "magically fixing" badly behaved applications. As the article states, no sensible application does 1-byte network write() syscalls. Software that does that should be fixed. It makes sense only in the case when you are the kernel sysadmin and somehow cannot fix the software that runs on the machine, maybe for team-political reasons. I claim that's pretty rare. For all other cases, it makes sane software extra complicated: You need to explicitly opt-out of odd magic that makes poorly-written software have slightly more throughput, and that makes correctly-written software have huge, surprising latency. John Nagle says here and in linked threads that Delayed Acks are even worse. I agree. But the Send/Send/Receive receive pattern that Nagle's Algorithm degrades is a totally valid and common use case, including anything that does pipelined RPC over TCP. Both Delayed Acks and Nagle's Algorithm should be opt-in, in my opinion. It should be called TCP_DELAY, which you can opt-into if you can't be asked to implement basic userspace buffering. People shouldn't /need/ to know about these. Make the default case be the unsurprising one.
- carterschonwald 2y agoOh hey! It’s been a while how’re you?!
- a_t48 2y agoThanks for the reminder to set this on the new framework I’m working on. :)
- jandrese 2y agoThe problem with making it opt in is that the point of the protocol was to fix apps that, while they perform fine for the developer on his LAN, would be hell on internet routers. So the people who benefit are the ones who don't know what they are doing and only use the defaults.
- inopinatus 2y agoWith some vendors you have to solve it like a policy problem, via a LD_PRELOAD shim.
- ww520 2y agoSame here. My first job out of college was at a database company. Queries at the client side of the client-server based database were slow. It was thought the database server was slow as hardware back then was pretty pathetic. I traced it down to the network driver and found out the default setting of TCP_NODELAY was off. I looked like a hero when turning on that option and the db benchmarks jumped up.
- pjc50 2y ago> I feel like the logic behind it is sound, but it just doesn't work for some workloads. The logic is only sound for interactive plaintext typing workloads. It should have been turned off by default 20 years ago, let alone now.
- p_l 2y agoRemember that IPv4 original "target replacement date" (as it was only an "experimental" protocol) was 1990... And a common thing in many more complex/advanced protocols was to explicitly delineate "messages", which avoids the issue of Nagle's algorithm altogether.
- immibis 2y agoNot when creating a socket - when sending data. When sending data, you should indicate whether this data block prefers high throughput or low latency.
- kazinator 2y ago> that an engineer needs to be forced to set while creating a socket Because there aren't enough steps in setting up sockets! Haha. I suspect that what would happen is that many of the programming language run-times in the world which have easier-to-use socket abstractions would pick a default and hide it from the programmer, so as not to expose an extra step.