4 ms·
Broadly speaking, UDP applications which rely on IP packet fragmentation are broken as designed. If you wanted reliable transport you would have used TCP or a
by ajross 11y ago
Broadly speaking, UDP applications which rely on IP packet fragmentation are broken as designed. If you wanted reliable transport you would have used TCP or a higher level abstraction. If you wanted simple transport of large data chunks, you would have chosen likewise. You picked UDP because you have latency requirements that cannot be met by TCP, and that means you need to know what your packets are actually doing, and that includes fragmentation.
If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes.
That's not to say that this isn't a bug. But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
- sunsu 11y agoWhile I (sort of) agree with the basic premise of what you're saying, in the real world, its not very helpful. We didn't "pick" UDP. We operate a VoIP related service that interoperates with many different carriers via SIP. Almost all of those carriers ONLY use UDP. SIP UDP packets can often be fairly large. This is especially problematic because GCE uses a non standard MTU size (1460 bytes). This does not make our app, or every other SIP related app that is forced to use UDP, "Buggy".
- jsolson 11y agoUpon further consideration, I've deleted the bulk of this comment. Rough summary of what I had here: I'm an engineer on GCE (in particular I built our current virtio-net device and a small fraction of the other fiddly bits that sit behind that) -- some details in the bug jumped out at me and I thought there might be a quick fix, but I hadn't processed all of the details and posted a bit prematurely. After further review my original post was essentially content free other than 'IP fragmentation works correctly between internal IPs', which is not germane to the actual customer-reported issue.
- sunsu 11y agoThanks for your comment. The vast majority of networks outside of GCE don't use an MTU of 1460. Even if we set the instance MTU size to 1460, we still have problems. If a UDP packet sent to a GCE instance from outside GCE is larger than GCE's MTU, the first fragment is received and no subsequent fragments ever make it to the instance. This is the bug. This is very common with protocols like SIP when the SDP gets large due to a lot of media attributes.
- jsolson 11y agoYup, sorry, I hadn't read all of the details before I commented. I've mostly deleted my original as it didn't apply. When I jumped on HN on my ride home I saw this and was a weeee bit over-eager to be helpful.
- wmf 11y agoI agree with this, but unfortunately one such broken protocol is IKE, widely used for VPNs. (Off topic: has anyone tried grafting a TLS handshake onto ESP?)
- kentonv 11y agoI'm not sure if I'd go so far as to say that apps which rely on UDP fragment reassembly are "buggy", but I definitely agree that the article title is exaggerating by calling UDP on GCE "unusable". Many UDP services (including one operated by my company, on GCE) will work just fine. Interestingly Linux will actually, by default, throw EMSGSIZE any time you try to send() a UDP datagram that is larger than the detected network MTU to the destination. As I understand it, you have to explicitly turn this behavior off to get fragmentation. http://man7.org/linux/man-pages/man7/udp.7.html http://man7.org/linux/man-pages/man7/udp.7.html
- ars 11y ago> I'm not sure if I'd go so far as to say that apps which rely on UDP fragment reassembly are "buggy" Isn't fragment reassembly essentially implying all the packets (fragments) will arrive, and will be rearranged in order? i.e. exactly what TCP does? For example imagine sending a single 1MB packet via UDP, letting it fragment and be reassembled. What distinguishes that from TCP?
- jblow 11y agoNo. If a fragment of a fragmented UDP packet is dropped, the packet is dropped. It is simple. If you understand why UDP exists you will get it.
- jacquesm 11y agoAfter a suitable time-out. Hm. That makes me wonder if you couldn't use a scheme such as fragment-fragment-dropped-fragment repeatedly to eat up router or IP stack resources.
- cbsmith 11y agoLet's just say it wouldn't chew up nearly the kind of resources as is achievable by selectively dropping IP packets for a TCP connection.
- 11y ago
- jws 11y agoNo. It's broken. The title should be longer and include "which fragmentation". Large UDP packets have a packets success rate of the single packet success rate raised to the number of fragments power, so don't get attached to them in situations with significant packet loss. But this behavior is just an unaddressed bug.
- cbsmith 11y agoYeah, exactly. When "higher chance of being dropped" is replaced with "guarantee they'll be dropped", you kind of have a huge problem.
- cbsmith 11y ago> If you wanted simple transport of large data chunks, you would have chosen likewise. That's not entirely fair. UDP is actually a very convenient transport for packetized data of 65,000 bytes or less. > You picked UDP because you have latency requirements that cannot be met by TCP. That is not really a good reason to use UDP, as you won't necessarily get better latency (as has oft been demonstrated). You should pick UDP because you have a connectionless protocol and/or one with data loss requirements that are different from those built in to TCP. > that means you need to know what your packets are actually doing, and that includes fragmentation. UDP is supposed to abstract IP fragmentation issues for you. > If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes. I've used UDP based protocols all the time. I'd agree that I've had to be a bit more aware of path MTU when it comes to diagnosing or debugging problems, but in terms of the application code reading and writing UDP packets, I've never done MTU discovery. > But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with. Considering there are bytes in the UDP header specifically designed to allow you send packets larger than MTU size, this is actually a huge problem. For many people, this requires reimplementing UDP's fragmentation & defragmentation logic up in layer-7. Isn't it bad enough that people are reimplementing TCP on top of UDP? Do we really need to tell them to reimplement UDP on top of UDP?
- Dylan16807 11y ago> If you wanted reliable transport you would have used TCP or a higher level abstraction. Nobody asked for that, just to send slightly larger packets that will probably get through. > If you wanted simple transport of large data chunks, you would have chosen likewise. Why? A 4KB data chunk doesn't need TCP more than a 1KB data chunk. You know the 1500 byte MTU is completely arbitrary and many networks set it higher, right? > You picked UDP because you have latency requirements that cannot be met by TCP, and that means you need to know what your packets are actually doing, and that includes fragmentation. Fragmentation itself is a cost of microseconds. The main problem with TCP is head of line blocking, and moderately large UDP packets don't have to deal with that. You don't need to trace every packet all the way through the network to get low latency and manually handle loss. > the apps for which GCE is "unusable" are buggy apps to start with Nothing you listed is a bug.
- cbsmith 11y agoI kind of wrote a lot but didn't summarize very well. A key principle of UDP is quite the contrary of what you are saying. UDP applications can have guarantees that they don't have to engage in MTU discovery or fragmentation issues. It provides a way to have an abstract, static contract about packets that is agnostic to layer-2. Because UDP is comparatively simple, it has been abused as a proxy to implement your own protocol on top of IP, and in that context you of course you really have to deal with al those concerns. However, it is a terrible mistake to think that is what UDP is about or how one should use UDP.
- chetanahuja 11y agoThat this comment is on top of this thread is saddening. You just called about a dozen broadly used protocols that depend on nothing more than the basic guarantee provided by UDP (unreliable datagram delivery-non/delivery) "broken by design". If I was Linus, now would be my cue to unleash a tirade about how you don't fing break userspace. Ever. Protocol layering works because higher level protocols depend on long standing contracts with the lower layers. Google decided to willy nilly go and break the basic contract of UDP across their entire cloud and the top comment here faults protocols built on those guarantees. Disappointing.