6 ms·
Issue 87 – google-compute-engine – UDP Packet Fragments cannot be reassembled
- eloff 11y agoWow, reported May 2014, and still no resolution for such a serious issue. GCE looks nice in theory, but I've heard no end of problems like this with shitty communication and support when something doesn't work. I'll stick with AWS for the time being. I really wish Google would get their act together and provide serious competition though. That's good for everybody who uses the cloud.
- toomuchtodo 11y agoAWS still doesn't support IPv6 except on internet facing ELBs. Pick your poison. https://forums.aws.amazon.com/thread.jspa?messageID=536049 https://forums.aws.amazon.com/thread.jspa?messageID=536049 https://www.reddit.com/r/aws/comments/3ccn5o/real_ipv6_support_in_aws/ https://www.reddit.com/r/aws/comments/3ccn5o/real_ipv6_suppo...
- kerr23 11y agoAWS de-prioritizes UDP packets making it not a great choice for UDP based applications as well.
- akent 11y agoDo you have a reference or a link for this?
- kerr23 11y agoAWS is always super closed mouthed about their infrastructure. The info I have came from a conversation with an AWS Solutions Architect. It's a couple of years old, but based on my experience it's still true today.
- majke 11y agoThat doesn't sound right. Consider DNS.
- dfc 11y agoWhenever people talk about behavior/treatment of UDP traffic I consider DNS as a special case. I have no idea how AWS handles UDP but I will never use DNS as a generalizable example of UDP traffic.
- cbsmith 11y agoAt least in the past, it really was true. I've been burned by this before using UDP in AWS. In AWS, I've learned to be skeptical of using protocols other than TCP. That said, it consequently has been a long time since I've tested UDP over AWS.
- kerr23 11y agoDNS was precisely the area where we got bit by it. We were using a Non-AWS DNS resolver (aka Google) and we would often get dns resolution errors despite our NAT not being remotely taxed by the traffic.
- cbsmith 11y agoYeah, I've run in to this. The point about NAT makes me wonder though if it is really de-prioritization or just the network straining to handle all that recalculation of checksums.
- dsl 11y agoThe last update really hits the nail on the head for most Google products: > Apparently the update from Google is to find another support channel to escalate, or only use TCP. I've never had a Google product issue ever resolved using an official channel I was directed to. It's only by back channels, friends of friends, posting to HN, etc.
- deleted 11y ago[deleted]
- kbuck 11y agoThey've likely blocked your VPN because they think you're trying to bypass region restrictions. In this case they don't care whether you're human or not: they simply want to make sure you can't watch videos that would not normally be available in your country. It's a ridiculous system, but it's unfortunately how media licensing still works. You could try getting a cheap VPN endpoint device and putting it between your router and your residential internet connection. Then you could VPN to that device to access the (usually-blocked) YouTube while still using your residential IP (assuming the blocking is done on your router).
- kuschku 11y agoWell, the router is also the DSL modem – so there’s nowhere to put it inbetween, and my parents hate modifying that stuff. So I’ll have to pay more for a VPS that’s in my country just to make sure I don’t get hit by this again? Well, then fuck it, I’ll just use a goddamn free VPN or Tor or whatever.
- scott00 11y agoI think your best bet is to solve the youtube filter/access problem in a different way. If you have a cable modem and/or wireless router, it may let you filter web traffic by MAC address (mine does). If your sister uses a different computer than you, you could block access for her MAC, and allow access for yours. If there's only a single computer, software that does web filtering with a password-based override would probably work.
- ajross 11y agoBroadly speaking, UDP applications which rely on IP packet fragmentation are broken as designed. If you wanted reliable transport you would have used TCP or a higher level abstraction. If you wanted simple transport of large data chunks, you would have chosen likewise. You picked UDP because you have latency requirements that cannot be met by TCP, and that means you need to know what your packets are actually doing, and that includes fragmentation. If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes. That's not to say that this isn't a bug. But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
- sunsu 11y agoWhile I (sort of) agree with the basic premise of what you're saying, in the real world, its not very helpful. We didn't "pick" UDP. We operate a VoIP related service that interoperates with many different carriers via SIP. Almost all of those carriers ONLY use UDP. SIP UDP packets can often be fairly large. This is especially problematic because GCE uses a non standard MTU size (1460 bytes). This does not make our app, or every other SIP related app that is forced to use UDP, "Buggy".
- jsolson 11y agoUpon further consideration, I've deleted the bulk of this comment. Rough summary of what I had here: I'm an engineer on GCE (in particular I built our current virtio-net device and a small fraction of the other fiddly bits that sit behind that) -- some details in the bug jumped out at me and I thought there might be a quick fix, but I hadn't processed all of the details and posted a bit prematurely. After further review my original post was essentially content free other than 'IP fragmentation works correctly between internal IPs', which is not germane to the actual customer-reported issue.
- sunsu 11y agoThanks for your comment. The vast majority of networks outside of GCE don't use an MTU of 1460. Even if we set the instance MTU size to 1460, we still have problems. If a UDP packet sent to a GCE instance from outside GCE is larger than GCE's MTU, the first fragment is received and no subsequent fragments ever make it to the instance. This is the bug. This is very common with protocols like SIP when the SDP gets large due to a lot of media attributes.
- oofabz 11y agoIt sounds like UDP packets that fit within the MTU work fine. If you need to transmit more than fits in one packet (1452 bytes), UDP is a bad choice. SCTP is ideal for this use case but it is not well supported by OSs or networking APIs. TCP works but adds overhead. TFTP works, is UDP-only and has less overhead than TCP, but it does not respond well to packet loss. UDT is like TFTP done right, and is a good solution if you can setup a dependency on its large C++ library.
- cbsmith 11y agoSCTP is a pretty good choice. It has been a bit since I used it, but it is a more complex protocol and I recall often run in to challenges getting it to perform as efficiently over various bits of network equipment. Last I checked UDT was far more complex than UDP, and since it is layered on top of UDP, I'd think it'd be vulnerable to this problem (although it had all kinds of logic for correctly sizing packets and windows, so maybe it correctly avoids this problem). Either way though, from an application perspective UDT looks much more like TCP than UDP, so I wouldn't think it'd be an obvious choice to replace UDP.
- addingnumbers 11y ago> If you need to transmit more than fits in one packet (1452 bytes) You must never make assumptions about what fits in one packet. The MTU could be 100, or less, or 8000, or more. As soon as you start doing math based on MTU values that you don't permanently have end-to-end control of yourself, you're setting yourself up for trouble.
- oofabz 11y agoThat's true, it's a bad idea to assume an MTU of 1500. Although there is no minimum MTU in IPv4, IPv6 specifies a minimum of 1280 bytes. So if you send your UDP packets over IPv6, you are guaranteed room for 1232 bytes of payload.
- kaa2102 11y agoI've run into several misfires while using Google Cloud/Compute Engine: MySQL database access, email and encryption. These features didn't work without either a Google or third-party service. I set up postfix and use SendGrid for email and Google's Cloud SQL. You can get tech support at Silver or Gold level. I think everyone starts at Bronze.
- dboreham 11y agoThis is certainly not the only case of "network subtly broken on cloud VMs". For example, every provider I have tested (including AWS, Rackspace, Digital Ocean, Linode) enables TCP segment reassembly offload, and provides no way to disable it (presumably because it is being done on the host not the VM). This will typically break TCP tunneling (e.g. using GRE) because PMTUD doesn't work under these conditions. fwiw a shout out to Soft Layer which is the only VM hosting provider I'm aware of that does not suffer from this blight (provided you pay for additional IP addresses routed to your box).
- halayli 11y agoI feel the title should be updated to: "for Applications Which Reply on UDP packets larger than 1500 bytes".
- fabulist 11y agoAny application might very reasonably choose to do this. Maybe version X never does, and you can satisfy yourself of this by peaking at the code, but there is no reason to believe that X+1 won't, or that another vendor's product/FOSS project you need to interoperate with won't, etc.
- api 11y agoWait... you mean there are protocols other than http? Somebody should tell Google, Amazon, and Microsoft.
- dang 11y agoPlease don't editorialize the titles of articles you submit to HN. The submitted title was "Warning: Google Compute Engine Unusable for Applications Which Rely on UDP", which several commenters have objected to as exaggerated.