13 ms·
Lying to TCP makes it a best-effort streaming protocol
- makira 9y agoSo, UDP.
- funkaster 9y agonot exactly: > If it’s used that way, it makes TCP a reliable, in-order protocol. the key here is "in-order".
- Dylan16807 9y agoIt's absolutely trivial to put a sequence number in UDP packets. The question is whether you want the other mechanisms of TCP: handshaking, (limited) retransmission, flow control...
- digi_owl 9y agoNever mind that most sensible streaming formats will include data in packets that allows the receiver to rebuild missing packets (up to a threshold).
- isostatic 9y agoA typical rtp stream will have smpte fec on top, allowing burst lost of say 20 packets, or random loss theoretically upto 5% In the last 10 years of streaming rtp over the Internet the vast majority of failures is bursty. Reordering is more rare than you'd expect. Looking at one sample from India to Europe over a period of 6 weeks, my 30mbit rtp stream was fine 99.998% with fec. The rest of the time that's why I dual stream, either timeshift (send the same packet again 100ms later - as I said most outages are time based), or dual path (although if the packets traverse the same router en route there are issues), or even both.
- mordero 9y agoI don't think that is quite true. I believe with UDP there is no promise of packets being received in order. I think this article is saying that you still get the benefits of processing the packets in the order received, but you don't have to worry about the latency of waiting for the re-transmission of any packets.
- vvanders 9y agoIt's so trivial to add ordering to UDP, it's really the right protocol to use here. Subverting TCP leads to all sorts of problems around congestion(which you can no longer filter on because which TCP streams are being non-compliant?) that it just should not be done.
- marcosdumay 9y ago> It's so trivial to add ordering to UDP Specially when compared with creating and using a user space IP protocol like was done here, or adding a new one into the kernel.
- JoeAltmaier 9y agoTCP layer also gives you pacing, congestion control and flow control. Which UDP, you'd have to do yourself?
- JoeAltmaier 9y agoAnd its not totally trivial to reorder UDP. Say you receive packets 1,2,3,5. How long to you wait for '4'? Maybe it was dropped; maybe its coming. Then you get 6,7. Is 4 still out there? You've got 3 packets in your pocket, waiting for 4. That adds up too. TCP gives you some idea of what packets you SHOULD have received, so you can respond better. UDP doesn't have any windowing etc so you have no idea.
- vvanders 9y agoI guess I don't really see the big issue with that. It's not like windowing and congestion control is some kind of black magic. It's spelled out pretty cleanly in the TCP RFC and pretty straight-forward to reimplement. Generally if you're hitting cases where TCP is causing you grief and you need to reach for UDP you've already got enough context to understand your congestion problems/etc. We've been doing this in game-dev for decades, ditto the voip space so it's not like you don't have a wealth of knowledge to draw from if you're really stumped.
- andreasvc 9y agoNo. UDP is connectionless, which makes it harder to use with NAT.
- xxpor 9y agoNot sure why you say that, UDP conntracking has been around for a long time.
- trav4225 9y agoI suspect that what andreasvc meant is that the default "NAT" configuration of most consumer-grade gear is such that it will block UDP (unless some other mechanism such as UPnP is used)...
- noselasd 9y agoBut it doesn't. Consumer gear NATs UDP pretty much as nice as it does TCP.
- trav4225 9y agoHmm, perhaps you are right... I switched from consumer gear to more "enterprise-y" gear a few years back, so my data points may be a bit outdated.
- testvox 9y agoI've been using normal consumer routers for NATed home connects since 2004 or so and I've never had an issue with outgoing UDP. It's required for basically every video game after all.
- lmns 9y agoThat's true for TCP as well. Almost all consumer-grade routers block all incoming connection attempts, not just UDP. From a stateful firewall's point of view both UDP and TCP have state.
- yarg 9y agoThe problem here is that it fails to take into account why any packets might be missing. If it's due to congestion, you've just subverted the mechanism TCP uses to relieve it - and the more people that are using the deviant implementation, the bigger the problem becomes, which impacts everyone.
- jrockway 9y agoI'm not convinced it matters in this case, lying only prevents the other end from being extra-smart. We'll stick with the VoIP example. Packets are going to be dropped independently of any information in the TCP headers. If there's too much load, some of them will disappear. If your client says "yeah I got that data" when it didn't actually get the data, it doesn't increase load any more -- if you're sending a real-time 256kbps audio stream, then you need to be able to send 256,000 bits every second regardless of whether or not the network is capable of that. By not retransmitting packets you don't care about, you're decreasing load. Of course, if you push the information all the way up the stack to the application, you can do interesting things. You can notice that packets are being dropped and switch to a codec that sounds worse but sends fewer bits per second, and maybe you'll get a better quality call going on with fewer dropouts. Now that I think about it, I'm surprised this isn't how we deal with degrading things over slow connections in general. I would much prefer to only get the "mobile friendly" version of a page if I'm actually on a mobile connection. Right now, the hueristic seems to be "if screen < desktop; send mobile page". That of course is silly because my home WiFi can happily pipe huge images into my phone faster than the web server can send them to me, while my 8 core laptop with a 4k screen tethered to my phone can't magically make 4G faster. Interesting interesting.
- jsnell 9y agoIt's how people deal with video (adaptive bitrate streaming). It also used to be how people dealt with images in the dark days of content rewriting middleboxes. So it's not that nobody's thought about the idea: it's that it's way harder to do once it's a question of really changing behavior rather than just being more aggressive with a quality/size trade-off knob. Also, mobile friendly versions of sites are at least as much about the user interface and rendering speed as about reducing bandwidth usage.
- dogecoinbase 9y agoThis, like many other current network flow innovations (see: QUIC), is greedy-algorithm short-sightedness that only works as a result of the improvements to regional/inter-datacenter networking over the last decade. They fall down miserably in the face of actual unreliability/variable latency/packet duplication, which still exist on many less developed networks worldwide.
- stingraycharles 9y agoFrom that point of view, UDP streaming is also short-sighted. Practically, this is nothing more than a type of "QoS" flag, where of course things would also break down if everything started giving themselves priority 1. So it depends more upon whether the use case is appropriate than it does about technical implementation.
- kevin_nisbet 9y agoSo this is an interesting idea, however, I suspect it may be quite problematic on the receive side if implemented. Once upon a time, I used to write network protocol sniffers. I basically simulated a partial TCP stack, in order to allow higher layer protocols decode traffic that was out of order. If a packet was lost in the capture, it's basically the same thing as the "lying ack" in the article, where the real endpoint would ack a frame the capture engine never saw. This becomes really difficult to process, because TCP simulates a stream, and if you lose a packet, and skip some bytes in the stream, where you pick up and continue may not be the beginning of a new message. So you also need to create heuristics, on how do you discard received data, until you get to a point where you can understand the stream again, and continue processing. As others have commented, UDP is likely a better choice here, as ordering semantics can be added, but on packet loss, the next received packet can always be parsed. Or look at SCTP, which allows turning on/off in order deliver, and I believe through an extension allows turning on/off reliability (but I'm not sure if this ever got implemented).
- adrianmonk 9y agoYes, the receive side would need to have a plan to interpret the data. One way to do this is to use a format where you can resync. For example, have special values that only occur in certain positions. You look for one of those values and can then enter a known state. Another possibility is to be selective about what you will allow to be discarded. Suppose you have a framing structure so that you send a byte count followed by a blob of data (like video or audio data). Then you can use these anticipatory acks to express that you can live without that blob. For example, if your protocol says "and now here's 123456 bytes of video data", you can ack with a sequence number that reflects having received those 123456 bytes but no more. Obviously, this limits your ability to skip ahead and may not be as useful.
- kevin_nisbet 9y agoSorry, I didn't mean to imply that this couldn't be done. I was trying to indicate that it is hard to do and adds a decent amount of complexity as someone who has written software that does this.
- lumisota 9y agoIt’s been shown that leaving holes in the sequence space isn’t good for middleboxes [0]. Wanting TCP to behave more like UDP is a reasonable goal (e.g., for reachability), but you throw the benefits away if 20% of connections are disrupted by middleboxes. Other approaches, like Minion [1] or TCP Hollywood [2], see deployability because they adhere to the assumptions made about TCP by middleboxes. [0]: “Is it Still Possible to Extend TCP?” - http://www0.cs.ucl.ac.uk/staff/ucacmha/papers/extend-tcp.pdf http://www0.cs.ucl.ac.uk/staff/ucacmha/papers/extend-tcp.pdf [1]: Minion - https://tools.ietf.org/html/draft-iyengar-minion-protocol-01 https://tools.ietf.org/html/draft-iyengar-minion-protocol-01 [2]: TCP Hollywood - http://ieeexplore.ieee.org/document/7497221/ http://ieeexplore.ieee.org/document/7497221/
- fulafel 9y agoIt would be very good for the net if something came up that penalised middleboxes though. Maybe this could be it.
- lumisota 9y agoQUIC demonstrates a better approach: encrypt everything, limiting the ability for middleboxes to make assumptions about behaviour.
- kevincox 9y agoEven QUIC found that the couple of bytes of "flag" fields in the header were enough for middleboxes to mess with it. It seems incredibly hard to devise a protocol which is extensible without having middleboxes mess it up.
- lumisota 9y agoIndeed. Though that issue sort of validated the approach: middlebox vendors will latch on to any unencrypted bits they can. The only way to stop that is to encrypt _everything_.
- 9y ago
- mixedbit 9y agoIn TCP if you ACK something that wasn't sent the ACK is dropped.
- kelnos 9y agoThat's not what's being talked about here. The idea is that you ACK packets that you believe have been sent, but have been dropped or delayed, because you've received a later packet.
- adrianmonk 9y agoGood point. Though that puts this somewhere in a middle ground between a best effort stream and a reliable stream. The ability to throw away data you don't care about is not absolute but is instead conditional upon successfully receiving more data. In a true best effort streaming system, the sender wouldn't retransmit. With this scheme, the sender might retransmit, sometimes, depending on whether certain packets get through and allow certain acks to get sent.
- adrianmonk 9y agoThat seems easy enough to work around. Send two acks, one for what you really received and one for what you are pretending you've received. If the data has been sent but lost, the sender will accept the aggressive ack. If it hasn't been sent yet, it will accept the less aggressive ack. For that matter, if you have some way of knowing the expected transmission rate (even approximately as long as you can resync to reality), you can blindly send a stream of periodic acks with steadily increasing sequence numbers.
- jacob019 9y agoCould be quite useful for TCP/TCP VPN type applications. It is widely accepted that TCP/TCP degrades performance, hence TCP/UDP is preferred. But UDP isn't always available.
- Animats 9y agoI suppose that's the answer to Random Early Drop. I used to argue that you want to use fair queuing and drop the newest packet in a stream, so all the old packets get delivered. But Random Early Drop, which is trivial to implement, caught on in routers. That means you lose random packets from the stream, and they have to be retransmitted. There's nothing awful about acknowledging a TCP sequence number for a valid packet received even if some previous packets are missing. You know the missing packets were sent, or you wouldn't have the later packet. If you don't need them, why ask for retransmission? The receiving TCP doing this should notice that packets are being lost and cut down the congestion window a bit, so the sender will slow down. This is purely a receive-side thing. It shouldn't bother upstream middleboxes. But, as someone pointed out, about 25% of middleboxes don't like it.
- deleted 9y ago[deleted]
- jgrip 9y agoSo basically redoing a less useful version of SCTP with the same issues in regards to middle boxes. I wish Microsoft would finally get their thumbs out and put SCTP into Windows.
- whiletrue84 9y agoWhy would anyone implement VoIP over TCP to then lie to TCP in order ot make it practically UDP?
- kazinator 9y agoI have read about this long ago. This was in the context not of ignoring don't-care data, but using premature ACKs as a way of garnering more bandwidth. The idea is to send ACKs ahead of time to provoke the sender into transmitting faster, defeating some of the congestion control mechanisms in TCP. The network has to be reliable to make it work (the pipe has to buffer all the excess spew), and the sending stack has to ignore situations when it gets a premature ACK, ahead of the sequence number it has actually sent.
- kazinator 9y agoProposed socket-level API: allow llseek on the descriptor, in the forward direction only.