6 ms·
IMO one of the worst mistakes made in IP development was not having made a standard protocol for the one thing that pretty much everyone seems to want: A relia
by dale_glass 2y ago
IMO one of the worst mistakes made in IP development was not having made a standard protocol for the one thing that pretty much everyone seems to want:
A reliable, unlimited-length, message-based protocol.
With TCP there's a million users that throw out the stream aspect and implement messages on top of it. And with UDP people implement reliability and the ability to transmit >1 MTU.
So much time wasted reinventing the wheel.
- supriyo-biswas 2y agoThere’s Homa[1]. [1] https://github.com/PlatformLab/Homa https://github.com/PlatformLab/Homa
- H8crilA 2y agoSorry for a useless comment but now I feel so stupid for never noticing that pretty much everything done on TCP is actually message exchange, not streams. The mismatch is indeed so unfortunate.
- eru 2y agoWell, you often have some dependency between messages. Some re-orderings don't matter, but some do. If you just keep everything in the same order, then you never have to worry about any re-ordering. I can see why people picked streams as the one-size-fits-all-(but-badly) abstraction.
- tliltocatl 2y agoBecause worse is better and if you try to be everything you (don't) get OSI. Also because everyone loves serial ports and dialup modems and TCP just emulates a serial port over packet network. And it wasn't supposed to be one-size-fits-all, rather «let us have telnet while we figuring out the rest» but then ossification happend.
- iainmerrick 2y agoIs it really that unfortunate? What’s the big downside? There’s a bit of reinvention needed, sure, but adding a framing layer is about the easiest task in networking. If you want a standard way of doing it, these days you can use a WebSocket. Edit to add: oh, I see, you want reliable but unordered messages. That would definitely be useful sometimes, but other times you do want ordering. If you don’t need ordering, isn’t that pretty much what QUIC does?
- p_l 2y agoSCTP did both reliable/unreliable and ordered/unordered, with QUIC arguably being at least a bit inspired by it, but going over UDP because internet is broken. EDIT: Internet being broken is also why there's a grease extension for QUIC
- taneq 2y agoWhat would ‘reliable but unordered’ even mean? What’s the difference between a dropped message and one that’s just late?
- saurik 2y agoThe difference is whether you get to process the ones which came after first or you must wait for the retransmission of the earlier ones. FWIW, SCTP provides for this difference.
- Aurornis 2y ago> I feel so stupid for never noticing that pretty much everything done on TCP is actually message exchange, not streams. This generalization isn’t true at all. Streaming data over TCP is extremely common in many applications. When you download a large file, you’re streaming it to disk, for example. I guess you could get pedantic and argue that an entire stream of something is actually one large message, but that’s really straining the definitions.
- adrian_b 2y agoBy numbers of communication actions, pretty much everything may actually be a message exchange, but by the amount of data that was transferred for many years the file downloads and uploads had been dominant on the Internet, which are the kind of thing for which TCP has been designed. For a big file more data may be transferred than for thousands or millions of message exchanges. Nowadays perhaps video streaming, which may be lossy, so TCP is not needed for it (i.e. what is now called streaming does not have the reliability requirements of the TCP streams), might transfer more data on the Internet than file downloads, but that is not a desirable technical evolution, the replacement of downloading with streaming is just a means to extract more money from those who are satisfied by such a service.
- tliltocatl 2y agoYou are looking for SCTP.
- lxgr 2y agoWhich you will absolutely not get past pretty much any middleboxes, unless you encapsulate it in UDP.
- kixelated 2y agoI tried using SCTP before QUIC. Unfortunately, while the API looks like you can mix reliable/unreliable messages, in reality the transport doesn't support it because of how FORWARD-TSN works. There were some other pretty fundamental blockers too.
- Ekaros 2y agoNow I wonder. Do we have any actual streaming use of TCP, with purely streaming protocol.
- eru 2y agoYes. When your messages need to be received and processed in the same order they are sent, then a stream is a good (or good enough) abstraction. For most applications some re-orderings of messages don't matter, and others would need special handling. So as a one-size-fits-all-(but-badly) abstraction you can use a stream. > Do we have any actual streaming use of TCP, with purely streaming protocol. But to give you a proper answer: the stream of keyboard inputs from the user to the server in Telnet (or SSH).
- Ekaros 2y agoOpens up the question that if you have "messages" are you a stream anymore? Can your stream of messages start mid message for example? Surely a stream can have this happen. Or are you instead messages send over a stream. In which case abstraction of stream instead of reliable message transportation is bit weird. Wouldn't each input be a single albeit too short message? But this level of granularity really makes little sense...
- eru 2y agoIf you have a reliable-byte-stream abstraction, you only need to add a way to serialise your messages into bytes, and you get a reliable-messages-in-order abstraction for 'free'. That's convenient! And you don't need to teach your lower level protocols about where your higher level messages start and end. You can also stick your higher level messages into a structure that's more complicated than a stream, eg you can stick them into a tree. Anything you can serialise, you can send. Of course, the downside to this is that when you don't need these strong guarantees, you are paying for stuff you don't need.
- nyc_pizzadev 2y agoHigh quality VOD (ie streaming a 4K movie). HTTP block file systems, each block needs to be streamed reliably to fulfill the read() call plus read ahead.
- jzwinck 2y agoHere is a good message framing protocol built on TCP: https://www.nasdaq.com/docs/SoupBinTCP%204.0.pdf https://www.nasdaq.com/docs/SoupBinTCP%204.0.pdf If you want large messages just redefine the length from 16 to 32 bits. It's been used for millions of messages per day for 25 years and hasn't been changed in a long time. Admittedly it isn't as common as TCP and that's a shame. But it's out there, it's public, it's minimal and it makes sense.
- eru 2y ago> SoupBinTCP guarantees that the client receives each sequenced message generated by the server in the correct order, [...] This protocol guarantees too much. It's still a stream.
- ajb 2y agoActually, the mistake in IP was in not encrypting the next layer, so that networks can block transport protocol innovation by filtering anything other than TCP and UDP. This is the mistake that quic has fixed.
- Ekaros 2y agoI think that one can be forgiven for pure lack of processing power... We forget just how under powered even host machines were and how resource hungry encryption is...
- d-z-m 2y agoQUIC still runs over UDP. Are you referring to MASQUE?
- ajb 2y agoI should perhaps have said "works round" rather than fixes - yes, it has to run over UDP but the quic team had an explicit goal of not allowing the network to interfere further : see sec 3.3 and 7.5 of https://dl.acm.org/doi/pdf/10.1145/3098822.3098842 https://dl.acm.org/doi/pdf/10.1145/3098822.3098842 This now means that innovations based on Quic can occur
- mycall 2y agoSince both QUIC and UDP are transport layers, does this mean that once switches and ISPs start supporting QUIC along side UDP and TCP, that a new version of QUIC can be released which doesn't require UDP?
- ajb 2y agoSwitches don't need to change anything to support QUIC, precisely because it uses UDP. So, the deployment of QUIC doesn't involve any technical changes that would enable a version of QUIC that worked directly over IP. It's possible that the defeat of middleboxes would change the motivation of ISPs to filter out alternate protocol numbers, meaning that there might be less resistance to getting them to allow that in the future - but doing so is still a big exercise. Also, the overhead of a UDP packet is 8 bytes total, in 4 16-bit fields: - source port - dest port - length - checksum So, we can save a max of 8 bytes per packet - how many of these can we practically save? The initial connection requires source and dest ports, but also sets up other connection IDs so theoretically you could save them on subsequent packets (but that's pure speculation, with zero due diligence on my part). It would require operational changes and would make any NATs tricky (so practically only for IPv6) Length and checksum maybe you can save - QUIC has multiple frames per UDP datagram so there must be another length field there (anyway, IETF is busily inventing a UDP options format on the basis that the UDP length field is redundant to packet length, so can be used to point at an options trailer). QUIC has integrity protection but it doesn't seem to apply to all packet types - however I guess a checksum could be retained for those that don't only. So in sum maybe you could save up to 8 bytes per packet, but it would still be a lot of work to do so (removing port numbers especially)
- jan_k_ 2y agoWebSocket?
- paulgb 2y agoWebSocket is over TCP, so still ordered. I think by message-based GP is talking not only about message framing (which WebSocket provides) but also reliable out-of-order delivery.
- ori_b 2y agoWhen does a long out of order delay look like a packet loss? Is it possible for an out of order packet to get delayed by a millisecond? A second? A minute? An hour? Have you written code to be robust to this?
- paulgb 2y agoOther people have probably thought about it more than I have, but the way I see it working is that a set of unacked messages are stored on each end of the connection. When a connection receives a message, it acks it by ID. When acks come out of the order the client sent them in (or after a timeout period), it re-sends the packet that may have dropped. If every message is getting dropped, the set of unacked messages fills up and the connection stops accepting messages from the application, similar to TCP in that situation. I mostly work with TCP so haven't had to deal with unreliable channels generally, but I do use a similar approach at the application level for reliable delivery across reconnects.
- ori_b 2y agoMy point is that a lost message looks the same as an arbitrary delay.
- p_l 2y agoThat's because TCP evolved as, essentially, abstraction of serial port. And due to various political & cultural & industrial lobbying, more flexible but complex on their face approaches (OSI) were rejected. TCP meant you could easily attach a serial terminal to a stream, a text protocol meant you could interact or debug a protocol by throwing together a sandwich, an undergrad, and a serial terminal (coffee too if they do a good job). You could also in theory use something like SMTP from a terminal attached over a TIP (which provided dial-in connection into ARPAnet where you told the TIP what host and what port you wanted). If you look through some of the old protocol definitions you'll note that a bunch of them actually refer to simplest mode of TELNET as base. Then in extension of arguably the same behaviour that blocked GOSIP mandate we have pretty much reduced internet to TCP, UDP, and bits of ICMP, because you can't depend on plain IP connectivity in a world of NAT and broken middleboxes.
- throwaway7ahgb 2y agoCorrect on the reliable UDP. I've seen a few implementations of a reliable UDP messaging stream (using multicast or broadcast). It uses a sequencer to ensure all clients receive the correct message and order. It can be extremely fast and reliable.
- taneq 2y agoIf it guarantees delivery and preserves order, what’s the difference vs. TCP?
- colanderman 2y agoDelimited messages vs. stream.
- throwaway7ahgb 2y agoMuch much faster, TCP is very slow and complicated (relatively). This can be sped up by not using the kernel using something talking directly to hardware.
- FpUser 2y agoI have implemented publish/subscribe middleware in the 90s using UDP and reliable IP multicast. It was not too hard to do and had taken me about two month. Yes I did reinvent the wheel but it had numerous positive sides: a) It added yet another area of expertise to my portfolio. b) I did it because it was needed to facilitate another product in the company where I worked. The alternative at the time would be paying $350,000 for middleware from Vendor-X. Not acceptable. c) Vendor-X people noticed us, loved what we did and made us a partner to resell their middleware and we've made a ton of money on deployment, configuring and consulting services.
- deleted 2y ago[deleted]
- _zoltan_ 2y agolibfabric implements msg on top of TCP, among others. I love that I can just switch between TCP and RDMA without any code change.
- bvrmn 2y ago> unlimited-length Where to keep in-progress frame data? I case of TCP it's an app and buffer for only a single frame or app even could make "stream processing" for in-progress frame.