17 ms·

We need a replacement for TCP in the datacenter [pdf]

• Joel_Mckay 4y ago
  "Standards" https://xkcd.com/927/ https://xkcd.com/927/
  • deleted 4y ago
    [deleted]
  • Dylan16807 4y ago
    They're not trying to make a universal standard. People are way too quick to dig out this xkcd.
    • Joel_Mckay 4y ago
      So, you are suggesting a different standard response. I think this cognitive trap neglects the lessons of NetBEUI. Enhance your calm.
      • Dylan16807 4y ago
        Cute, but I am suggesting not having a standard response.
        • Joel_Mckay 4y ago
          And "not having a standard response" is the antithesis of global communication. I appreciate your perspective though. Have a wonderful day.
          • Dylan16807 4y ago
            I usually find that standard responses impede real communication.
            • Joel_Mckay 4y ago
              The ISO OSI stratifies communication into layers, with lower layers tasked with the role of encapsulating meaningful human content for delivery across the networks. The protocol itself is unaware of its payload content in most circumstances, and this is why most network hardware has minimal complexity yet remains interoperable. I assure you every packet of content is encapsulated in several standard responses. Have a nice day =)
• defrost 4y ago
  Jumping to the end: > TCP is the wrong protocol for datacenter computing. > Every aspect of TCP’s design is wrong: there is no part worth keeping. I cannot disagree and Ousterhout argues well. > Homa offers an alternative that appears to solve all of TCP’s problems. I'm well behind the curve on protocols and now I have something to learn more about. > The best way to bring Homa into widespread usage is integrate it with the RPC frameworks that underly most large-scale datacenter applications. More or less the case for whatever replaces TCP in a tight computing warehouse setup.
  • colechristensen 4y ago
    > Every aspect of TCP’s design is wrong The driver of most of a global network of computers which has been wildly successful beyond dreams before it was real… probably deserves a better deal than “every aspect is wrong”. It has worked fanatically well and chasing the long tail of performance improvements isn’t equivalent to determining what has gotten us here is wrong.
    • Spivak 4y ago
      Everything about the protocol being wrong for the specific case of machines directly wired to one another over a high speed reliable network is not an admonishment of the protocol in general. And the protocol, being an abstract concept, doesn’t have feeling to hurt.
    • hinkley 4y ago
      My Distributed Computing professor said, “now we are going to discuss why Ethernet is a terrible protocol but we use it anyway.” Like democracy, everything else we’ve tried is even worse.
    • teraflop 4y ago
      You're cherry-picking an interpretation of a single sentence, when it should be read in the context of the preceding one: Ousterhout says every aspect of TCP's design is wrong for (modern) datacenter computing. He's not saying bad decisions were made at the time it was designed, nor even that it's badly designed for other use cases today. The first few paragraphs of the article give even more context: > The TCP transport protocol has proven to be phenomenally successful and adaptable. [...] It is an extraordinary engineering achievement to have designed a mechanism that could survive such radical changes in underlying technology. > However, datacenter computing creates unprecedented challenges for TCP. [...] The datacenter environment, with millions of cores in close proximity and individual applications harnessing thousands of machines that interact on microsecond timescales, could not have been envisioned by the designers of TCP, and TCP does not perform well in this environment
• kevin_thibedeau 4y ago
  If only we had a stream control transport protocol and were allowed to use it.
  • wmf 4y ago
    You can use SCTP in a datacenter. Ousterhout el al. are surely aware of SCTP so I assume Homa is better in some way.
    • intelVISA 4y ago
      Forgive my ignorance but why isn't SCTP more frequently used in DCs? I know it misbehaves with home routers etc. but shouldn't be a factor here.
      • kevin_nisbet 4y ago
        I suspect there are a couple of contributors. TCP is prevalent on the internet, so you need a fairly strong motivation and benefits to adopt a second protocol. A lot of engineering also doesn't get underlying networking, so one of the successes of TCP is it's a file descriptor that you either write to or read from and magic makes it come out the other side. I've seen tech leadership on networking centric products know nothing more than you read and write and magic makes the data appear on the other side. Even on implementations that use SCTP, I've seen products that only using a single stream and mark every message as requiring in order delivery. So it was effectively what TCP offers using the SCTP protocol. At the time TCP was also far higher performance than SCTP. This wasn't so much a protocol thing, but because TCP was getting more engineering attention, it got a lot more scheduler optimization, kernel optimizations, and hardware offload support. So in many ways I think TCP scaled better due to these optimizations, which work both on the internet and internally. And then for multi-path, most data centers didn't get truly isolated networks. So if I'm running a mixture of TCP and SCTP, I still need L2 failover everywhere, which means my multi-homed SCTP connection isn't actually path diverse. And then where beneficial over the internet, there are a few success cases of using multipath TCP extensions. SCTP is still used quite a bit in the telco networks, but due to the above, it was quite a waste of time.
• Makobado1 4y ago
• ggm 4y ago
  Out of order delivery is fine in TCP within the window. It might be inefficient but it's not impossible, reassembly could be moved to userspace if userspace TCP was used. I have no problem with alternates to TCP in the DC with a crossbar fabric and far less loss, seems sensible. I wonder how it would play with QUIC and the session like behaviours now emerging.
  • kevin_nisbet 4y ago
    There was a time when out of order packets triggered congestion handling in TCP stacks which drastically reduces performance. This is where the concern comes from. I think it's a bit out dated though, I think the newer schedulers ignore out of order delivery. I've also seen problems on some embedded stacks, but that could easily be argued that the implementation is wrong. But I've seen things like credit card terminals break due to packet reordering.
    • ggm 4y ago
      I can't fault Ousterhout for writing in support of a new(ish) idea but his language here went to "forbidden" when in fact it's just "strongly disliked" TCP the protocol knows how to re-assemble out of order. What I think he's doing is making it a higher task to do it, outside of the protocol, or else providing some mechanism in user process space, amenable to threading. I can believe an async model of "tell me when this is complete" would work well with a bitmap/bloom filter type gate on what "has to be complete" to proceed. I like his writing. I was a fan of tcl/tk and used expect heavily back in the past.
    • bewo001 4y ago
      IBM AIX's TCP can either do selective ACKs or handle out-of-order packets. Which we were told when a firewall started reordering packets. It simply dropped out of order packets and therefore triggered congestion handling.
  • hinkley 4y ago
    > • In-order packet delivery This is a bit disingenuous, since it’s not the wire protocol but the kernel API that maintains the in order abstraction. With Jumbo packets you can still push a mountain of data without tripping up on “in order packet delivery” As developers we like this in order delivery to userspace because it vastly simplifies the code. We make up for the inefficiencies by processing dozens of hundreds of streams in parallel. We aren’t going to give that up just because the wire protocol changes.
• manv1 4y ago
  It's interesting that everyone (including the author) talks about UDP as a lossy protocol, but it doesn't seem that UDP drops actually occur on a routine basis anywhere. The UDP-based DDOS attacks seem to prove that; if UDP really was being dropped, those DDOS attacks wouldn't be so problematic. That said, it's an interesting read. TCP is inefficient, but that inefficiency has been patched/masked by hardware solutions. It's nice to see that someone's still thinking about this this. I remember the days when there were tons of non-IP protocols floating around (IPX, DecNET, AppleTalk, etc). TCP/IP won, which was not an obvious thing at the time.
  • vore 4y ago
    I think re UDP the point regarding it being unreliable is that you have to design your applications to take the unreliability into account, because it does happen even if it may be infrequent: assuming that it is reliable when you can get unreliable behavior will result in correctness issues.
    • PixelOfDeath 4y ago
      I see UDP more like a low level interface allowing you to build your own on top. Where you decide what packages need to be revived 100% and which one can be dropped. Basically the foundation of your very own TCP with hookers and blackjack.
      • deleted 4y ago
        [deleted]
      • russellbeattie 4y ago
        The new digital television broadcasting system in the US (ATSC 3.0) is exactly this. It's all UDP, but wrapped in another layer which allows multiple virtual streams, and that's all encoded in a CDMA wireless protocol. It's bundled up at the broadcast center, sent out via the big towers, and then unwrapped and decoded on the receiver. The end result is that once the receiver chipset has stripped off the wrapper, the OS of whatever client device is consuming the broadcast just gets regular looking UDP packets filled with MPEG-TS or DASH media streams, plus web pages, ads, games, or whatever. A.k.a. blackjack and hookers. Think of it as a giant one-way WiFi network using just UDP for the packets. It's honestly pretty cool.
• annoyingnoob 4y ago
  Homa https://dl.acm.org/doi/abs/10.1145/3230543.3230564 https://dl.acm.org/doi/abs/10.1145/3230543.3230564
• BirAdam 4y ago
  Well, the key would be to develop and deploy Homa in a DC and test in implementation at scale. If it actually ameliorates the perceived shortcomings of TCP that make nothing in TCP worth keeping as this author says, then cool. My only complain with issues like this is the cost of implementation. Someone has to pay to build a DC around it or increase the cost of maintenance for several years to a decade while supporting two completely different and incompatible networks.
  • adobrawy 4y ago
    Homa protocl can be deployed on the basis of existing switches (not all of them, but some of them will play well). See also https://github.com/PlatformLab/HomaModule https://github.com/PlatformLab/HomaModule . In addition, a closed environment and the SDN's popularity in large data centers is a significant cost-reducing factors compared to typical IPv6 deployment.
    • MuffinFlavored 4y ago
      Dumb question, there's no way to talk to a PC over the Internet with Homa, right? Since our home ISPs + routers are all only doing UDP/TCP over IPv4/IPv6? Homa is mainly for "LAN"?
      • wmf 4y ago
        Yes.
        • MuffinFlavored 4y ago
          Will we ever see the day where computers over the Internet talk to each other over something other than UDP/TCP over IPv4/IPv6?
          • wmf 4y ago
            Given Google and Apple's efforts to de-ossify the Internet, maybe. But no time soon.
• ShroudedNight 4y ago
  At a glance, this sounds very similar to AWS' SRD protocol [1] - I'm curious how they compare but I see no mention of SRD in this paper. [1] https://assets.amazon.science/a6/34/41496f64421faafa1cbe301c007c/a-cloud-optimized-transport-protocol-for-elastic-and-scalable-hpc.pdf https://assets.amazon.science/a6/34/41496f64421faafa1cbe301c...
  • 29athrowaway 4y ago
    Sounds very familiar to a lot of ideas that have been proposed as improvements to TCP over the years.
  • deanCommie 4y ago
    Strange how this is so far down the list but a more recent "nobody in the industry wants to invest in making things better" rant is higher upvoted. Clearly, AWS knows a thing or two about the datacenter industry, wanted to invest in making things better, and DID invest in making things better - and even published a paper on it.
• derrickrburns 4y ago
  Funny. I worked at AT&T Bell Labs in the 80s. All of these insights seem eerily familiar.
  • tptacek 4y ago
    Sure; they're also similar to the SCTP insights, from the late 1990s.
    • Animats 4y ago
      Yup. Also QNX native networking protocol. QNX's basic networking primitive is a remote procedure call. So there's a message-oriented network protocol underneath. It can be run either on top of UDP or directly at the IP level.
• DonHopkins 4y ago
  At first glance though he was calling for the replacement of TCL in the Datacenter! That's not what I'd Expect from John Ousterhout. ;) https://wiki.tcl-lang.org/page/Expect https://wiki.tcl-lang.org/page/Expect
  • blacksqr 4y ago
    Why not, he's had a Raft of good ideas.
• tptacek 4y ago
  "We hypothesize that flow-consistent routing is responsible for virtually all of the congestion that occurs in the core of datacenter networks". Flow-consistent routing is the constraint that packets for a given TCP 4-tuple get routed through the same network path, rather than balanced across all viable paths; locking a flow to a particular path makes it unlikely that segments will be received out of order on the destination, which TCP handles poorly.
  • hinkley 4y ago
    Or, by sending the traffic over all routes, there is no way to keep one server from monopolizing all traffic, because each route is oblivious to the stress currently being experienced by all its peers. It has to set a policy using local data, not global data. The usual failure mode for clever people thinking about software is taking their third person omniscient view of the system status and thinking they can write software that replicated what a human would do in that situation. We are still so very far from human level intuition and reasoning.
    • wmf 4y ago
      Ultimately one server cannot inject more than one link worth of traffic (e.g. 100 Gbps) into the network which is a tiny fraction of total capacity. Researchers have gotten really good results with "spray and pray" for sub-RTT flows combined with latency and queue depth feedback for multi-RTT flows.
      • hinkley 4y ago
        Spray and pray sounds like a reasonable fit for UDP, no? We’ve had these sorts of bottlenecks before, and they didn’t last. It’s always possible something fundamental changed, but it’s also possible that we are doing something wrong as the motherboard or OS levels and adopting new solutions puts us right back in that space where a couple of servers can easily saturate a network. If a network card can move data as fast or faster than the main memory bus on a computer then what are we even doing? Should we be treating each subsystem as a special purpose computer and turn the bus into a network switch?
• mlhpdx 4y ago
  > … it should be possible to bring it into widespread usage by integrating it with RPC frameworks. That’s a powerful word, “should”. Many software in the datacenter are almost as old as TCP itself, in whole or part. Difficult but working, they will continue to linger unless something more than six letters of aspiration is applied to reimagining and rebuilding that considerable bulk.
  • tptacek 4y ago
    It's theoretically much easier to introduce a new transport inside of a DC, since you're inside the network perimeter and you'll generally have control over policy-based filtering decisions.
    • KerrAvon 4y ago
      The context here is TFA advocating for use of higher-level message-oriented frameworks instead of raw socket APIs so that they can use a non-TCP transport without changing the application code.
      • vlovich123 4y ago
        Within a DC/cloud provider network. This isn’t about protocols you’d see traversing the public internet until the cloud providers see value in a protocol and start to push it out through IETF (eg see QUIC which was done by a company that owned both the browser and the data center). If there’s a “small” SW improvement that lets you use your HW 10x more efficiently that’s totally worth it given the end of scaling. You either invest in SW or pay for custom ASIC development. You’re not getting a free lunch anymore by just waiting a few years and getting that 10x gain for “free”.
• deleted 4y ago
  [deleted]
• gduzan 4y ago
  Anyone for TIPC? http://tipc.io/ http://tipc.io/
• DonHopkins 4y ago
  I'd tell you a UDP joke, but you probably wouldn't get it. So here's a TCP joke: Hello, would you like to hear a TCP joke? Yes, I'd like to hear a TCP joke. OK, I'll tell you a TCP joke. OK, I'll hear a TCP joke. Are you ready to hear a TCP joke? Yes, I am ready to hear a TCP joke. OK, I'm about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline. OK, I'm ready to hear the TCP joke that will last 10 seconds, has two characters, does not have a setting and will end with a punchline. I'm sorry, your connection has timed out... ...Hello, would you like to hear a TCP joke?
  • pstuart 4y ago
    Ack.
    • DonHopkins 4y ago
      Fin!
      • egberts1 4y ago
        Rst
        • imoverclocked 4y ago
          Ack
          • 29athrowaway 4y ago
            There's no ACK for RST. RST is the equivalent of hanging up.
            • egberts1 4y ago
              RST comes at anytime and is sent anytime.
• a-dub 4y ago
  spdy for local rpc. interesting idea. seems the easiest way to get it out there would be support for it in grpc?
• jeffbee 4y ago
  There's a little misleading point on page 2, or just a mistake. It says "Driving a 100 Gbps network at 80% utilization in both directions consumes 10–20 cores just in the networking stack" and it cites Google's Snap paper from 2019. But the Snap paper quite clearly says nothing like that. It says that Snap can drive a 100gbps NIC to 80gpbs with just 1.05 cores (Table 1) and that the whole-machine CPU load at 80gpbs in an RPC benchmark is 4 CPUs per side (Figure 6(b)). Aside from that I completely agree that TCP is trash.
  • jimmySixDOF 4y ago
    Yes that stat stood out for me too and I was wondering how to actually test this without breaking anything in the process.
    • signa11 4y ago
      > Yes that stat stood out for me too and I was wondering how to actually test this without breaking anything in the process. dpdk has been doing just that for quite some time now. perhaps you can try that and see ?
      • jimmySixDOF 4y ago
        Thanks I will look into it but on first glance their testing still seems like its under lab conditions. Thanks for the tip!
• throwaway892238 4y ago
  Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer problems! It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. And then there's all the non-backend problems! And that's just TCP. We still lack any way to communicate up and down the stack of an entire transaction, for example for debugging purposes. We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years.
  • fragmede 4y ago
    What is QUIC in your book? Given, say, $50 million of dev time, what would you go about fixing? And in what way?
    • Areading314 4y ago
      Doesn't QUIC still run over TCP? I thought it was a replacement for HTTP not TCP (Edit: looks like it replaces TCP and HTTP)
      • jhardy54 4y ago
        Nope, UDP.
      • stevewatson301 4y ago
        QUIC runs over UDP, and provides streams and encryption. HTTP/3 is designed to take advantage of QUIC streams (replacing HTTP/2 streams which were problematic due to TCP head of line blocking). The RFCs are a bit elaborate so folks interested might want to look at this instead[1], which has one of the RFC authors explaining the basics of QUIC and HTTP/3. [1] https://www.youtube.com/watch?v=cdb7M37o9sU https://www.youtube.com/watch?v=cdb7M37o9sU
• mlerner 4y ago
  I wrote a summary of one of the approaches for replacing TCP mentioned in the paper (called Homa) here: https://www.micahlerner.com/2021/08/15/a-linux-kernel-implementation-of-the-homa-transport-protocol.html https://www.micahlerner.com/2021/08/15/a-linux-kernel-implem...
• fragmede 4y ago
  I mean, part of the envoy/grpc thing is to paper over some of the shortcomings of TCP so that yes, it's still TCP underneath, but you're not setting up connections the same way. Furthermore, any improvements in the space, the envoy sidecar is well positioned to do that upgrade.
• _jvqe 4y ago
  I think protocol churn extends well past datacenters and homa, with the need for 21st century domestic protocols to remove all of the baggage from tcp/ip completely. TCP/IP could be offshored where the undersea fiber meets the shore and setup perimeter dataedge servers as the only TCP/IP nodes on domestic boundaries. For a domestic protocol, it does not need to be routable and saves even more overhead (I liked the tcp joke in the comments btw). Encapsulation over a domestic protocol would still make subscriptions to a tcp/ip service available, but IMO, domestic boundaries would be better served with a modern domestic protocol. Hypothetically speaking, if every TLD was established on its own unique protocol in the original design, Internet 1.0 would have matured much differently. Every TLD is a .com nowadays anyways, too ambiguous to be of any use (and the .org debacle still makes me laugh. They forgot what an organization was and just blended into a .com reject. That mission statement was just toilet paper after the tug-of-war). As it turns out, one size does not fit all with protocols. Multiprotocol networks within domestic borders without tcp/ip wouldn't change any of the benefits of a data network.
• cortesoft 4y ago
  I think the main thing working against any alternative is that it is easier to keep consistency at all levels, instead of trying to TCP with external connections and something else with internal.
• userbinator 4y ago
  So... basically another reinvention of UDP? I'm not entirely sure what "dataceter computing" is supposed to mean, given that a datacenter can be hosting machines for a very wide array of applications with very different network requirements.
• everybodyknows 4y ago
  Could we get the title replaced to match the document, rather than the cliched, bombastic "It's time to ..." formulation? Ousterhout actually wrote: > We Need a Replacement for TCP in the Datacenter
• cryptonector 4y ago
  > Although Homa is not API-compatible with TCP, it should be possible to bring it into widespread usage by integrating it with RPC frameworks. Not being sockets API compatible kinda sucks. Ok, we could use a new connect() variation that allows for earlier data send, but the API being mostly similar would help -- there's a ton of socket code out there! As for RPC, well, RPC is mostly a thing of the past with most everything today is HTTPS, but usually through libraries that do the I/O, so it's possible to retrofit a new transport protocol into them. But why not just use QUIC within the datacenter?
  • ElevenLathe 4y ago
    I don't think they mean RPC in the sense of rpc(3), but rather things like gRPC or protobuf. The point is that if both sides really just want to talk gRPC to each other (and never want to talk plain TCP to each other), then that use case is "easy" to meet by implementing gRPC on top of Homa.
• rkeene2 4y ago
  I wrote a blog post that might be interesting to those wanting an introduction into some of the basics of the problems called "02-FEB-2011: Why is there packet loss ?" [0] [0] https://rkeene.org/projects/info/wiki/176 https://rkeene.org/projects/info/wiki/176
  • ghshephard 4y ago
    Great post. I like the closing: Ultimately, we decided the best thing to do was to do nothing and hope for the best. All of the experienced network engineers that I've worked with who have run into issues that feel like they can be improved by tweaking QoS somewhat, always end up saying to me, "Nah. Just get a bigger/more pipes." I've never been in a position like yours to make a cogent argument as to why they were wrong, and laid out the details as well as you did there.
• hinkley 4y ago
  Often these “revolutionary” changed get deconstructed and co-opted. Not always a bad thing. Looking through a video he did about this paper, I see a few things that could be popped out. SRPT (shortest remaining processing time) is there any reason this couldn’t be implemented as a LAN protocol? Receiver driven congestion control: isn’t this what the transmit window does? Are we not just talking about setting a more aggressive starting value? “sends packets in any order” and “can only send the first few packets without a grant (ack)” are fighting each other. Especially if you’re building a message oriented protocol, which tends to have shorter conversations. This reads as confused or schizophrenic. I think of what us being said here is that the Berkeley socket protocol sucks, and that a different one can get from user space to response faster. Great. But do you have to change the wire protocol for that or just introduce a better system call library? This part in particular reads a lot like , “what if Erlang was right and we implemented it at the kernel level?” Which is not a bad question to ask.
• DeathArrow 4y ago
  Time to return to IPX protocol ?
• Animats 4y ago
  The assumptions here seem to be no speed of light lag, no packet loss, and no security. The only problem is congestion. That's more like the interconnection fabric of a single-purpose supercomputer than a general-purpose data center. Which is probably why they mention Infiniband, a hardware interconnect for supercomputers, so much. Would this break down if you had to start talking to a remote machine in another data center? That's how outages and overloads are handled, after all. This is an interesting idea, but it's for a relatively narrow use case.
  • ninth_ant 4y ago
    It’s a narrow use case sure, but at the same time it’s an important one to a number of companies with significant engineering resources.
• Cupertino95014 4y ago
  My take: this is the making of a great essay question for a graduate course in networking. The chance of TCP being thrown out wholesale is zero. What might happen is some slow, incremental improvements in certain aspects of networking, like some other comments suggest. Until eventually, a networking student picks up a dusty old TCP doc and says to his teacher, innocently, "But we're not doing hardly any of this stuff anymore!"
• lobota 4y ago
  Ok I watched the Playlist movie about Spotify and they mentioned they fork TCP IP and made it better, that was a incorrect statement because under the hood everything on the internet still runs TCP/IP they probably just improved their application level or move to UDP
• ng55QPSK 4y ago
  Do we know, where it is submitted?
• curious_cat_163 4y ago
  "Although Homa is not API-compatible with TCP, it should be possible to bring it into widespread usage by integrating it with RPC frameworks." I was about to rant that Prof. Ousterhout should just deploy some of his students and get that transport - RPC integration done and prove out his point. But, then I tried to look for it first and found this: https://www.usenix.org/system/files/atc21-ousterhout.pdf https://www.usenix.org/system/files/atc21-ousterhout.pdf Has anybody tried it in an actual data-center?
• jl2718 4y ago
  The ultimate dream protocol is one in which a sender just encodes bits in a certain way such that the receiver will get them, and puts them on the line without any handshaking or synchronization. I don’t think this is impossible. The space of orthogonal codes across time and frequency could be chosen to be practically infinite, therefore, any random selection of two such codes would look like white noise to each other. The receiver would have to listen on a large subset of such channels all at once, which is not practical in real-time, but could be practical looking backward at the stored waveform from some carrier channel that all such possibilities have in common. It would commonly miss single bits and large chunks of data, so it would have to have FEC across multiple scales of code, frequency, and time. This works for large messages, but smaller messages would have to be sent over a channel with bandwidth narrowed to consume the time window of detection. Thus you should have a fair guarantee that either every message will be received during the detection window, or no message will be received, and this could be the job of network infrastructure to monitor and buffer as necessary. If that fails, then, well, maybe let the application layer deal with it.
  • cryptonector 4y ago
    > The ultimate dream protocol is one in which a sender just encodes bits in a certain way such that the receiver will get them, and puts them on the line without any handshaking or synchronization. This is a recipe for DDoS. Some handshaking is always necessary. You can minimize it, but you can't get rid of it.
    • zokier 4y ago
      > This is a recipe for DDoS. Inside a datacenter?
      • cryptonector 4y ago
        Oh, right, in a datacenter probably not. EDIT: But you know, UDP meets your bill.
        • 4y ago
• josephg 4y ago
  > The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport. But, because we can't have nice things, message framing has to be re-implemented on top of TCP in a different, custom way by every single protocol. I've basically got message framing-over-TCP in muscle memory at this point, in each of the variants you commonly see. The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). In my opinion, the real problem is the idea that "everything is a file". Byte streams aren't a very useful abstraction. "Everything is a stream of messages" would be a far better base metaphor for computing.
  • langsoul-com 4y ago
    What do you make to require implementing or making network protocols?
    • josephg 4y ago
      Multiplayer video games, realtime updating webpages, database bindings, sync protocols (I play around around with CRDTs a lot), p2p distributed systems, whatever really! Implementing a wire protocol seems to come up about once every couple of years. And for context, I've been programming now for about 30 years.
  • cryptonector 4y ago
    > The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). No, HTTP/2 and QUIC do not change the semantics of HTTP. Also, you can have endless streams with HTTP/1.1: just use chunked encoding to POST/PUT and use Range: bytes=0- and chunked encoding for GET and chunked encoding for POST response bodies. In HTTP/2 there's only the equivalent of chunked encoding -- there's no definite content length in HTTP/2.
• trustingtrust 4y ago
  The original darpa paper is what the TCP/IP stack is still based on right ? It feels like it was never even intended to be at the scale its deployed on. Which amazes me to be honest that people have somehow gotten it to work to this scale.
  • topranks 4y ago
    The congestion control algorithms have been significantly improved since then. But yes it’s a testament to how good the original design is that it’s still what we use today. Protocols like SCTP and QUIC work similarly but can avoid head-of-like blocking.
• Luker88 4y ago
  A few years ago (when QUIC was coming out) I was developing the theory of a new transport/encryption/authentication protocol. The focus was as much on transport as in the built-in federated authentication. There was not much interest in the field and I had a lot of the theory and formal proofs, but no implementation. This month I found a cofounder and we are reordering a lot of the information and presentation, we should start asking for funds in more or less a month. I still believe my solution to be much more complete than anything in use today (again: on paper), but since there seems to be some interest today, I'll ask here: Can anyone suggest some seed funds to check for a starttup? We will be based half US, half EU. For more details, fenrirproject.org (again: old stuff there, ignore the broken code)
  • kanwisher 4y ago
    Why don’t you actually build gasp a prototype before asking for money
    • Luker88 4y ago
      yeah, thank you for the kind comments about not needing money (aka: my time as no value) and asking me to build the prototype with irony. As I said, the project was started a few years back, and since I did not have the time to work on it, maybe it means my life does not give me the time and money to build this on the side. But I'll always find it funny how half of the people go "you need to have solid theory proofs before" and the other half goes "where is the working code". As I said we just started some housekeeping and are not ready to start, and as many point out, it's hard to make money on infrastructure. I know, I did not ask how to make money on this. The idea is to keep the base as open as possible and make money on other service built on this. I was only asking on pointers to funds interested on tech loosely connected to this. And if they don't like our current state or something else fine, no need for you to do their job, and in a witty way, too.
      • kortilla 4y ago
        > yeah, thank you for the kind comments about not needing money (aka: my time as no value) and asking me to build the prototype with irony The world (read: the investment community) doesn’t care about your time though. If you are going to pitch an infrastructure project that should work in software without a working demo, you’re going to have to take a huge haircut on valuation. If you want to be successful with a theory and proofs, join academia and publish them. If you want to get investors for infrastructure, make something that works. Academia churns out “protocols on paper” every year that go absolutely nowhere. You need to differentiate yourself if you’re looking for more than a research grant.
• kreetx 4y ago
  What about RINA: https://en.wikipedia.org/wiki/Recursive_Internetwork_Architecture https://en.wikipedia.org/wiki/Recursive_Internetwork_Archite...
• pulse7 4y ago
  "Homa demonstrates that it is possible to create a transport protocol that avoids all of TCP’s problems." - This is a huge statement... It assumes the author knows all of TCP’s problems...
• Matthias247 4y ago
  > Every significant element of TCP, from its stream orientation to its requirement of in-order packet delivery, is wrong for the datacenter. It is time to recognize that TCP’s problems are too fundamental and interrelated to be fixed; This seems like a pretty bad way to start a paper. It throws an extremely strong assumption into the room without backing it up by data. Having worked in the Cloud/Datacenter space for lots of years, I really have a hard time describing any situations where TCP limited the performance of applications and not anything else. It doesn't matter that much if a slightly different networking stack could lower RTT for from 50us to 10us if the P99.9 latency of the overall stack is determined by garbage collection, a process being CPU starved, or being blocked on disk IO or another major fault. Those things can all be in the 100ms to 1s region, and are the real common sources of latency in distributed systems. The main TCP latency problem that I've experienced over the years is SYN or SYN-ACK packets getting dropped to due overloaded links or CPU starvation, and the retry from the client only happening after 1s. Annoying, but one can work around a bit racing multiple connections. Besides the TCP handshake time there's also another round trip for setting up a TLS connection - sure. But both of those latencies are in practice worked around with connection pooling. Speaking of TLS - I can't find a single reference to it in the paper. And talking about datacenter networking without mentioning TLS sees to miss something. Pretty much every security team in a bigger company will push on TLS by default - even if the datacenter is deemed a trusted zone. How does it matter if the TCP connection state is 2kB and the HOMA state is less, if my real TCP connection state is anyway at least another 32kB for TLS buffers plus probably megabytes for send and receive buffers plus whatever the application needs to buffer. Last thing I would like to mention is that datacenter workloads are not "just messaging", and the boundary from messsaging to streaming is pretty fluid. What happens if the RPC calls fetches the content of a 10MB file? Is that still a message? If we treat it as such, it means that the stack needs to buffer the full messsage in memory at once, whereas with TCP (and e.g. HTTP on top) it can be streamed, with only the send buffer sizes being in memory. What about if it is 1MB? We could certainly argue that some applications just transfer a few bytes here and there, but I'm seriously not sure if I would label those as the majority of datacenter applications. And with the typical practice of placing a lot of metadata into each RPC call (> 5kB auth headers, logging data, etc) even the smallest RPC calls are not that small anyore.
• WiSaGaN 4y ago
  I don't really get the hate for TCP here. No one is forcing others to use TCP. It is perfectly fine to just use UDP if TCP is not the right fit for your scenario, and build whatever semantics you need on top of that. QUIC did it. What can't the author?
• pantulis 4y ago
  I am not really sure what Ousterhout means when talking "datacenter" but one of the key aspects of TCP networked applications is that it really doesn't matter if they are located in a datacenter, at the edge, in a mobile device, in a Raspberry at home or in your car and, better, they can be more or less moved from one hosting to another. Will this mean that application developers need to work with different network stacks at the same time?
  • wmf 4y ago
    Yes, you would have multiple protocols at some points. If you look at microservice architectures they already have an internal service mesh and external API gateways.
• jokoon 4y ago
  Why not just use UDP instead? I feel TCP is just designed to send files and data segments, but it doesn't work well for other things.
• rossmohax 4y ago
  > It uses several techniques for this, of which the most notable is that it takes advantage of the priority queues provided by modern switches. I'd understand if they said routers, but switches? Do L2 switches have any notion of priority and if yes how does it work?
  • kijiki 4y ago
    Typically you map a VLAN to a priority via PFC (Priority Flow Control). You can do it with vconfig on Linux. Switch OSes have their own CLI for this. Some switches can do PFC for untagged packets. They classify based on DSCP, and map that to a PFC priority. I've never used that though.
• netfortius 4y ago
  Homa over EVPN-VXLAN, across multiple data centers - what could possibly go wrong? See [1] (4) [1] https://www.rfc-editor.org/rfc/rfc1925 https://www.rfc-editor.org/rfc/rfc1925
• sriku 4y ago
  Ourtserhout's paper perhaps comes from his vision for RAMCloud as well ... in which he bet on network latency going ultra low over time that accessing memory on another machine was fast enough to enable whole new categories of applications to emerge. https://dl.acm.org/doi/10.1145/2806887 https://dl.acm.org/doi/10.1145/2806887
• sylware 4y ago
  until this replacement is as stable in time and simple to implement, no worries. But if it is a stability joke or need 748937493874943 devs to code an alternative...
• KaiserPro 4y ago
  I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would move to RDMA over infiniband. For shared memory clusters, thats what's been done for years. but for general purpose computing its pretty rare. Most of the time its not worth the effort. Infiniband is cheap now, so its not that hard to deploy RDMA[1] type interconnects. Having a reliable layer2 with inbuilt flow control solves a number of issues, even if you are just slamming IP over the top. shit, even 25/100gig is cheap now. so most of your problems can be solved by putting extra nics in your servers and have fancypants distributed LACP type setups on your top of rack/core network. The biggest issue is that its not the network that's constraining throughput, it either processing or some other non network IO. [1]I mean it is hard, but not as hard as implementing a brand new protocol and expecting it to be usable and debuggable.
  • amluto 4y ago
    The paper explicitly addresses Infiniband.
    • KaiserPro 4y ago
      not really. they conflate infiniband with RoCE which given they have different semantics on congestion control, I'd say is a bit of a whoopsey. if they are using RoCE, are they using DCB to avoid loss(well make it "lossless")? the paper implies otherwise.
      • birdyrooster 4y ago
        For those who don’t know, RoCE is somewhat of a failure in the marketplace right now.
        • throw0101a 4y ago
• nickdothutton 4y ago
  23 years ago I sat in a meeting with Sun, Intel, Mellanox, and 1 or 2 others. In that meeting we discussed putting an RDMA interface on individual hard drives, trays of RAM, CPUs, and other more exotic devices (like battery backed RAM, no conventional SSD in those days of course). You’d install RAM 1 42U rack at a time, disks likewise, CPUs in another rack and so on. All partitioned, controlled, managed, and of course billed-for by a “data center OS”.
  • EricE 4y ago
    The network is the computer....
  • tristor 4y ago
    I was in the room for similar discussions as part of the OpenCompute project, which was around remote IO and resource disaggregation. There are systems like this, and they may make sense for certain use cases, but generally speaking hyperscalers today are built around virtualization which doesn't align well to this model.
  • wmf 4y ago
    Disaggregation costs an absolute fortune. The network is 10% of the total datacenter cost and the network does not carry memory and PCIe traffic. If you make the network 10x faster to carry that, now it's 90% of the cost.
• andersced 4y ago
  We have been testing out various protocols to overcome in our case TCP head-of-line blocking by using the protocols-> SRT: https://github.com/Haivision/srt https://github.com/Haivision/srt (C++ wrapper https://github.com/andersc/cppSRTWrapper https://github.com/andersc/cppSRTWrapper) RIST: https://code.videolan.org/rist/rist-cpp https://code.videolan.org/rist/rist-cpp KCP: https://github.com/Unit-X/kcp-cpp https://github.com/Unit-X/kcp-cpp We wrap all data in a common container format https://github.com/agilecontent/efp https://github.com/agilecontent/efp To decouple the data from the transport. Yes the above solutions are media centric but can be used for almost any arbitrary data. The protocols are not 'fare' so starvation may happen, and must be handled on the application level. /A
• lizknope 4y ago
  For those unfamiliar with the author. https://en.wikipedia.org/wiki/John_Ousterhout https://en.wikipedia.org/wiki/John_Ousterhout He is probably most famous for having created the Tcl language and Tk GUI library. He also worked on the Sprite distributed operating system, the Magic VLSI design tool, and a bunch of other things.
  • eigenrick 4y ago
    Also, more recently, one of the inventors of the RAFT algorithm.
• brainzap 4y ago
  I always assumed google datacenter has their own protocol already
• 752963e64 4y ago
• rafaelturk 4y ago
  UDP?
• JoeAltmaier 4y ago
  Google hasn't used TCP in the datacenter for years. What they use I don't know. But it's even custom switches with custom chips. My son did work in graduate school for a clean-slate network implementation of a network for the datacenter. Maybe Google, I don't remember. One issue I remember they addressed was, scheduling bandwidth for VM migration within their datacenter cloud. See, some customer reserves a 'machine' for their services but really they get something like a VM slice of a ginormous machine (Multiple TB memory, 100 cores or whatnot). Each customer gets some of that and thinks it's a machine of their own. That customer slice shares the larger machine with maybe 10-100 other customers. Then somebody's slice starts to use more resources and has to be moved to a machine with more 'room'. That wants to be fast and seamless. It can be maybe 1TB of stuff. Their slice doesn't want to be interrupted for long. So this machine needs bandwidth that isn't subscribed for the migration. So does the target machine. So does the cloud network. Then all the addresses have to be re-homed. Another issue: those competing slices need a virtual network adapter. They each think they own one (each is running a copy of linux or whatnot), but it has to be a physically shared and rationed device. All while using the TCP abstraction on a network-adapter abstraction on a driver abstraction, but really on their new network hardware that's actually present on the ginormous machine. This includes all the TCP features plus the bandwidth reservations the cloud needs etc. So yes it's abundantly obvious that the datacenter needs (has) a new network.
  • delroth 4y ago
    > Google hasn't used TCP in the datacenter for years. That's absolutely false. I don't have any sources except for having worked at Google from 2013-2022, but it's not like you quoted any sources either, so... There's a reason why Google is still releasing stuff like TCP BBR (2017).
    • JoeAltmaier 4y ago
      Well, the first link googling 'google datacenter hardware' is google's article on how they don't use standard TCP hardware or software in their datacenters. But I guess that was too much to ask...
• HippoBaro 4y ago
  I think that was an interesting read. I have worked on an userspace implementation of TCP via DPDK so I have sympathy for the limitations mentioned (the load balancing and thread scheduling argument are very accurate). However I would have liked a section dedicated as to why hardware accelerated UDP wouldn’t be an adequate solution vs a whole new protocol. It seems to me it provides a solid basis for achieving the results the author wants to bring about?
• LinuxBender 4y ago
  I believe something missed in these discussions is that pertaining to "the data-center". A data-center is not a technology, it is a grouping of assets. Those assets and their associated services need to communicate not just with each other, but with other assets and services on the internet. Regardless of what incredible technical solution one creates, it will have to allow for simultaneous existence of current IP protocols in parallel with whatever proposed replacements to exist seamlessly with one another or significant adoption would never occur. A data-center is not an isolated bubble, at least not any more unless one wants to translate said protocols through a single point of success gateway. Should such a replacement ever occur it will have to be done piece by piece until there is nothing left using current IP protocols. So I believe people should all create their proposed protocols and give businesses a low-friction path to adoption one service at a time. As more applications adopt said protocols, the most popular, highest reliable, most performant least friction path will likely win and if successful then at some distant point in the future perhaps most existing IP protocols could be deprecated. As a reminder, each application will have to adopt libraries to speak on this protocol and know how to utilize it. There will be a "battle hardening" period to work out the bugs and security controls. All of the network gear in the entire path between data-centers and clients will need OS/firmware/Asic updates to understand this protocol. Given the transition speed to IPv6 as an example, this could be a very long road. There is also some discussion of QUIC and SPDY. Those are not new IP protocols. Those are new standards within an existing L7 application protocol HTTP that still utilize existing L3/L4 protocols. Replacing TCP and UDP? in the data-center means a new protocol in /etc/protocols not encapsulated in an existing protol such as protol 6 TCP or protocol 17 UDP. The network gear and OS on every device in the path will need to understand this new protocol. Tools such as tcpdump and libraries such as libpcap would need to be updated to understand these new protocols before they would even be used in a development environment. Could it be that I misunderstood the intent and perhaps we just want yet another new L7 application protocol on top of UDP?
• jmull 4y ago
  I get the issue with TCP, but I'm not sure about Homa... e.g., why not UDP with some homa-like semantics on top? That might really ease the "Getting there from here" issue. In fact, what's actually being suggested is for applications to replace calls to TCP-based APIs with calls to gRPC APIs (or other high-level RCP APIs), where the transport layer becomes an implementation detail. Fair enough, but this is a very round-about way to go about it.
  • eigenrick 4y ago
    It would be interesting to see a write-up from them on Homa's benefits over UDP. UDP has been used to work-around issues with TCP both in the datacenter and in unreliable WANs. Skimming over the paper, I think the magic of Homa is in its RPC calls and its short-lived connections. When they're handled at layer 3 and 4, they can provide a significant hint to switches, routers and hosts regarding prioritization and congestion control. If you agree upon the prioritization algorithms as part of the protocol, then both sending and receiving hardware can coordinate much more easily. If we instead just implemented something like Homa on top of UDP, it would basically mean that the top of the OSI stack would have to somehow inform the lower layers of the stack about these "sessions". You'd also have to hope that 3rd party peers decide to implement the hints in the same way. This would result in much more complexity.
    • jmull 4y ago
      Here's what I'm thinking: You add a homa-like header inside a UDP packet. Inside the data center you use switches, nics, etc that know and understand the home-like protocol and can implement homa-like behavior... as needed. Anywhere else, you can fall back to UDP, due to its ubiquity. Yes, various things would have to somehow know the homa-like protocol was being used... just like various things would have to somehow know the homa protocol was being used. Yes, different vendors would have to have compatible homa-like implementations... just like different vendors would have to have compatible homa implementations. I think the complexity you mention is inherent in anything that actually gets more widely deployed and used.
• GartzenDeHaes 4y ago
  1 security vendor flagged this URL as malicious https://www.virustotal.com/gui/url/43f33fe70cb4ef9fcc23704606499f2714abb17d4c20316ae481af067307dd7e https://www.virustotal.com/gui/url/43f33fe70cb4ef9fcc2370460...
• citizenpaul 4y ago
  TCP is basically exists to deal with an unsolvable problem, "the two generals" Its easy to look at it and say it could be better because you have tunnel vision for your use case. However its also easy to forget that the protocol literally has a billion edge cases. Why do you think your NIC has a new driver update every couple of weeks even though they have been running the same protocols for the last 50 years?
• throw0101a 4y ago
  One of the reasons we're 'stuck' with TCP and UDP on the Internet is because most middle-boxes (firewalls) don't really understand anything else. If you're strictly operating inside a DC, presumably with minimal/fewer firewalls, could alternatives like SCTP and DCCP be an option? * https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr... * https://en.wikipedia.org/wiki/Datagram_Congestion_Control_Protocol https://en.wikipedia.org/wiki/Datagram_Congestion_Control_Pr... * https://en.wikipedia.org/wiki/Transport_layer https://en.wikipedia.org/wiki/Transport_layer
• deleted 4y ago
  [deleted]
• dragontamer 4y ago
  Hmm, haven't read the paper yet, but I immediately did "Ctrl+F sctp" and didn't find anything. I know that sctp was the next generation stream-oriented protocol designed to fix the out-of-order message problem in commnications, as well as a whole bunch of connection issues (4-way handshake instead of 3-way for better open/close. Datagram oriented in-order stream, so that every packet has a proper size involved. Etc. etc.) As far as I know, sctp should solve all the requirements in section 2 of this paper (except "load balancing", which might be solved by lower-level protocol sharing of some kind?). So its weird to not see sctp discussed. -------- Yeah, sctp ain't popular, but these exact sets of problems / requirements and issues with TCP have been known for decades. SCTP, is also a decades-old protocol (though not as old as TCP), and is the most obvious solution to the problem (and already supported by Linux).
• musicale 4y ago
  Haven't data centers switched to RDMA already? Why are we still wasting time with this networking nonsense when most of the time we're just copying data from one memory or cache to another, over a private interconnect? ;-) It seems that Osterhout's complaints about RDMA are about current RDMA implementations. I expect many of them are fixable. Osterhout's complaints about TCP are all valid, though he doesn't mention my pet peeve with TCP, which is that connectivity breaks when your IP address changes. And requiring apps to deal with IP addresses in user-level APIs seems like a mistake. Simple request-response RPC protocols were a good idea in the 1980s, and they're still a good idea today. I should probably read the Homa paper(s) regarding congestion control though, as it isn't covered in this PDF.
• westurner 4y ago
  FWIU, barring FTL; superluminal communication breakthroughs and controls, Deep Space Networking needs a new TCP, as well: From https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_timer.c https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_t... : void tcp_retransmit_timer(struct sock *sk) { /* Increase the timeout each time we retransmit. Note that * we do not increase the rtt estimate. rto is initialized * from rtt, but increases here. Jacobson (SIGCOMM 88) suggests * that doubling rto each time is the least we can get away with. * In KA9Q, Karn uses this for the first few times, and then * goes to quadratic. netBSD doubles, but only goes up to *64, * and clamps at 1 to 64 sec afterwards. Note that 120 sec is * defined in the protocol as the maximum possible RTT. I guess * we'll have to use something other than TCP to talk to the * University of Mars. * * PAWS allows us longer timeouts and large windows, so once * implemented ftp to mars will work nicely. We will have to fix * the 120 second clamps though! */ /? "tp-planet" "tcp-planet" https://www.google.com/search?q=%22tp-planet%22+%22tcp-planet%22+%22SCPS-TP%22 https://www.google.com/search?q=%22tp-planet%22+%22tcp-plane... https://scholar.google.com/scholar?q=%22tp-planet%22+%22tcp-planet%22+%22SCPS-TP%22 https://scholar.google.com/scholar?q=%22tp-planet%22+%22tcp-...