9 ms·

Is it time to replace TCP in data centers?

104 points by zdw 4y ago

LinuxBender 4y ago
Is It Time to Replace TCP in Data Centers? Probably not? Maybe? One would have to either integrate support for new protocol into every server, switch, router, IoT oh and of course all the 3rd party clouds and clients one is speaking to. OR everything leaving the datacenter would have to be dual-stack and/or funneled through some WAN optimizer/gateway/proxy device that can translate new protocol into TCP/IP creating a single point of success bottle-neck. Dual-stack brings up some security issues that this protocol will have to address not to mention more cabling complexity. I think the best place to start this conversation would be with architects at Microsoft, IBM/Redhat, maybe even Meta since they acquired several kernel developers and let them see how cost effective the gains are. If the big players buy into this and they have kernel developers that can integrate seamless support into Linux and Microsoft to start with and a few big players try it out then maybe it would share the same market cap as Infiniband. Let them deploy a proof of concept pod for free for a year and see what they can do with it. I think this would have to be successful first before other vendors start adding support for new protocols. If the plan is to have one vendor to rule them all then it will not succeed as there would be no competition and it would be too expensive for mass adoption. This would end up being another proprietary thing that IBM or some other big company acquires and sits on. At least that is my opinion based on my experience deploying SAN switches, proprietary memory inter-connect buses, proprietary storage and storage clustering, proprietary mini-mainframes and server clusters. Speed improvements will impress technical people but businesses ultimately look into TCO/ROI and reliability. Complexity is a factor in reliability and the ability to hire people to support said new thing. If anything I have seen datacenters going the opposite direction; that is, keeping things as generic as possible and using open source solutions to scale first to their vertical limits and then horizontally. Ceph is a great example of this.
- blibble 4y ago
  you can run a new protocol on top of IP "relatively" easily: it routes fine across the internet as switches and routers don't care about TCP[1] (switches don't even care about IP) linux will even let you do it from userspace (with net_admin cap) there are some exceptions: like if your endpoint is some a crappy cloud provider (e.g. azure) that provides you something that looks like an ethernet network but really is a glorified TCP/unicast UDP proxy [1]: ignoring NAT and IGMP/MLD snooping (not that anyone does those outside of internal networks... right?)
  - xxpor 4y ago
    routers care about tcp (or udp) because of flow hashing. write a new IP proto and see what happens when you try to push >100 gbit
    - blibble 4y ago
      you have 3 out of 5 parts of the standard tuple not perfect but not unworkable either
      - toast0 4y ago
        It can be necessary to get flow hashing on high volume server A : server B communications (especially if you're running aggregated interfaces and are using multiple connections). If you need that in a 3-tuple hashing environment, you're going to need to apply many addresses to one or both servers, which is going to cause problems with arp/ndp cache sizes. Maybe that's ok if you just route a /64 to each server and don't use NDP inside the /64; but you probably need to work on the server OS to work with many addresses well, things get a little shifty there.
    - pezezin 4y ago
      Isn't that why IPv6 includes an explicit flow label in the header?
jmclnx 4y ago
I say no, and I do not see how this could ever happen. Just look at IPv4 to IPv6 to see an example on how this would go :)
- Dylan16807 4y ago
  Anything with packets would still be based on normal IP so you wouldn't have those issues. In the worst case, you just build on top of UDP and don't care that you're wasting four bytes.
paxys 4y ago
Unless the alternative is a 100% backwards compatible drop-in replacement for TCP, even starting to discuss it is pointless. And if it is compatible at that level, it will just bring with it all the pitfalls that we want to replace in the first place. The TCP/IP stack works because I don't need to care about what environment the two processes that need to communicate are running in. They could both be on my local machine, or in my home network, or communicating over the internet, or some random intranet, in a data center, across continents, on any OS or any kind of device...it simply does not matter. "Just do a ground-up rewrite of your entire software stack and you'll get a guaranteed 5% efficiency gain" isn't the bullet proof argument that people who come up with these alternatives seem to think.
- foota 4y ago
  This is only really true of external connections, for orgs with large degrees of control over their internal stack this can make sense.
- readingnews 4y ago
  Correct. I agree. Well said.
- mike_d 4y ago
  Suggested alternative title: "Is it time to replace oxygen in the atmosphere?"
- ithkuil 4y ago
  I wonder if we need to replace TCP in the data centers or if what data centers really need is RPC (which ia covered well with QUIC)
  - StillBored 4y ago
    Right, I've been saying this for a while, it seems that many of the hyperscalers/etc made the mistake of trying to utilize TCP (and maybe HTTP/etc) for their (generic) RPC mechanisms. Which is hardly the best plan, when what is actually needed is just a reliable datagram protocol without a bunch of connection state. So, while the actual RPC protocol has its own issues, the one thing they got right was the ability for the portmapper to indicate UDP vs TCP as the transport on a service basis. There have been a few improved generic RPC mechanisms, and I don't really understand why some of these places feel the need to "replace TCP" when really what they need is a more formalized RPC mechanism that can set/detect the datagram reliability and pick varying levels of protocol retry/etc as needed.
omani 4y ago
Betteridge's law of headlines.
- omani 4y ago
  I love it when I know I will be downvoted. that's why I posted this. you know who I am.
  - ben0x539 4y ago
    We've all heard of the law. It's a cute observation but everybody is capable of making it independently and commenting with a reminder on every article with a yes/no question as the title is just noise.
    - omani 4y ago
      understandable. and I agree. it was more like a test to see how the bots/npcs would react (with their downvotes).
likeabbas 4y ago
Doesn't HTTP3 solve this with QUIC?
- exabrial 4y ago
  It does until you try to tcpdump :/ 1) It's not TCP 2) It's a PITA on a 'trusted' network because there is not "encryption off" mode
  - olodus 4y ago
    Those are very solvable problems. I work with non-TCP traffic in pcaps very often and while I agree encrypted traffic is annoying to deal with sometimes because of the extra step it can be solved with good tooling.
  - zamadatix 4y ago
    tcpdump works fine with it, and any other ethernet or IP based protocol despite the commands name, it's just 2) that's a problem (for certain perspectives of what is a problem).
  - erik_seaberg 4y ago
    As long as governments attack networks with impunity, it’s a mistake to trust them.
    - shawnz 4y ago
      See this well-known slide from the PRISM leaks as a reminder of this: https://upload.wikimedia.org/wikipedia/commons/f/f2/NSA_Muscular_Google_Cloud.jpg https://upload.wikimedia.org/wikipedia/commons/f/f2/NSA_Musc...
    - KaiserPro 4y ago
      Its a question of choosing when and how you encrypt. For internal datacentre traffic, it depends on the value of the data. But you will need to do application level encryption. For links between datacentres, encrypting links based on rules is a sensible and not horrifically challenging thing to do. If you're big enough to have to worry about the volume of encrypted traffic, then you have rules based bandwidth priorities. for virtually everyone else, just use wireguard or some other VPN with encryption.
shanemhansen 4y ago
This seems like a review of John Ousterhout's work w/ Homa. I highly recommend reading the original. https://arxiv.org/abs/2210.00714 https://arxiv.org/abs/2210.00714 For those who don't know Ousterhout created the first log structured filesystem and created TCL (used heavily in hardware verification but also forming the backbone of the some of the first large web servers: aolserver). I was actually surprised to find out he co-founded a company with the current CTO of Cloudflare. https://en.wikipedia.org/wiki/John_Ousterhout https://en.wikipedia.org/wiki/John_Ousterhout He has both a candidate replacement as well as benchmarks showing 40% performance improvements with grpc on homa compared to grpc on TCP. https://github.com/PlatformLab/grpc_homa https://github.com/PlatformLab/grpc_homa With that in mind I think nobody will replace TCP and I doubt anything not IP compatible will be able to get off the ground. His argument is essentially that for low latency RPC protocols TCP is a bad choice. We've already seen people build a number of similar systems on UDP including HTTP replacements that have delivered value for clients doing lots of parallel requests on the WAN. I think many big tech companies are essentially already choosing to bypass TCP. I recall facebook doing alot of work with memcache on udp. I can't find any public docs on whether or not Google's internal RPC uses TCP. I wouldn't be surprised at all if in the near future something like grpc/capnproto/twirp/etc had an in-datacenter TCP-free fast path. It would be cool if it was built on Homa.
- zarkov99 4y ago
  TCP is rarely used in HFT data centers except as required by exchanges.
- tyingq 4y ago
  Ousterhout was also one of the co-authors of the Raft consensus paper.
- killingtime74 4y ago
  He also wrote one of my favorite software books https://web.stanford.edu/~ouster/cgi-bin/book.php https://web.stanford.edu/~ouster/cgi-bin/book.php. Presentation https://youtu.be/bmSAYlu0NcY https://youtu.be/bmSAYlu0NcY
- wpietri 4y ago
olodus 4y ago
Since one of the suggestions the article brings up is QUIC then won't it happen almost automatically? Atleast the distributed systems I work with usually have http between their parts (a few with some other RPC solution). Then when the switch happens to http 3 QUIC will come with it. There will probably be some problems along the way sure and will take time for people to tune thibgs, but that must be expected right?
- topranks 4y ago
  QUIC involves encryption though. If that’s not needed in a given scenario the overhead might not be desirable. But otherwise, yes, it addresses many of these problems.
WesolyKubeczek 4y ago
There has been a few of TCP would-be killers out there, SCTP and DCCP come to mind. And yet, here we are. I even remember that there have been Linux kernel security vulnerabilities in those protocol somehow exploitable even when you don't actively use them, so... I don't compile my own kernels often these days, but when I do, I usually say N to any and all protocols I'm not likely to be using in my remaining life.
deleted 4y ago
[deleted]
nickdothutton 4y ago
I see we’re doing this again. I last worked on whole-DC designs around 1999 with Sun, Intel, and of course… Mellanox since we were to use RDMA/InfiniBand. The plan was to embed InfiniBand semiconductors on disks, RAM (individual sticks or trays of sticks) and CPUs. There was to be an orchestration/provisioning layer allowing you to reserve a composable machine (CPU(s), RAM, storage, other IO, etc) at will, and some kind of supervisory layer was to ensure that these things were as local as possible to one-another in any given rack, aisle, floor.
- Traubenfuchs 4y ago
  So you were pretty much working on a more hardware based predecessor of kubernetes? Why did it not work out?
  - SgtBastard 4y ago
    It wasn’t written in Go </snark> I’d also love to hear more about the concept.
    - sacnoradhq 4y ago
      Go peaked in 2017. Rust is the new hotness. 3 years from now, it will be something else.
  - bobleeswagger 4y ago
    > Why did it not work out? Software won, it seems.
    - KaiserPro 4y ago
      kinda, It was cheaper to use fibrechannel/ethernet and plain old sparc/intel machines This was about the time that grid engines promised mainframe computing without the cost. Only that it required people to understand how to parallelise their workloads. That and modify programmes to run in on many machines at once.
  - 4y ago
manv1 4y ago
"Is it time to spend hundreds of billions of dollars on new networking equipment for marginal gains? Pundits say yes."
- dokem 4y ago
  This is our punishment for violating the osi model. Why should network gear need to know about above the ip layer?
  - manv1 4y ago
    The OSI model was standardized in 1984. The first TCP RFC came out in 1980, with versions existing in real life before that. The OSI model isn't, and has never been, the be-all end-all of networking models, and it's bizarre to me that people continue to believe that the OSI is the only valid model to follow. Maybe you can explain why you believe that the OSI model is canonical?
    - dokem 4y ago
      What other model(s) are you referring to? TCP/IP? that still has identical ip/transport separation so can you explain why it's a good idea to so blantly violate fundamental layers of abstraction?
- wmf 4y ago
  Homa uses existing equipment.
Animats 4y ago
There are many ways to improve on TCP for specific use cases. You can get maybe another 10%-20% of performance, as Google did with QUIC. Used outside that use case, performance may be worse. The design goal of TCP was that it should Just Work. It needed to be interoperable between different vendors, provide reasonably good performance, work over a wide range of networks speeds, work over possibly flaky networks, continue to perform under overload, and not have manual tuning parameters. It does well against those criteria. This new protocol the sort of thing Osterholt is known for. He came up with "log" file systems, which are optimized for the case where disk writes outnumber reads. Reading then requires more seeks on mechanical drives. If you're mostly writing logs that are seldom read, it's a win. If you're doing more reading than writing, it's a lose.
- toast0 4y ago
  > There are many ways to improve on TCP for specific use cases. You can get maybe another 10%-20% of performance, as Google did with QUIC. Most of the gains of QUIC are in exchanging data with the first round trip, which you can do with TCP fast open as long as middleboxes don't mess it up; different congestion control, which you could do in TCP if you control the OS on the sending side as long as the middle doesn't mess it up, which shouldn't happen in a data center; and maybe some benefits for connection switching, which should hopefully not be necessary in a data center (although I've certainly seen cases it would help!); multi-path TCP also addresses this, but has probably worse CPU efficiency problems: if a logical flow is coming in on multiple NIC queues, there's going to be cross-cpu communication on the data structures. Unless it's changed, QUIC also simply won't handshake if the path MTU is too small, so you don't get to have the excitement of path MTU discovery failures; but you really shouldn't have that problem in a datacenter.
floatinglotus 4y ago
Replacing TCP in the DC is a lot easier now with the advent of smartnic/DPUs. You can make the server think they are using a perfect TCP connection even if the protocol is different. And the opposite, the server/app can use a new protocol like json-stream-whatever and the smartnic can proxy it onto a TCP session. It turns out that cheap single purpose NICs were great for standardizing on thr lowest common denominator (tcp/udp) but now that it’s clear that the network protocol is the bottleneck, we can imagine and implement new protocols while maintaining backward compatibility.
- wmf 4y ago
  You can make the server think they are using a perfect TCP connection even if the protocol is different. I wouldn't go this route because by the time the data gets to the DPU you have already burned the cycles and paid the price of TCP. Offloading anything is tricky in general because the protocol for managing the offload needs to be leaner than the thing being offloaded.
- eklitzke 4y ago
  It is way more complicated than this because in any heterogeneous network environment you need to be really careful about what happens then you have new protocols like this interacting for shared resources. There are even TCP congestion control algorithms that aren't mutually compatible (e.g. under congestion situations one algorithm will be more aggressive and dominate the available bandwidth).
wmf 4y ago
Previous Homa discussions: https://news.ycombinator.com/item?id=34871710 https://news.ycombinator.com/item?id=34871710 (on the front page) https://news.ycombinator.com/item?id=28440542 https://news.ycombinator.com/item?id=28440542 https://news.ycombinator.com/item?id=28204808 https://news.ycombinator.com/item?id=28204808 https://news.ycombinator.com/item?id=33738711 https://news.ycombinator.com/item?id=33738711 https://news.ycombinator.com/item?id=33401480 https://news.ycombinator.com/item?id=33401480
ardel95 4y ago
An example of this already happening in a way that is transparent to the applications: https://aws.amazon.com/about-aws/whats-new/2022/11/elastic-network-adapter-ena-express-amazon-ec2-instances/ https://aws.amazon.com/about-aws/whats-new/2022/11/elastic-n...
ChuckMcM 4y ago
Harsh, but accurate ... I can’t take Homa seriously based on the arguments made in this position paper – why should I trust the claimed benefits of a proposed solution if the position paper favoring it misidentified the problems and ignored all prior art? Had Infiniband been developed in a way that avoided all of the patent encumbrances it would have replaced IP networking in large data centers (certainly at the Rack level, and likely at the Cluster level as well).
deleted 4y ago
[deleted]
fpanzer 4y ago
In the real world, we should start by replacing UDP first
javier_e06 4y ago
For satellite links TCP is too "chatty". Once the session enters the companies hardware it is faked. This gives some companies an edge. The trade-off here is weather is a session within a mega-bandwith data center or trickle-up-down high orbit sat-link the session is the-session. Bottom-up designs can be advantageous inflections. Software trickery.

104 points by zdw 4y ago

Is It Time to Replace TCP in Data Centers? Probably not? Maybe? One would have to either integrate support for new protocol into every server, switch, router, IoT oh and of course all the 3rd party clouds and clients one is speaking to. OR everything leaving the datacenter would have to be dual-stack and/or funneled through some WAN optimizer/gateway/proxy device that can translate new protocol into TCP/IP creating a single point of success bottle-neck. Dual-stack brings up some security issues that this protocol will have to address not to mention more cabling complexity. I think the best place to start this conversation would be with architects at Microsoft, IBM/Redhat, maybe even Meta since they acquired several kernel developers and let them see how cost effective the gains are. If the big players buy into this and they have kernel developers that can integrate seamless support into Linux and Microsoft to start with and a few big players try it out then maybe it would share the same market cap as Infiniband. Let them deploy a proof of concept pod for free for a year and see what they can do with it. I think this would have to be successful first before other vendors start adding support for new protocols. If the plan is to have one vendor to rule them all then it will not succeed as there would be no competition and it would be too expensive for mass adoption. This would end up being another proprietary thing that IBM or some other big company acquires and sits on. At least that is my opinion based on my experience deploying SAN switches, proprietary memory inter-connect buses, proprietary storage and storage clustering, proprietary mini-mainframes and server clusters. Speed improvements will impress technical people but businesses ultimately look into TCO/ROI and reliability. Complexity is a factor in reliability and the ability to hire people to support said new thing. If anything I have seen datacenters going the opposite direction; that is, keeping things as generic as possible and using open source solutions to scale first to their vertical limits and then horizontally. Ceph is a great example of this.

blibble 4y ago
you can run a new protocol on top of IP "relatively" easily: it routes fine across the internet as switches and routers don't care about TCP[1] (switches don't even care about IP) linux will even let you do it from userspace (with net_admin cap) there are some exceptions: like if your endpoint is some a crappy cloud provider (e.g. azure) that provides you something that looks like an ethernet network but really is a glorified TCP/unicast UDP proxy [1]: ignoring NAT and IGMP/MLD snooping (not that anyone does those outside of internal networks... right?)
- xxpor 4y ago
  routers care about tcp (or udp) because of flow hashing. write a new IP proto and see what happens when you try to push >100 gbit
  - blibble 4y ago
    you have 3 out of 5 parts of the standard tuple not perfect but not unworkable either
    - toast0 4y ago
      It can be necessary to get flow hashing on high volume server A : server B communications (especially if you're running aggregated interfaces and are using multiple connections). If you need that in a 3-tuple hashing environment, you're going to need to apply many addresses to one or both servers, which is going to cause problems with arp/ndp cache sizes. Maybe that's ok if you just route a /64 to each server and don't use NDP inside the /64; but you probably need to work on the server OS to work with many addresses well, things get a little shifty there.
  - pezezin 4y ago
    Isn't that why IPv6 includes an explicit flow label in the header?

jmclnx 4y ago

I say no, and I do not see how this could ever happen. Just look at IPv4 to IPv6 to see an example on how this would go :)

Dylan16807 4y ago
Anything with packets would still be based on normal IP so you wouldn't have those issues. In the worst case, you just build on top of UDP and don't care that you're wasting four bytes.

paxys 4y ago

Unless the alternative is a 100% backwards compatible drop-in replacement for TCP, even starting to discuss it is pointless. And if it is compatible at that level, it will just bring with it all the pitfalls that we want to replace in the first place. The TCP/IP stack works because I don't need to care about what environment the two processes that need to communicate are running in. They could both be on my local machine, or in my home network, or communicating over the internet, or some random intranet, in a data center, across continents, on any OS or any kind of device...it simply does not matter. "Just do a ground-up rewrite of your entire software stack and you'll get a guaranteed 5% efficiency gain" isn't the bullet proof argument that people who come up with these alternatives seem to think.

foota 4y ago
This is only really true of external connections, for orgs with large degrees of control over their internal stack this can make sense.
readingnews 4y ago
Correct. I agree. Well said.
mike_d 4y ago
Suggested alternative title: "Is it time to replace oxygen in the atmosphere?"
ithkuil 4y ago
I wonder if we need to replace TCP in the data centers or if what data centers really need is RPC (which ia covered well with QUIC)
- StillBored 4y ago
  Right, I've been saying this for a while, it seems that many of the hyperscalers/etc made the mistake of trying to utilize TCP (and maybe HTTP/etc) for their (generic) RPC mechanisms. Which is hardly the best plan, when what is actually needed is just a reliable datagram protocol without a bunch of connection state. So, while the actual RPC protocol has its own issues, the one thing they got right was the ability for the portmapper to indicate UDP vs TCP as the transport on a service basis. There have been a few improved generic RPC mechanisms, and I don't really understand why some of these places feel the need to "replace TCP" when really what they need is a more formalized RPC mechanism that can set/detect the datagram reliability and pick varying levels of protocol retry/etc as needed.

omani 4y ago

Betteridge's law of headlines.

omani 4y ago
I love it when I know I will be downvoted. that's why I posted this. you know who I am.
- ben0x539 4y ago
  We've all heard of the law. It's a cute observation but everybody is capable of making it independently and commenting with a reminder on every article with a yes/no question as the title is just noise.
  - omani 4y ago
    understandable. and I agree. it was more like a test to see how the bots/npcs would react (with their downvotes).

likeabbas 4y ago

Doesn't HTTP3 solve this with QUIC?

exabrial 4y ago
It does until you try to tcpdump :/ 1) It's not TCP 2) It's a PITA on a 'trusted' network because there is not "encryption off" mode
- olodus 4y ago
  Those are very solvable problems. I work with non-TCP traffic in pcaps very often and while I agree encrypted traffic is annoying to deal with sometimes because of the extra step it can be solved with good tooling.
- zamadatix 4y ago
  tcpdump works fine with it, and any other ethernet or IP based protocol despite the commands name, it's just 2) that's a problem (for certain perspectives of what is a problem).
- erik_seaberg 4y ago
  As long as governments attack networks with impunity, it’s a mistake to trust them.
  - shawnz 4y ago
    See this well-known slide from the PRISM leaks as a reminder of this: https://upload.wikimedia.org/wikipedia/commons/f/f2/NSA_Muscular_Google_Cloud.jpg https://upload.wikimedia.org/wikipedia/commons/f/f2/NSA_Musc...
  - KaiserPro 4y ago
    Its a question of choosing when and how you encrypt. For internal datacentre traffic, it depends on the value of the data. But you will need to do application level encryption. For links between datacentres, encrypting links based on rules is a sensible and not horrifically challenging thing to do. If you're big enough to have to worry about the volume of encrypted traffic, then you have rules based bandwidth priorities. for virtually everyone else, just use wireguard or some other VPN with encryption.

shanemhansen 4y ago

This seems like a review of John Ousterhout's work w/ Homa. I highly recommend reading the original. https://arxiv.org/abs/2210.00714 https://arxiv.org/abs/2210.00714 For those who don't know Ousterhout created the first log structured filesystem and created TCL (used heavily in hardware verification but also forming the backbone of the some of the first large web servers: aolserver). I was actually surprised to find out he co-founded a company with the current CTO of Cloudflare. https://en.wikipedia.org/wiki/John_Ousterhout https://en.wikipedia.org/wiki/John_Ousterhout He has both a candidate replacement as well as benchmarks showing 40% performance improvements with grpc on homa compared to grpc on TCP. https://github.com/PlatformLab/grpc_homa https://github.com/PlatformLab/grpc_homa With that in mind I think nobody will replace TCP and I doubt anything not IP compatible will be able to get off the ground. His argument is essentially that for low latency RPC protocols TCP is a bad choice. We've already seen people build a number of similar systems on UDP including HTTP replacements that have delivered value for clients doing lots of parallel requests on the WAN. I think many big tech companies are essentially already choosing to bypass TCP. I recall facebook doing alot of work with memcache on udp. I can't find any public docs on whether or not Google's internal RPC uses TCP. I wouldn't be surprised at all if in the near future something like grpc/capnproto/twirp/etc had an in-datacenter TCP-free fast path. It would be cool if it was built on Homa.

zarkov99 4y ago
TCP is rarely used in HFT data centers except as required by exchanges.
tyingq 4y ago
Ousterhout was also one of the co-authors of the Raft consensus paper.
killingtime74 4y ago
He also wrote one of my favorite software books https://web.stanford.edu/~ouster/cgi-bin/book.php https://web.stanford.edu/~ouster/cgi-bin/book.php. Presentation https://youtu.be/bmSAYlu0NcY https://youtu.be/bmSAYlu0NcY
wpietri 4y ago

olodus 4y ago

Since one of the suggestions the article brings up is QUIC then won't it happen almost automatically? Atleast the distributed systems I work with usually have http between their parts (a few with some other RPC solution). Then when the switch happens to http 3 QUIC will come with it. There will probably be some problems along the way sure and will take time for people to tune thibgs, but that must be expected right?

topranks 4y ago
QUIC involves encryption though. If that’s not needed in a given scenario the overhead might not be desirable. But otherwise, yes, it addresses many of these problems.

WesolyKubeczek 4y ago

There has been a few of TCP would-be killers out there, SCTP and DCCP come to mind. And yet, here we are. I even remember that there have been Linux kernel security vulnerabilities in those protocol somehow exploitable even when you don't actively use them, so... I don't compile my own kernels often these days, but when I do, I usually say N to any and all protocols I'm not likely to be using in my remaining life.

deleted 4y ago

[deleted]

nickdothutton 4y ago

I see we’re doing this again. I last worked on whole-DC designs around 1999 with Sun, Intel, and of course… Mellanox since we were to use RDMA/InfiniBand. The plan was to embed InfiniBand semiconductors on disks, RAM (individual sticks or trays of sticks) and CPUs. There was to be an orchestration/provisioning layer allowing you to reserve a composable machine (CPU(s), RAM, storage, other IO, etc) at will, and some kind of supervisory layer was to ensure that these things were as local as possible to one-another in any given rack, aisle, floor.

Traubenfuchs 4y ago
So you were pretty much working on a more hardware based predecessor of kubernetes? Why did it not work out?
- SgtBastard 4y ago
  It wasn’t written in Go </snark> I’d also love to hear more about the concept.
  - sacnoradhq 4y ago
    Go peaked in 2017. Rust is the new hotness. 3 years from now, it will be something else.
- bobleeswagger 4y ago
  > Why did it not work out? Software won, it seems.
  - KaiserPro 4y ago
    kinda, It was cheaper to use fibrechannel/ethernet and plain old sparc/intel machines This was about the time that grid engines promised mainframe computing without the cost. Only that it required people to understand how to parallelise their workloads. That and modify programmes to run in on many machines at once.
- 4y ago

manv1 4y ago

"Is it time to spend hundreds of billions of dollars on new networking equipment for marginal gains? Pundits say yes."

dokem 4y ago
This is our punishment for violating the osi model. Why should network gear need to know about above the ip layer?
- manv1 4y ago
  The OSI model was standardized in 1984. The first TCP RFC came out in 1980, with versions existing in real life before that. The OSI model isn't, and has never been, the be-all end-all of networking models, and it's bizarre to me that people continue to believe that the OSI is the only valid model to follow. Maybe you can explain why you believe that the OSI model is canonical?
  - dokem 4y ago
    What other model(s) are you referring to? TCP/IP? that still has identical ip/transport separation so can you explain why it's a good idea to so blantly violate fundamental layers of abstraction?
wmf 4y ago
Homa uses existing equipment.

Animats 4y ago

There are many ways to improve on TCP for specific use cases. You can get maybe another 10%-20% of performance, as Google did with QUIC. Used outside that use case, performance may be worse. The design goal of TCP was that it should Just Work. It needed to be interoperable between different vendors, provide reasonably good performance, work over a wide range of networks speeds, work over possibly flaky networks, continue to perform under overload, and not have manual tuning parameters. It does well against those criteria. This new protocol the sort of thing Osterholt is known for. He came up with "log" file systems, which are optimized for the case where disk writes outnumber reads. Reading then requires more seeks on mechanical drives. If you're mostly writing logs that are seldom read, it's a win. If you're doing more reading than writing, it's a lose.

toast0 4y ago
> There are many ways to improve on TCP for specific use cases. You can get maybe another 10%-20% of performance, as Google did with QUIC. Most of the gains of QUIC are in exchanging data with the first round trip, which you can do with TCP fast open as long as middleboxes don't mess it up; different congestion control, which you could do in TCP if you control the OS on the sending side as long as the middle doesn't mess it up, which shouldn't happen in a data center; and maybe some benefits for connection switching, which should hopefully not be necessary in a data center (although I've certainly seen cases it would help!); multi-path TCP also addresses this, but has probably worse CPU efficiency problems: if a logical flow is coming in on multiple NIC queues, there's going to be cross-cpu communication on the data structures. Unless it's changed, QUIC also simply won't handshake if the path MTU is too small, so you don't get to have the excitement of path MTU discovery failures; but you really shouldn't have that problem in a datacenter.

floatinglotus 4y ago

Replacing TCP in the DC is a lot easier now with the advent of smartnic/DPUs. You can make the server think they are using a perfect TCP connection even if the protocol is different. And the opposite, the server/app can use a new protocol like json-stream-whatever and the smartnic can proxy it onto a TCP session. It turns out that cheap single purpose NICs were great for standardizing on thr lowest common denominator (tcp/udp) but now that it’s clear that the network protocol is the bottleneck, we can imagine and implement new protocols while maintaining backward compatibility.

wmf 4y ago
You can make the server think they are using a perfect TCP connection even if the protocol is different. I wouldn't go this route because by the time the data gets to the DPU you have already burned the cycles and paid the price of TCP. Offloading anything is tricky in general because the protocol for managing the offload needs to be leaner than the thing being offloaded.
eklitzke 4y ago
It is way more complicated than this because in any heterogeneous network environment you need to be really careful about what happens then you have new protocols like this interacting for shared resources. There are even TCP congestion control algorithms that aren't mutually compatible (e.g. under congestion situations one algorithm will be more aggressive and dominate the available bandwidth).

wmf 4y ago

Previous Homa discussions: https://news.ycombinator.com/item?id=34871710 https://news.ycombinator.com/item?id=34871710 (on the front page) https://news.ycombinator.com/item?id=28440542 https://news.ycombinator.com/item?id=28440542 https://news.ycombinator.com/item?id=28204808 https://news.ycombinator.com/item?id=28204808 https://news.ycombinator.com/item?id=33738711 https://news.ycombinator.com/item?id=33738711 https://news.ycombinator.com/item?id=33401480 https://news.ycombinator.com/item?id=33401480

ardel95 4y ago

An example of this already happening in a way that is transparent to the applications: https://aws.amazon.com/about-aws/whats-new/2022/11/elastic-network-adapter-ena-express-amazon-ec2-instances/ https://aws.amazon.com/about-aws/whats-new/2022/11/elastic-n...

ChuckMcM 4y ago

Harsh, but accurate ... I can’t take Homa seriously based on the arguments made in this position paper – why should I trust the claimed benefits of a proposed solution if the position paper favoring it misidentified the problems and ignored all prior art? Had Infiniband been developed in a way that avoided all of the patent encumbrances it would have replaced IP networking in large data centers (certainly at the Rack level, and likely at the Cluster level as well).

deleted 4y ago

[deleted]

fpanzer 4y ago

In the real world, we should start by replacing UDP first

javier_e06 4y ago

For satellite links TCP is too "chatty". Once the session enters the companies hardware it is faked. This gives some companies an edge. The trade-off here is weather is a session within a mega-bandwith data center or trickle-up-down high orbit sat-link the session is the-session. Bottom-up designs can be advantageous inflections. Software trickery.