3 ms·

Is It Time to Replace TCP in Data Centers? Probably not? Maybe? One would have to either integrate support for new protocol into every server, switch, router

Is It Time to Replace TCP in Data Centers? Probably not? Maybe? One would have to either integrate support for new protocol into every server, switch, router, IoT oh and of course all the 3rd party clouds and clients one is speaking to. OR everything leaving the datacenter would have to be dual-stack and/or funneled through some WAN optimizer/gateway/proxy device that can translate new protocol into TCP/IP creating a single point of success bottle-neck. Dual-stack brings up some security issues that this protocol will have to address not to mention more cabling complexity. I think the best place to start this conversation would be with architects at Microsoft, IBM/Redhat, maybe even Meta since they acquired several kernel developers and let them see how cost effective the gains are. If the big players buy into this and they have kernel developers that can integrate seamless support into Linux and Microsoft to start with and a few big players try it out then maybe it would share the same market cap as Infiniband. Let them deploy a proof of concept pod for free for a year and see what they can do with it. I think this would have to be successful first before other vendors start adding support for new protocols. If the plan is to have one vendor to rule them all then it will not succeed as there would be no competition and it would be too expensive for mass adoption. This would end up being another proprietary thing that IBM or some other big company acquires and sits on. At least that is my opinion based on my experience deploying SAN switches, proprietary memory inter-connect buses, proprietary storage and storage clustering, proprietary mini-mainframes and server clusters. Speed improvements will impress technical people but businesses ultimately look into TCO/ROI and reliability. Complexity is a factor in reliability and the ability to hire people to support said new thing. If anything I have seen datacenters going the opposite direction; that is, keeping things as generic as possible and using open source solutions to scale first to their vertical limits and then horizontally. Ceph is a great example of this.

blibble 4y ago
you can run a new protocol on top of IP "relatively" easily: it routes fine across the internet as switches and routers don't care about TCP[1] (switches don't even care about IP) linux will even let you do it from userspace (with net_admin cap) there are some exceptions: like if your endpoint is some a crappy cloud provider (e.g. azure) that provides you something that looks like an ethernet network but really is a glorified TCP/unicast UDP proxy [1]: ignoring NAT and IGMP/MLD snooping (not that anyone does those outside of internal networks... right?)
- xxpor 4y ago
  routers care about tcp (or udp) because of flow hashing. write a new IP proto and see what happens when you try to push >100 gbit
  - blibble 4y ago
    you have 3 out of 5 parts of the standard tuple not perfect but not unworkable either
    - toast0 4y ago
      It can be necessary to get flow hashing on high volume server A : server B communications (especially if you're running aggregated interfaces and are using multiple connections). If you need that in a 3-tuple hashing environment, you're going to need to apply many addresses to one or both servers, which is going to cause problems with arp/ndp cache sizes. Maybe that's ok if you just route a /64 to each server and don't use NDP inside the /64; but you probably need to work on the server OS to work with many addresses well, things get a little shifty there.
  - pezezin 4y ago
    Isn't that why IPv6 includes an explicit flow label in the header?

blibble 4y ago

you can run a new protocol on top of IP "relatively" easily: it routes fine across the internet as switches and routers don't care about TCP[1] (switches don't even care about IP) linux will even let you do it from userspace (with net_admin cap) there are some exceptions: like if your endpoint is some a crappy cloud provider (e.g. azure) that provides you something that looks like an ethernet network but really is a glorified TCP/unicast UDP proxy [1]: ignoring NAT and IGMP/MLD snooping (not that anyone does those outside of internal networks... right?)

xxpor 4y ago
routers care about tcp (or udp) because of flow hashing. write a new IP proto and see what happens when you try to push >100 gbit
- blibble 4y ago
  you have 3 out of 5 parts of the standard tuple not perfect but not unworkable either
  - toast0 4y ago
    It can be necessary to get flow hashing on high volume server A : server B communications (especially if you're running aggregated interfaces and are using multiple connections). If you need that in a 3-tuple hashing environment, you're going to need to apply many addresses to one or both servers, which is going to cause problems with arp/ndp cache sizes. Maybe that's ok if you just route a /64 to each server and don't use NDP inside the /64; but you probably need to work on the server OS to work with many addresses well, things get a little shifty there.
- pezezin 4y ago
  Isn't that why IPv6 includes an explicit flow label in the header?