Consider how Lustre RPC works. It's mainly based on RDMA-capable protocols, though it has a TCP binding as well. For the TCP binding Lustre uses separate conn

Consider how Lustre RPC works. It's mainly based on RDMA-capable protocols, though it has a TCP binding as well. For the TCP binding Lustre uses separate connections for RPC headers and bulk data, and even in the TCP case it's built around RDMA as a concept. RDMA essentially means that before you do a bulk I/O you negotiate acquisition of buffers for that bulk I/O, which imposes discipline on resource usage, which is pretty great for HPC and... less great on the public Internet. For writes this means an extra round trip at least once to set up the buffers you will use. Now, the thing about RDMA is that it greatly benefits from HW NIC support -think InfiniBand-, but for that to be feasible the RDMA protocol needs to be its own ULP running over IP or -even better- a protocol at the same layer as IP. (If it's not run over IP, however, it can't be an Internet protocol.) This is where the real need for new protocols other than TCP or UDP should come in. And there are some such protocols, like RoCEv2. It's totally possible to build an RDMA protocol over UDP that is performant. One might worry about the UDP header overhead, but it's minimal, and besides, you'll want cryptography, which will add more overhead for authentication tags. Don't allow fragmentation, meaning you'll need PMTUD, and you don't need your own length fields, so the only UDP overhead that could be avoided is the src/dst port numbers (4 bytes!)... which you might have anyways in a new ULP.