11 ms·

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better.

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer problems! It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. And then there's all the non-backend problems! And that's just TCP. We still lack any way to communicate up and down the stack of an entire transaction, for example for debugging purposes. We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years.

fragmede 4y ago
What is QUIC in your book? Given, say, $50 million of dev time, what would you go about fixing? And in what way?
- Areading314 4y ago
  Doesn't QUIC still run over TCP? I thought it was a replacement for HTTP not TCP (Edit: looks like it replaces TCP and HTTP)
  - jhardy54 4y ago
    Nope, UDP.
  - stevewatson301 4y ago
    QUIC runs over UDP, and provides streams and encryption. HTTP/3 is designed to take advantage of QUIC streams (replacing HTTP/2 streams which were problematic due to TCP head of line blocking). The RFCs are a bit elaborate so folks interested might want to look at this instead[1], which has one of the RFC authors explaining the basics of QUIC and HTTP/3. [1] https://www.youtube.com/watch?v=cdb7M37o9sU https://www.youtube.com/watch?v=cdb7M37o9sU
  - klabb3 4y ago
    It replaces TCP+TLS, and runs multiple streams on the same conn, supports transition from eg wifi to ethernet on at least one of the nodes. And since it's over udp implementations are mostly in user space. Which is good if you want it now, but not great for performance. Ip packets are very small so you gotta have either kernel support for quic or batch IO, otherwise it's often CPU limited (yes, really). In addition congestion control is wonky, unfortunately. In my experience (quic-go), it's too shy in the presence of TCP streams, which ends up getting more bandwidth. But that depends on the algorithm used, implementation and God knows what else.
  - notpushkin 4y ago
    I guess you were thinking about another clever name protocol, SPDY :-) SPDY → HTTP/2 QUIC → HTTP/3
- OmarAssadi 4y ago
  In addition to QUIC, KCP [1] is another reliable low-latency protocol that sits on top of UDP that might be interesting. And unlike RFC 9000/9001 (QUIC), encryption is optional. I haven't really seen it mentioned much outside of primarily China-focused projects, like V2Ray [2], but there is also some English information in their Git repo [2]. [1]: <https://github.com/skywind3000/kcp https://github.com/skywind3000/kcp> [2]: <https://www.v2fly.org/en_US/ https://www.v2fly.org/en_US/> [3]: <https://github.com/skywind3000/kcp/blob/master/README.en.md https://github.com/skywind3000/kcp/blob/master/README.en.md>
  - xfs 4y ago
    KCP uses a brute force congestion control algorithm that is unfair and inefficient. It is also poorly specified, which is probably why it is less commonly used outside circumvention circles.
    - ronsor 4y ago
      KCP is notably used by the popular mobile game Genshin Impact.
    - fomine3 4y ago
      Still is looks interesting for some use cases, even though it's not fair if it's fully utilized on the internet.
- Luker88 4y ago
  IMHO QUIC is nice, but a disappointment, since it could have been so much more. Does not handle unreliable messages, still only (multi)streaming, no direct support for multicast, 0-rtt which need a lot of stuff to be manually done TheRightWay or risk amplification attacks, the (imho) under-researched (and removed) forward error correction, and more. I just restarted working on what I consider to be the solution to this, federated authentication and a bit more, but $50M is too far to be even a dream since I am not google.
ajross 4y ago
Surely most of your ideas are already being deployed in QUIC/HTTP3. It just happens inside a UDP datagram, for compatibility. Really you're not going to see any new IP protocol layers, there's too much quirky hardware on the network that wouldn't be able to handle it. If we can't even get IPv6 to work all the way to the client, we're never seeing new values for the protocol byte.
- vlovich123 4y ago
  Don’t the hyperscaled cloud providers run totally segmented networks? What’s stopping them from using something proprietary internally and just exposing TCP at the end for termination of client connections?
  - wmf 4y ago
    Google already does that.
    - vlovich123 4y ago
      I’m not aware of them using something other than TCP internally (I’m sure by now they’ve migrated to QUIC but I’m not sure that QUIC necessarily solves some of the scaling challenges / optimizes for gRPC and low latency).
      - uluyol 4y ago
        Google is using remote memory accesses rather than TCP for at least some classes of traffic (e.g. a caching system). They've been publishing details about how it all works too. Also, they have a transport (Pony express) developed specifically for RPCs, rather than byte streams or datagrams. Links: https://research.google/pubs/pub51341/ https://research.google/pubs/pub51341/, https://research.google/pubs/pub50590/ https://research.google/pubs/pub50590/, https://research.google/pubs/pub48630/ https://research.google/pubs/pub48630/, more generally https://research.google/pubs/?area=networking https://research.google/pubs/?area=networking
bigDinosaur 4y ago
Your ideas are interesting, can you link to or explain a concrete example though? The idea of everything magically debugging itself doesn't apply to a single piece of software I've ever seen, so I'm curious what kind of design would lead to that being possible.
- Areading314 4y ago
  Heres an example of an improvement to sending large files over long distances -- Tsunami protocol. It tries to get a best of both worlds to limit the detrimental effect of synchronous roundtrips in the TCP protocol for file transfers: https://tsunami-udp.sourceforge.net/ https://tsunami-udp.sourceforge.net/
alexgartrell 4y ago
> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is ridiculous. Hyperscalars see an immediate ROI from efficiency/reliability improvements and actively invest in TCP alternatives all of the time. It's just really hard. Networking companies see an ability to differentiate their products from their peers and work on this kind of thing as well. I did a 3 second google for "QUIC acceleration Mellanox" and got a hit on Nvidia's blog right away. You just can't trivially replace something with an investment totally 50 years of clock time and thousands of years of engineer time. It will either take a long time or a massive shift in needs/technology. FWIW, I wouldn't be surprised if the high-performance RDMA networks being put together for AI workloads were the thing that grew into the "next" thing.
- oconnor663 4y ago
  > 50 years of clock time and thousands of years of engineer time It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.
  - kebman 4y ago
    Yes you can. Just offer a better product, and people will buy it instead of the old or bad product. Better yet, make the new product backwards compatible, and fewer people will have qualms about forking out for it. Better yet, do an aggressive takeover, like Microsoft did, and just force the entire industry to adopt your stuff...
    - scantydolt 4y ago
      Great! When do you think you'll have it done?
      - kebman 4y ago
        Done? What do you mean "done"? Consulting hours are much better on projects that cannot ever be finished!1
peter_retief 4y ago
At last hopefully there is light at the end of the tunnel. Big question for me is who is going to build it?
bsder 4y ago
> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly. See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc. My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is zero mention of SCTP and only compares to TCP and not UDP. The problem with RPC is that multiple organizations must agree on meaning. And that's just not going to fly. It is damn near a miracle that a huge number of institutions all agree on the Ethernet/IP command "Please take this bag of bytes closer to the machine named: <string of bytes>."
- idlehand 4y ago
  Never thought about that before. Ethernet supports extremely high levels of data transmission. ISB C for intrgrated charging ND data transfer makes sense, but why are there HDMI cables?
  - GTP 4y ago
    I think it would be an overkill to use a networking protocol to connect exactly two devices. Plus if you have something specific to video stream transfer you could maybe do some optimization specific to that use case, although I can't think of any at the moment.
    - zrail 4y ago
      Excepting HDMI the parents examples are all networks with more than one peer. Thunderbolt and USB3 can both have arbitrary trees of nodes.
      - GTP 4y ago
        The parent comment was specifically wondering about HDMI, the other examples are given saying that those have some reason to exist, while he didn't see any reason for having HDMI instead of recycling some other protocol.
duped 4y ago
Every major corporation has multiple research organizations doing nothing but invest in things that don't have immediate shareholder value. What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. The scale at which these changes needs to happen to be practical are at the cluster level in a single data center. If you can propose something that fits that bill there are a few companies willing to pay you millions in salary as an engineering fellow to do it.
- simplotek 4y ago
  > What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. I'd put it differently: it's paying up hundreds of billions in infrastructure to have some sort of gain. And which gain is that exactly? I see a lot of "the world is dumb but I am smart" comments in this thread but I saw no one presenting any clear advantage or performance improvement claim about hypothetical replacements. I see a lot of "we need to rewrite things" comments but not a single case being made with a clear tradeoff being presented. Every single criticism of TCP/IP in this thread sounds like change for the sake of change, and changes that aren't even presented with tangible improvements in mind or a clear performance gain. Wouldn't that explain why TCP is so prevalent, and no one in their right mind thinks of replacing it?
  - fragmede 4y ago
    I mean the goal is more performance, especially if you can get more performance out of the same hardware. Faster setup times, faster connections, more connections, maybe faster teardown. Lower contention on saturated links. Inside of the datacenter is a controlled environment where something like that could work. Replacing TCP over the Internet at large is going to be an uphill battle. Still, if we're replacing the whole thing, then simpler code on the client and server end would be nice.
    - simplotek 4y ago
      > I mean the goal is more performance, especially if you can get more performance out of the same hardware. Are there actual numbers demonstrating this? I mean, people are advocating wasting billions revamping infrastructure. What kind of performance are you hoping to buy with those billions? And are those gains worth it, or is just sake for the sake of change? Sometimes things are indeed good enough.
deleted 4y ago
[deleted]
simplotek 4y ago
> Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. If this was true then how do you explain that the likes of AWS, the same company who ended up investing in developing their own processor line, doesn't seem to agree that none of the pet peeves you mentioned are worth fixing?
- emn13 4y ago
  It's not obvious to me that replacing TCP really is harder than designing your "own" chip. Scarequotes here because those graviton chips (that's what you're referring to, I think?) are of course ARM chips, so they're not designing something fresh; they're adapting a very mature design to their own needs. In terms of interoperability, a custom chip based on a standard design is probably a simpler, more locally addressable problem than new network protocols. Isn't it plausible that graviton was designed yet TCP retained simply because graviton as a project is easier to complete successfully?
  - deleted 4y ago
    [deleted]
- xxpor 4y ago
  https://aws.amazon.com/blogs/hpc/in-the-search-for-performance-theres-more-than-one-way-to-build-a-network/ https://aws.amazon.com/blogs/hpc/in-the-search-for-performan...
robertlagrant 4y ago
> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is just not true. Stuff needs to be funded and worth doing, and the internet, like almost everything, is built on making things worth paying for, but there are also loads of improvements everywhere are being made.
primeblue 4y ago
friendzis 4y ago
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols I think this is more of an artefact of horizontal scaling and port-contention. De-facto standard discovery mechanism DNS does not work with ports, so "well-known port" abstraction kinda fails. Http as tunnel mostly avoids/sidesteps this problem. > We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. This is weird take or I don't understand it. If you can communicate with an edge node in another network, but the edge node has issues communicating with some inner node (on your behalf), then, as a user, you have no hope of fixing that connectivity issue anyway, regardless of whether layered approach is used or not. This may be related to previous point about http as universal tunnel. Yes, this is a problem, but in a way that communications are effectively terminated at the edge node and monstrosity of stuff happens behind the scenes
- patrec 4y ago
  > De-facto standard discovery mechanism DNS does not work with ports Yes, it does, see SRV records.
  - friendzis 4y ago
    I meant DNS A/AAAA queries with preconfigured/well-known ports being the default. While some applications/protocols/services do use some port discovery mechanism, I would argue it is nowhere close to being de-facto standard.
  - bewo001 4y ago
    So true, but how many developers know about them? The API situation does not help either.
- AtNightWeCode 4y ago
  I would say ports are mainly a problem on layers below transport even though some tech overuse ports.
kortilla 4y ago
> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is severe bullshit on two fronts: - there is an immediate return on value - Google was driving this a decade+ ago for improvements in the data center (things like doubled+cancelable rpc, tcp cubic, quic, etc) - academia constantly attempts to make these improvements as well because researchers are super incentivized to dethrone tcp for the glory. There are constant attempts to re-invent various layers (IP, tcp, the non existent upper layers of the OSI, etc) that come out of academic conferences every year. The reason we’re still here is because our current stacks have been heavily optimized and tooled for production workloads. NICs can transparently re-assemble TCP segments for the OS and they can segment before transmit. You have to have a damn good value prop to throw away everything from software and hardware to careers and curriculum. It has to be a shitload better than the security nightmare of “return back each layer of the stack”.
gjulianm 4y ago
> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years. I work in a company that builds network troubleshooting/observability tools and we have some pretty experienced analysts to tell you what's wrong with the network. With that context, your idea of having any tool automatically diagnosing network issues is a pipe dream. The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners, failures aren't always easily reproducible, and with human configuration (and therefore errors) almost every step of the way. Even if a tool that "returns each layer of the stack" would be useful, it still would be far from enough to diagnose issues.
- EricE 4y ago
  "The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners" Ah, how people forget the early days of networking. I remember vividly the early days of the Networld/Interop trade show - Interop was in the name because if, as a vendor, your equipment couldn't integrate with the show network they would throw your booth off the show floor. That's how bad interoperability in the early days was!
cdogl 4y ago
I’ll defer to experts on the network-layer problems but im not sure what you see as the problem with converging on HTTP. It’s awkward and inelegant, but as an a backend application developer I never feel like it gets in my way.
jstimpfle 4y ago
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP If you have ever used multiple TCP or UDP connections in parallel on a single machine (doesn't matter if server or client) then you should realize that ports are required. Apart from that, you can run HTTP on other ports than 80. You can also use HTTP to load balance or do service discovery by means of redirects. (Caveat, I don't work in this field and can't say how solid the approach works in practice).
twawaaay 4y ago
I don't think you realise why this is so hard. The basic reason is that software at every level expects TCP/IP. And you can't drop in a translation layer because it will require at least the same amount of overhead as "real" TCP/IP. It is not a local problem, it is a global problem that affects basically every single piece of non-trivial software in existence. Even if you construct your datacenter with the new protocol you will run into problems that you can't run anything in it. Want Python? Sorry, have to rewrite it. And every Python library. And every Python application. Then you need to deal with problems that people who can run their scripts on their machines can't run them in datacenter. And so on. The reason nobody wants to do this is that they would be investing huge amount of money to solve a problem for everybody else. Because the only way to make TCP/IP replacement work is to make it completely free and available to everybody. There are much better ways to allocate your funds and precious top level engineers that let them distance themselves from competition temporarily.
starfallg 4y ago
> We still lack any way to communicate up and down the stack of an entire transaction, for example for debugging purposes. We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years. This violates the principle of encapsulation that the entire field of networking is based on, not to mention an massive security hole.
KaiserPro 4y ago
> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues Thats virtual networking. but that introduces latency if its not well configured. > But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf not really, assuming you have the right fabric, its nowhere near as hard as that. Plus you seem to be forgetting that there is more to the network than TCP. There is a whole physical layer that has lots of semantics that greatly affect how easy it is to debug higher levels.
arka2147483647 4y ago
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. And then there's all the non-backend problems! The paper argues the in '3.1 Stream orientation' section, that stream orientation is a problem for TCP, and says that most apps send messages instead, and the better protocol should handle messages, natively, etc. Which is a good point I think. But back to TCP. What do you do, if you need to send Messages between applications in TCP? Preferably those Messages would be encrypted also. You could make up your own protocol, but you probably would rather not! So you use something that is readily available, and does messages, encryption, etc. Would be nice if there were also a ready to use load balancers, caches, tools to debug it, etc Now, what would be such a protocol. Why HTTPS, of course. So I kind of think that the lack of a low level Message Protocol has lead us, as an industry, to coalesce these features bit-by-bit on top of HTTP. It's not perfect by any means, but it does the job.
- pclmulqdq 4y ago
  HTTPS adds a tremendous amount of overhead to give you messaging. It's a lot better from a hyperscaler's perspective to replace TCP and not use the byte stream abstraction. After all, networks send messages. It's silly to throw that away at one layer and try to get it back at the next layer.
guenthert 4y ago
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. Nobody forces them though. It would be much easier to publish a standard port number mapping than to develop a (or multiple) new protocols. Now you just need to motivate people to use it.
still_grokking 4y ago
> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. I would affirm that. It's imho true for almost everything in IT tech. How computers "work" today is just pure madness when looked anyhow closer. Everything's a result of some "historic accidents" back in the days, and from that the usual race to the bottom caused by market powers. Nobody is willing to touch any of the lower layers no matter how crazy they are form today's viewpoint. We just shovel new layers on top to paper over the mistakes of the past. Nothing gets repaired, or actually what would be more more important, rethought form the ground up in light of new technological possibilities and changed requirements. I understand from the economic standpoint how this comes. But I'm also quite sure we didn't make any fundamental improvements in the last 50 years of computing. That's a very bad sign when everything in a field that's not even really 100 years old is frozen in time since 50 years because everything's so fragile and complex that fundamental changes aren't possible. This looks like a text book example of a house of cards… Given how vital IT tech is to modern life I fear that this will crash at some point in the worst way possible. And even if it won't crash, which is really strongly hope, we will never have nice things again as nothing of the old rotten things can be reasonably changed.

fragmede 4y ago

What is QUIC in your book? Given, say, $50 million of dev time, what would you go about fixing? And in what way?

Areading314 4y ago
Doesn't QUIC still run over TCP? I thought it was a replacement for HTTP not TCP (Edit: looks like it replaces TCP and HTTP)
- jhardy54 4y ago
  Nope, UDP.
- stevewatson301 4y ago
  QUIC runs over UDP, and provides streams and encryption. HTTP/3 is designed to take advantage of QUIC streams (replacing HTTP/2 streams which were problematic due to TCP head of line blocking). The RFCs are a bit elaborate so folks interested might want to look at this instead[1], which has one of the RFC authors explaining the basics of QUIC and HTTP/3. [1] https://www.youtube.com/watch?v=cdb7M37o9sU https://www.youtube.com/watch?v=cdb7M37o9sU
- klabb3 4y ago
  It replaces TCP+TLS, and runs multiple streams on the same conn, supports transition from eg wifi to ethernet on at least one of the nodes. And since it's over udp implementations are mostly in user space. Which is good if you want it now, but not great for performance. Ip packets are very small so you gotta have either kernel support for quic or batch IO, otherwise it's often CPU limited (yes, really). In addition congestion control is wonky, unfortunately. In my experience (quic-go), it's too shy in the presence of TCP streams, which ends up getting more bandwidth. But that depends on the algorithm used, implementation and God knows what else.
- notpushkin 4y ago
  I guess you were thinking about another clever name protocol, SPDY :-) SPDY → HTTP/2 QUIC → HTTP/3
OmarAssadi 4y ago
In addition to QUIC, KCP [1] is another reliable low-latency protocol that sits on top of UDP that might be interesting. And unlike RFC 9000/9001 (QUIC), encryption is optional. I haven't really seen it mentioned much outside of primarily China-focused projects, like V2Ray [2], but there is also some English information in their Git repo [2]. [1]: <https://github.com/skywind3000/kcp https://github.com/skywind3000/kcp> [2]: <https://www.v2fly.org/en_US/ https://www.v2fly.org/en_US/> [3]: <https://github.com/skywind3000/kcp/blob/master/README.en.md https://github.com/skywind3000/kcp/blob/master/README.en.md>
- xfs 4y ago
  KCP uses a brute force congestion control algorithm that is unfair and inefficient. It is also poorly specified, which is probably why it is less commonly used outside circumvention circles.
  - ronsor 4y ago
    KCP is notably used by the popular mobile game Genshin Impact.
  - fomine3 4y ago
    Still is looks interesting for some use cases, even though it's not fair if it's fully utilized on the internet.
Luker88 4y ago
IMHO QUIC is nice, but a disappointment, since it could have been so much more. Does not handle unreliable messages, still only (multi)streaming, no direct support for multicast, 0-rtt which need a lot of stuff to be manually done TheRightWay or risk amplification attacks, the (imho) under-researched (and removed) forward error correction, and more. I just restarted working on what I consider to be the solution to this, federated authentication and a bit more, but $50M is too far to be even a dream since I am not google.

ajross 4y ago

Surely most of your ideas are already being deployed in QUIC/HTTP3. It just happens inside a UDP datagram, for compatibility. Really you're not going to see any new IP protocol layers, there's too much quirky hardware on the network that wouldn't be able to handle it. If we can't even get IPv6 to work all the way to the client, we're never seeing new values for the protocol byte.

vlovich123 4y ago
Don’t the hyperscaled cloud providers run totally segmented networks? What’s stopping them from using something proprietary internally and just exposing TCP at the end for termination of client connections?
- wmf 4y ago
  Google already does that.
  - vlovich123 4y ago
    I’m not aware of them using something other than TCP internally (I’m sure by now they’ve migrated to QUIC but I’m not sure that QUIC necessarily solves some of the scaling challenges / optimizes for gRPC and low latency).
    - uluyol 4y ago
      Google is using remote memory accesses rather than TCP for at least some classes of traffic (e.g. a caching system). They've been publishing details about how it all works too. Also, they have a transport (Pony express) developed specifically for RPCs, rather than byte streams or datagrams. Links: https://research.google/pubs/pub51341/ https://research.google/pubs/pub51341/, https://research.google/pubs/pub50590/ https://research.google/pubs/pub50590/, https://research.google/pubs/pub48630/ https://research.google/pubs/pub48630/, more generally https://research.google/pubs/?area=networking https://research.google/pubs/?area=networking

bigDinosaur 4y ago

Your ideas are interesting, can you link to or explain a concrete example though? The idea of everything magically debugging itself doesn't apply to a single piece of software I've ever seen, so I'm curious what kind of design would lead to that being possible.

Areading314 4y ago
Heres an example of an improvement to sending large files over long distances -- Tsunami protocol. It tries to get a best of both worlds to limit the detrimental effect of synchronous roundtrips in the TCP protocol for file transfers: https://tsunami-udp.sourceforge.net/ https://tsunami-udp.sourceforge.net/

alexgartrell 4y ago

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is ridiculous. Hyperscalars see an immediate ROI from efficiency/reliability improvements and actively invest in TCP alternatives all of the time. It's just really hard. Networking companies see an ability to differentiate their products from their peers and work on this kind of thing as well. I did a 3 second google for "QUIC acceleration Mellanox" and got a hit on Nvidia's blog right away. You just can't trivially replace something with an investment totally 50 years of clock time and thousands of years of engineer time. It will either take a long time or a massive shift in needs/technology. FWIW, I wouldn't be surprised if the high-performance RDMA networks being put together for AI workloads were the thing that grew into the "next" thing.

oconnor663 4y ago
> 50 years of clock time and thousands of years of engineer time It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.
- kebman 4y ago
  Yes you can. Just offer a better product, and people will buy it instead of the old or bad product. Better yet, make the new product backwards compatible, and fewer people will have qualms about forking out for it. Better yet, do an aggressive takeover, like Microsoft did, and just force the entire industry to adopt your stuff...
  - scantydolt 4y ago
    Great! When do you think you'll have it done?
    - kebman 4y ago
      Done? What do you mean "done"? Consulting hours are much better on projects that cannot ever be finished!1

peter_retief 4y ago

At last hopefully there is light at the end of the tunnel. Big question for me is who is going to build it?

bsder 4y ago

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly. See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc. My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is zero mention of SCTP and only compares to TCP and not UDP. The problem with RPC is that multiple organizations must agree on meaning. And that's just not going to fly. It is damn near a miracle that a huge number of institutions all agree on the Ethernet/IP command "Please take this bag of bytes closer to the machine named: <string of bytes>."

idlehand 4y ago
Never thought about that before. Ethernet supports extremely high levels of data transmission. ISB C for intrgrated charging ND data transfer makes sense, but why are there HDMI cables?
- GTP 4y ago
  I think it would be an overkill to use a networking protocol to connect exactly two devices. Plus if you have something specific to video stream transfer you could maybe do some optimization specific to that use case, although I can't think of any at the moment.
  - zrail 4y ago
    Excepting HDMI the parents examples are all networks with more than one peer. Thunderbolt and USB3 can both have arbitrary trees of nodes.
    - GTP 4y ago
      The parent comment was specifically wondering about HDMI, the other examples are given saying that those have some reason to exist, while he didn't see any reason for having HDMI instead of recycling some other protocol.

duped 4y ago

Every major corporation has multiple research organizations doing nothing but invest in things that don't have immediate shareholder value. What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. The scale at which these changes needs to happen to be practical are at the cluster level in a single data center. If you can propose something that fits that bill there are a few companies willing to pay you millions in salary as an engineering fellow to do it.

simplotek 4y ago
> What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. I'd put it differently: it's paying up hundreds of billions in infrastructure to have some sort of gain. And which gain is that exactly? I see a lot of "the world is dumb but I am smart" comments in this thread but I saw no one presenting any clear advantage or performance improvement claim about hypothetical replacements. I see a lot of "we need to rewrite things" comments but not a single case being made with a clear tradeoff being presented. Every single criticism of TCP/IP in this thread sounds like change for the sake of change, and changes that aren't even presented with tangible improvements in mind or a clear performance gain. Wouldn't that explain why TCP is so prevalent, and no one in their right mind thinks of replacing it?
- fragmede 4y ago
  I mean the goal is more performance, especially if you can get more performance out of the same hardware. Faster setup times, faster connections, more connections, maybe faster teardown. Lower contention on saturated links. Inside of the datacenter is a controlled environment where something like that could work. Replacing TCP over the Internet at large is going to be an uphill battle. Still, if we're replacing the whole thing, then simpler code on the client and server end would be nice.
  - simplotek 4y ago
    > I mean the goal is more performance, especially if you can get more performance out of the same hardware. Are there actual numbers demonstrating this? I mean, people are advocating wasting billions revamping infrastructure. What kind of performance are you hoping to buy with those billions? And are those gains worth it, or is just sake for the sake of change? Sometimes things are indeed good enough.

deleted 4y ago

[deleted]

simplotek 4y ago

> Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. If this was true then how do you explain that the likes of AWS, the same company who ended up investing in developing their own processor line, doesn't seem to agree that none of the pet peeves you mentioned are worth fixing?

emn13 4y ago
It's not obvious to me that replacing TCP really is harder than designing your "own" chip. Scarequotes here because those graviton chips (that's what you're referring to, I think?) are of course ARM chips, so they're not designing something fresh; they're adapting a very mature design to their own needs. In terms of interoperability, a custom chip based on a standard design is probably a simpler, more locally addressable problem than new network protocols. Isn't it plausible that graviton was designed yet TCP retained simply because graviton as a project is easier to complete successfully?
- deleted 4y ago
  [deleted]
xxpor 4y ago
https://aws.amazon.com/blogs/hpc/in-the-search-for-performance-theres-more-than-one-way-to-build-a-network/ https://aws.amazon.com/blogs/hpc/in-the-search-for-performan...

robertlagrant 4y ago

> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is just not true. Stuff needs to be funded and worth doing, and the internet, like almost everything, is built on making things worth paying for, but there are also loads of improvements everywhere are being made.

primeblue 4y ago

friendzis 4y ago

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols I think this is more of an artefact of horizontal scaling and port-contention. De-facto standard discovery mechanism DNS does not work with ports, so "well-known port" abstraction kinda fails. Http as tunnel mostly avoids/sidesteps this problem. > We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. This is weird take or I don't understand it. If you can communicate with an edge node in another network, but the edge node has issues communicating with some inner node (on your behalf), then, as a user, you have no hope of fixing that connectivity issue anyway, regardless of whether layered approach is used or not. This may be related to previous point about http as universal tunnel. Yes, this is a problem, but in a way that communications are effectively terminated at the edge node and monstrosity of stuff happens behind the scenes

patrec 4y ago
> De-facto standard discovery mechanism DNS does not work with ports Yes, it does, see SRV records.
- friendzis 4y ago
  I meant DNS A/AAAA queries with preconfigured/well-known ports being the default. While some applications/protocols/services do use some port discovery mechanism, I would argue it is nowhere close to being de-facto standard.
- bewo001 4y ago
  So true, but how many developers know about them? The API situation does not help either.
AtNightWeCode 4y ago
I would say ports are mainly a problem on layers below transport even though some tech overuse ports.

kortilla 4y ago

> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is severe bullshit on two fronts: - there is an immediate return on value - Google was driving this a decade+ ago for improvements in the data center (things like doubled+cancelable rpc, tcp cubic, quic, etc) - academia constantly attempts to make these improvements as well because researchers are super incentivized to dethrone tcp for the glory. There are constant attempts to re-invent various layers (IP, tcp, the non existent upper layers of the OSI, etc) that come out of academic conferences every year. The reason we’re still here is because our current stacks have been heavily optimized and tooled for production workloads. NICs can transparently re-assemble TCP segments for the OS and they can segment before transmit. You have to have a damn good value prop to throw away everything from software and hardware to careers and curriculum. It has to be a shitload better than the security nightmare of “return back each layer of the stack”.

gjulianm 4y ago

> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years. I work in a company that builds network troubleshooting/observability tools and we have some pretty experienced analysts to tell you what's wrong with the network. With that context, your idea of having any tool automatically diagnosing network issues is a pipe dream. The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners, failures aren't always easily reproducible, and with human configuration (and therefore errors) almost every step of the way. Even if a tool that "returns each layer of the stack" would be useful, it still would be far from enough to diagnose issues.

EricE 4y ago
"The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners" Ah, how people forget the early days of networking. I remember vividly the early days of the Networld/Interop trade show - Interop was in the name because if, as a vendor, your equipment couldn't integrate with the show network they would throw your booth off the show floor. That's how bad interoperability in the early days was!

cdogl 4y ago

I’ll defer to experts on the network-layer problems but im not sure what you see as the problem with converging on HTTP. It’s awkward and inelegant, but as an a backend application developer I never feel like it gets in my way.

jstimpfle 4y ago

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP If you have ever used multiple TCP or UDP connections in parallel on a single machine (doesn't matter if server or client) then you should realize that ports are required. Apart from that, you can run HTTP on other ports than 80. You can also use HTTP to load balance or do service discovery by means of redirects. (Caveat, I don't work in this field and can't say how solid the approach works in practice).

twawaaay 4y ago

I don't think you realise why this is so hard. The basic reason is that software at every level expects TCP/IP. And you can't drop in a translation layer because it will require at least the same amount of overhead as "real" TCP/IP. It is not a local problem, it is a global problem that affects basically every single piece of non-trivial software in existence. Even if you construct your datacenter with the new protocol you will run into problems that you can't run anything in it. Want Python? Sorry, have to rewrite it. And every Python library. And every Python application. Then you need to deal with problems that people who can run their scripts on their machines can't run them in datacenter. And so on. The reason nobody wants to do this is that they would be investing huge amount of money to solve a problem for everybody else. Because the only way to make TCP/IP replacement work is to make it completely free and available to everybody. There are much better ways to allocate your funds and precious top level engineers that let them distance themselves from competition temporarily.

starfallg 4y ago

> We still lack any way to communicate up and down the stack of an entire transaction, for example for debugging purposes. We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years. This violates the principle of encapsulation that the entire field of networking is based on, not to mention an massive security hole.

KaiserPro 4y ago

arka2147483647 4y ago

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. And then there's all the non-backend problems! The paper argues the in '3.1 Stream orientation' section, that stream orientation is a problem for TCP, and says that most apps send messages instead, and the better protocol should handle messages, natively, etc. Which is a good point I think. But back to TCP. What do you do, if you need to send Messages between applications in TCP? Preferably those Messages would be encrypted also. You could make up your own protocol, but you probably would rather not! So you use something that is readily available, and does messages, encryption, etc. Would be nice if there were also a ready to use load balancers, caches, tools to debug it, etc Now, what would be such a protocol. Why HTTPS, of course. So I kind of think that the lack of a low level Message Protocol has lead us, as an industry, to coalesce these features bit-by-bit on top of HTTP. It's not perfect by any means, but it does the job.

pclmulqdq 4y ago
HTTPS adds a tremendous amount of overhead to give you messaging. It's a lot better from a hyperscaler's perspective to replace TCP and not use the byte stream abstraction. After all, networks send messages. It's silly to throw that away at one layer and try to get it back at the next layer.

guenthert 4y ago

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. Nobody forces them though. It would be much easier to publish a standard port number mapping than to develop a (or multiple) new protocols. Now you just need to motivate people to use it.

still_grokking 4y ago

> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. I would affirm that. It's imho true for almost everything in IT tech. How computers "work" today is just pure madness when looked anyhow closer. Everything's a result of some "historic accidents" back in the days, and from that the usual race to the bottom caused by market powers. Nobody is willing to touch any of the lower layers no matter how crazy they are form today's viewpoint. We just shovel new layers on top to paper over the mistakes of the past. Nothing gets repaired, or actually what would be more more important, rethought form the ground up in light of new technological possibilities and changed requirements. I understand from the economic standpoint how this comes. But I'm also quite sure we didn't make any fundamental improvements in the last 50 years of computing. That's a very bad sign when everything in a field that's not even really 100 years old is frozen in time since 50 years because everything's so fragile and complex that fundamental changes aren't possible. This looks like a text book example of a house of cards… Given how vital IT tech is to modern life I fear that this will crash at some point in the worst way possible. And even if it won't crash, which is really strongly hope, we will never have nice things again as nothing of the old rotten things can be reasonably changed.