9 ms·

What would happen if we didn't use TCP or UDP?

215 points by Hawzen 2y ago

FuriouslyAdrift 2y ago
Infiniband works great...
pkkm 2y ago
You don't need to make up your own for this experiment. There's already a pretty old protocol that's far superior to TCP, but failed to get adoption because of network hardware dropping everything other than TCP and UDP. It's called SCTP.
- nubinetwork 2y ago
  I thought the point was that they wanted to use something that didn't exist yet (either in RL use or in RFC form)...
- valorzard 2y ago
  SCTP is really cool, I first found out about it because it’s the basis for WebRTC data channels. It’s basically reliable UDP, but you can turn off the reliability if you want. Makes me wonder why QUIC exists when SCTP does…
  - 15155 2y ago
    > why QUIC exists when SCTP does Because QUIC uses UDP, which is supported by most/all intermediate routing equipment.
    - noam_k 2y ago
      Is this a real issue? SCTP runs over IP, so unless your talking about firewalls and such, the support should be there. Edit: a quick search showed that NAT traversal is an issue (of course!)
      - evrimoztamur 2y ago
        Hole punching is perhaps why UDP is de facto?
      - Karrot_Kream 2y ago
        Yes this is called protocol ossification [1] or ossification for short. Other transport layer protocol rollouts have been stymied by ossification such as MPTCP. QUIC specifically went with UDP to prevent ossification yet if you hang out in networking forums you'll still find netops who want to block QUIC if they can. [1]: https://en.m.wikipedia.org/wiki/Protocol_ossification https://en.m.wikipedia.org/wiki/Protocol_ossification
        crims0n 2y ago
        Because from an enterprise security perspective, it breaks a lot of tools. You can’t decrypt, IDS/IPS signatures don’t work, and you lose visibility to what is going on in your network.
        Dylan16807 2y ago
        Wrapping everything in UDP breaks the same tools but it's more obnoxious for everyone involved.
        Karrot_Kream 2y ago
        Yes I know why netops want to block QUIC but that just shows the tension between the folks who want to build new functionality and the folks who are in charge of enterprise security. I get it, I've held SRE-like roles in the past myself. When you're in charge of security and maintenance, you have no positive incentive to allow innovation. New functionality gives you nothing. You never get called into a meeting and congratulated for new functionality you help unlock. You only get called in if something goes wrong, and so you have every incentive to monitor, lock down, and steer traffic as best as you can so things don't go wrong on your watch. IMO it's a structural problem that blocks a lot of innovation. The same thing happens when a popular open source project that's author led switches to an external maintainer. When the incentives to block innovation are stronger than the incentives to allow it, you get ossification.
    - saurik 2y ago
      WebRTC runs SCTP over DTLS over UDP.
    - nly 2y ago
      The whole point of UDP is to allow alternative protocols to be implemented on top. SCTPs mistake was it wasn't implemented as a userland library on top of UDP to begin with.
    - Imustaskforhelp 2y ago
      I am now genuinely wondering Maybe its me being stupid but why don't we use quic always instead of tcp? I think it has to do with something that I read that tcp can do upto 1000 connections simultaneously no worries and they won't interfere with each other's bandwidth / impact each other , but udp does make it possible for one service being very high to impact other. There was this latest test by anton putra with udp vs tcp and the difference was IIRC negligible. Someone said that he should probably use udp in kernel mode to essentially get insane performance I amnot sure
      - WorldMaker 2y ago
        > Maybe its me being stupid but why don't we use quic always instead of tcp? A big reason is because QUIC is a lot younger than TCP and it will take a while for all the use cases of TCP to decide (if they are actively maintained and looking at possible upgrades) if QUIC is a good option worth testing. QUIC's rollout so far hasn't been entirely without bugs/controversies/quirks/obstacles/challenges. You still see a lot more HTTP/2 than HTTP/3 connections in the current wild and that doesn't seem to be changing near as fast as major providers upgraded HTTP/1.x to HTTP/2. There's still a bunch of languages and contemporary OSes without strong QUIC support. (Just the other day on HN was a binding for Erlang to msquic, IIRC, for a first pass at QUIC support in that language.) Some point soon QUIC might start feeling as rock solid as TCP, but today TCP is (decades of) rock solid and QUIC is still a lot new and a little quirky.
        walth 2y ago
        Safari on IOS still has a ton lingering HTTP/3 / QUIC bugs. I think it is to the point that if your user base doesn't warrant it, (i.e. you are targeting well connected devices with minimal latency/packetloss) it's not even worth turning HTTP/3 on
        Imustaskforhelp 2y ago
        so quic just lacks the decades of experience but is a better protocol than tcp overall ? That is kind of nice to know actually. The support will come considering its built on top of UDP. You just need people pushing and google is already pushing it hard . The main problem is quic's support in languages. But support will come.So after reading this comment of yours , I am pretty optimistic about quic overall
    - lttlrck 2y ago
      That's not the reason, SCTP over UDP was already standardized
  - tepmoc 2y ago
    https://news.ycombinator.com/item?id=41488416 https://news.ycombinator.com/item?id=41488416
  - bdd8f1df777b 2y ago
    Others have mentioned protocol ossification which is indeed the primary reason. A secondary reason is that QUIC fuses TLS so its latency is further reduced by one RTT. For high latency networks, the difference is palpable.
  - signa11 2y ago
    > It’s basically reliable UDP, ... more importantly though, it transmits multiple independent streams of message chunks in parallel. similarity with UDP ends at message oriented nature of the protocol. closest equivalent for TCP would be MPTCP I suppose ?
  - 0x457 2y ago
    Because pure SCTP can't survive outside your LAN, thanks to everything in-between you and your destination. Why not use SCTP on top of UDP? Well, because one of the main benefits of QUIC is TLS being at its core. SCTP you're talking about runs on top of DTLS on top of UDP. DTLS has issues on its own, but even if it didn't it wouldn't beat QUIC in TTFB.
  - ianburrell 2y ago
    SCTP can run over UDP. QUIC is supposed to be faster than SCTP by combining layers and eliminate round trips. Also, QUIC is a stream protocol like TCP. SCTP makes messages explicit. Both have multiplexing which is why seem different.
- jeroenhd 2y ago
  SCTP is fascinating because it's one of the backbone technologies that makes communication possible for most people on the planet (as the mobile network stack pretty much relies on it), yet it's effectively unsupported on almost every consumer device. If you want to use it, you're probably going to have to ship a userland implementation that needs privileges to open a raw network socket, because kernel implementations are rare (and often slow). We could've had it as a QUIC replacement if it weren't for terrible middleboxes and IPv4 NAT screwing everyone over once again. Hell, NAT wouldn't even have been an issue had SCTP support been widespread before consumer NAT devices started being implemented. It's a prime example of how new network protocols now have to lie and deceive to work over the internet. TLS needs to pretend to be TLS 1.2, QUIC needs to pretend to be an application on top of UDP while reimplementing SCTP and TCP, and even things like secure DNS are now piped over HTTPS just in case a shitty middlebox can't deal with raw TLS.
  - tsimionescu 2y ago
    While the gist of your post is spot on, I do feel it should be noted that DoH is preferred over DoT not to protect from middleboxes that don't work properly, but from middleboxes that are actively trying to outright censor encrypted DNS, but can't afford to snoop on/prevent all HTTPS traffic. It's an anti-censorship measure, not a compatibility measure.
    - capitainenemo 2y ago
      It's also about user privacy with ISPs as well as anti-censorship. https://www.cloudflare.com/learning/dns/dns-over-tls/ https://www.cloudflare.com/learning/dns/dns-over-tls/
      - tsimionescu 2y ago
        DoT (DNS over TLS) would have been enough for privacy from your ISP, using a dedicated port. It's only when you want to protect from censorship that you need to hide the (encrypted) DNS traffic among other traffic that can't be easily blocked.
- immibis 2y ago
  It's actually universal within a certain niche. I think phone networks are doing just about everything over SCTP internally. When SS7 links get replaced they get replaced with something that uses SCTP. Not sure of the details because I don't work there. Related: there's a parallel Internet with a different root of number allocation called GRX/IPX (GPRS Roaming Exchange/Internetwork Packet Exchange)
- AnotherGoodName 2y ago
  IPX is another that was very common just 20years ago. Many old games only support ipx networking and you need to run an ipx over tcp emulator to play them multiplayer nowadays.
- dakra137 2y ago
  TCP would be fine if it had the concept of message, in addition to stream. The sender sends a message. It flows to the recipient. The recipient program can specify that a receive receives the entire message, no more and no less, as long as the application's target input buffer is large enough. SCTP does this. A shim on top of TCP socket receive could also do this also, as long as there is a convention to prefix each message with a length field, say 16 bits, with the MSB indicating that the message is incomplete and is continued in the next length delimited segment.
globular-toast 2y ago
What you want is a packet socket: sock_raw = socket(AF_PACKET , SOCK_RAW , htons(ETH_P_ALL)); IP networks should forward anything, but NAT is a major problem. Would be interesting to try with IPv6.
- Hawzen 2y ago
  That's very interesting! I'll try out IPv6
  - globular-toast 2y ago
    If you want to play with packet sockets you might find my notebook useful where I was going to implement TCP/IP from scratch (well, from layer 2 up): https://github.com/georgek/notebooks/blob/master/internet.ipynb https://github.com/georgek/notebooks/blob/master/internet.ip... I did ICMP over IP but got bored before I got to TCP, though (it's way more complicated than IP). You could drop in your custom protocol in place of ICMP.
- ay 2y ago
  To second this and expand on the reasons behind: The way the NATs (network address translators) are sharing the scarce public IPv4 addresses is by multiplexing on the transport level fields (ports in case of TCP/UDP and IDs or inner packet transport level fields in case of ICMP). Since they are unaware of your protocol, they get into a “special case mode”, which on a naive translator might consume a whole IP address (so you would really make a network admin with a few of those, because you exhaust all their available addresses :-) ; but on the carrier grade NAT there are safeguards against it and the packets are just dropped.
  - geokon 2y ago
    This is outside my area of expertise.. so naiive question.. but ports aren't tied to the protocol .. right? If you open a raw socket, it's still on some associated port number. NAT traversal multiplexes ports.. so why would that preclude using any arbitrary protocol?
megadata 2y ago
[flagged]
casey2 2y ago
It's weird to me that the definition of portable has changed from "can be moved" to "can be moved to another machine" to "can be moved to another environment" to "can be moved to another program" We had the option to make every object addressable and we chose not to. Why keep sweating over it, just accept that software only works on the devs machine and ship that.
paweladamczuk 2y ago
I always assumed any TCP/UDP packets would get captured by the OS network stack in order to be sent only to the processes listening on specific ports. I guess this is a security feature, since a process cannot even listen on some ports without having elevated privileges. I wouldn't expect another process being able to capture all this traffic anyway. This would also require a mechanism of sending the same stream to multiple processes (TCP listeners and all-protocol listeners). But I didn't even know it was possible to capture traffic from multiple transport layer protocols using a syscall, perhaps that syscall requires elevated privileges itself..?
- Hawzen 2y ago
  > perhaps that syscall requires elevated privileges itself..? You are exactly right
- globular-toast 2y ago
  It requires elevated privileges, but this is how programs like tcpdump and wireshark work. On Linux it's also possible to give a program these permissions for any user by setting "capabilities", specifically cap_net_admin and cap_net_raw.
- netbsdusers 2y ago
  DHCP services require this ability to receive and send UDP packets on raw sockets, barring a few advanced systems like Solaris that provide them with necessary facilities. Usually they install a BPF module on the socket to filter out uninteresting packets. https://kb.isc.org/docs/aa-00379 https://kb.isc.org/docs/aa-00379
Emjayen 2y ago
As someone who has implemented various transport protocols (both standard and custom) the biggest hurdle in layering atop IP will not be the gauntlet of WAN routers, surprisingly, rather consumer NAT devices. One interesting case for a particular Netgear family of routers was that, while the traffic would survive end-to-end it was not without corruption - very specific corruption which took the form of zeroing the first 4 bytes. Given this aligns with where the src/dst ports would normally be, I suspect it was being treated as TCP/UDP albeit without the actual translation path taken.
LinuxBender 2y ago
What would happen if we didn't use TCP or UDP? One would have a hard time communicating with anyone. The internet has standardized around TCP and UDP. There are too many devices in most paths that will not be able to handle other protocols. Replacing all the hardware on the internet would take even longer than deprecating IPv4 in my pessimistic opinion. To get around this there would have to be some significant gains of the new protocol that would warrant big expenses in every corporation and government and all the hardware manufacturers would all have to agree on implementing support and further agree on interpretation of the new RFC.
- mrbluecoat 2y ago
  Agreed. This statement sums up the obvious: > Granted, the server was just two hops away from the client, and it didn't have to pass through the scary sea of the internet
  - immibis 2y ago
    Actually the Internet at large is fine with any protocol on top of IP. It's your home router's NAT function that can only handle UDP and TCP. Set it to bridge mode and use your computer as the router (if you need one) and you can send anything from that computer. If you have CGNAT you're still screwed. Get one of those free ipv6 tunnels.
jmclnx 2y ago
We would all be on UUCP. That was doing similar things before TCP/UDP.
- kjs3 2y ago
  UUCP was only a couple years before TCP, and isn't really an equivalent (it's fancy file transfer with a little remote command exec sprinkled on in the end) and natively its dialup oriented; it usually requires some other L3 protocol to run over networks. There was a time when IPX/SPX was a contender. Xerox pitched XNS directly at TCP. DECNet/OSI was around. There were a lot of others...lot's of experimenting going on at the time.
  - immibis 2y ago
    UUCP's primary distinction is that it's store-and-forward.
- Karrot_Kream 2y ago
  If you're interested in a more "modern" UUCP, there's NNCP [1] (HTTPS ver here [2].) It continues to mostly be a file-transfer protocol with a bit of signaling added on. [1]: http://www.nncpgo.org/ http://www.nncpgo.org/ [2]: https://nncp.mirrors.quux.org/ https://nncp.mirrors.quux.org/
MisterTea 2y ago
There's also Bell Labs IL - Internet Link - used to speed up 9P links for Plan 9 over LANs. https://doc.cat-v.org/plan_9/4th_edition/papers/il/ https://doc.cat-v.org/plan_9/4th_edition/papers/il/ And this brings me to why I love networking on Plan 9. First off the dial string, net!address!service, passes the network along with the address and service port letting you easily switch protocols as needed. e.g. a program could listen on IL port 666 using the dial string il!*!666 and the client dials il!10.1.1.1!666. Second, that lets you dial and listen on any protocol from any program. If one wanted to use raw Ethernet you use the dial string ether0!aabbccddeeff. If I wanted to add a protocol like quic or a dead one like ipx/spx I just need to mount it to /net and keep the semantics the same as other services in net and any program can dial that protocol - ezpz user space networking stacks. Powerful abstraction.
- kjs3 2y ago
  Reminds me a bit of UUCP or AT&T Datakit addressing.
  - MisterTea 2y ago
    Datakit was in fact one of the early supported networks in Plan 9. You could dial into a 9 machine over datakit, mount an outward facing IP stack over your /net and your on-line: http://man.postnix.pw/plan_9_2e/3/datakit http://man.postnix.pw/plan_9_2e/3/datakit
    - kjs3 2y ago
      Interesting, and I suppose not surprising given the source. AT&T kept on with Datakit long after circuit switched data networking became passe.
pjmlp 2y ago
We would be using one of the other protocols that were common at the time, before TCP/UDP/IP took over everything.
jumperabg 2y ago
We would be sending data via rockets guided by pigeons inside of them.
fancyfredbot 2y ago
It feels like the article ends with a cliffhanger! Why did a single packet of the custom protocol get through with all later packets dropped? Does anyone know?
- wrs 2y ago
  Yeah! I am baffled that even one got through, but given that it did, why only one? And I would’ve immediately tried the “every protocol” version at that point…
- kazen44 2y ago
  my guess is the first packet got through a firewall, which created a flow. subsequent packets then got blocked because the firewall has no way of matching this to an existing flow.
  - m3talsmith 2y ago
    Definitely the most likely
sema4hacker 2y ago
I think a more interesting question is: what if internet protocols and routing equipment were designed from scratch today? Besides much larger packets, I'm guessing something basic in the style of UDP would be chosen to replace HTTP to simplify query-response lookups, a much simpler streaming protocol would be chosen to replace TCP and support all the video playing going on, and those two protocols would more efficiently handle the vast majority of traffic.
- wmf 2y ago
  QUIC is redesigned TCP.
- ddejohn 2y ago
  There have been a few of these types of discussions on HN -- search phrases like "internet from scratch" -- really fun reading.
theLegionWithin 2y ago
ah, the "what would happen if we reinvented the wheel?" postulate.
dakra137 2y ago
We might use Ethernet WAN and LAN. See https : / / neosnetworks . com / resources / blog / what-is-ethernet-wan/

215 points by Hawzen 2y ago

Infiniband works great...

pkkm 2y ago

You don't need to make up your own for this experiment. There's already a pretty old protocol that's far superior to TCP, but failed to get adoption because of network hardware dropping everything other than TCP and UDP. It's called SCTP.

nubinetwork 2y ago
I thought the point was that they wanted to use something that didn't exist yet (either in RL use or in RFC form)...
valorzard 2y ago
SCTP is really cool, I first found out about it because it’s the basis for WebRTC data channels. It’s basically reliable UDP, but you can turn off the reliability if you want. Makes me wonder why QUIC exists when SCTP does…
- 15155 2y ago
  > why QUIC exists when SCTP does Because QUIC uses UDP, which is supported by most/all intermediate routing equipment.
  - noam_k 2y ago
    Is this a real issue? SCTP runs over IP, so unless your talking about firewalls and such, the support should be there. Edit: a quick search showed that NAT traversal is an issue (of course!)
    - evrimoztamur 2y ago
      Hole punching is perhaps why UDP is de facto?
    - Karrot_Kream 2y ago
      Yes this is called protocol ossification [1] or ossification for short. Other transport layer protocol rollouts have been stymied by ossification such as MPTCP. QUIC specifically went with UDP to prevent ossification yet if you hang out in networking forums you'll still find netops who want to block QUIC if they can. [1]: https://en.m.wikipedia.org/wiki/Protocol_ossification https://en.m.wikipedia.org/wiki/Protocol_ossification
      - crims0n 2y ago
        Because from an enterprise security perspective, it breaks a lot of tools. You can’t decrypt, IDS/IPS signatures don’t work, and you lose visibility to what is going on in your network.
        Dylan16807 2y ago
        Wrapping everything in UDP breaks the same tools but it's more obnoxious for everyone involved.
        Karrot_Kream 2y ago
        Yes I know why netops want to block QUIC but that just shows the tension between the folks who want to build new functionality and the folks who are in charge of enterprise security. I get it, I've held SRE-like roles in the past myself. When you're in charge of security and maintenance, you have no positive incentive to allow innovation. New functionality gives you nothing. You never get called into a meeting and congratulated for new functionality you help unlock. You only get called in if something goes wrong, and so you have every incentive to monitor, lock down, and steer traffic as best as you can so things don't go wrong on your watch. IMO it's a structural problem that blocks a lot of innovation. The same thing happens when a popular open source project that's author led switches to an external maintainer. When the incentives to block innovation are stronger than the incentives to allow it, you get ossification.
  - saurik 2y ago
    WebRTC runs SCTP over DTLS over UDP.
  - nly 2y ago
    The whole point of UDP is to allow alternative protocols to be implemented on top. SCTPs mistake was it wasn't implemented as a userland library on top of UDP to begin with.
  - Imustaskforhelp 2y ago
    I am now genuinely wondering Maybe its me being stupid but why don't we use quic always instead of tcp? I think it has to do with something that I read that tcp can do upto 1000 connections simultaneously no worries and they won't interfere with each other's bandwidth / impact each other , but udp does make it possible for one service being very high to impact other. There was this latest test by anton putra with udp vs tcp and the difference was IIRC negligible. Someone said that he should probably use udp in kernel mode to essentially get insane performance I amnot sure
    - WorldMaker 2y ago
      > Maybe its me being stupid but why don't we use quic always instead of tcp? A big reason is because QUIC is a lot younger than TCP and it will take a while for all the use cases of TCP to decide (if they are actively maintained and looking at possible upgrades) if QUIC is a good option worth testing. QUIC's rollout so far hasn't been entirely without bugs/controversies/quirks/obstacles/challenges. You still see a lot more HTTP/2 than HTTP/3 connections in the current wild and that doesn't seem to be changing near as fast as major providers upgraded HTTP/1.x to HTTP/2. There's still a bunch of languages and contemporary OSes without strong QUIC support. (Just the other day on HN was a binding for Erlang to msquic, IIRC, for a first pass at QUIC support in that language.) Some point soon QUIC might start feeling as rock solid as TCP, but today TCP is (decades of) rock solid and QUIC is still a lot new and a little quirky.
      - walth 2y ago
        Safari on IOS still has a ton lingering HTTP/3 / QUIC bugs. I think it is to the point that if your user base doesn't warrant it, (i.e. you are targeting well connected devices with minimal latency/packetloss) it's not even worth turning HTTP/3 on
      - Imustaskforhelp 2y ago
        so quic just lacks the decades of experience but is a better protocol than tcp overall ? That is kind of nice to know actually. The support will come considering its built on top of UDP. You just need people pushing and google is already pushing it hard . The main problem is quic's support in languages. But support will come.So after reading this comment of yours , I am pretty optimistic about quic overall
  - lttlrck 2y ago
    That's not the reason, SCTP over UDP was already standardized
- tepmoc 2y ago
  https://news.ycombinator.com/item?id=41488416 https://news.ycombinator.com/item?id=41488416
- bdd8f1df777b 2y ago
  Others have mentioned protocol ossification which is indeed the primary reason. A secondary reason is that QUIC fuses TLS so its latency is further reduced by one RTT. For high latency networks, the difference is palpable.
- signa11 2y ago
  > It’s basically reliable UDP, ... more importantly though, it transmits multiple independent streams of message chunks in parallel. similarity with UDP ends at message oriented nature of the protocol. closest equivalent for TCP would be MPTCP I suppose ?
- 0x457 2y ago
  Because pure SCTP can't survive outside your LAN, thanks to everything in-between you and your destination. Why not use SCTP on top of UDP? Well, because one of the main benefits of QUIC is TLS being at its core. SCTP you're talking about runs on top of DTLS on top of UDP. DTLS has issues on its own, but even if it didn't it wouldn't beat QUIC in TTFB.
- ianburrell 2y ago
  SCTP can run over UDP. QUIC is supposed to be faster than SCTP by combining layers and eliminate round trips. Also, QUIC is a stream protocol like TCP. SCTP makes messages explicit. Both have multiplexing which is why seem different.
jeroenhd 2y ago
SCTP is fascinating because it's one of the backbone technologies that makes communication possible for most people on the planet (as the mobile network stack pretty much relies on it), yet it's effectively unsupported on almost every consumer device. If you want to use it, you're probably going to have to ship a userland implementation that needs privileges to open a raw network socket, because kernel implementations are rare (and often slow). We could've had it as a QUIC replacement if it weren't for terrible middleboxes and IPv4 NAT screwing everyone over once again. Hell, NAT wouldn't even have been an issue had SCTP support been widespread before consumer NAT devices started being implemented. It's a prime example of how new network protocols now have to lie and deceive to work over the internet. TLS needs to pretend to be TLS 1.2, QUIC needs to pretend to be an application on top of UDP while reimplementing SCTP and TCP, and even things like secure DNS are now piped over HTTPS just in case a shitty middlebox can't deal with raw TLS.
- tsimionescu 2y ago
  While the gist of your post is spot on, I do feel it should be noted that DoH is preferred over DoT not to protect from middleboxes that don't work properly, but from middleboxes that are actively trying to outright censor encrypted DNS, but can't afford to snoop on/prevent all HTTPS traffic. It's an anti-censorship measure, not a compatibility measure.
  - capitainenemo 2y ago
    It's also about user privacy with ISPs as well as anti-censorship. https://www.cloudflare.com/learning/dns/dns-over-tls/ https://www.cloudflare.com/learning/dns/dns-over-tls/
    - tsimionescu 2y ago
      DoT (DNS over TLS) would have been enough for privacy from your ISP, using a dedicated port. It's only when you want to protect from censorship that you need to hide the (encrypted) DNS traffic among other traffic that can't be easily blocked.
immibis 2y ago
It's actually universal within a certain niche. I think phone networks are doing just about everything over SCTP internally. When SS7 links get replaced they get replaced with something that uses SCTP. Not sure of the details because I don't work there. Related: there's a parallel Internet with a different root of number allocation called GRX/IPX (GPRS Roaming Exchange/Internetwork Packet Exchange)
AnotherGoodName 2y ago
IPX is another that was very common just 20years ago. Many old games only support ipx networking and you need to run an ipx over tcp emulator to play them multiplayer nowadays.
dakra137 2y ago
TCP would be fine if it had the concept of message, in addition to stream. The sender sends a message. It flows to the recipient. The recipient program can specify that a receive receives the entire message, no more and no less, as long as the application's target input buffer is large enough. SCTP does this. A shim on top of TCP socket receive could also do this also, as long as there is a convention to prefix each message with a length field, say 16 bits, with the MSB indicating that the message is incomplete and is continued in the next length delimited segment.

globular-toast 2y ago

What you want is a packet socket: sock_raw = socket(AF_PACKET , SOCK_RAW , htons(ETH_P_ALL)); IP networks should forward anything, but NAT is a major problem. Would be interesting to try with IPv6.

Hawzen 2y ago
That's very interesting! I'll try out IPv6
- globular-toast 2y ago
  If you want to play with packet sockets you might find my notebook useful where I was going to implement TCP/IP from scratch (well, from layer 2 up): https://github.com/georgek/notebooks/blob/master/internet.ipynb https://github.com/georgek/notebooks/blob/master/internet.ip... I did ICMP over IP but got bored before I got to TCP, though (it's way more complicated than IP). You could drop in your custom protocol in place of ICMP.
ay 2y ago
To second this and expand on the reasons behind: The way the NATs (network address translators) are sharing the scarce public IPv4 addresses is by multiplexing on the transport level fields (ports in case of TCP/UDP and IDs or inner packet transport level fields in case of ICMP). Since they are unaware of your protocol, they get into a “special case mode”, which on a naive translator might consume a whole IP address (so you would really make a network admin with a few of those, because you exhaust all their available addresses :-) ; but on the carrier grade NAT there are safeguards against it and the packets are just dropped.
- geokon 2y ago
  This is outside my area of expertise.. so naiive question.. but ports aren't tied to the protocol .. right? If you open a raw socket, it's still on some associated port number. NAT traversal multiplexes ports.. so why would that preclude using any arbitrary protocol?

megadata 2y ago

[flagged]

casey2 2y ago

It's weird to me that the definition of portable has changed from "can be moved" to "can be moved to another machine" to "can be moved to another environment" to "can be moved to another program" We had the option to make every object addressable and we chose not to. Why keep sweating over it, just accept that software only works on the devs machine and ship that.

paweladamczuk 2y ago

I always assumed any TCP/UDP packets would get captured by the OS network stack in order to be sent only to the processes listening on specific ports. I guess this is a security feature, since a process cannot even listen on some ports without having elevated privileges. I wouldn't expect another process being able to capture all this traffic anyway. This would also require a mechanism of sending the same stream to multiple processes (TCP listeners and all-protocol listeners). But I didn't even know it was possible to capture traffic from multiple transport layer protocols using a syscall, perhaps that syscall requires elevated privileges itself..?

Hawzen 2y ago
> perhaps that syscall requires elevated privileges itself..? You are exactly right
globular-toast 2y ago
It requires elevated privileges, but this is how programs like tcpdump and wireshark work. On Linux it's also possible to give a program these permissions for any user by setting "capabilities", specifically cap_net_admin and cap_net_raw.
netbsdusers 2y ago
DHCP services require this ability to receive and send UDP packets on raw sockets, barring a few advanced systems like Solaris that provide them with necessary facilities. Usually they install a BPF module on the socket to filter out uninteresting packets. https://kb.isc.org/docs/aa-00379 https://kb.isc.org/docs/aa-00379

Emjayen 2y ago

As someone who has implemented various transport protocols (both standard and custom) the biggest hurdle in layering atop IP will not be the gauntlet of WAN routers, surprisingly, rather consumer NAT devices. One interesting case for a particular Netgear family of routers was that, while the traffic would survive end-to-end it was not without corruption - very specific corruption which took the form of zeroing the first 4 bytes. Given this aligns with where the src/dst ports would normally be, I suspect it was being treated as TCP/UDP albeit without the actual translation path taken.

LinuxBender 2y ago

What would happen if we didn't use TCP or UDP? One would have a hard time communicating with anyone. The internet has standardized around TCP and UDP. There are too many devices in most paths that will not be able to handle other protocols. Replacing all the hardware on the internet would take even longer than deprecating IPv4 in my pessimistic opinion. To get around this there would have to be some significant gains of the new protocol that would warrant big expenses in every corporation and government and all the hardware manufacturers would all have to agree on implementing support and further agree on interpretation of the new RFC.

mrbluecoat 2y ago
Agreed. This statement sums up the obvious: > Granted, the server was just two hops away from the client, and it didn't have to pass through the scary sea of the internet
- immibis 2y ago
  Actually the Internet at large is fine with any protocol on top of IP. It's your home router's NAT function that can only handle UDP and TCP. Set it to bridge mode and use your computer as the router (if you need one) and you can send anything from that computer. If you have CGNAT you're still screwed. Get one of those free ipv6 tunnels.

jmclnx 2y ago

We would all be on UUCP. That was doing similar things before TCP/UDP.

kjs3 2y ago
UUCP was only a couple years before TCP, and isn't really an equivalent (it's fancy file transfer with a little remote command exec sprinkled on in the end) and natively its dialup oriented; it usually requires some other L3 protocol to run over networks. There was a time when IPX/SPX was a contender. Xerox pitched XNS directly at TCP. DECNet/OSI was around. There were a lot of others...lot's of experimenting going on at the time.
- immibis 2y ago
  UUCP's primary distinction is that it's store-and-forward.
Karrot_Kream 2y ago
If you're interested in a more "modern" UUCP, there's NNCP [1] (HTTPS ver here [2].) It continues to mostly be a file-transfer protocol with a bit of signaling added on. [1]: http://www.nncpgo.org/ http://www.nncpgo.org/ [2]: https://nncp.mirrors.quux.org/ https://nncp.mirrors.quux.org/

MisterTea 2y ago

There's also Bell Labs IL - Internet Link - used to speed up 9P links for Plan 9 over LANs. https://doc.cat-v.org/plan_9/4th_edition/papers/il/ https://doc.cat-v.org/plan_9/4th_edition/papers/il/ And this brings me to why I love networking on Plan 9. First off the dial string, net!address!service, passes the network along with the address and service port letting you easily switch protocols as needed. e.g. a program could listen on IL port 666 using the dial string il!*!666 and the client dials il!10.1.1.1!666. Second, that lets you dial and listen on any protocol from any program. If one wanted to use raw Ethernet you use the dial string ether0!aabbccddeeff. If I wanted to add a protocol like quic or a dead one like ipx/spx I just need to mount it to /net and keep the semantics the same as other services in net and any program can dial that protocol - ezpz user space networking stacks. Powerful abstraction.

kjs3 2y ago
Reminds me a bit of UUCP or AT&T Datakit addressing.
- MisterTea 2y ago
  Datakit was in fact one of the early supported networks in Plan 9. You could dial into a 9 machine over datakit, mount an outward facing IP stack over your /net and your on-line: http://man.postnix.pw/plan_9_2e/3/datakit http://man.postnix.pw/plan_9_2e/3/datakit
  - kjs3 2y ago
    Interesting, and I suppose not surprising given the source. AT&T kept on with Datakit long after circuit switched data networking became passe.

pjmlp 2y ago

We would be using one of the other protocols that were common at the time, before TCP/UDP/IP took over everything.

jumperabg 2y ago

We would be sending data via rockets guided by pigeons inside of them.

fancyfredbot 2y ago

It feels like the article ends with a cliffhanger! Why did a single packet of the custom protocol get through with all later packets dropped? Does anyone know?

wrs 2y ago
Yeah! I am baffled that even one got through, but given that it did, why only one? And I would’ve immediately tried the “every protocol” version at that point…
kazen44 2y ago
my guess is the first packet got through a firewall, which created a flow. subsequent packets then got blocked because the firewall has no way of matching this to an existing flow.
- m3talsmith 2y ago
  Definitely the most likely

sema4hacker 2y ago

I think a more interesting question is: what if internet protocols and routing equipment were designed from scratch today? Besides much larger packets, I'm guessing something basic in the style of UDP would be chosen to replace HTTP to simplify query-response lookups, a much simpler streaming protocol would be chosen to replace TCP and support all the video playing going on, and those two protocols would more efficiently handle the vast majority of traffic.

wmf 2y ago
QUIC is redesigned TCP.
ddejohn 2y ago
There have been a few of these types of discussions on HN -- search phrases like "internet from scratch" -- really fun reading.

theLegionWithin 2y ago

ah, the "what would happen if we reinvented the wheel?" postulate.

dakra137 2y ago

We might use Ethernet WAN and LAN. See https : / / neosnetworks . com / resources / blog / what-is-ethernet-wan/