6 ms·

I find it interesting how computing is starting to mature. It used to be something that only the latest generation was privy of. Everything was state of the art

I find it interesting how computing is starting to mature. It used to be something that only the latest generation was privy of. Everything was state of the art and always changing. My parents didn't understand me playing video games, it was not something they could grasp. Now, my daughter will probably be there when the Internet is a 100 years old, some ancient invention from a generation long gone. Computing will feel very different from how it feels to me.

dale_glass 3y ago
Some things about that suck though. Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already. TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. Of course people came up with workarounds, but that's mostly adding a bunch of different layers on top. So on the surface everything looks stable, but in reality there's a huge amount of protocols piggybacking on top of TCP and UDP.
- scrollaway 3y ago
  What happened to QUIC? Didn’t google want to have a replacement for tcp?
  - dale_glass 3y ago
    QUIC runs on top of UDP, so it inherits all the limitations of the stack.
- Bluecobra 3y ago
  Jumbo/9000 byte frames have been a thing for a long time now and a lot of switches enable it by default so you can definitely reap the benefits of 100G/400G. The problem is that every L3 hop also needs to support a higher MTU if you want to route this traffic without fragmentation.
  - nicholasbraker 3y ago
    And as such is mostly used within datacenter fabrics.
    - allarm 3y ago
      And that’s reasonable, because loss of a large datagram has larger impact. Thus jumbos in data centers, but not in the wild Internet where losses are more common.
  - toast0 3y ago
    This would be good enough, and allow for progressive enhancement, except that path MTU detection is somewhere between fragile and broken on the internet. So maybe you run 9000 packets on internal interfaces, and 1500 externally, or more often, that's too hard and 1500 is it. There are 9000 internet exchanges, so maybe there is hope, but I haven't seen any progress reports.
- throw0101c 3y ago
  > Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already. How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is? I can see some server applications, but we've had offloading for years now: * https://en.wikipedia.org/wiki/TCP_offload_engine https://en.wikipedia.org/wiki/TCP_offload_engine Netflix is hitting (encrypted) 800 Gb/s: * https://www.youtube.com/watch?v=36qZYL5RlgY https://www.youtube.com/watch?v=36qZYL5RlgY * https://news.ycombinator.com/item?id=32519881 https://news.ycombinator.com/item?id=32519881 > TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. Tallk to the vendors of middleware boxes (e.g., (CPE) firewalls): we've had (say) SCTP and DCCP for twenty years that offer variations, but they're never allowed to pass through. No new Layer 4 can really be allowed through because of ossification, and so kludging everything to TCP/UDP port 80/443 is what we're left with. NAT and a lack of end-to-end also doesn't help: one has to futz around with things like STUN and TURN.
  - dale_glass 3y ago
    > How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?\ Well, here's my personal experience for instance. If you take a Linux server and try to do put a 10G interface into a bridge, that disables a good amount of the offload capability of the network card. As a result you don't get 10G anymore even on fairly decent hardware. So VMs in a bridge with a 10G interface is already a configuration that doesn't quite work. VFIO does the trick fortunately. Also, surely even for ASICs there's a cost that comes with having to handle 150M packets/s. And yeah, you can do those speeds on switches and devices with limited functionality, but maybe I want some iptables rules on my server hooked up to 100G. And all of those are really workarounds that shouldn't be necessary. Having more than a hundred million packets per second is something that only happens because things got fossilized on a tiny packet size, not because there's actually some point to splitting up a 2 GB download into more than a million tiny bits.
- MisterTea 3y ago
  > TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. The stream is an atom of data - it's a block of serialized memory. How else do we transfer arbitrary amounts of data?
- allarm 3y ago
  > Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. What do you mean?
  - dale_glass 3y ago
    TCP sends a contiguous stream of data. Lots and lots of programs work by building some sort of message system on top, with some sort of message type/length/payload system. Thousands of slightly different versions of reinventing that particular wheel.
    - allarm 3y ago
      Not sure I understand, sorry. By “on top” do you mean L5-L7 protocols? Like, for simplicity, FTP or HTTP? Or you mean reimplementing the transport layer itself, like QUIC?
    - allarm 3y ago
      Regardless, I don’t see that it’s possible to get rid of TCP/IP in any meaningful perspective really. There’s millions of devices around that support TCP offload, thousands of thousands of network equipment with routing processors and ASICs onboard. See ipv6 transition as a good example, and we’re much, much more ready to ipv6 than to a new tcp. There’s a ton of artifacts and limitations in the stack that don’t make sense today - tcp ports as connection ids being one of them - but that’s something that’s taken care of on L7. Transport layer is a different beast though.
- eternityforest 3y ago
  The entire network stack and the whole OSI concept is awful... The way DNS and TLS and certs are done kind of sucks, ARP is pretty insane... not much in networking is all that nice, there's not really any piece of it that is as awesome as something like a nice Python library. Encryption could have been IP layer, DNS could have always been secure, certificates don't need to exist if IP addresses and DNS were secure by themselves, separate IP and MAC addresses might not really be needed ... But the fact that there is a global standard internet protocol is amazing!
msla 3y ago
Some things are the same because they've changed considerably. Ethernet used to be a collision domain on coaxial cable, not a switched full-duplex medium on twisted pair. It changed piece by piece into something that's conceptually fairly different from what it was, especially when it ceased being a collision domain. https://en.wikipedia.org/wiki/Collision_domain https://en.wikipedia.org/wiki/Collision_domain > Early Ethernet variants (10BASE5, 10BASE2) were based on a shared wire and inherently half-duplex, representing a single, potentially large collision domain. Collision domains are also found in an Ethernet hub or repeater environment where each host segment connects to a hub, and all segments represent only one collision domain within one broadcast domain. Collision domains are also found in other shared medium networks, e. g. wireless networks such as Wi-Fi. > Modern wired networks use a network switch to reduce or eliminate collisions. By connecting each device directly to a port on the switch, either each port on a switch becomes its own collision domain (in the case of half-duplex links), or the possibility of collisions is eliminated entirely in the case of full-duplex links. For Gigabit Ethernet and faster, no hubs or repeaters exist and all devices require full-duplex links.
- throw0101c 3y ago
  > Ethernet used to be a collision domain on coaxial cable, not a switched full-duplex medium on twisted pair. And it's come back full-circle to (radio) collision domains (à la AlohaNet) with Wifi, which I think the vast majority of people use on a day-to-day basis rather than plugging in.
  - 1ncorrect 3y ago
    The original Ethernet spec uses CSMA/CD[0] 802.11 uses CSMA/CA[1] [0] https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_with_collision_detection https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_... [1] https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_with_collision_avoidance https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_...
  - kloch 3y ago
    > the vast majority of people use on a day-to-day basis rather than plugging in. For last 100 feet yes, but for the entire rest of the network it's either fiber or copper. Notably, Ethernet framing sometimes with additional encapsulation is used everywhere along the path.

dale_glass 3y ago

Some things about that suck though. Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already. TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. Of course people came up with workarounds, but that's mostly adding a bunch of different layers on top. So on the surface everything looks stable, but in reality there's a huge amount of protocols piggybacking on top of TCP and UDP.

scrollaway 3y ago
What happened to QUIC? Didn’t google want to have a replacement for tcp?
- dale_glass 3y ago
  QUIC runs on top of UDP, so it inherits all the limitations of the stack.
Bluecobra 3y ago
Jumbo/9000 byte frames have been a thing for a long time now and a lot of switches enable it by default so you can definitely reap the benefits of 100G/400G. The problem is that every L3 hop also needs to support a higher MTU if you want to route this traffic without fragmentation.
- nicholasbraker 3y ago
  And as such is mostly used within datacenter fabrics.
  - allarm 3y ago
    And that’s reasonable, because loss of a large datagram has larger impact. Thus jumbos in data centers, but not in the wild Internet where losses are more common.
- toast0 3y ago
  This would be good enough, and allow for progressive enhancement, except that path MTU detection is somewhere between fragile and broken on the internet. So maybe you run 9000 packets on internal interfaces, and 1500 externally, or more often, that's too hard and 1500 is it. There are 9000 internet exchanges, so maybe there is hope, but I haven't seen any progress reports.
throw0101c 3y ago
> Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already. How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is? I can see some server applications, but we've had offloading for years now: * https://en.wikipedia.org/wiki/TCP_offload_engine https://en.wikipedia.org/wiki/TCP_offload_engine Netflix is hitting (encrypted) 800 Gb/s: * https://www.youtube.com/watch?v=36qZYL5RlgY https://www.youtube.com/watch?v=36qZYL5RlgY * https://news.ycombinator.com/item?id=32519881 https://news.ycombinator.com/item?id=32519881 > TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. Tallk to the vendors of middleware boxes (e.g., (CPE) firewalls): we've had (say) SCTP and DCCP for twenty years that offer variations, but they're never allowed to pass through. No new Layer 4 can really be allowed through because of ossification, and so kludging everything to TCP/UDP port 80/443 is what we're left with. NAT and a lack of end-to-end also doesn't help: one has to futz around with things like STUN and TURN.
- dale_glass 3y ago
  > How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?\ Well, here's my personal experience for instance. If you take a Linux server and try to do put a 10G interface into a bridge, that disables a good amount of the offload capability of the network card. As a result you don't get 10G anymore even on fairly decent hardware. So VMs in a bridge with a 10G interface is already a configuration that doesn't quite work. VFIO does the trick fortunately. Also, surely even for ASICs there's a cost that comes with having to handle 150M packets/s. And yeah, you can do those speeds on switches and devices with limited functionality, but maybe I want some iptables rules on my server hooked up to 100G. And all of those are really workarounds that shouldn't be necessary. Having more than a hundred million packets per second is something that only happens because things got fossilized on a tiny packet size, not because there's actually some point to splitting up a 2 GB download into more than a million tiny bits.
MisterTea 3y ago
> TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. The stream is an atom of data - it's a block of serialized memory. How else do we transfer arbitrary amounts of data?
allarm 3y ago
> Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. What do you mean?
- dale_glass 3y ago
  TCP sends a contiguous stream of data. Lots and lots of programs work by building some sort of message system on top, with some sort of message type/length/payload system. Thousands of slightly different versions of reinventing that particular wheel.
  - allarm 3y ago
    Not sure I understand, sorry. By “on top” do you mean L5-L7 protocols? Like, for simplicity, FTP or HTTP? Or you mean reimplementing the transport layer itself, like QUIC?
  - allarm 3y ago
    Regardless, I don’t see that it’s possible to get rid of TCP/IP in any meaningful perspective really. There’s millions of devices around that support TCP offload, thousands of thousands of network equipment with routing processors and ASICs onboard. See ipv6 transition as a good example, and we’re much, much more ready to ipv6 than to a new tcp. There’s a ton of artifacts and limitations in the stack that don’t make sense today - tcp ports as connection ids being one of them - but that’s something that’s taken care of on L7. Transport layer is a different beast though.
eternityforest 3y ago
The entire network stack and the whole OSI concept is awful... The way DNS and TLS and certs are done kind of sucks, ARP is pretty insane... not much in networking is all that nice, there's not really any piece of it that is as awesome as something like a nice Python library. Encryption could have been IP layer, DNS could have always been secure, certificates don't need to exist if IP addresses and DNS were secure by themselves, separate IP and MAC addresses might not really be needed ... But the fact that there is a global standard internet protocol is amazing!

msla 3y ago

Some things are the same because they've changed considerably. Ethernet used to be a collision domain on coaxial cable, not a switched full-duplex medium on twisted pair. It changed piece by piece into something that's conceptually fairly different from what it was, especially when it ceased being a collision domain. https://en.wikipedia.org/wiki/Collision_domain https://en.wikipedia.org/wiki/Collision_domain > Early Ethernet variants (10BASE5, 10BASE2) were based on a shared wire and inherently half-duplex, representing a single, potentially large collision domain. Collision domains are also found in an Ethernet hub or repeater environment where each host segment connects to a hub, and all segments represent only one collision domain within one broadcast domain. Collision domains are also found in other shared medium networks, e. g. wireless networks such as Wi-Fi. > Modern wired networks use a network switch to reduce or eliminate collisions. By connecting each device directly to a port on the switch, either each port on a switch becomes its own collision domain (in the case of half-duplex links), or the possibility of collisions is eliminated entirely in the case of full-duplex links. For Gigabit Ethernet and faster, no hubs or repeaters exist and all devices require full-duplex links.

throw0101c 3y ago
> Ethernet used to be a collision domain on coaxial cable, not a switched full-duplex medium on twisted pair. And it's come back full-circle to (radio) collision domains (à la AlohaNet) with Wifi, which I think the vast majority of people use on a day-to-day basis rather than plugging in.
- 1ncorrect 3y ago
  The original Ethernet spec uses CSMA/CD[0] 802.11 uses CSMA/CA[1] [0] https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_with_collision_detection https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_... [1] https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_with_collision_avoidance https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_...
- kloch 3y ago
  > the vast majority of people use on a day-to-day basis rather than plugging in. For last 100 feet yes, but for the entire rest of the network it's either fiber or copper. Notably, Ethernet framing sometimes with additional encapsulation is used everywhere along the path.