13 ms·
Oh, I was hoping this would be something built more directly over Ethernet, rather than on top of UDP/IP (if I'm understanding the layer diagram correctly). I'
by mintplant 3y ago
Oh, I was hoping this would be something built more directly over Ethernet, rather than on top of UDP/IP (if I'm understanding the layer diagram correctly).
I've been working with Ethernet devices a lot lately, using the network as a communication bus, essentially. I find that there's a lot of complexity that we simply don't need: ARP, DHCP, DNS... So many points of failure. We know all the devices on our LAN and their unique MAC addresses, and could do everything we need to addressing-wise at Layer 2. But everything's built on Layer 3 and up, so we're effectively working backward to map devices to IP addresses and vice versa. It's unsatisfying.
- wmf 3y ago1. Your forwarding table would have to be larger because Ethernet uses exact match instead of longest prefix match. For example, you might be limited to 128K servers total while Google has millions. 2. The Ethernet header has less entropy for ECMP than a UDP/IP header. Maybe you could add entropy somewhere but ASICs may not support it. 3. You're breaking compatibility with... everything. Maybe Google could afford this but no one else could.
- Dylan16807 3y ago1. In a single data center scale computing system, though? 2. Use some bytes outside of the header? 3. I get the impression this needs hardware to really use well anyway.
- kortilla 3y ago2. Don’t do this. It causes reordering between flows and software doesn’t like out of order packets.
- tuetuopay 3y ago1. Especially in the datacenter. When you add VMs to the mix you get LOADS of devices to address. Add on top that a single device has multiple connections (management, internet, storage, etc), you'd run out of capacity almost instantly. 2. The point is to find some bytes that are constant for a given logical stream of related packets. Taking bytes outside of the header means taking bytes from the payload, that is by definition not deterministic. That's why everything identifies flows using the IPs + ports + protocol.
- Dylan16807 3y ago> Especially in the datacenter. Okay, but my point there was saying "Google has millions of servers" isn't relevant, we're not looking at the entire company. Even with a few addresses per VM, how many racks do you need to put into the same shared-compute mass? One data center is the upper limit, but it doesn't have to be the entire data center.
- tuetuopay 3y agoBack of the envelope math using modern hypervisors that can fit loads of vms in a single U - let’s put 500 vms on a single one. with 128c256t CPUs it’s easy - say you can fit 30 of those in a single rack (the common rack is 42U) due to power constraints - and place 10 of those racks That’s 500 x 30 x 10 = 150000 nodes to address. With 10 racks you already blow past the scaling limits of the common datacenter switch when it comes to MAC addresses. Here are the limits for Cisco’s Nexus 9000 series, a very common datacenter switch: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/92x/scalability/guide_923/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_923.html#id_91721 https://www.cisco.com/c/en/us/td/docs/switches/datacenter/ne...
- cereal_cable 3y agoPlus layer 2 switches when they don't know the destination port will flood. With so many hosts that would be absolutely horrific. I've seen computers at moderately sized LAN parties (talking 100 nodes, far from the large or even massive events) that were literally crippled by the broadcast traffic. At some point the flooding and layer 2 discovery (ARP) would do the same as well. Limiting the broadcast domains with layer 3 really makes the Internet possible. Sure, you can have less overhead and simply do layer 2 only, and really it is completely possible. It's just such a rare use case that it in practice isn't important enough to actually do.
- Bluecobra 3y ago“Layers are only ever added, not removed.” This is a long read but you might find it enjoyable: https://apenwarr.ca/log/20170810 https://apenwarr.ca/log/20170810
- stonegray 3y agoAdding is favoured over subtracting in problem solving https://www.researchgate.net/publication/350711603_Adding_is_favoured_over_subtracting_in_problem_solving https://www.researchgate.net/publication/350711603_Adding_is...
- deleted 3y ago[deleted]
- genmud 3y agoBut you can't route MACs... you can't associate domain names to MACs... Maybe for a LAN that's fine, but for connecting devices between broadcast domains, it just doesn't cut it.
- MiddleMan5 3y agoWhy though? You can't route MAC because... ? Because ipv4 provides a higher entropy address? Because MAC is self-assigned and reduplication would require a higher level system? or just because we just don't use MAC addresses that way? I'm certain there are reasons IP came to live alongside/on top of MAC, but saying you can't do multi-hop routing with it just isn't true. If all the technologies of the Internet were reset tomorrow, how might you design the perfect layer 2 addressing and routing system?
- stingraycharles 3y agoYou would need to structure mac addresses in such a way that they can be easily grouped for routing a-la IP subnets. It just isn’t suitable for this.
- freedom-fries 3y agoYou can't route MAC because there is no prefix matching - only exact matching. That's exactly why you need to "switch" them... and incidentally this is what your proposal accomplishes – it's equivalent to a fully-switched network. Switches (especially L3 switches) maintain port-MAC association tables to switch packets between ports and they're available off the shelf.
- blown_gasket 3y ago0C:F9:31:D2:DB:51 AB:33:C6:C6:19:74 I used a MAC address generator to get those two, but I think two is enough to make the discussion. Current reality aside, would you be able to identify those with binary math as being on the same network device, different network devices, across the world? MAC addresses on physical NICs are provided by the manufacturer, sure you can adjust them but I think that leaves the good-faith portion of this discussion. So if you wanted to have those to communicate no matter what you would have to have a network device state: "I'm network device A, I have this device 0C:F9:31:D2:DB:51" then another state: "I'm network device B, I have this device AB:33:C6:C6:19:74". Then whenever 0C:F9:31:D2:DB:51 wants to talk with AB:33:C6:C6:19:74 it's network device will have to just send it to the next upstream network device or if there are multiple network devices that could be upstream you could send it to them all which is just not great for security whatsoever or you now have to do a recursive lookup for whatever n devices might yet be upstream and wait for a response to see if one of those has it. Overall trying to send ethernet frames globally without an IP network sounds like not a great idea.
- freedom-fries 3y agoNot first, but eventually the layers are needed to solve problems that evolve when your software needs to operate in someone else's network. That said, it's not very hard to directly talk "ethernet" using raw sockets. Here's an example if you're interested: https://gist.github.com/austinmarton/1922600 https://gist.github.com/austinmarton/1922600.
- mdekkers 3y agoWe used to have something like that and didn’t like it https://networkencyclopedia.com/netbios/ https://networkencyclopedia.com/netbios/
- vidarh 3y agoThe fact that there used to be a lot of alternative lower level network layers is incidentally also the best argument for IP: We needed a common shared layer because it was shit trying to gateway between multiple different protocols that user-level software had to know about. And as much as ethernet is dominant now, it's still not the only thing.
- wayfinder 3y agoAh yes, instead of going to google.com or 192.168.1.1 or adding a printer connected to my Wi-Fi, let me open my big yellow pages of globally unique MAC addresses…
- allanrbo 3y agoI think op was just saying as a bus on a local network for some very specialized application.
- wayfinder 3y agoBut you can do that already. DNS, ARP, TCP/UDP, and IP are all completely optional. Just write a program to send and receive Ethernet frames. I believe it doesn’t have to contain IP.
- otabdeveloper4 3y ago> let me open my big yellow pages of globally unique MAC addresses Yeah, well, you've basically described IPv6.
- Animats 3y agoIt's been done. There's XNS, and there's QNX networking over raw Ethernet. Both worked well. Look them up. You probably don't want to go that route, but it is technically possible on a LAN.
- xorcist 3y agoThere's also Audio-over-Ethernet, which is still used for professional audio where latency must be held low. I believe there are other broadcast protocols that runs on Ethernet for the same reason. Fibre Channel-over-Ethernet used to be a thing too, but I haven't seen it for a while. Perhaps latency wasn't as much of an issue as people thought and it lost to iSCSI. I don't think it necessarily is bad idea to run protocols directly on the data link layer. The fewer parts the better. It's just that somewhere someone probably wants to route it, and the more general usage tends to win. So it's always going to be a niche market where latency is really important.
- dezgeg 3y agoeCPRI, or essentially radio signal over ethernet (ie. from base station indoors to radio module at top of the tower/roof) is another interesting use case (with even stricter latency requirements than audio).
- mgaunard 3y agoAll of cloud/web tech is about reinventing the wheel at upper layers. These days they're rebuilding everything on top of HTTP.
- eecc 3y agoHeh, what a coincidence. Just about yesterday I was chatting with a friend about the similarities between Service Mesh and the ESB of yore.
- MichaelMoser123 3y agoThere are systems like this: Fibre Channel has it's own data link layer, actually they do reliability at the data link layer! I think InfiniBand is similar to this respect. Actually it's interesting that google didn't choose any of these, for their high bandwidth storage needs. They have the money to do their own thing, but why should they?
- pkhuong 3y agoGoogle migrated away from IB years ago. IIUC, the failure modes at the time (e.g., fully locked up fabric) were too painful at the time, and they preferred to work with a mostly vanilla Linux kernel for userspace networking.
- pclmulqdq 3y agoInfiniband and most other specialized protocols have data-link-layer reliable messaging. The downside of this is that a congested switch cannot drop packets, so you end up backpressuring your network to death unless the people writing software really know what they're doing. Google was not able to make this work at scale.
- NavinF 3y agoIf you're flexible enough to forgo UDP/IP, why not use infiniband instead of ethernet? That gets rid of all the complexity you mentioned but still gives you ordered streams
- MichaelZuo 3y agoI agree, it's hard to see what real world use case would benefit from dumping UDP/IP while holding on to Ethernet, and not moving over to an Infiniband or similar solution. That also can't be satisfied by one of the existing specialized solutions that another user mentioned, such as EtherCAT.
- fmajid 3y agoInfiniband is basically a nVidia monopoly since they bought Mellanox, and the hyperscalers who already chafe at nVidia's GPU pricing power don't like it one bit, which is why they are working so hard on getting rid of it. I don't understand enough about niche high-performance interconnects to know if CXL is a viable alternative for Infiniband where Ethernet-based solutions have too much latency.
- ascar 3y agoWhen I read the title I thought this is going to be an ethernet competitor similar to Infiniband.
- hydroid7 3y agoYou're my man, I'm thinking about the same! E.g. Look at TIPC.
- deleted 3y ago[deleted]
- fragmede 3y agoATA over Ethernet (AoE) was one technology that forewent those layers for performance reasons.
- traxys 3y agoI think in High-Performance Computing a lot of the networking NICs behave in that way, because you are 100% sure that you know the fabric layout. Stuff like InfiniBand, HPE Slingshot, Atos BXI, ... There is a consortium that's building a specification for those kinds of things: https://ultraethernet.org/ https://ultraethernet.org/
- DanielHB 3y agotake a look at CBOR if you want a little more power in your payloads. It is basically binary space-efficient JSON
- NavinF 3y agoDid you reply to the wrong thread?
- elcritch 3y agoMove to ipv6 and drop ARP and DHCP, which eliminates a good chunk of the older cruft. IPv6 builds everything on top of multicast support which is required for IPv6 switches/routers. It's so much cleaner. You could even avoid DNS if you really want.