4 ms·

You can get 10 Gb networking for fairly affordable prices with say, Mikrotik hardware. I think the reason why it costs a lot is that packets are still tiny, an

You can get 10 Gb networking for fairly affordable prices with say, Mikrotik hardware. I think the reason why it costs a lot is that packets are still tiny, and ports are many. A quality 16 port, 10 Gb switch is expected to have no internal bottleneck, able to handle every port being used to full capacity at once. And that works out to 320 Gbps bandwidth in the whole switch. Then there's that thanks to the Internet, we're still stuck with 1500 byte packets, which means best case we're dealing with 812,744 pps, and worst case, we're dealing with 14,880,952 pps. And that's on one port. With the later number, your time budget per packet is 67ns. And if you start looking, context switches, cache misses, locks, etc all have times also measured in nanoseconds. It's getting very, very tight and requires extremely careful programming to actually make this work. And of course in modern times, we have firewalls, NAT, statistics, routing, etc all of which cost precious time out of this budget, and which are all user-configurable, so it's something that's difficult to just shove into a purpose-built chip to take that load off the CPU. PCIe has the advantage of that it doesn't have to deal with half the crazy stuff networking involves.

jjoonathan 5y ago
PCIe is a packet switched network with point-to-point routing, firewalls, and link aggregation. Everything is done on dedicated silicon because it simply can't be done in software -- forget having a cache miss, the PCIe bus itself might be the mechanism to serve the cache miss. Or ship a bunch of 400Gbps ethernet traffic... speaking of which, picking a bandwidth fight with PCI express is probably not the best move ever :) > 16 port, 10 Gb ... 320 Gbps bandwidth in the whole switch That's a nice switch. Expensive, but nice. Let's do the math on a modern low-end consumer CPU: PCIe: 20x 32Gb/s x2 = 1280Gbps Chipset: 8x 16Gb/s x2 = 256Gbps Display: 4x 4x 12Gb/s = 192Gbps ...and if we compared the switch to a CPU in its own price bracket, this would go from silly to ludicrous. Look, I'm not asking for a battery of 400GbE ports on every laptop, but USB can deliver 10Gb/s across a few meters for a few dollars while Ethernet wants a few hundred dollars to ship the same bits over the same distance, and that's unfortunate. People keep trying to make excuses for Ethernet, but I don't think it deserves their efforts. I think something's broken in the consumer networking industry -- it feels like they gave up on consumers entirely after 1GbE, and I think we could all benefit if someone breathed some life back into that corner of the market.

jjoonathan 5y ago

PCIe is a packet switched network with point-to-point routing, firewalls, and link aggregation. Everything is done on dedicated silicon because it simply can't be done in software -- forget having a cache miss, the PCIe bus itself might be the mechanism to serve the cache miss. Or ship a bunch of 400Gbps ethernet traffic... speaking of which, picking a bandwidth fight with PCI express is probably not the best move ever :) > 16 port, 10 Gb ... 320 Gbps bandwidth in the whole switch That's a nice switch. Expensive, but nice. Let's do the math on a modern low-end consumer CPU: PCIe: 20x 32Gb/s x2 = 1280Gbps Chipset: 8x 16Gb/s x2 = 256Gbps Display: 4x 4x 12Gb/s = 192Gbps ...and if we compared the switch to a CPU in its own price bracket, this would go from silly to ludicrous. Look, I'm not asking for a battery of 400GbE ports on every laptop, but USB can deliver 10Gb/s across a few meters for a few dollars while Ethernet wants a few hundred dollars to ship the same bits over the same distance, and that's unfortunate. People keep trying to make excuses for Ethernet, but I don't think it deserves their efforts. I think something's broken in the consumer networking industry -- it feels like they gave up on consumers entirely after 1GbE, and I think we could all benefit if someone breathed some life back into that corner of the market.