3 ms·

> How far we've come. I remember having a semi-decent on board 100mbps LAN was only reserved for highest end. These cpus are much more than 10x faster than the

by nullc 5y ago

> How far we've come. I remember having a semi-decent on board 100mbps LAN was only reserved for highest end. These cpus are much more than 10x faster than the CPUs you would have used then. Networking speeds in desktop devices haven't kept pace. Not just with cpu speeds but with storage speeds-- which is particularly obnoxious when you want to use network attached storage. I assume this is due to a mixture of internet and wireless creating an extremely low bandwidth least common denominator and that running >1GB over copper is kinda problematic (finicky, power hungry, etc) -- and the industry seems to reason (perhaps correctly) the the customers in this segment can't handle fiber. I personally have 40GBE attached desktops (and 100gbe attached stuff within the server rack)-- thankfully quite economically due to surplus enterprise hardware, so I'm well aware that it can be done... but to do it in a small formfactor system is still a problem, e.g. for any of these non-apu systems your normal mini itx systems will use all their slot space for a graphics card and not have room for a mellanox adapter.

bayindirh 5y ago
> These cpus are much more than 10x faster than the CPUs you would have used then. Actually, it's not only about the CPU performance, because NIC performance drew a cosine wave w.r.t. processor power over the years. Earlier NICs had much more machinery inside them and were expensive. Then Realtek came and we witnessed the era of (almost) software LAN adapters, then the silicon became cheap, and some of the stuff moved back into the LAN controller. So modern, higher end cards do not do significantly more offloading, but they handle other things like virtualization of cards between many VMs, or other higher level tasks. From what I've seen, making a faster CPU is easy, but a faster and wider fabric is harder to weave due to distance, switching related noise and other physics related phenomena. Cache and good memory controllers are also relatively easy, but expensive. Also, RAM and higher voltage doesn't go well together, because RAM uses a lot of energy for its size, so it comes with heat and stability problems and other lifetime related problems (yes, you can fry RAMs). Storage is another hard technology. Spinning rust is limited by physics, vibration, rotation speed and other mechanical stuff. SSDs came crashing, but before they were cheap and Sun didn't got swallowed by Oracle, they did nice ZFS arrays with some very expensive SSD caches and spinning drives combined. I've seen whattheheckisthat amount of speed from an array with a size of 6-8U total and a couple of IB interfaces fitted to it at the factory. Currently we get that amount of speed from Lustre arrays with some SSDs and a lot of spinning disks. With the current backbone capacity of a standard desktop computer, even on the high end, 40Gbps network is overkill unless you're going to ingest that data directly at the CPU. Yes, absorbing the data at the disks are possible, for what cost and what use case? 10Gbps is understandable, but I still consider 1Gbps as a sweet spot for general purpose computing. If you're going to transfer vast amounts of data from a storage array to a local system, yes you can go higher, but it's still a niche. Also as you've said, going over 1Gbps is problematic from a signal integrity point of view, and fiber is too fragile and expensive for average customer. These Mellanox cards get quite hot when they're constantly utilized, so in a little ITX box, both the card and the system will be cooked. The process becomes faster if you use active fiber cables, because connectors sink the heat inside the case via heat sinks attached to the connectors on the cards. Even if you use them in a system room, building a large network with them requires expensive equipment which has adequate bandwidth and cable length. Also, running an IB network needs other software machinery to keep the network up. You can directly run ethernet, or TCP over IB, which kinda defeats the purpose and adds additional penalty and reduces the performance. All in all, higher speed network is still doesn't bring too much of a value to the home of the average user, and is not still that cheap for the enthusiast. Also the hardware is not scaled down for the home. Cards are big & hot, connectors/cables are bulky and the fabric gear is noisy, bulky and high maintenance for home. Yes they work rather nicely, but I'd not want them at home. I have enough of them at the system room already.
- sliken 5y ago
  Mostly agree, and just bought a ZFS server for home, 5 disks (RAIDz2) and 2xNVMe 2TB for root (256GB) and cache (the rest). GigE is pretty annoying in that it can't keep up with a single spinning disk (let alone 5), so large files (family videos, movies, system images, backups, etc) can take annoying long. Hell even wifi access points are becoming bottle necked by GigE. Fortunately 2.5Gbe looks like a reasonable compromise. Rarely a bottleneck and support is common on cable modems (motorola among others), access points, motherboards (pretty much anything newish from AMD/Intel), and even small SBCs if you run one as a firewall, router, or pi-hole. There are some switches and routers around and are often fanless till the port counts get high. Even quality 10G cards seem to run pretty hot, have huge heat sinks, and the sfp+ I've worked with tend to get pretty hot. Seems like there is a newer gen 10G chip, but they are rare and expensive but do run substantially cooler. Boards like the Intel quad port 10G X710-T4L (the cool one) vs the X710-T4 (the hot board) run at about half the power (14 vs 29 watts). I'm shopping around for the dual port version (X710-TSL) which peaks at 9.6 watts and uses 8.2 watts typically. No idea how other (non-intel) 10G boards do on power use.
- nullc 5y ago
  40GB is overkill, but 10GB still bottlenecks for me. I'm surprised to see anyone consider 1GB a sweet spot-- it can't keep up with a single conventional disk, and it's causes obvious slowdowns for fairly conventional operations. And indeed, the heat from higher speed cards is a nuisance... though 10G copper is awful in that respect. > too fragile and expensive for average customer I think that's a bit overstated. It's more fragile but not enormously so. Terminations are a nuisance, which favors pre-made patch cables. I'd pretty much always prefer to use 10g over fiber than copper. But I'll freely admit that having spent more than a decade as a network engineer my expectations may be weird. I will say that after many years of using fiber connected equipment at home for the desktop/server systems, my spouse has never broken a cable or connector (nor have I, of course). I've never-- professionally nor personally-- had good luck with copper 10g and try hard to avoid it. If I have to use well made/tested pre-made patch cables to make it work I might as well use fiber (or dac cables) and trust that literally anything I buy will just work. > Also, running an IB network needs other software machinery to keep the network up. I used to use IB because the IB only nics were super cheap on ebay-- but I don't anymore. It's easy enough to use 10/40/100GBE. The extra software was a nuisance compared to the value it provided. > and is not still that cheap for the enthusiast. I mean, you can get 40gbe capable cards on ebay for like $50ish (usually generation or two old mellanox dual ib/eth cards)-- main cost to them beyond their power consumption is knowing how to use the utility to put them in ethernet mode. Dac cables and LR2 optics at similar prices. Particularly if most of your usage can be covered by dac cables, I think it's pretty affordable-- not 1gbe affordable, which is close to free on on a cost per port basis, but when you're bottlenecked on network bandwidth there isn't really an alternative, except for keeping data locally on the systems.

by nullc 5y ago

bayindirh 5y ago

> These cpus are much more than 10x faster than the CPUs you would have used then. Actually, it's not only about the CPU performance, because NIC performance drew a cosine wave w.r.t. processor power over the years. Earlier NICs had much more machinery inside them and were expensive. Then Realtek came and we witnessed the era of (almost) software LAN adapters, then the silicon became cheap, and some of the stuff moved back into the LAN controller. So modern, higher end cards do not do significantly more offloading, but they handle other things like virtualization of cards between many VMs, or other higher level tasks. From what I've seen, making a faster CPU is easy, but a faster and wider fabric is harder to weave due to distance, switching related noise and other physics related phenomena. Cache and good memory controllers are also relatively easy, but expensive. Also, RAM and higher voltage doesn't go well together, because RAM uses a lot of energy for its size, so it comes with heat and stability problems and other lifetime related problems (yes, you can fry RAMs). Storage is another hard technology. Spinning rust is limited by physics, vibration, rotation speed and other mechanical stuff. SSDs came crashing, but before they were cheap and Sun didn't got swallowed by Oracle, they did nice ZFS arrays with some very expensive SSD caches and spinning drives combined. I've seen whattheheckisthat amount of speed from an array with a size of 6-8U total and a couple of IB interfaces fitted to it at the factory. Currently we get that amount of speed from Lustre arrays with some SSDs and a lot of spinning disks. With the current backbone capacity of a standard desktop computer, even on the high end, 40Gbps network is overkill unless you're going to ingest that data directly at the CPU. Yes, absorbing the data at the disks are possible, for what cost and what use case? 10Gbps is understandable, but I still consider 1Gbps as a sweet spot for general purpose computing. If you're going to transfer vast amounts of data from a storage array to a local system, yes you can go higher, but it's still a niche. Also as you've said, going over 1Gbps is problematic from a signal integrity point of view, and fiber is too fragile and expensive for average customer. These Mellanox cards get quite hot when they're constantly utilized, so in a little ITX box, both the card and the system will be cooked. The process becomes faster if you use active fiber cables, because connectors sink the heat inside the case via heat sinks attached to the connectors on the cards. Even if you use them in a system room, building a large network with them requires expensive equipment which has adequate bandwidth and cable length. Also, running an IB network needs other software machinery to keep the network up. You can directly run ethernet, or TCP over IB, which kinda defeats the purpose and adds additional penalty and reduces the performance. All in all, higher speed network is still doesn't bring too much of a value to the home of the average user, and is not still that cheap for the enthusiast. Also the hardware is not scaled down for the home. Cards are big & hot, connectors/cables are bulky and the fabric gear is noisy, bulky and high maintenance for home. Yes they work rather nicely, but I'd not want them at home. I have enough of them at the system room already.

sliken 5y ago
Mostly agree, and just bought a ZFS server for home, 5 disks (RAIDz2) and 2xNVMe 2TB for root (256GB) and cache (the rest). GigE is pretty annoying in that it can't keep up with a single spinning disk (let alone 5), so large files (family videos, movies, system images, backups, etc) can take annoying long. Hell even wifi access points are becoming bottle necked by GigE. Fortunately 2.5Gbe looks like a reasonable compromise. Rarely a bottleneck and support is common on cable modems (motorola among others), access points, motherboards (pretty much anything newish from AMD/Intel), and even small SBCs if you run one as a firewall, router, or pi-hole. There are some switches and routers around and are often fanless till the port counts get high. Even quality 10G cards seem to run pretty hot, have huge heat sinks, and the sfp+ I've worked with tend to get pretty hot. Seems like there is a newer gen 10G chip, but they are rare and expensive but do run substantially cooler. Boards like the Intel quad port 10G X710-T4L (the cool one) vs the X710-T4 (the hot board) run at about half the power (14 vs 29 watts). I'm shopping around for the dual port version (X710-TSL) which peaks at 9.6 watts and uses 8.2 watts typically. No idea how other (non-intel) 10G boards do on power use.
nullc 5y ago
40GB is overkill, but 10GB still bottlenecks for me. I'm surprised to see anyone consider 1GB a sweet spot-- it can't keep up with a single conventional disk, and it's causes obvious slowdowns for fairly conventional operations. And indeed, the heat from higher speed cards is a nuisance... though 10G copper is awful in that respect. > too fragile and expensive for average customer I think that's a bit overstated. It's more fragile but not enormously so. Terminations are a nuisance, which favors pre-made patch cables. I'd pretty much always prefer to use 10g over fiber than copper. But I'll freely admit that having spent more than a decade as a network engineer my expectations may be weird. I will say that after many years of using fiber connected equipment at home for the desktop/server systems, my spouse has never broken a cable or connector (nor have I, of course). I've never-- professionally nor personally-- had good luck with copper 10g and try hard to avoid it. If I have to use well made/tested pre-made patch cables to make it work I might as well use fiber (or dac cables) and trust that literally anything I buy will just work. > Also, running an IB network needs other software machinery to keep the network up. I used to use IB because the IB only nics were super cheap on ebay-- but I don't anymore. It's easy enough to use 10/40/100GBE. The extra software was a nuisance compared to the value it provided. > and is not still that cheap for the enthusiast. I mean, you can get 40gbe capable cards on ebay for like $50ish (usually generation or two old mellanox dual ib/eth cards)-- main cost to them beyond their power consumption is knowing how to use the utility to put them in ethernet mode. Dac cables and LR2 optics at similar prices. Particularly if most of your usage can be covered by dac cables, I think it's pretty affordable-- not 1gbe affordable, which is close to free on on a cost per port basis, but when you're bottlenecked on network bandwidth there isn't really an alternative, except for keeping data locally on the systems.