4 ms·

40GB is overkill, but 10GB still bottlenecks for me. I'm surprised to see anyone consider 1GB a sweet spot-- it can't keep up with a single conventional disk,

by nullc 5y ago

40GB is overkill, but 10GB still bottlenecks for me. I'm surprised to see anyone consider 1GB a sweet spot-- it can't keep up with a single conventional disk, and it's causes obvious slowdowns for fairly conventional operations. And indeed, the heat from higher speed cards is a nuisance... though 10G copper is awful in that respect. > too fragile and expensive for average customer I think that's a bit overstated. It's more fragile but not enormously so. Terminations are a nuisance, which favors pre-made patch cables. I'd pretty much always prefer to use 10g over fiber than copper. But I'll freely admit that having spent more than a decade as a network engineer my expectations may be weird. I will say that after many years of using fiber connected equipment at home for the desktop/server systems, my spouse has never broken a cable or connector (nor have I, of course). I've never-- professionally nor personally-- had good luck with copper 10g and try hard to avoid it. If I have to use well made/tested pre-made patch cables to make it work I might as well use fiber (or dac cables) and trust that literally anything I buy will just work. > Also, running an IB network needs other software machinery to keep the network up. I used to use IB because the IB only nics were super cheap on ebay-- but I don't anymore. It's easy enough to use 10/40/100GBE. The extra software was a nuisance compared to the value it provided. > and is not still that cheap for the enthusiast. I mean, you can get 40gbe capable cards on ebay for like $50ish (usually generation or two old mellanox dual ib/eth cards)-- main cost to them beyond their power consumption is knowing how to use the utility to put them in ethernet mode. Dac cables and LR2 optics at similar prices. Particularly if most of your usage can be covered by dac cables, I think it's pretty affordable-- not 1gbe affordable, which is close to free on on a cost per port basis, but when you're bottlenecked on network bandwidth there isn't really an alternative, except for keeping data locally on the systems.

bayindirh 5y ago
> But I'll freely admit that having spent more than a decade as a network engineer my expectations may be weird. I think you're correct in that regard. I'm an HPC administrator for ~15 years, and I manage quite a few systems with a sizeable storage and related network machinery up to a certain point. While I can understand your point, I've written my comment from the eyes of a bona fide standard consumer which doesn't do anything higher end on their homes. > 40GB is overkill, but 10GB still bottlenecks for me. I'm surprised to see anyone consider 1GB a sweet spot-- it can't keep up with a single conventional disk, and it's causes obvious slowdowns for fairly conventional operations. My sweet spot comment assumes streaming content consumption, occasional system to system file transfers, in home entertainment streaming and some gaming maybe. If I'd have to move my photography or video archive to a NAS instead of internal drives, a fatter network would be indeed required. To be bottlenecked by 10gbps, you really need to have a busy network with a lot of going on. None of my personal networks carry that amount of data any given time, and they don't need to do it to be honest. Moving 10gbps fairly constantly is not fairly conventional, TBH. > I think that's a bit overstated. It's more fragile but not enormously so. Terminations are a nuisance, which favors pre-made patch cables. I'd pretty much always prefer to use 10g over fiber than copper. Yes, modern fibers can endure relatively unbelievable amount of abuse, but it's still well inside the boundaries of a normal user's limits, and the threshold is still lower compared to fiber. Many people I know can damage a fiber without intending or knowing it. > I used to use IB because the IB only nics were super cheap on ebay-- but I don't anymore. It's easy enough to use 10/40/100GBE. The extra software was a nuisance compared to the value it provided. When you're doing HPC, the value provided by the features are indispensable. Latency, MPI offloading, DMA are all essential things you need for a really performing cluster. So we prefer IB rather than Ethernet. So horses for courses, obviously. I'd also rather not use IB for a home network. > I mean, you can get 40gbe capable cards on ebay for like $50ish... Consider the price as the money + setup + maintenance. For you, the last two items have no cost virtually. For me they'd be negligible. For a regular consumer, they're much bigger items to wrestle with. So, as a people who knows how to play with bigger tools is really a privilege and luxury. However, I have enough servers at work, and I don't want to manage stuff at home. So if that lower and, dumb stuff works as I want and doesn't make my life worse, I'm a happy camper. As a result, I over-engineer my home stuff very minimally.

by nullc 5y ago

bayindirh 5y ago

> But I'll freely admit that having spent more than a decade as a network engineer my expectations may be weird. I think you're correct in that regard. I'm an HPC administrator for ~15 years, and I manage quite a few systems with a sizeable storage and related network machinery up to a certain point. While I can understand your point, I've written my comment from the eyes of a bona fide standard consumer which doesn't do anything higher end on their homes. > 40GB is overkill, but 10GB still bottlenecks for me. I'm surprised to see anyone consider 1GB a sweet spot-- it can't keep up with a single conventional disk, and it's causes obvious slowdowns for fairly conventional operations. My sweet spot comment assumes streaming content consumption, occasional system to system file transfers, in home entertainment streaming and some gaming maybe. If I'd have to move my photography or video archive to a NAS instead of internal drives, a fatter network would be indeed required. To be bottlenecked by 10gbps, you really need to have a busy network with a lot of going on. None of my personal networks carry that amount of data any given time, and they don't need to do it to be honest. Moving 10gbps fairly constantly is not fairly conventional, TBH. > I think that's a bit overstated. It's more fragile but not enormously so. Terminations are a nuisance, which favors pre-made patch cables. I'd pretty much always prefer to use 10g over fiber than copper. Yes, modern fibers can endure relatively unbelievable amount of abuse, but it's still well inside the boundaries of a normal user's limits, and the threshold is still lower compared to fiber. Many people I know can damage a fiber without intending or knowing it. > I used to use IB because the IB only nics were super cheap on ebay-- but I don't anymore. It's easy enough to use 10/40/100GBE. The extra software was a nuisance compared to the value it provided. When you're doing HPC, the value provided by the features are indispensable. Latency, MPI offloading, DMA are all essential things you need for a really performing cluster. So we prefer IB rather than Ethernet. So horses for courses, obviously. I'd also rather not use IB for a home network. > I mean, you can get 40gbe capable cards on ebay for like $50ish... Consider the price as the money + setup + maintenance. For you, the last two items have no cost virtually. For me they'd be negligible. For a regular consumer, they're much bigger items to wrestle with. So, as a people who knows how to play with bigger tools is really a privilege and luxury. However, I have enough servers at work, and I don't want to manage stuff at home. So if that lower and, dumb stuff works as I want and doesn't make my life worse, I'm a happy camper. As a result, I over-engineer my home stuff very minimally.