4 ms·

I never thought I'd actually shop for a 10Gbe switch here in canada, but I went from a 100mbps trash-tier docsis (ugh) connection to an almost too-good-to-be-t

I never thought I'd actually shop for a 10Gbe switch here in canada, but I went from a 100mbps trash-tier docsis (ugh) connection to an almost too-good-to-be-true symmetric 1.5gbs FTTH with Bell Fibe. For the same price (!), around 60$/month with a permanent discount that Bell seems to hand out to everyone who asks. The issue is that it seems to be very hard to find affordable switches with 10gbe ethernet copper ports, instead of SFP. The few that exist either cost more than what I pay for my connection in an entire year, or don't interact well with the ISP provided hub/modem (it has no SFP). Bell also seems to have locked down their new hardware so it is impossible to bypass their hub. And while I have a home server rack, even the more affordable second hand enterprise switches are usually ancient powerhogs. So for now wifi 6e is basically just as fast as my wired setup, which is weird. I guess I'm not used to being at the edge of home networking tech as a canadian... but I'd be happy to hear about recommendations and what to look for/avoid with that type of gear either (new or used)

selectodude 4y ago
Do you need to have 10GbE? That is very high end kit, especially with copper. The power consumption is extreme and the distances are never going to be very good compared to fiber. 2.5GbE ethernet is super accessible now from Netgear, Trendnet, TPLink etc and doesn't require new NICs on any newer desktops. All my shit that 2.5GbE is useful supports 2.5GbE.
- mardifoufs 4y ago
  I'm not sure actually, I have a local NAS/Server rack where I store work related datasets (think a few hundred/thousands 12-15GB .h5 files). And while I definitely don't need instant or even super fast access to them, I'm sure it wouldn't hurt either. We already have tons of optimizations and usually process datasets in a cloud environment with "good" IOPS/Network[1] speeds, but it is still faster to use locally in some situations. 10Gbe gear would be a nice to have to avoid bottlenecks within the LAN... but I just realized that I'm making a good case for just shelling out more money instead of cheaping out haha. [1] As good as it gets on Azure, I guess. Which is not much at all, even on Ultra SSDs so I keep a local cache for smaller scale stuff
cortesoft 4y ago
Since you only have 1.5gbs, why don't you just get a 2.5g switch? Those are much easier to find and can work with plain cat 6 Ethernet
- mardifoufs 4y ago
  I guess to me 2.5gbe just seems like a weird in between with less support, but I agree that it's probably going to be my only option otherwise. Though a 10Gbe switch would also be awesome for local lan, since my servers' NVME SSDs are constantly "throttled" by my 1Gbe (and would be at 2.5Gbe too, but to a lesser extent).
seanherron 4y ago
For about $50 US, you can get a 10Gbase-T SFP+ module that gives you a copper port at 10gig within a SFP+ form factor. That, coupled with a cheap Mikrotik switch, is plenty to get started. Second hand Brocade ICX switches are also plentiful and not too power hungry (but they can be loud).
- vladvasiliu 4y ago
  Depending on how many ports you need, you may get away with an ICX 7150-12. It has 14 fixed Gb copper ports and 2 10 Gb SFP ports. It's fanless and has 12 GbE PoE ports.
- mardifoufs 4y ago
  Woah, the brocade switches might be a very good option. I don't mind it being loud, since my server rack is in a well insulated part of my basement and have had no noise issues even with a couple 2U servers. Thanks for the suggestion!
- paranoidrobot 4y ago
  Just a heads up to anyone who does intend to go buy a bunch of those modules. They run absurdly hot, and they eat a lot of power. If you don't have very good ventilation then you may find these modules running at well north of 80C. I ended up having to mount a 120MM PC fan and attach some mini-headsinks to them to keep them under 50C. Just sitting on a desk they were too hot to touch. You may find that the power budget for your small 8-12 port SFP+ Switch isn't enough to run more than a few of these at once. I have an 8 Port Mikrotik SFP+ switch that can only run about 4 of them. The Quad-Port SFP+ NIC I bought from fs.com (Intel XL710-BM1) can only do two. Expect to also run into issues if you use old, poor quality or mistreated cables. If you bend them too far (trying to get around a sharp corner) - they will work fine at 1Gbit, but just not work at all at 10Gb. You may find it's a whole lot cheaper (and more reliable) to just get some optical cable and transceivers if you want to go more than a few meters. For distances less than a few meters DAC cables are often reasonable value. From personal experience, I'm now leaving the 10G-BaseT transceivers for runs where I can't run optical (like existing in-wall wiring).
mastax 4y ago
10Gbe is really best with DAC or Optical cables.
- mardifoufs 4y ago
  Even for really short runs? I'd mostly use it for the link between my ISP modem to the switch, and wouldn't mind slower speeds for the rest of my devices. But that's definitely something I'll have to keep in mind.
  - newman314 4y ago
    TwinAX has lower latency and power requirements than fiber. https://www.arista.com/assets/data/pdf/Copper-Faster-Than-Fiber-Brief.pdf https://www.arista.com/assets/data/pdf/Copper-Faster-Than-Fi...
  - namibj 4y ago
    10GBASE-T is the worst within-building 10Gbit networking choice, unless one can't use fiber when the distance exceeds the baseband twinax copper DACs.
stordoff 4y ago
I also never thought I'd bother with 10GbE until I stumbled on some used Mellanox ConnectX-3 cards that were just cheap enough to play around with (~$30/card including DAC cables). Seeing file transfer speeds of >700MBps between my server and NAS convinced me it was worth keeping them installed, though initially just as a point-to-point link between those two systems.[1] A few months later, I was upgrading to a 24-port managed switch, and found one with four SPF+ ports (Aruba 1930) for little more than I could find a purely gigabit switch (~£180). I moved the 10Gbps NAS/server link to the main network, and added 10GbE to my main PC (with another ConnectX-3), and it's been a really nice system to use. A little overkill for most of my usage, but it's nice to have. I'm not convinced it's worth the cost of moving most of my other devices to 10GbE, even though I have a >1Gbps (down) Internet connection, as switches with more SPF+ ports or any 10G-BaseT ports (which some of my devices would require IIRC) seem rather expensive. Like you, I'm also open to suggestions if there's a cheap(ish) option I've overlooked. [1] Side note: I actually moved a USB drive from the server to the NAS as it had significantly faster (by about 30%) sustained read/write speeds that way. I'm presuming the NAS is caching the requests in memory, but both are on a UPS and it's only temporary working data anyway so I'm not too worried if the data doesn't actually hit the disc immediately.
throw0101c 4y ago
> […] almost too-good-to-be-true symmetric 1.5gbs FTTH with Bell Fibe. One issue I have with Bell Fibe is the new HomeHub 4000 hardware is mandatory, and the GPON optic is not removable (the one in the HH 3000 is). I'm not exactly enamoured at having to use Bell's hardware (though I'm sure it gets the job done for most people).
- mardifoufs 4y ago
  I think you can still get the HH3000 if you manage to escalate a request past the first layers of consumer support. I also heard about buying special optics that you can completely set up to mimic the original bell GPON, down to the SN but those are expensive. It's so weird to "downgrade" such a useful feature (that does not even hurt any other part of their business, and only serve to make 1.5gbps+ speeds more attractive for those who would actually need that much speed), but hey it's still bell after all

selectodude 4y ago

Do you need to have 10GbE? That is very high end kit, especially with copper. The power consumption is extreme and the distances are never going to be very good compared to fiber. 2.5GbE ethernet is super accessible now from Netgear, Trendnet, TPLink etc and doesn't require new NICs on any newer desktops. All my shit that 2.5GbE is useful supports 2.5GbE.

mardifoufs 4y ago
I'm not sure actually, I have a local NAS/Server rack where I store work related datasets (think a few hundred/thousands 12-15GB .h5 files). And while I definitely don't need instant or even super fast access to them, I'm sure it wouldn't hurt either. We already have tons of optimizations and usually process datasets in a cloud environment with "good" IOPS/Network[1] speeds, but it is still faster to use locally in some situations. 10Gbe gear would be a nice to have to avoid bottlenecks within the LAN... but I just realized that I'm making a good case for just shelling out more money instead of cheaping out haha. [1] As good as it gets on Azure, I guess. Which is not much at all, even on Ultra SSDs so I keep a local cache for smaller scale stuff

cortesoft 4y ago

Since you only have 1.5gbs, why don't you just get a 2.5g switch? Those are much easier to find and can work with plain cat 6 Ethernet

mardifoufs 4y ago
I guess to me 2.5gbe just seems like a weird in between with less support, but I agree that it's probably going to be my only option otherwise. Though a 10Gbe switch would also be awesome for local lan, since my servers' NVME SSDs are constantly "throttled" by my 1Gbe (and would be at 2.5Gbe too, but to a lesser extent).

seanherron 4y ago

For about $50 US, you can get a 10Gbase-T SFP+ module that gives you a copper port at 10gig within a SFP+ form factor. That, coupled with a cheap Mikrotik switch, is plenty to get started. Second hand Brocade ICX switches are also plentiful and not too power hungry (but they can be loud).

vladvasiliu 4y ago
Depending on how many ports you need, you may get away with an ICX 7150-12. It has 14 fixed Gb copper ports and 2 10 Gb SFP ports. It's fanless and has 12 GbE PoE ports.
mardifoufs 4y ago
Woah, the brocade switches might be a very good option. I don't mind it being loud, since my server rack is in a well insulated part of my basement and have had no noise issues even with a couple 2U servers. Thanks for the suggestion!
paranoidrobot 4y ago
Just a heads up to anyone who does intend to go buy a bunch of those modules. They run absurdly hot, and they eat a lot of power. If you don't have very good ventilation then you may find these modules running at well north of 80C. I ended up having to mount a 120MM PC fan and attach some mini-headsinks to them to keep them under 50C. Just sitting on a desk they were too hot to touch. You may find that the power budget for your small 8-12 port SFP+ Switch isn't enough to run more than a few of these at once. I have an 8 Port Mikrotik SFP+ switch that can only run about 4 of them. The Quad-Port SFP+ NIC I bought from fs.com (Intel XL710-BM1) can only do two. Expect to also run into issues if you use old, poor quality or mistreated cables. If you bend them too far (trying to get around a sharp corner) - they will work fine at 1Gbit, but just not work at all at 10Gb. You may find it's a whole lot cheaper (and more reliable) to just get some optical cable and transceivers if you want to go more than a few meters. For distances less than a few meters DAC cables are often reasonable value. From personal experience, I'm now leaving the 10G-BaseT transceivers for runs where I can't run optical (like existing in-wall wiring).

mastax 4y ago

10Gbe is really best with DAC or Optical cables.

mardifoufs 4y ago
Even for really short runs? I'd mostly use it for the link between my ISP modem to the switch, and wouldn't mind slower speeds for the rest of my devices. But that's definitely something I'll have to keep in mind.
- newman314 4y ago
  TwinAX has lower latency and power requirements than fiber. https://www.arista.com/assets/data/pdf/Copper-Faster-Than-Fiber-Brief.pdf https://www.arista.com/assets/data/pdf/Copper-Faster-Than-Fi...
- namibj 4y ago
  10GBASE-T is the worst within-building 10Gbit networking choice, unless one can't use fiber when the distance exceeds the baseband twinax copper DACs.

stordoff 4y ago

I also never thought I'd bother with 10GbE until I stumbled on some used Mellanox ConnectX-3 cards that were just cheap enough to play around with (~$30/card including DAC cables). Seeing file transfer speeds of >700MBps between my server and NAS convinced me it was worth keeping them installed, though initially just as a point-to-point link between those two systems.[1] A few months later, I was upgrading to a 24-port managed switch, and found one with four SPF+ ports (Aruba 1930) for little more than I could find a purely gigabit switch (~£180). I moved the 10Gbps NAS/server link to the main network, and added 10GbE to my main PC (with another ConnectX-3), and it's been a really nice system to use. A little overkill for most of my usage, but it's nice to have. I'm not convinced it's worth the cost of moving most of my other devices to 10GbE, even though I have a >1Gbps (down) Internet connection, as switches with more SPF+ ports or any 10G-BaseT ports (which some of my devices would require IIRC) seem rather expensive. Like you, I'm also open to suggestions if there's a cheap(ish) option I've overlooked. [1] Side note: I actually moved a USB drive from the server to the NAS as it had significantly faster (by about 30%) sustained read/write speeds that way. I'm presuming the NAS is caching the requests in memory, but both are on a UPS and it's only temporary working data anyway so I'm not too worried if the data doesn't actually hit the disc immediately.

throw0101c 4y ago

> […] almost too-good-to-be-true symmetric 1.5gbs FTTH with Bell Fibe. One issue I have with Bell Fibe is the new HomeHub 4000 hardware is mandatory, and the GPON optic is not removable (the one in the HH 3000 is). I'm not exactly enamoured at having to use Bell's hardware (though I'm sure it gets the job done for most people).

mardifoufs 4y ago
I think you can still get the HH3000 if you manage to escalate a request past the first layers of consumer support. I also heard about buying special optics that you can completely set up to mimic the original bell GPON, down to the SN but those are expensive. It's so weird to "downgrade" such a useful feature (that does not even hurt any other part of their business, and only serve to make 1.5gbps+ speeds more attractive for those who would actually need that much speed), but hey it's still bell after all