6 ms·

Speaking of which, what's a reasonably cheap solution for a little 10 GBit/s home LAN (Linux)? A 4 port Microtik switch and which cards? Nothing fancy. Just

Speaking of which, what's a reasonably cheap solution for a little 10 GBit/s home LAN (Linux)? A 4 port Microtik switch and which cards? Nothing fancy. Just something that won't bottleneck like crazy the PCIe 3.0 or 4.0 x4 lanes NVMe SSDs my machines have. I'd rather go to 10 GBit/s directly instead of 2.5 but I'll go 2.5 if it's really easier/cheaper.

cpgxiii 5y ago
Mikrotik makes a 4xSFP+ switch and Ubiquiti makes a 4x10GBase-T switch. The Mikrotik one can take SFP+ copper transceivers, so it allows you to mix and match twisted pair and fiber optics if you need it. Asus makes a pair of Aquantia (now Marvell) 10G PCIe x4 cards, one with SFP+ and the other with 1/2.5/5/10G Base-T. Other vendors have similar cards in PCIe x4 and Thunderbolt 2/3 formats.
- zrail 5y ago
  I just picked up a Mikrotik CRS305 from Amazon and a pair of used Intel X520T dual SPF+ cards from eBay, plus a couple DAC (direct attach copper) cables for maybe $300 total. I'm very happy with the performance of this setup. One thing to keep in mind about the CRS305 is that it can't handle more than one or maybe two SFP+ copper modules. They use a lot of power and put out a lot of heat. I think the CRS309 has fewer restrictions.
  - csdvrx 5y ago
    Could you recommend a PCIe adapter to receive a Nokia 3fe4960ac SFP+ transceiver (AT&T fiber) and maybe one or two 2.5G or 10G modules?
    - zrail 5y ago
      Oh I have no idea. You'd need to do some digging. I picked the Intel card because it was decently priced and I knew it would be reasonably compatible with what I was doing. Sounds like maybe you want a quad SFP+ card?
    - ComputerGuru 5y ago
      You can’t bypass AT&T’s absolutely craptastic fiber modem, even if you got the bidi transceiver you needed. The connection is encrypted with keys hardcoded into the modem.
      - csdvrx 5y ago
        And this thing doesn't have a JTAG?
      - d136o 5y ago
        For their craptastic router you could downgrade the firmware, use an old exploit to root, extract the keys, and then use those keys on your own Linux box to authenticate as the router… For the fiber ONT (what I realize now you may be calling their fiber modem) I would love to see a similar attempt to get their hardware out of the loop.
        ComputerGuru 5y ago
        Yeah, I’m referring to the ONT - which is now additionally bundled into their router + wifi box. There’s similar exploit to bypass the router part of the equation but you still need the ONT to negotiate the connection.
        csdvrx 5y ago
        Do you have a link to this exploit?
      - toast0 5y ago
        Is it actually encrypted? or is it just 802.1x like it used to be (I moved away from the service area when they were still doing separate ONT and crappy 'internet gateway' that turned ethernet into ethernet.
        ComputerGuru 5y ago
        The latter. The session negotiation is encrypted. You can technically shelve the unit after that’s been negotiated but it would need to be plugged back in each time your connection reset for some reason.
  - geerlingguy 5y ago
    The transceivers get up to 90-95°C (enough to leave a little burn!) if you plug them next to each other. They can work in that configuration (with up to 4 in that enclosure) but I'd only do it if you have a fan on the thing. It relies entirely on passive ventilation, and the enclosure gets hot, too, like 60°C or more!
    - zrail 5y ago
      Oh hey I just read the post about transceivers being slow only in one direction. I have the same problem on a different scale. Two X520T cards, one in a Dell T20 and the other in a Dell T30. Both connected with DACs to the CRS305. 9.something Gbps one direction, 7.something Gbps the other. I haven't bothered to do much actual diagnosis on this because I'm satisfied with the speed as-is, but it's another illustration of weirdness where you'd least expect it.
      - geerlingguy 5y ago
        Yeah; I've noticed asymmetric speeds in certain situations with the CRS305 as well... I thought it was just me, but I'm glad (sad?) to know someone else sees the same thing too :)
  - Xamayon 5y ago
    I find having a 120mm pc fan sitting on top blowing down through the case's top holes keeps everything sane. A 120mm fan matches the size almost perfectly, and there are even some usb powered ones available online. Running the fan slow and with rubber standoffs to help air move freely allows it to be nearly silent. Without that, even a single sfp+ to rj45 copper module can overheat and become unstable under load. Other than that the switches work great, perfect option for small scale cheapish 10G networking.
dragontamer 5y ago
PCIe 2.0 cards (such as ConnectX 3) are $30 used on ebay. The modern versions (ie: ConnectX 6) are much more expensive of course, but the older hardware might do better with FreeBSD / Linux anyway. Of course, your mileage may vary so do some research before buying (or buy for $30 and hope for the best). You'll also either want Optical Transceivers + Fiber, or Active Fiber (aka: Fiber that has transceivers soldered onto them already), or DAC (direct-attach copper, the cheapest but only works over small distances). Optical Transceivers + Fiber is the most flexible of course, different transceivers are spec'd for different wires and different costs though, so it gets a bit complicated. Active Fiber and DACs are is probably easiest, but if you buy a 10m cable but need like 12m, you need to buy a new cable + transceiver combo (rather than just purchasing a bunch of different fiber lengths). Length and cost is the big difference from Active Fiber and DAC, but otherwise are very similar in use. You plug the modules into the SFP+ port, hope its compatible and let them rip.
- geerlingguy 5y ago
  ConnectX3 and Intel SFP cards are pretty reliable and easy enough to get working on Linux. On my Windows 10 Pro machine, all the cards I plugged in just worked immediately, no driver install required. I'd start there and if you find you need other features, think about spending more.
  - fomine3 5y ago
    My ConnectX-3 on Windows 10/11 machine takes 15 secs to reconnect on resume from sleep. It's not serious issue but a bit annoying. Anyone had similar issue?
- consp 5y ago
  ConnectX-3 are PCIe3 x4/x8 cards, even the SFP+ ones. The older connectx-2 cards are PCIe2. If you want to go all-out fast and not need a switch you can use a QSFP+ <-> 4xSFP+ cable or splitter by getting a QSFP+ card on the server end.
bravetraveler 5y ago
The major decision to make is, what kind of connector do you want? Ethernet or SFP? Ethernet is more costly, SFP is the more approachable-but-unusual way. Lengths of the runs and how many clients changes the decision making a bit, so keep that in mind. I didn't mind cost so much but I wanted to stick with typical Ethernet. Mainly a comfort thing, I don't know the limits of SFP well. I went with this switch: Mikrotik CRS312-4C+8XG ... and these cards: ASUS XG-C100C I upgraded my home lab to 10GbE around a year ago mainly to speed up my SSD-based backups. All told I think it was about $800 to upgrade my storage cluster and a couple clients.
- OJFord 5y ago
  Ethernet is more costly?! Over what distance? And surely ignoring the 'endpoint hardware', switches or whatever, (or equivalently considering only stuff high enough end that it has SFP 'for free' anyway) whatever the distance?
  - Sebb767 5y ago
    You can get a lot of cheap, used SFP+ stuff on eBay and the likes. For short range (up to ~3m) you can go with DACs, which are 5-10€ a piece; for longer ranges, you can go to fiber, where you'll pay 20-30€ per cable. Ethernet cables are a bit cheaper, but you need high quality ones (cat6 shielded and up) and the receivers are a lot more expensive, plus they get hotter and use more power. Unless you get a really great deal on ethernet hardware or need backwards compatibility to 2.5/1 gbit, SFP+ will be a lot cheaper.
  - zamadatix 5y ago
    In the case of 10G sure, over any distance. Servers use cheap and simple passive copper DACs in racks and switches use optical transceivers for uplinks (usually of a higher speed RJ45 doesn't support anyways). 10G RJ45 is more complicated to implement for both the NIC and switch and requires more power, meanwhile it's not mass produced because of the aforementioned DACs and optical uplinks and lack of consumer penetration.
  - icelancer 5y ago
xxpor 5y ago
I have a server that I'm already running a bunch of vms on running opnsense, with an Intel 82599 passed through. I'd use SR-IOV but the freebsd driver is buggy when using vlans. There are some "cheap" ($250-400) switches for 10g now. HW routers are still spendy.
hnaccount141 5y ago
I'm running a bunch of old Mellanox ConnectX-3 cards, they've been plug and play in all of my Linux systems. Cheap too.
sigstoat 5y ago
as was mentioned, used mellanox cards off ebay. then hit fs.com for optics and cables.

cpgxiii 5y ago

Mikrotik makes a 4xSFP+ switch and Ubiquiti makes a 4x10GBase-T switch. The Mikrotik one can take SFP+ copper transceivers, so it allows you to mix and match twisted pair and fiber optics if you need it. Asus makes a pair of Aquantia (now Marvell) 10G PCIe x4 cards, one with SFP+ and the other with 1/2.5/5/10G Base-T. Other vendors have similar cards in PCIe x4 and Thunderbolt 2/3 formats.

zrail 5y ago
I just picked up a Mikrotik CRS305 from Amazon and a pair of used Intel X520T dual SPF+ cards from eBay, plus a couple DAC (direct attach copper) cables for maybe $300 total. I'm very happy with the performance of this setup. One thing to keep in mind about the CRS305 is that it can't handle more than one or maybe two SFP+ copper modules. They use a lot of power and put out a lot of heat. I think the CRS309 has fewer restrictions.
- csdvrx 5y ago
  Could you recommend a PCIe adapter to receive a Nokia 3fe4960ac SFP+ transceiver (AT&T fiber) and maybe one or two 2.5G or 10G modules?
  - zrail 5y ago
    Oh I have no idea. You'd need to do some digging. I picked the Intel card because it was decently priced and I knew it would be reasonably compatible with what I was doing. Sounds like maybe you want a quad SFP+ card?
  - ComputerGuru 5y ago
    You can’t bypass AT&T’s absolutely craptastic fiber modem, even if you got the bidi transceiver you needed. The connection is encrypted with keys hardcoded into the modem.
    - csdvrx 5y ago
      And this thing doesn't have a JTAG?
    - d136o 5y ago
      For their craptastic router you could downgrade the firmware, use an old exploit to root, extract the keys, and then use those keys on your own Linux box to authenticate as the router… For the fiber ONT (what I realize now you may be calling their fiber modem) I would love to see a similar attempt to get their hardware out of the loop.
      - ComputerGuru 5y ago
        Yeah, I’m referring to the ONT - which is now additionally bundled into their router + wifi box. There’s similar exploit to bypass the router part of the equation but you still need the ONT to negotiate the connection.
        csdvrx 5y ago
        Do you have a link to this exploit?
    - toast0 5y ago
      Is it actually encrypted? or is it just 802.1x like it used to be (I moved away from the service area when they were still doing separate ONT and crappy 'internet gateway' that turned ethernet into ethernet.
      - ComputerGuru 5y ago
        The latter. The session negotiation is encrypted. You can technically shelve the unit after that’s been negotiated but it would need to be plugged back in each time your connection reset for some reason.
- geerlingguy 5y ago
  The transceivers get up to 90-95°C (enough to leave a little burn!) if you plug them next to each other. They can work in that configuration (with up to 4 in that enclosure) but I'd only do it if you have a fan on the thing. It relies entirely on passive ventilation, and the enclosure gets hot, too, like 60°C or more!
  - zrail 5y ago
    Oh hey I just read the post about transceivers being slow only in one direction. I have the same problem on a different scale. Two X520T cards, one in a Dell T20 and the other in a Dell T30. Both connected with DACs to the CRS305. 9.something Gbps one direction, 7.something Gbps the other. I haven't bothered to do much actual diagnosis on this because I'm satisfied with the speed as-is, but it's another illustration of weirdness where you'd least expect it.
    - geerlingguy 5y ago
      Yeah; I've noticed asymmetric speeds in certain situations with the CRS305 as well... I thought it was just me, but I'm glad (sad?) to know someone else sees the same thing too :)
- Xamayon 5y ago
  I find having a 120mm pc fan sitting on top blowing down through the case's top holes keeps everything sane. A 120mm fan matches the size almost perfectly, and there are even some usb powered ones available online. Running the fan slow and with rubber standoffs to help air move freely allows it to be nearly silent. Without that, even a single sfp+ to rj45 copper module can overheat and become unstable under load. Other than that the switches work great, perfect option for small scale cheapish 10G networking.

dragontamer 5y ago

PCIe 2.0 cards (such as ConnectX 3) are $30 used on ebay. The modern versions (ie: ConnectX 6) are much more expensive of course, but the older hardware might do better with FreeBSD / Linux anyway. Of course, your mileage may vary so do some research before buying (or buy for $30 and hope for the best). You'll also either want Optical Transceivers + Fiber, or Active Fiber (aka: Fiber that has transceivers soldered onto them already), or DAC (direct-attach copper, the cheapest but only works over small distances). Optical Transceivers + Fiber is the most flexible of course, different transceivers are spec'd for different wires and different costs though, so it gets a bit complicated. Active Fiber and DACs are is probably easiest, but if you buy a 10m cable but need like 12m, you need to buy a new cable + transceiver combo (rather than just purchasing a bunch of different fiber lengths). Length and cost is the big difference from Active Fiber and DAC, but otherwise are very similar in use. You plug the modules into the SFP+ port, hope its compatible and let them rip.

geerlingguy 5y ago
ConnectX3 and Intel SFP cards are pretty reliable and easy enough to get working on Linux. On my Windows 10 Pro machine, all the cards I plugged in just worked immediately, no driver install required. I'd start there and if you find you need other features, think about spending more.
- fomine3 5y ago
  My ConnectX-3 on Windows 10/11 machine takes 15 secs to reconnect on resume from sleep. It's not serious issue but a bit annoying. Anyone had similar issue?
consp 5y ago
ConnectX-3 are PCIe3 x4/x8 cards, even the SFP+ ones. The older connectx-2 cards are PCIe2. If you want to go all-out fast and not need a switch you can use a QSFP+ <-> 4xSFP+ cable or splitter by getting a QSFP+ card on the server end.

bravetraveler 5y ago

The major decision to make is, what kind of connector do you want? Ethernet or SFP? Ethernet is more costly, SFP is the more approachable-but-unusual way. Lengths of the runs and how many clients changes the decision making a bit, so keep that in mind. I didn't mind cost so much but I wanted to stick with typical Ethernet. Mainly a comfort thing, I don't know the limits of SFP well. I went with this switch: Mikrotik CRS312-4C+8XG ... and these cards: ASUS XG-C100C I upgraded my home lab to 10GbE around a year ago mainly to speed up my SSD-based backups. All told I think it was about $800 to upgrade my storage cluster and a couple clients.

OJFord 5y ago
Ethernet is more costly?! Over what distance? And surely ignoring the 'endpoint hardware', switches or whatever, (or equivalently considering only stuff high enough end that it has SFP 'for free' anyway) whatever the distance?
- Sebb767 5y ago
  You can get a lot of cheap, used SFP+ stuff on eBay and the likes. For short range (up to ~3m) you can go with DACs, which are 5-10€ a piece; for longer ranges, you can go to fiber, where you'll pay 20-30€ per cable. Ethernet cables are a bit cheaper, but you need high quality ones (cat6 shielded and up) and the receivers are a lot more expensive, plus they get hotter and use more power. Unless you get a really great deal on ethernet hardware or need backwards compatibility to 2.5/1 gbit, SFP+ will be a lot cheaper.
- zamadatix 5y ago
  In the case of 10G sure, over any distance. Servers use cheap and simple passive copper DACs in racks and switches use optical transceivers for uplinks (usually of a higher speed RJ45 doesn't support anyways). 10G RJ45 is more complicated to implement for both the NIC and switch and requires more power, meanwhile it's not mass produced because of the aforementioned DACs and optical uplinks and lack of consumer penetration.
- icelancer 5y ago

xxpor 5y ago

I have a server that I'm already running a bunch of vms on running opnsense, with an Intel 82599 passed through. I'd use SR-IOV but the freebsd driver is buggy when using vlans. There are some "cheap" ($250-400) switches for 10g now. HW routers are still spendy.

hnaccount141 5y ago

I'm running a bunch of old Mellanox ConnectX-3 cards, they've been plug and play in all of my Linux systems. Cheap too.

sigstoat 5y ago

as was mentioned, used mellanox cards off ebay. then hit fs.com for optics and cables.