4 ms·
Incredible. Traditional 10Gb network gear is very expensive to buy and to operate. $50 is a lot cheaper.
by nickstinemates 3y ago
Incredible. Traditional 10Gb network gear is very expensive to buy and to operate. $50 is a lot cheaper.
- Palomides 3y agonot really true, a similar setup with 40Gbe (three nodes, no switch) would run you less than $100 for pcie cards and cables
- hamandcheese 3y agoThe only NICs I see in that price range are old Mellanox cards. Intel NICs are 5-10x that price. I'm not sure why, but my suspicion is that it has to do with driver support and/or interoperability.
- Palomides 3y agoI run mellanox connectx3 cards, they work immediately with no extra drivers on windows 10/11 and every linux I've tried mellanox is/was quite good at getting code upstreamed maybe I need to do my own blog post about my pile of computers...
- TacticalCoder 3y ago> maybe I need to do my own blog post about my pile of computers... Yes, several of us love to read about that! I haven't switched to 10 Gbit/s yet...
- thefz 3y ago> I run mellanox connectx3 cards, they work immediately with no extra drivers on windows 10/11 and every linux I've tried Own three, can confirm for Windows and Linux.
- gregors 3y agowould love to read this!
- Hikikomori 3y agoMellanox works fine.
- hamandcheese 3y agoGotcha. Why the price discrepancy then?
- Hikikomori 3y agoAre you comparing new Intel cards to old mellanox cards on ebay? If not idk why, I have not compared them myself, some feature maybe? Cost doesn't always make sense either.
- hamandcheese 3y agoIm just comparing the prices I see when I search ebay for "40gbe nic" vs "40gbe nic intel", making no effort to compare features.
- petronio 3y agoI can't answer definitively, but I was looking for SFP cards recently and the older cards don't really support ASPM. The cards themselves aren't power hogs, but they keep the CPU from going into lower states during idle. The cheapest one I found that others related had ASPM actually working was the Intel X710, and those are much more expensive than the ConnectX-3.
- ApolIllo 3y agoI avoiding older 10gbe NICs due to power consumption. Maybe that's why?
- olavgg 3y agoAnd with Mellanox you get working RMDA/ROCE.
- nickstinemates 3y agoCan you provide links? I'll upgrade my home lab from 10g to 40g immediately at that price.
- Hikikomori 3y agoMellanox connect-x 3 probably.
- willglynn 3y agoIn 2024 I would suggest deploying 2x25G instead, via e.g. MCX4121. Pricing is similar (<$30 NICs), but: * 2x25G throughput is higher than 40G, * 25G latency is lower than 40G, * you can use 25G ports as 10G ports, and * you can use DACs to connect 4x25G <=> 100G That last point is particularly relevant given the existence of switches like the Mikrotik CRS504, providing 4x100G ports on 25W.
- Palomides 3y agothose are all reasonable points, if I were doing mine again I would spend a little more and go up to 100gbe if you run all older mellanox gear the cx3 can do the kinda nonstandard 56gbe as well
- bombela 3y agoWhat are you doing that you need 100gbe? I am still on 1gbe... I guess I don't transfer anything bigger than a few GiB time to time.
- nickstinemates 3y agoNo individual connection other than like a cr tral storage server needs 100gbe, at least for me, but a 100gbe backplane is good for a lot of 1gbe poe devices as an example. With residential fiber/coax reaching 5gb, 1gb is not enough.
- Palomides 3y agocx353a and cx354a, prefer the fcbt models, but you can reflash them all to fcbt