5 ms·

You use CWDM4 for switch to switch. Cheaper/easier to use DAC cables for switch to server connections. Though I don't know anyone doing 100G to server yet, so y

by erentz 10y ago

You use CWDM4 for switch to switch. Cheaper/easier to use DAC cables for switch to server connections. Though I don't know anyone doing 100G to server yet, so you'd more likely be using DAC breakout cables to deliver 25G to the server.

ericd 10y ago
Cool, thanks for the info. Wondering if this is remotely worth it for a home computing cluster yet.
- shouldbworking 10y ago
  Linux can route traffic over 40G on regular hardware as long as packet sizes are fairly large >~150 bytes. I would guess that serving static assets from ram would be able to saturate that on a single machine
  - gonzo 10y ago
    150 bytes is a fairly small payload. I think you dropped a 0. kernel bypass will get you line rate 40gbps w/64 byte packets. We've been showing IPsec over 40g links during the past week. https://twitter.com/netgateusa/status/853694461456646144 https://twitter.com/netgateusa/status/853694461456646144
    - shouldbworking 10y ago
      Neat! I've heard you can get line rate on slightly bigger packets without bypass using IPVS with direct return since it bypasses part of handling. In a lot of scenarios the big pipe is right at the load balancer so this is useful.
  - shaklee3 10y ago
    You can do 40gbe even if packet sizes are small using any kernel bypass technique and multiple cores. Mellanox claims up to 130Mpps.
    - shouldbworking 10y ago
      Is this just blind forwarding or does that include some decent routing logic? I've looked into kernel bypass but I keep shying away because it seems to proprietary, especially the virtualized flavor which seems to be Intel only
      - shaklee3 10y ago
        Blind forwarding. It's left to the application to do everything.
- en4bz 10y ago
  100G, probably not unless you are extremely wealthy and you have a burning desire to be destitute. 40G is definitely doable for a homelab. Used 40Gbe adapters are roughly $200 and DAC cables are < $100. QSPF+ Switches are much harder to find used and/or cheap though. This means that p2p solutions or ring networks are generally preferred unless you are wealthy. If you want fiber, which you don't need for a homelab, you pay the prices listed above.
- jsjohnst 10y ago
  Agree with other commenters, not sure what the 2.5x increase in traffic would open for you in a home setting. I use 40gb fiber between switches on opposite ends of my place and 10gb connections between switch and computers and I've always found something else to be my bottleneck, not the internal network itself. I can saturate my wifi network easily though (I've got a $20k retail Aruba wifi network, yet still not enough). I buy most of my gear used so it's not that expensive (would've been very expensive retail), but still overkill unless your a network nerd (like me) or are doing something out of your home you likely should be using a DC.
  - ericd 10y ago
    Yeah, I think 10 G will be more than good enough for now. Thanks for the info!

by erentz 10y ago

ericd 10y ago

Cool, thanks for the info. Wondering if this is remotely worth it for a home computing cluster yet.

shouldbworking 10y ago
Linux can route traffic over 40G on regular hardware as long as packet sizes are fairly large >~150 bytes. I would guess that serving static assets from ram would be able to saturate that on a single machine
- gonzo 10y ago
  150 bytes is a fairly small payload. I think you dropped a 0. kernel bypass will get you line rate 40gbps w/64 byte packets. We've been showing IPsec over 40g links during the past week. https://twitter.com/netgateusa/status/853694461456646144 https://twitter.com/netgateusa/status/853694461456646144
  - shouldbworking 10y ago
    Neat! I've heard you can get line rate on slightly bigger packets without bypass using IPVS with direct return since it bypasses part of handling. In a lot of scenarios the big pipe is right at the load balancer so this is useful.
- shaklee3 10y ago
  You can do 40gbe even if packet sizes are small using any kernel bypass technique and multiple cores. Mellanox claims up to 130Mpps.
  - shouldbworking 10y ago
    Is this just blind forwarding or does that include some decent routing logic? I've looked into kernel bypass but I keep shying away because it seems to proprietary, especially the virtualized flavor which seems to be Intel only
    - shaklee3 10y ago
      Blind forwarding. It's left to the application to do everything.
en4bz 10y ago
100G, probably not unless you are extremely wealthy and you have a burning desire to be destitute. 40G is definitely doable for a homelab. Used 40Gbe adapters are roughly $200 and DAC cables are < $100. QSPF+ Switches are much harder to find used and/or cheap though. This means that p2p solutions or ring networks are generally preferred unless you are wealthy. If you want fiber, which you don't need for a homelab, you pay the prices listed above.
jsjohnst 10y ago
Agree with other commenters, not sure what the 2.5x increase in traffic would open for you in a home setting. I use 40gb fiber between switches on opposite ends of my place and 10gb connections between switch and computers and I've always found something else to be my bottleneck, not the internal network itself. I can saturate my wifi network easily though (I've got a $20k retail Aruba wifi network, yet still not enough). I buy most of my gear used so it's not that expensive (would've been very expensive retail), but still overkill unless your a network nerd (like me) or are doing something out of your home you likely should be using a DC.
- ericd 10y ago
  Yeah, I think 10 G will be more than good enough for now. Thanks for the info!