6 ms·

I very much want to transition to open hardware models for networking gear - for both my own home use and for rsync.net. However, at no single site (especially

by rsync 12y ago

I very much want to transition to open hardware models for networking gear - for both my own home use and for rsync.net. However, at no single site (especially my home) do I need a 6U chassis full of switch ports. Is there a 1U version of this on the horizon ?

aaronbee 12y ago
Facebook's Wedge 1U switch is what this builds on. https://code.facebook.com/posts/681382905244727/introducing-wedge-and-fboss-the-next-steps-toward-a-disaggregated-network/ https://code.facebook.com/posts/681382905244727/introducing-...
- mey 12y ago
  Is anyone building a copy of that to sell to other people besides Facebook? Similar to how you can buy a Backblaze Pod ( https://www.backblaze.com/blog/backblaze-storage-pod-4/ https://www.backblaze.com/blog/backblaze-storage-pod-4/ ) from http://www.45drives.com/ http://www.45drives.com/
  - wmf 12y ago
    I haven't seen anyone selling Wedge specifically, but it's not that different from what you can get from Quanta, Edge-Core, or Penguin Computing.
  - stock_toaster 12y ago
    Apparently juniper is with their OCX1100[1] model. [1]: http://forums.juniper.net/t5/Data-Center-Technologists/Juniper-OCX1100-48SX-Technical-Deep-Dive/ba-p/265370 http://forums.juniper.net/t5/Data-Center-Technologists/Junip...
    - virtuallynathan 12y ago
      All of these things are based on the Broadcom Trident II chip. If you want it cheap, don't go to juniper - get one from Quanta. You can have a 32x40G switch for $6,000.
      - stock_toaster 12y ago
        thanks, good to know!
      - e12e 12y ago
        On a similar note, I just recently became aware of Cumulus[1] Debian based switches (but the good bits are closed source) from among others edge-core[2] (via a presentation by PaaS-provider http://zetta.io http://zetta.io) -- eg: http://whiteboxswitch.com/products/edge-core-as4600-54t http://whiteboxswitch.com/products/edge-core-as4600-54t [1] http://cumulusnetworks.com/support/faq/ http://cumulusnetworks.com/support/faq/ [2] http://www.edge-core.com/ http://www.edge-core.com/
        epistasis 12y ago
        We've been loving Cumulus + Quanta for 10Gb and 40Gb, in that it's more manageable than Cisco (for our environment) and a fraction of the cost. We end up using it at 1Gb too, but it's just a price match there, instead of a win.
        kijiki 12y ago
        To clarify, the only part that is closed is "switchd", which is a userspace program that watches the kernel data structures (route tables, neighbor tables, bridges, ports, vlans, etc) and programs the hardware to match. It links against proprietary silicon vendor SDKs, and programs registers whose description were given to us under NDA. Without this part, everything works the same, but is of course not hardware accelerated. So the 100% open source parts of Cumulus Linux would still make a great Network OS for a router/switch VM. We don't yet have an official VM version, but that is something we will have in the future. - nolan co-founder/CTO Cumulus Networks
        MichaelGG 12y ago
        What is the flexibility with "open" switches? To get linerate switching, I'm guessing you're still limited by the hardware? Is the benefit that you can more easily setup routing tables (instead of depending on the switch vendor's capabilities), vlans, etc. just by creating them in userspace then pushing them over to the hardware part? Or can you actually get fairly low level, like implementing your own algorithms for channel bonding? A while back I wanted to do some L7 inspection, but could only get like 10G per server, and we had 40G coming in. EtherChannel didn't acceptably balance out the traffic. Doing so would have required dealing with one of the network processor vendors and all that mess. Would an open switch platform make this a straightforward exercise?
  - virtuallynathan 12y ago
    I'd imagine folks like Quanta or Hyve Solutions will be selling something like this soon.
gonzo 12y ago
You probably don't need 16 40Gbps channels at home, either.
- ericd 12y ago
  Heh maybe not need, but want? I definitely want. Looks like a 40 GbE NIC is just $500: http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313A-BCCT/dp/B00L50BAVW/ http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313... Just need an SSD RAID to feed it...
  - virtuallynathan 12y ago
    Less than $500 actually, you can get them for about $350.
  - e12e 12y ago
    While I wasn't aware you could get them that cheap, I think I'd still look into second hand infiniband for home use. Allows the use of copper for short (pc-to-pc, point-to-point) distances. While optic is absolutely cool, as far as I can tell a single 1m patch cable will cost ~80 USD -- quite a lot considering the cost of the NIC... And even in IBM's sales brochure for their optical switch, infiband comes out a little ahead: http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=SA&subtype=WH&htmlfid=QCW03034USEN&appname=STGE_QC_AA_USEN_WH#loaded http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=S...
    - wmf 12y ago
      40GbE uses approximately the same copper cables as Infiniband. There's probably more used IB equipment floating around than used 40GbE, but otherwise I'd go with Ethernet.
      - e12e 12y ago
        Ah, of course. I was misled by wikipedia[1], but it should've been obvious that the same connection could be used for both inifinband and ethernet. So, indeed, there are copper interconnects: http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Copper/dp/B00CA2A7XQ/ref=sr_1_2?ie=UTF8&qid=1423702311&sr=8-2&keywords=qsfp++40gb+copper+cable http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Cop... Actually seems the price is finally coming down a bit (compared to what I remember these used to cost, years ago -- but maybe I've just upped my budget ;-). [1] http://en.wikipedia.org/wiki/QSFP http://en.wikipedia.org/wiki/QSFP "The Quad Small Form-factor Pluggable (QSFP) is a compact, hot-pluggable transceiver used for data communications applications. It interfaces networking hardware to a fiber optic cable." But it also interfaces hardware to copper cables, as I gather.
actaeon169 12y ago
http://onie.org/ http://onie.org/ For example: http://www.edge-core.com/ProdDtl.asp?sno=425&AS4600-54T http://www.edge-core.com/ProdDtl.asp?sno=425&AS4600-54T or http://www.edge-core.com/ProdDtl.asp?sno=423&AS5600-52X http://www.edge-core.com/ProdDtl.asp?sno=423&AS5600-52X
kijiki 12y ago
Basically anything on here: http://cumulusnetworks.com/support/linux-hardware-compatibility-list/ http://cumulusnetworks.com/support/linux-hardware-compatibil... Some easy ways to buy are: https://bm-switch.com/ https://bm-switch.com/ http://whiteboxswitch.com/ http://whiteboxswitch.com/ http://ipfabrics.net/ http://ipfabrics.net/ http://www.penguincomputing.com/products/network-switches/ http://www.penguincomputing.com/products/network-switches/ - nolan co-founder/CTO Cumulus
- tiernano 12y ago
  Are there any that are, say, 48 port GigE? 10G and 40G are a bit much for home use...
  - zymhan 12y ago
    10G and 40G are a bit much for my office, even.
  - skuhn 12y ago
    There are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and $700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain support in order to continue using the product (legally). I can't imagine getting enough utility from a switch for this to make sense for home use. It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly low end 1 gig switch (single power supply, limited L3 capability), and you can absolutely negotiate Juniper / Cisco / etc. down to be close to or even below the pricing of the Cumulus solution. That is tougher for 10 gig or 40 gig equipment, which is where the value proposition kicks in for Cumulus. Maybe Cumulus's approach is enough of a value add for it to make sense to pay a premium, but everyone I talk to is interested in cost savings first and better manageability a distant second. You'll still have to have Juniper / Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they don't have a full range of switch models.
    - kijiki 12y ago
      Stay tuned for low-end 1G platforms running Cumulus Linux. - nolan
sschueller 12y ago
http://www.pcengines.ch/apu.htm http://www.pcengines.ch/apu.htm
pingswept 12y ago
By "open hardware," do you mean open source hardware, or something else? From the responses, it looks like "open hardware" means something like "can install any OS on it" in the network switch world.

by rsync 12y ago

aaronbee 12y ago

Facebook's Wedge 1U switch is what this builds on. https://code.facebook.com/posts/681382905244727/introducing-wedge-and-fboss-the-next-steps-toward-a-disaggregated-network/ https://code.facebook.com/posts/681382905244727/introducing-...

mey 12y ago
Is anyone building a copy of that to sell to other people besides Facebook? Similar to how you can buy a Backblaze Pod ( https://www.backblaze.com/blog/backblaze-storage-pod-4/ https://www.backblaze.com/blog/backblaze-storage-pod-4/ ) from http://www.45drives.com/ http://www.45drives.com/
- wmf 12y ago
  I haven't seen anyone selling Wedge specifically, but it's not that different from what you can get from Quanta, Edge-Core, or Penguin Computing.
- stock_toaster 12y ago
  Apparently juniper is with their OCX1100[1] model. [1]: http://forums.juniper.net/t5/Data-Center-Technologists/Juniper-OCX1100-48SX-Technical-Deep-Dive/ba-p/265370 http://forums.juniper.net/t5/Data-Center-Technologists/Junip...
  - virtuallynathan 12y ago
    All of these things are based on the Broadcom Trident II chip. If you want it cheap, don't go to juniper - get one from Quanta. You can have a 32x40G switch for $6,000.
    - stock_toaster 12y ago
      thanks, good to know!
    - e12e 12y ago
      On a similar note, I just recently became aware of Cumulus[1] Debian based switches (but the good bits are closed source) from among others edge-core[2] (via a presentation by PaaS-provider http://zetta.io http://zetta.io) -- eg: http://whiteboxswitch.com/products/edge-core-as4600-54t http://whiteboxswitch.com/products/edge-core-as4600-54t [1] http://cumulusnetworks.com/support/faq/ http://cumulusnetworks.com/support/faq/ [2] http://www.edge-core.com/ http://www.edge-core.com/
      - epistasis 12y ago
        We've been loving Cumulus + Quanta for 10Gb and 40Gb, in that it's more manageable than Cisco (for our environment) and a fraction of the cost. We end up using it at 1Gb too, but it's just a price match there, instead of a win.
      - kijiki 12y ago
        To clarify, the only part that is closed is "switchd", which is a userspace program that watches the kernel data structures (route tables, neighbor tables, bridges, ports, vlans, etc) and programs the hardware to match. It links against proprietary silicon vendor SDKs, and programs registers whose description were given to us under NDA. Without this part, everything works the same, but is of course not hardware accelerated. So the 100% open source parts of Cumulus Linux would still make a great Network OS for a router/switch VM. We don't yet have an official VM version, but that is something we will have in the future. - nolan co-founder/CTO Cumulus Networks
        MichaelGG 12y ago
        What is the flexibility with "open" switches? To get linerate switching, I'm guessing you're still limited by the hardware? Is the benefit that you can more easily setup routing tables (instead of depending on the switch vendor's capabilities), vlans, etc. just by creating them in userspace then pushing them over to the hardware part? Or can you actually get fairly low level, like implementing your own algorithms for channel bonding? A while back I wanted to do some L7 inspection, but could only get like 10G per server, and we had 40G coming in. EtherChannel didn't acceptably balance out the traffic. Doing so would have required dealing with one of the network processor vendors and all that mess. Would an open switch platform make this a straightforward exercise?
- virtuallynathan 12y ago
  I'd imagine folks like Quanta or Hyve Solutions will be selling something like this soon.

gonzo 12y ago

You probably don't need 16 40Gbps channels at home, either.

ericd 12y ago
Heh maybe not need, but want? I definitely want. Looks like a 40 GbE NIC is just $500: http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313A-BCCT/dp/B00L50BAVW/ http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313... Just need an SSD RAID to feed it...
- virtuallynathan 12y ago
  Less than $500 actually, you can get them for about $350.
- e12e 12y ago
  While I wasn't aware you could get them that cheap, I think I'd still look into second hand infiniband for home use. Allows the use of copper for short (pc-to-pc, point-to-point) distances. While optic is absolutely cool, as far as I can tell a single 1m patch cable will cost ~80 USD -- quite a lot considering the cost of the NIC... And even in IBM's sales brochure for their optical switch, infiband comes out a little ahead: http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=SA&subtype=WH&htmlfid=QCW03034USEN&appname=STGE_QC_AA_USEN_WH#loaded http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=S...
  - wmf 12y ago
    40GbE uses approximately the same copper cables as Infiniband. There's probably more used IB equipment floating around than used 40GbE, but otherwise I'd go with Ethernet.
    - e12e 12y ago
      Ah, of course. I was misled by wikipedia[1], but it should've been obvious that the same connection could be used for both inifinband and ethernet. So, indeed, there are copper interconnects: http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Copper/dp/B00CA2A7XQ/ref=sr_1_2?ie=UTF8&qid=1423702311&sr=8-2&keywords=qsfp++40gb+copper+cable http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Cop... Actually seems the price is finally coming down a bit (compared to what I remember these used to cost, years ago -- but maybe I've just upped my budget ;-). [1] http://en.wikipedia.org/wiki/QSFP http://en.wikipedia.org/wiki/QSFP "The Quad Small Form-factor Pluggable (QSFP) is a compact, hot-pluggable transceiver used for data communications applications. It interfaces networking hardware to a fiber optic cable." But it also interfaces hardware to copper cables, as I gather.

actaeon169 12y ago

http://onie.org/ http://onie.org/ For example: http://www.edge-core.com/ProdDtl.asp?sno=425&AS4600-54T http://www.edge-core.com/ProdDtl.asp?sno=425&AS4600-54T or http://www.edge-core.com/ProdDtl.asp?sno=423&AS5600-52X http://www.edge-core.com/ProdDtl.asp?sno=423&AS5600-52X

kijiki 12y ago

Basically anything on here: http://cumulusnetworks.com/support/linux-hardware-compatibility-list/ http://cumulusnetworks.com/support/linux-hardware-compatibil... Some easy ways to buy are: https://bm-switch.com/ https://bm-switch.com/ http://whiteboxswitch.com/ http://whiteboxswitch.com/ http://ipfabrics.net/ http://ipfabrics.net/ http://www.penguincomputing.com/products/network-switches/ http://www.penguincomputing.com/products/network-switches/ - nolan co-founder/CTO Cumulus

tiernano 12y ago
Are there any that are, say, 48 port GigE? 10G and 40G are a bit much for home use...
- zymhan 12y ago
  10G and 40G are a bit much for my office, even.
- skuhn 12y ago
  There are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and $700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain support in order to continue using the product (legally). I can't imagine getting enough utility from a switch for this to make sense for home use. It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly low end 1 gig switch (single power supply, limited L3 capability), and you can absolutely negotiate Juniper / Cisco / etc. down to be close to or even below the pricing of the Cumulus solution. That is tougher for 10 gig or 40 gig equipment, which is where the value proposition kicks in for Cumulus. Maybe Cumulus's approach is enough of a value add for it to make sense to pay a premium, but everyone I talk to is interested in cost savings first and better manageability a distant second. You'll still have to have Juniper / Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they don't have a full range of switch models.
  - kijiki 12y ago
    Stay tuned for low-end 1G platforms running Cumulus Linux. - nolan

sschueller 12y ago

http://www.pcengines.ch/apu.htm http://www.pcengines.ch/apu.htm

pingswept 12y ago

By "open hardware," do you mean open source hardware, or something else? From the responses, it looks like "open hardware" means something like "can install any OS on it" in the network switch world.