9 ms·
Introducing “6-pack”: the first open hardware modular switch
- sudioStudio64 12y agoThe work going into whitebox switching and routing is super exciting. I really dig this stuff.
- rsync 12y agoI very much want to transition to open hardware models for networking gear - for both my own home use and for rsync.net. However, at no single site (especially my home) do I need a 6U chassis full of switch ports. Is there a 1U version of this on the horizon ?
- aaronbee 12y agoFacebook's Wedge 1U switch is what this builds on. https://code.facebook.com/posts/681382905244727/introducing-wedge-and-fboss-the-next-steps-toward-a-disaggregated-network/ https://code.facebook.com/posts/681382905244727/introducing-...
- mey 12y agoIs anyone building a copy of that to sell to other people besides Facebook? Similar to how you can buy a Backblaze Pod ( https://www.backblaze.com/blog/backblaze-storage-pod-4/ https://www.backblaze.com/blog/backblaze-storage-pod-4/ ) from http://www.45drives.com/ http://www.45drives.com/
- wmf 12y agoI haven't seen anyone selling Wedge specifically, but it's not that different from what you can get from Quanta, Edge-Core, or Penguin Computing.
- stock_toaster 12y agoApparently juniper is with their OCX1100[1] model. [1]: http://forums.juniper.net/t5/Data-Center-Technologists/Juniper-OCX1100-48SX-Technical-Deep-Dive/ba-p/265370 http://forums.juniper.net/t5/Data-Center-Technologists/Junip...
- virtuallynathan 12y agoAll of these things are based on the Broadcom Trident II chip. If you want it cheap, don't go to juniper - get one from Quanta. You can have a 32x40G switch for $6,000.
- stock_toaster 12y agothanks, good to know!
- e12e 12y agoOn a similar note, I just recently became aware of Cumulus[1] Debian based switches (but the good bits are closed source) from among others edge-core[2] (via a presentation by PaaS-provider http://zetta.io http://zetta.io) -- eg: http://whiteboxswitch.com/products/edge-core-as4600-54t http://whiteboxswitch.com/products/edge-core-as4600-54t [1] http://cumulusnetworks.com/support/faq/ http://cumulusnetworks.com/support/faq/ [2] http://www.edge-core.com/ http://www.edge-core.com/
- epistasis 12y agoWe've been loving Cumulus + Quanta for 10Gb and 40Gb, in that it's more manageable than Cisco (for our environment) and a fraction of the cost. We end up using it at 1Gb too, but it's just a price match there, instead of a win.
- kijiki 12y agoTo clarify, the only part that is closed is "switchd", which is a userspace program that watches the kernel data structures (route tables, neighbor tables, bridges, ports, vlans, etc) and programs the hardware to match. It links against proprietary silicon vendor SDKs, and programs registers whose description were given to us under NDA. Without this part, everything works the same, but is of course not hardware accelerated. So the 100% open source parts of Cumulus Linux would still make a great Network OS for a router/switch VM. We don't yet have an official VM version, but that is something we will have in the future. - nolan co-founder/CTO Cumulus Networks
- virtuallynathan 12y agoI'd imagine folks like Quanta or Hyve Solutions will be selling something like this soon.
- gonzo 12y agoYou probably don't need 16 40Gbps channels at home, either.
- ericd 12y agoHeh maybe not need, but want? I definitely want. Looks like a 40 GbE NIC is just $500: http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313A-BCCT/dp/B00L50BAVW/ http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313... Just need an SSD RAID to feed it...
- virtuallynathan 12y agoLess than $500 actually, you can get them for about $350.
- e12e 12y agoWhile I wasn't aware you could get them that cheap, I think I'd still look into second hand infiniband for home use. Allows the use of copper for short (pc-to-pc, point-to-point) distances. While optic is absolutely cool, as far as I can tell a single 1m patch cable will cost ~80 USD -- quite a lot considering the cost of the NIC... And even in IBM's sales brochure for their optical switch, infiband comes out a little ahead: http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=SA&subtype=WH&htmlfid=QCW03034USEN&appname=STGE_QC_AA_USEN_WH#loaded http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=S...
- wmf 12y ago40GbE uses approximately the same copper cables as Infiniband. There's probably more used IB equipment floating around than used 40GbE, but otherwise I'd go with Ethernet.
- e12e 12y agoAh, of course. I was misled by wikipedia[1], but it should've been obvious that the same connection could be used for both inifinband and ethernet. So, indeed, there are copper interconnects: http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Copper/dp/B00CA2A7XQ/ref=sr_1_2?ie=UTF8&qid=1423702311&sr=8-2&keywords=qsfp++40gb+copper+cable http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Cop... Actually seems the price is finally coming down a bit (compared to what I remember these used to cost, years ago -- but maybe I've just upped my budget ;-). [1] http://en.wikipedia.org/wiki/QSFP http://en.wikipedia.org/wiki/QSFP "The Quad Small Form-factor Pluggable (QSFP) is a compact, hot-pluggable transceiver used for data communications applications. It interfaces networking hardware to a fiber optic cable." But it also interfaces hardware to copper cables, as I gather.
- actaeon169 12y agohttp://onie.org/ http://onie.org/ For example: http://www.edge-core.com/ProdDtl.asp?sno=425&AS4600-54T http://www.edge-core.com/ProdDtl.asp?sno=425&AS4600-54T or http://www.edge-core.com/ProdDtl.asp?sno=423&AS5600-52X http://www.edge-core.com/ProdDtl.asp?sno=423&AS5600-52X
- kijiki 12y agoBasically anything on here: http://cumulusnetworks.com/support/linux-hardware-compatibility-list/ http://cumulusnetworks.com/support/linux-hardware-compatibil... Some easy ways to buy are: https://bm-switch.com/ https://bm-switch.com/ http://whiteboxswitch.com/ http://whiteboxswitch.com/ http://ipfabrics.net/ http://ipfabrics.net/ http://www.penguincomputing.com/products/network-switches/ http://www.penguincomputing.com/products/network-switches/ - nolan co-founder/CTO Cumulus
- tiernano 12y agoAre there any that are, say, 48 port GigE? 10G and 40G are a bit much for home use...
- zymhan 12y ago10G and 40G are a bit much for my office, even.
- skuhn 12y agoThere are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and $700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain support in order to continue using the product (legally). I can't imagine getting enough utility from a switch for this to make sense for home use. It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly low end 1 gig switch (single power supply, limited L3 capability), and you can absolutely negotiate Juniper / Cisco / etc. down to be close to or even below the pricing of the Cumulus solution. That is tougher for 10 gig or 40 gig equipment, which is where the value proposition kicks in for Cumulus. Maybe Cumulus's approach is enough of a value add for it to make sense to pay a premium, but everyone I talk to is interested in cost savings first and better manageability a distant second. You'll still have to have Juniper / Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they don't have a full range of switch models.
- kijiki 12y agoStay tuned for low-end 1G platforms running Cumulus Linux. - nolan
- sschueller 12y agohttp://www.pcengines.ch/apu.htm http://www.pcengines.ch/apu.htm
- pingswept 12y agoBy "open hardware," do you mean open source hardware, or something else? From the responses, it looks like "open hardware" means something like "can install any OS on it" in the network switch world.
- rustyconover 12y agoI get the feeling this is a bit like Bugatti building the Veyron, its super cool and impressive but the vast majority of us will never be able to or have a need to drive it. It is still fun watch this switch go around the proverbial track, but I'm happy knowing that I'll never have to configure, build and test a switch of this complexity unless I really absolutely have to, with my largest caveat being AWS disappearing from the face of the earth.
- AceJohnny2 12y agoYou're not the target market. You shouldn't be making an analogy with a Veyron, which remains a luxury car for an individual. You should rather be comparing it with a firetruck, a vehicle custom-built for a purpose you as an individual you will never need. I currently work in the network hardware industry (think Cisco, Juniper, etc). Our boxes sell in the 6-figure price range, each [1]. We sell to your ISP, wireless carrier, datacenter constructor. We're the competition this kind of box is aiming at. [1] it's highly specific hardware and software for a low-volume market. Individual chips used in the hardware can cost multiple thousands of dollars each.
- es09 12y agoGood points. I doubt if they are aiming at any competition. Their stated goal is to reduce the cost of the hardware for themselves by standardizing it. Their hope is that the other big guys (amazon, google etc) will pick this solution up driving up adoption and lowering the cost. If this happens, Cisco, Juniper et al gets disrupted and that's just a side effect.
- gonzo 12y ago> Individual chips used in the hardware can cost multiple thousands of dollars each. and 10 years ago, the compute power of a QC i7 would have tens of thousands of dollars.
- AceJohnny2 12y agoRight, but we don't care about past chips. Moore's law applies to the whole market at the same time, so today's special network chips are still much better at what they do than today's general-purpose chips. It's kind of stupid to say that today's general-purpose hardware can do what yesterday's special hardware could do at a faction of the cost, because the specialist market has also moved on and wants the performance of today's special hardware.
- deleted 12y ago[deleted]
- cordite 12y agoI wonder if youtube will ever notice that an uploaded video is quiet and only on one channel, and allow for normalized mono listening..
- gdooraccess 12y agoDoes anyone know which team in FB is building software for these products? I did glance FB company info but could not get the information.
- aristus 12y agoI no longer work there but I can ask the usual suspects. Email is in my profile.
- gdooraccess 12y agoThanks and sent you an email. For anybody who is looking for similar details on the team who are behind these products, just FYI - there is not much info on FB site or on the internet. If I get any info later, will report back.
- hamburglar 12y agoThe only datapoint I have is that it's in the SF bay area, since I am acquainted with someone who got poached from Amazon specifically to work on this and had to relocate from Seattle to there.
- VLM 12y agoI was expecting a link to something like this, but more advanced http://opencores.org/project,mac_layer_switch http://opencores.org/project,mac_layer_switch and some links to digilent dev boards for hardware, however the linked project is a bit more ambitious, all full of custom ASICs and such. An open source switch made out of COTS FPGA dev boards would be interesting. So you'd use something like http://www.digilentinc.com/Products/Detail.cfm?NavPath=2,400,1228&Prod=NETFPGA-1G-CML http://www.digilentinc.com/Products/Detail.cfm?NavPath=2,400... But what to use as a COTS "just unpack it all, plug it all together, upload the firmware and go" backplane?
- virtuallynathan 12y agoThat NIC seems quite overpriced for 4x1G...
- wmf 12y agoIt's not a NIC; it's an FPGA that can be programmed to do things that NICs have never dreamed of.
- virtuallynathan 12y agoI understand that, but you can get more powerful ones for not much more money.
- detaro 12y agoNICs or FPGA boards with NICs? If the latter, I'm all ears
- virtuallynathan 12y agoNetronome, http://www.mellanox.com/related-docs/prod_adapter_cards/PB_Programmable_ConnectX-3_Pro_%20Card_VPI.pdf http://www.mellanox.com/related-docs/prod_adapter_cards/PB_P..., there are others for more $$ using the Altera Arria 10, or other Xilinx FPGAs with 2x40G or even 4x100G.
- fidotron 12y agoIsn't this just wrapping for the COTS switching ASICs or am I missing something big? Almost all "open" projects seem to hit a point where the line between open and closed is quite arbitrary, even if they choose not to see it that way.
- wmf 12y ago"Wrapping commodity ASICs" is worth $4B: http://www.arista.com/ http://www.arista.com/ If Facebook wants to give me that value for free I won't complain.
- trsohmers 12y agoAnd the founder/chairman/chief development officer of Arista, Andy Bechtolsheim, is a board member and founder of the Open Compute Project.
- SEJeff 12y agoFWIW, all Arista hardware is AWESOME. They run a lightly modified very minimal linux operating systems that is very clearly (not even really hidden) Fedora. I got the privilege / opportunity to reverse engineer one of these to see if it was hackable (ie: add custom hardware into them or run apps directly on the switches). Pretty much the entirety of the Arista userspace is python with all of the magic really in their ASICS. Truly, I've not seen many better engineered pieces of equipment. At the time (I did this > 4 years ago), the Arista switches had dual core AMD Athlons with a few G of ram. They were willing to put more RAM in one, or let us upgrade it. For a so called "vendor product" Arista clearly gets it. They have some really solid Linux guys working for them. When you get the pleasure of using their python apis or pull one apart, it shows that they know what they're doing. It isn't completely insane to think of running apps directly on the switches when they are simply x86_64 Fedora boxes with some fancy asics and a lot of interfaces.
- spydum 12y agopretty sure the big win here is: 1) low individual component cost (which is great when you need 100's of them for a datacenter build out) 2) no frills, high throughput non-blocking backplane which are normally only available from the enterprise grade network vendors at top dollar per individual component, and come with a mountain of features and bugs you DONT NEED 3) software defined networking stack: if you can imagine a scenario/feature that would improve your life, nothing but development costs will get in your way. contrast with enterprise vendors, which requires explaining the problem to begin with, and dangling a wad of money explaining how it will be worth their while to develop it
- korethr 12y agoThis is rather cool, and I'm glad to see some work being done in the networking gear space utilizing open designs and firmware like this. Unfortunately, unless I find myself needing to network a datacenter in the near future, it's not immediately useful to me. That said, I do hope other developments in network gear that will be useful in other markets emerge from this effort. From my perspective, there's a gap in network gear between the unmanaged, low port-count switches in plastic enclosures, targeted to home and small office consumers, and the lower tiers of Cisco's catalog, targeted toward top-of-rack or wiring-closet-of-a-larger-building type uses. I would love to see a managed switch with say, 8-24 ports, supporting features such 802.11Q VLANs. I would love to be able segment my network at home so different devices with different performance and security needs aren't all stepping on each other's toes. And I'd like the firmware and hardware designs to be open source, so it can readily patched when bugs are found, and easily adapted to new use cases. I realize that I'm an outlier and that my needs are not common, or there'd probably be equipment on the market that met them. But it is my hope that as a result of Facebook's work here, and similar efforts, that building such a device will become feasible.
- gh02t 12y agoI also wanted something in the category you're describing and after shopping around a bit I stumbled across Mikrotik's products. I bought one of their 24 port smart switches and it was exactly what I was looking for. Not a lot more expensive than some of the nicer consumer-targeted gear, but it has way more functionality. http://routerboard.com/CRS125-24G-1S-IN http://routerboard.com/CRS125-24G-1S-IN
- pdkl95 12y agoWow. That switching fabric is so much smaller than the HIPPI crossbar backplanes we used in the 100baseT/pre-GigE switches that I used to write firmware for... almost 20 years ago now. Of course, complexity means higher cost, so it's obvious why nobody bought our stuff. Hopefully, this open hardware project will have better luck! /* it's easy to forget just how many iterations of Moore's Law have happened since the mid-90s */
- btoptical 12y agoThese big switch boxes typically end up being about thermal management and this box looks like thermal design was an afterthought. Also I'm not sure the power entry design is really all that smart. Generally speaking if you're in the market for a 640Gb/s switch (or 3.8Tb/s switch), does your data center really not have access to 48V power? The AC/DC conversion wastes power and space. The QSFP's are "spaced for optimal airflow." However this spacing seems to neglect cooling the QSFPs themselves. Belly-to-belly mounting of modules is usually the most thermally challenging way to arrange them. The heat dissipated by the QSFP's is generally directed towards the top of the module. By placing open air channels between modules, they have effectively ensured that little to no air flows over the QSFP heat sinks (which is not shown). So there is probably a limitation on which reach codes are supported. My guess is that because of the thermal limitations of this design, it's not truly non-blocking in all reach configurations.
- wmf 12y agodoes your data center really not have access to 48V power? No, they really don't. I've never heard of 48V outside of telco.
- btoptical 12y agoI've been to lots of them all over the world, names you'd recognize, and they've all had 48V.
- DiabloD3 12y agoFacebook's Open Rack project is based around 48v.
- teh_klev 12y agoYes they do. My old employer TeleCity installed 48v as standard, for their telco customers.
- jacquesm 12y agoIt's pretty common here (nl), you have to ask for it but it is usually available. I'd advise against it because you're deviating from the bulk but if you need it you can usually get it, especially at the larger colocation facilities.
- aaronsnoswell 12y agoAm I the only one who read the title and thought it referred to a 'pushbutton' style electronics hardware switch? I would have been quite excited to see that!
- MrZipf 12y agoThis looks great. They also have the advantage that James Hamilton mentioned AMZ got from building their own hardware - the get to do a cleansheet s/w design and not have to support all the gazillion options that Cisco does. The talk was on HN a little while ago, worth watching: https://www.youtube.com/watch?v=JIQETrFC_SQ https://www.youtube.com/watch?v=JIQETrFC_SQ The scale at FB means they're probably saving hundreds of millions doing this.
- arca_vorago 12y agoI just want a fully open source switch that's ready for enterprise, from the h/w to s/w. Linux preferred but bsd based is acceptable. I've been a "cisco guy" since 2001 or so, but I am so tired of them. Licensing fees kill budgets that could be used on other things, and you end up surrounded by consultants that only ever touch Lozedoze systems insisting that "nobody ever got fired for buying Cisco". Smartnet is a must for some equipment, yes, but I'm so ready for a paradigm shift in networking. I've really been watching ubiquiti and their switching/routing products, they seem very promising but not quite prod ready. I am impressed with Dells open switches too. Edit: The Microtik stuff being linked elsewhere here is looking pretty awesome too. Not quite FOSS but still.
- charlesnw 12y agoSo this is very interesting. As someone who runs a 6509 (almost fully populated with line cards and specialized controllers) as my HOME core switch (with a second one on the way for full redundancy), an open modular switch is very cool. (I'm currently hacking on OpenFlow -> NETCONF bridging, to bring typical SDN capabilities to legacy Cisco environments). Hence my rather..... extensive home network. This is very similar to what Cisco is doing with Nexus and the "fabric extender" TOR replacement kit. I'll have to see if Facebook has any of this stuff in GIT and stand it up in a VM environment and play with it (I already do a bunch of OpenFlow stuff on OpenWRT and am looking at implementing an open southbound API on FPGA on the parallela board).