7 ms·
Sysadmin friendly high speed Ethernet switching
- redleader55 3y agoI'm curious what did this guy connect to several 100 GbE ports and how does the upstream connections he mentions look like, and from which provider. The device is second hand, so the likely use-case is a Home(?)Lab type of setup.
- benjojo12 3y agoHi, I am the guy. > what did this guy connect to several 100 GbE They have 100GBASE-PLR4 optics in them, that allow the 100G ports to be split up into 4x25G ports (or actually in this switch, 2x25G ports due to a hardware limitation with this switch) > how does the upstream connections he mentions look like, and from which provider They are just normal 10G-LR Single mode optics, in a data center. > The device is second hand, so the likely use-case is a Home(?)Lab type of setup. Nope, this device now runs my business bgp.tools
- redleader55 3y agoThat suddenly makes a lot more sense! Thank you for the explanations and for the writeup!
- ComputerGuru 3y agoStupid question, why does the datacenter provide you with LR smf instead of SR mmf?
- benjojo12 3y agoIn my eyes, MultiModeFibre is on it's way out. The high speed (25G+) does not have good solutions for multimode, and the length limits that physics enforces with multimode mean that it's not "no-brainer" applicable for going any more than between the same rack row. So if you are dealing with datacenter cross connects that can exceed the max distance for MultiMode then you might spend hours debugging broken stuff for no real gain. MultiMode is slightly cheaper, but it's a false economy the moment stuff does not work correctly. I've spoken to people with DC cross connects that go into the 5km+ of cable distance. So it's easier to just stock one kind of optic per speed, and call it a day. Equinix I think already phased out MMF XCs
- ComputerGuru 3y agoThanks for the info.
- tssva 3y agoI stopped dealing with large data center providers in 2014. By that point we had switched to using single mode within our colocation spaces and most of the colocation data center providers we dealt with had stopped providing any multimode cross connects prior to that.
- karma_pharmer 3y agoFriends don't let friends use multi-mode fiber. Single-mode fiber is future-proof. Utilities bury that stuff and depreciate it over a 30-year lifetime. There has been like one spec change since the 1970s. Single-mode fiber. Always single-mode fiber. Nothing else, ever.
- gorkish 3y agoStupid answer: Multimode sucks. The only reason to use it is if the person before you failed to know that it sucks and now it cannot be replaced.
- somat 3y agoFor them unfamiliar with fiber optics. Single mode fiber is better than multi mode in nearly every category. The only advantage multimode has is when terminating it. The looser tolerances and large fibers make it easier to attach an end.
- deleted 3y ago[deleted]
- PreInternet01 3y agoYeah, the whole "(hyper)converged" network gear experience has been... let's say less than overwhelming. I'm happy that this person managed to get an experience they're happy with, but keep in mind that: -The SN2010 retailed for about 10K (Euro/Dollars) -It has never been truly available, as in: you could go somewhere, order it, and expect it to turn up in 2-3 days -Even though small, these units tend to be loud, with at least 2 tiny fans making a lot of high-pitched noise But the most salient point is that, even with "Linux on a router/switch", there's no guarantee that you'll get decent performance, as that entirely depends on how well the kernel understands the (proprietary) onboard chipset, which usually means that you're squarely in "well, here's this blob that works on certain kernel versions, and good luck with that!" territory.
- mattpallissard 3y agoHyperconverged infra is basically bundled networking, storage, and hypervisor. Think nutanix, vxrail, Cisco ucs.
- benjojo12 3y ago> But the most salient point is that, even with "Linux on a router/switch", there's no guarantee that you'll get decent performance As long as the ASIC is programmed the performance is the same between kernel versions. If the ASIC does not get programmed then it's like the entire device does not work at all (so you likely need to roll back the update you made). > which usually means that you're squarely in "well, here's this blob that works on certain kernel versions, and good luck with that!" territory. As mentioned in the post, the switch in question has blobless drivers, unlike the broadcom stuff you likely have experience with based on what you are saying
- wmf 3y agoYou're FUDding yourself pretty hard. All of these points have been addressed in various comments in this thread.
- jacknews 3y agoWoah these are around $5k. Not sure that's a fit for home network enthusiasts, and for business, just pay the bucks to be able to pass the buck - ie get a packaged solution not homebrew.
- benjojo12 3y ago> Woah these are around $5k. For what it's worth I did not pay 5k USD for it, that is significantly overpriced for it on the 2nd hand market. > for business, just pay the bucks to be able to pass the buck - ie get a packaged solution not homebrew There is a significant cost difference here, The overall point of the post is that the "packaged solution" is not viable when most switch vendors have what I can describe as "crap" software quality and/or support. So if you are in the land of simple L2 switching and L3 routing, this switch is amazing because you can escape the crap vendor software.
- hinkley 3y agoSort of an openwrt for hard lines.
- bradfa 3y agoeBay shows me a whole bunch of these SN2010-series switches in the $1000-2000 USD range. Totally reasonable for such a switch. You can step up to the SN2700 mentioned in the article for prices in the $2000-3000 USD range. All used from sellers claiming "tested". If you're really into homelab stuff or trying to run a small business where networking is critical, they're quite a good deal. The "packaged solutions" as a new product from a known vendor with a support contract (because honestly if you're not buying the support contract you're going to spend a LOT of time messing with the switch no matter what vintage it is) are at least 10x the price, which very well could be much too much for a home user or small business.
- candiddevmike 3y agoI've seen a few chinese knock off switches on Amazon and Alibaba that are cheap and layer 3/"managed". I messaged a few of the alibaba sellers telling them I'm "building a custom switch OS and want their hardware to prototype" and they seemed amicable to helping me get a custom firmware installed, but never pursued it more. I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with...
- greyface- 3y agoReally tempting price/performance, and I love the idea of getting away from questionable switch vendor OSes. How disruptive are switch config changes? If I edit /etc/network/interfaces to add a new port, vlan, etc - does a `systemctl restart networking` (or whatever equivalent) bounce ports or halt switching for a moment while changes are applied?
- treffer 3y agoIt just depends on what you use for management. IIRC the /etc/network/interfaces does a reconfiguration that's pretty disruptive. Things like brctl and ethtool worked on the fly without issues (note though that I mostly used Arista years ago). It is usually non-disruptive if it gets applied as deltas. If your config tool does a teardown/recreate then that's disruptive. Within the bounds of ethernet and routing protocols (OSPF DR/DBR changes are disruptive, STP can be fun, ....).
- benjojo12 3y ago> How disruptive are switch config changes? If I edit /etc/network/interfaces to add a new port, vlan, etc They are seamless as long as your configuration does not do something stupid like tear down interfaces to reconfigure them. The switch takes no noticeable time to program from the Linux space to the ASIC > - does a `systemctl restart networking` (or whatever equivalent) bounce ports or halt switching for a moment while changes are applied? "systemctl restart networking" will typically blow away even the well most well configured systems in my experience. In the post I suggest using ifupdown (the first one), since it's the most "easy" to debug, but I'm sure networkctl works too, with a healthy amount of systemd restraint
- bitbckt 3y agoI have an SN2700 in my rack, next to a pair of Arista 7060CXs (as a point of comparison). These are wildly under-rated devices outside of the STH fanbase. You may be surprised at how quiet and low power these Mellanox switches can be.
- benjojo12 3y agoQuiet in the terms of DC grade switching, I'm not sure you could get away with such a switch in a home environment. The 100G optics need plenty of airflow to keep cool so it's not just a case of swapping the fans for something smaller
- bitbckt 3y agoThe rack I’m referring to is in my home. YMMV, of course.
- dgacmu 3y agoYou can use a 100g DAC within a single rack (which is also cheaper). I only have a tiny bit of 100g at home and just do 10g optics for the connection out of the rack. Of course, that's my weird setup and it won't work once I want 100g to my office.
- ComputerGuru 3y agoThis is really, really cool and I didn't know the platform was open to the extent that you could install your own upstream Linux and just get going. I am curious though what configuration option prevents this from ending up with software switching. I understand the mellanox kernel module was compiled and loaded, but certainly that doesn't mean that anything you do in the network stack gets converted to switch fabric code and uses hardware packet switching. How do you make sure that you don't errantly wind up with poor latency and capped throughput? But also, short of making and selling your own networked device for whatever reason, what are the real benefits of going this approach? I can see crazy use cases for where you have full control of the network stack (but again, see my point above — how do you guarantee you are not doing this in software?) but for most purposes, especially with qsfp and fiber, how much are you really gaining by doing it on-device? What is the killer use case here? EDIT Upon rereading, it seems the switch is hard-coded to be hardware-switched and cannot end up in a situation where you are accidentally using software packet switching in the first place (i.e. it does not just optimize to a hardware packet switching state). But that limits what you can do considerably, to the point that an off-the-rack Juniper or CSCO or whatever probably has more features than you can do here without writing your own code to hook into the mellanox sdk?
- benjojo12 3y ago> But that limits what you can do considerably, to the point that an off-the-rack Juniper or CSCO or whatever probably has more features than you can do here without writing your own code to hook into the mellanox sdk? I mean, I'm not touching any mellanox sdk here, I am using the a very similar stack that someone on a "software router" would use, on a switch that can automatically accelerate it to 800G+ throughputs, while hitting a 60W power target. You can hit some of those performance/power numbers in vendor hardware like Juniper/Cisco/Arista, however you have to also put up with their software, I (and others in my group of peers) have not had great experiences with vendor software, and in this setup I am able to patch/fix the software on my own terms. If there is a security vuln in one section, I can fix that, and call it a day, I won't be forced to upgrade parts of the system I do not want to. I cannot do this with Juniper/Cisco/Arista always.
- 3y ago
- p1esk 3y agoThe author mentioned another similar write up which provided more useful explanations and context to me: https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html
- jmbwell 3y agoI want this and I want to put VyOS on it.
- pa7ch 3y agoI wish the edgerouter line by ubiquiti continued as another way to use vyatta.
- cvalka 3y agoVyOS is a great piece of software. I wish more people knew about it!
- gorkish 3y agoHaving experience with both; I am confident this would not work as well as you would want it to if even at all. VyOS has no native awareness of switchdev interfaces or their limitations
- jmbwell 3y agoI began wondering about this and planning to investigate. My first impulse was that Vyos would know about anything Debian would know about, so if I could get Debian to deal with it, so also could I vyos. Until I can buy one of these things or something similar, which I am now highly motivated to do, if you have any experiences or insights you can share, I’d love to see them.
- gorkish 3y agoThe thing that makes vyos "not Linux" is the configuration shell and monolithic configuration file. Ultimately vyos/vtysh consumes this single configuration file and uses it to template commands or configuration files for other underlying software. For example if you configure ipsec in the vyos configuration file, it will produce a configuration file from a template for FreeSWAN to implement the options you have set. I don't personally know offhand how things like bridges are configured in vyos and how vyos maps bridge configuration onto the physical interfaces, but I do know that there is more than one way to do it in Linux, and not all ways of doing it are compatible with switchdev interfaces. Without vyos having native awareness of switchdev device types or support in its configuration templates for applying features in ways that are tested and compatible with switchdev interfaces there's no way to know if a particular vyos configuration will result in something that works with switchdev or not. Yes, you can always drop to running manual commands to perform fixups or additional configuration but at that point you're losing most of the sauce that vyos provides and you might as well just go back to vanilla linux where you have full control of everything. It is my understanding that all the linux-based switch os's (Cumulus, OS10, etc.) work similarly -- custom shell + custom configuration file mapping to templated backend configurations. It would be nice to see vyos gain official templates tests and support for hardware accelerated switchdev
- neilv 3y agoI like the photo of smuggling gear out of the data center in a backpack.
- logifail 3y agoI have two Mikrotik CRS305-1G-4S+IN (10GbE) plus one CRS504-4XQ-IN (100GbE) switches in my office. Along with a load of other Mikrotik gear. The 10GbE ones are silent, and the 100GbE one isn't exactly loud, unlike lots of second-hand kit which has come from a DC...
- eqvinox 3y agoBenjojo wasn't trying to make something for home or office (or even lab) use, he runs DC installations. The point of the article was to have an as-open-as-possible switch. Mikrotik doesn't qualify for that.
- wolverine876 3y ago> as close to stock Debian as possible Is there an OS or another Linux distribution that matches Debian's performance in this respect, without the complexity of an entire Linux system? Could Debian be stripped down (and then how are updates applied)?
- benjojo12 3y agoSure you can install basically any EFI+amd64 distro you want, that also has the correct kernel module. I don't really know why you would want do that (though, RHEL based would be a reasonable 2nd option) Performance of the distro isnt really a big deal because the OS has so little to do with the day to day packet shuffling, My switch runs at a load average (crappy metric I know, but to give you a idea) of 0.04. I apply updates with apt update and apt upgrade, as you would a normal Debian server. Only the kernel is pinned since it is special for the switch, however you can rebuild the kernel deb's when you want, as the driver is in the mainline kernel repo.
- gamepsys 3y agoI think you are misunderstanding what the author means by "stock Debian." The issue here is that the hardware is so unique that we need to add kernel modules that are not included in the standard Linux kernel that ships with Debian. So we need a custom Linux kernel. Close to stock might also be referring to any config changes the author needed to make in order for the switch to function properly. This could be done with any Linux. I'm not sure what 'complexity' you are referring to, but the complex open modular nature of the Linux kernel allows us the flexibility to build a custom kernel for this interesting application without unnecessary modules. To be frank all modern operating systems are complex because modern hardware is complex and user expectations are high. Some just do a great job of hiding the complexity from the user. Linux makes an attempt to expose it's complexity, but it's complexity is about the same as iOS, macOS, Windows, etc. If you wanted to do without Debian or any distribution you could compile your own kernel, your own user space tools, and then install it on the hardware. At this point updates would need to be applied by pulling code from upstream and compiling it yourself. Look into Linux From Scratch to get an idea of how Linux without a distribution works. However, I don't think the juice is worth the squeeze in this situation. EDIT: My answer to "is there another OS", to which the answer is yeah, probably a dozen. I think a blog post of doing this with openBSD would be interesting because I'm not sure what the exact steps are to install custom drivers on a BSD and openBSD has a very high standard for security. I think the reason it's done on Linux is because there is a lot of expertise about Linux and this type of project is relativity straightforward with Linux.
- mschuster91 3y ago> So, I was very happy to learn that a friend had a Mellanox SN2010 that they were not using and were willing to sell to me. That thing retails new for 10k. You got awesome friends if they have such a thing lying around unused and willing to sell it to you at a discount! For those of us with less fortunate bank accounts: what's the smallest and reasonably affordable Mellanox model that has a similar featureset in terms of native Linux support?
- benjojo12 3y ago> That thing retails new for 10k. You got awesome friends if they have such a thing lying around unused and willing to sell it to you at a discount! Tactical eBay (or whatever is the robust 2nd handmarket in your region) can yield similar discounts on such hardware. Retail price is often list price, and that is often a large mark up regardless. The 32x100G port version of the same switch goes for around £2000 in the UK 2nd hand market > what's the smallest and reasonably affordable Mellanox model that has a similar featureset in terms of native Linux support? The SN2010 is likely the smallest, the SN2700 is likely the cheapest
- 0xbadcafebee 3y agoWait a few years and then root around in the datacenter dumpster once they toss the old gear after tech refresh (between 3-10yrs depending on the gear)
- mschuster91 3y agoNo such things around any more, unless you know people who're willing to risk their jobs. Everything's gotta be environmentally certified and whatnot... no one's putting stuff in a dumpster, it all goes to some commercial reseller that either refurbishes and sells on old equipment or hands it to a certified scrapyard.
- KaiserPro 3y agoI commend Ben for pushing the boundary of open Software networky-type things. Even though I have been running HPC clusters for years, I still really don't like core switching infra[1]. Its just a pain in the arse to investigate and run. [1] I realise that this is very much not a core switch, or even a TOR switch.
- zylent 3y agoI encourage playing with these + cumulus linux via the nvidia air lab environment. It may seem limiting at first, but being able to upload a graphviz topology as part of a CI/CD pipeline is extremely powerful. See: https://github.com/na-son/nvidia-air https://github.com/na-son/nvidia-air for bootstrapping a non-EVPN topology.
- frzen 3y agoI really enjoyed that writeup. I wonder has anyone played with DPUs in a similar way? I have been trying to think of how to hairpin traffic through some standalone DPUs like an nvidia bluefield or pensando. To make my own >100G east west stateful firewall for a small fraction of the cost of a real option. Switch like OPs with all traffic passing to and from a DPU to make a poor man's Aruba CX10000
- wmf 3y agoBluefield can also run regular Linux distros and it can run standalone without a host (but you have to power it somehow). https://www.servethehome.com/zfs-without-a-server-using-the-nvidia-bluefield-2-dpu-nvme-arm-aic-iscsi/ https://www.servethehome.com/zfs-without-a-server-using-the-... All other DPUs seem to be NDA-encrusted.
- MrBrobot 3y agoCool project… for homelab stuff though, I lean towards simpler, cheaper solutions…. Like Mikrotik hardware, and low power mini PCs. I’m not hosting a business out of my house, it all functions perfectly fine.
- deleted 3y ago[deleted]
- sophacles 3y agoGreat, seems like a setup that works for you. I'm considering going to find one of these switches for my lab - it'll be like the 5th switch in my rack. I like playing with high performance networking, and your mikrotik isn't going to cut it for the stuff I like to experiment/play around with. (Remember... this is a hobby where people persue their interests and experiment with things they want to know more about, not a proscribed set of exercises for just one path towards devops engineer).
- aftbit 3y agoPersonally I don't like Mikrotik very much. It's just too easy to turn on some feature that disables offloading. I run Ruckus/Brocade ICX6610 and ICX7150 in my lab currently, though the 6610 uses more power than makes sense.
- Asmod4n 3y agoWould buy something like this as a device you can connect via pcie to your server. No need to have switches and servers in your rack anymore, every server is a switch and every switch is a server with a 192 threads CPU. Insane.
- wmf 3y agoIt ends up crazy expensive and complex to configure but go for it.
- Asmod4n 3y agoI bet Nvidia is selling something like that but with only 2-4 ports for the same price as a 48 port switch...
- wmf 3y agoAFAIK DPUs are only ~$2,000 but if you put one in each server it adds up to far more money than a traditional network. They're really not intended to replace TOR switches.
- karma_pharmer 3y agoAll the switches in this series except the one in this review actually have two separate boards inside with a PCIe-over-cable connection between them. For example the SN2700 here (the sleeved cable in the third photo; you can't see the connector from the angle it's taken from): https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html The cable has a SAS (SFF-8087) connector on each end. I bet you can replace it with an SFF-8087-to-Oculink cable: https://www.amazon.com/chenyang-SFF-8611-SFF-8087-PCI-Express-Slimline/dp/B07DFHH4SK https://www.amazon.com/chenyang-SFF-8611-SFF-8087-PCI-Expres... and a PCIe-to-oculink card like one of these: https://www.amazon.com/Ableconn-PEX-OL153-OCuLink-SFF-8612-Adapter/dp/B08DR17F5F https://www.amazon.com/Ableconn-PEX-OL153-OCuLink-SFF-8612-A... (none of the links are affiliate links)
- chgs 3y agoNo mention of ptp, I’m curious how the switch handles it (and how is it connfiged from Linux) I find understanding how Linux networking really helpful in uunderstanding mikrotiks (which I use a lot in prod, although tend to shy away from for the most critical and demanding of services)
- wmf 3y agohttps://github.com/Mellanox/mlxsw/wiki/Precision-Time-Protocol https://github.com/Mellanox/mlxsw/wiki/Precision-Time-Protoc...
- gorkish 3y agoIIRC the half-width version of these switches are not advertised as having PTP support. I wouldn't have the first clue if it's straightforward to configure it to work with upstream linux but I do use PTP with Dell SmartFabric OS10 so there is likely some way to achieve it. As impressive as it is, switchdev is a big departure from "Linux networking" though. This is not a great platform if that is your main objective. The interfaces are not "normal" in that respect and huge subsystems of the Linux network stack do not apply.
- karma_pharmer 3y agoHaving gone through this struggle myself, here's the cheat sheet. You want a device that uses the Linux switchdev driver and is supported by dentOS whether or not you actually choose to run dentOS on it (I run NixOS on my switch): https://www.kernel.org/doc/html/latest/networking/switchdev.html https://www.kernel.org/doc/html/latest/networking/switchdev.... https://github.com/dentproject/dentOS https://github.com/dentproject/dentOS Switchdev support means you don't need hardware-specific userspace tools (with their own bizarre syntax to learn) in order to configure the switch. DentOS support means the device uses a sane bootloader (uboot or grub) and the only binary blobs on the device will be the ones built into the bootloader (IntelME, Arm Trusted Firmware) and the switch firmware which will be part of linux-firmware (and therefore very easy to manage/update). In particular, looking for these two keywords is how you make sure that the hardware vendor is staying on "their side of the line" between hardware and software. Violations of this line are endemic to 10G+ switching.
- mongol 3y agoI had no idea there were switches you could run NixOS on. What would be an example of such a switch?
- wmf 3y agoThat's what the blog post is about. Mellanox Spectrum switches can run any Linux distro.
- karma_pharmer 3y agoFor 10gbit+: Mellanox SN2xxx. Probably the SN3xxx, SN4xxx, and SN5xxx too but I can't afford those. For 1gbit: Ubiquiti Edgerouter ER-4 and ER-6; both have SFP cages for fiber optic transcievers https://github.com/NixOS/nixpkgs/pull/194153 https://github.com/NixOS/nixpkgs/pull/194153 https://store.ui.com/us/en/products/er-4 https://store.ui.com/us/en/products/er-4
- mongol 3y agoReally useful, thanks a bunch!
- m463 3y agocool "100000baseSR4" I remember 10base*, so I find 100000base* pretty amazing.
- hi-v-rocknroll 3y agoAll I need now is a 1U switch that has: 1. 48-port 10/40 GbE SFP+ with mostly copper UPoE/++ 1.5. 2-4 100 GbE QSFP28 uplinks 2. Doesn't sound like a jet engine 3. Doesn't phone home to a cloud in another country 4. Doesn't use an obscure, fragile configuration language 5. Doesn't cost $5k
- snvzz 3y agoOverriding your points, I'd be happy with: 1. 8-port nbase-t (2.5/5/10G supported in all ports) 5. Doesn't cost $500
- hi-v-rocknroll 3y agoThat's easy. I think I have a 10 GbE POE++ switch that does that.
- snvzz 3y agoWould appreciate some pointers. This would suffice for the fast part of my home LAN... the rest is handled well by a gigabit ethernet switch with one 10g uplink.
- hi-v-rocknroll 3y agoArr.. there be lots of treasures on the high fleas. https://www.ebay.com/itm/386453004153 https://www.ebay.com/itm/386453004153 Arista DCS-7050SX-64-R 48P 10GbE SFP+ 4P 40GbE QSFP+ RA Switch $180
- snvzz 3y agoActually looks really nice for the price, but I don't think it does support nbase-t.
- hi-v-rocknroll 3y agoReplying to myself (minus 2): Edgecore AS5610-52X on FleaBay. It even has dual hot-swap PSUs. Nothing a little resistor or different fan can't solve.