9 ms·
Making a Linux-managed network switch
- teddyh 2y agoDoes it support rSTP?
- MartijnBraam 2y agoI don't think this support it, only regular STP and apparently MSTP
- teddyh 2y ago> and apparently MSTP Where did you find that?
- gerdesj 2y agoSTP is "slow" STP, and RSTP is a degenerate case of MSTP! MSTP is Multiple spanning trees ie you can group VLANs and prefer paths for those groups of VLANs. That means if you have say two links between two bridges (switches) you can prefer some to use one link and the rest to use the other, that means you are not "wasting" a standby link. They will fail over to the surviving link on failure. STP and RSTP will only consider one link as a whole, so two ports are "wasted" when not in use: in the case of a two bridge, two links example. Old school STP without the Rapid part hasn't really been a thing for several decades. I can't think of why you wouldn't use RSTP in general but if you need to make best use of your forwarding capacity then a 50/50 MSTP may be indicated. That's where you look at your traffic flows across VLANs and try to bundle them up into a 50/50% collection. One lot prefers link A and the rest get link B. Obviously you can get really creative as the number of VLANs and links mount up. Bear in mind that dot1Q is a simple version of QinQ! Sorry, got a bit carried away there. For nearly all intents and purposes, RSTP is STP. If you plug in a network cable between two devices and it does not start working within say five seconds then you are living in the 1990s.
- deleted 2y ago[deleted]
- mito88 2y ago> If you plug in a network cable between two devices and it does not start working within say five seconds then you are living in the 1990s. a living in a hub... :)
- teddyh 2y agoI meant, where did you find that this specific Linux-managed network switch supports MSTP? Because the Linux bridge networking code, last time I looked, only supported STP; you had to run a separate daemon to even get RSTP.
- thelastparadise 2y agoWhat's the advantage of doing this over plugging multiple gigabit adapters into a linux machine and adding them all to a bridge? I'm guessing performance might be better with the hardware, but I don't know --has anyone done tests to show the difference?
- MartijnBraam 2y agoThis is more efficient if most of the traffic will get switched, seperate adapters is more efficient if most of it needs to be routed.
- simcop2387 2y agoIt'll be a combination of: 1. Throughput - say you use usb adapters, in a lot of ways usb is a shared bus so you'll run into max bandwidth quickly. This is especially because data will have to go in and then out, all the way to the cpu 2. Latency - because you're using software to do switching, it'll add time to process each packet and send it back out tne right place. You've also go any other interface latency adding to it 3. Power usage - eacj adapter will have it's own full network phy and hardware, which will increase the power draw. Combined with all the extra processing above and now your power usage is even higher. This means you also loae out on hardware offloading and other performance enhancememts that generally reduce power usage because less of the system is involved in mocing packets around 4. Features (potentially) - this will depend a lot on the hardware you choose, some of those cheap gigabit usb adapters i've tried didn't work with vlans and other features properly. But if you say load up a bunch of nice pcie cards with 1 or more ports thst support everything (never had issues with pcie ones) then you can now actually get a lot of features that are otherwise difficult or impossible on simpler hardware (though at that point you're doing routing more tham switching, but thay flexibility is why you'd potentially do this).
- Fnoord 2y agoWe're only talking 1 gbit ethernet here, so you can have multiple of those ports on PCIe. I have a PCIe card here with two 2.5 gbit on it (don't remember exactly how much it was on Ali but between 20 and 40 EUR) and I can saturate both with iperf3. Since the example only uses 4 ports, it should be easy to make a simple router with just two PCIe cards. But there's probably 4x 1 gbit PCIe out there, too. And if you use 1 gbit fiber, that wouldn't cost much power nor would it need much speed. If your uplink is DSL, you could use a VigorNIC 132.
- buccal 2y agoInteresting project. For simpler use you can get an OpenWRT capable router which in most cases uses a managed switch chip. OpenWRT provides a nice interface to configure VLANs and other options.
- tomatocracy 2y agoOpenWRT these days can also be installed on some switches eg the Zyxel GS1900 series (though support for things like PoE and 10Gb/SFP+ ports might be limited, I’m not sure where things stand there).
- stragies 2y agoLooking at the OpenWrt forum, work is coming along nicely w.r.t supporting FasterThan1G on RTL9X. Many/Most configurations seem to be working. RTL8X is AFAIC done, feature-complete. https://svanheule.net/switches/ https://svanheule.net/switches/ POE on those devices is mostly two types: Broadcom (well supported), and Realteks inhouse solution, which uses a 'dialect' of the Broadcom protocol. There is a git branch/PR, where the 'dialect' differences have been moved to individual modules. But it's not released yet. https://github.com/Hurricos/realtek-poe/pull/35 https://github.com/Hurricos/realtek-poe/pull/35 Unfortunately, there seem to be hard problems migrating up from Linux 5.15 to 6.1 or 6.6.
- znpy 2y agoWTF, i had no idea that was possible. I have that exact switch (GS1900-24E)... I need to look into that
- stragies 2y agoNice article, Thank you for the write-up. First time I saw somebody 'creatively using' an RTL83something switch was https://spritesmods.com/?art=rtl8366sb https://spritesmods.com/?art=rtl8366sb, and there were others since then, but yours was the first 'build my own managed switch', instead of 'adding an external brain to an unmanaged switch'
- MartijnBraam 2y agoAh yes the legendary sprite_tm, I've come across this while writing my own raspberry pi firmware to manage the switch over USB.
- gerdesj 2y ago"Network switches are simple devices, packets go in, packets go out. Luckily people have figured out how to make it complicated instead and invented managed switches." Expensive switches involve some pretty fancy ASICs. For example I have a pair of fairly elderly Dell OS9 switches with 48 x 10Gb/s ports and four x 40Gb/s QSFP+ fibre ports. These are "old skool" stacked jobbies. Each switch can shuffle up to 1.28Tb/s (1). That's quite a lot. You can get those for £1800 including VAT (2) these days and they will last nearly forever. I love to see efforts like this but do bear in mind that say Netgear will do a eight port 1Gb switch with Power over Ethernet on all ports for about £125. If you cost your time at somewhere between £20-50 per hour when evaluating whether a project is financially viable, then an off the shelf box might be indicated. However, if the actual purpose is the project itself then sod the price! (1) https://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell-Force10-S4820T-SpecSheet.pdf https://i.dell.com/sites/doccontent/shared-content/data-shee... (2) https://www.etb-tech.com/dell-force10-s4820t-10gbe-switch-os9-normal-airflow-sw00239.html https://www.etb-tech.com/dell-force10-s4820t-10gbe-switch-os...
- bithead 2y agoIgnorance is bliss.
- immibis 2y agoI worked with one of these ASICs, from Broadcom. Not 40, but something like 4x10+24x1Gbps + PCIe to CPU. The ASIC cost as much as you would expect (I don't know the actual number - a couple hundred bucks?). The software interface to it was very poorly documented, and was a library that supported all Broadcom switch ASICs, so was a few hundred megabytes .a file and was full of functions that would just return "not supported on this device" errors, which you wouldn't know until you tried it.
- zokier 2y agoNot Broadcom, but Microchip has list prices straight on their website (how refreshing!), and yeah few hundred bucks sounds about right; for example this random 128Gbps switch chip is about $120 in single quantities, $80 in volume https://www.microchip.com/en-us/product/vsc7552#purchase-from-store https://www.microchip.com/en-us/product/vsc7552#purchase-fro... It feels bit bonkers that a actual switch based on that probably costs thousands. The software side is crap as usual. They sell their Linux-based package at $75000, with maintenance (basically updates) at $17500. So even though the chip is cheap, you need to burn six figures to do anything with them :(
- wesapien 2y agoWhat ever happened to Openflow? Wouldn't this be a perfect device for it? Setting up a lab for it was almost impossible thats why I lost interest.
- wmf 2y agoOpenFlow was kind of the wrong solution to the problem. (The right solution is switchdev or SAI.)
- wesapien 2y agoI'll look them up. Cheers.
- SSLy 2y agoSAI being what?
- wmf 2y agoSwitch Abstraction Interface
- p_l 2y agoOpenFlow is a protocol, switchdev is a driver API for exposing access to essentially the same operation model as OpenFlow targets. You don't replace OpenFlow with switchdev, you might use OpenFlow to integrate a switchdev with an SDN
- wmf 2y agoYou want to write code that runs on the switch, whether it's a routing protocol or an intent reconciliation loop or whatever. You don't want to send low-level OpenFlow commands over the network because it ends up being slow and chatty.
- p_l 2y agoA considerable portion of SDN craze involved pretty much this. Even systems that didn't use OpenFlow were incredibly chatty, creating complex chains of NextHop routes from centralized or at least semi-centralized servers to the actual switches. Even in more sane setups, it's not actually uncommon to have control plane talk over network to actual forwarding plane - IIRC Cisco Nexus operates this way, with actual forwarding engines talking with control plane over Ethernet link. I like the code to run on the switches directly, but SDNs very often eschewed that for dumber forwarding engines controlled OpenFlow way, even if actual protocols used weren't OpenFlow.
- protocolture 2y agoNo offense intended, I may have misunderstood something here. You use a routerboard block diagram as your model to demonstrate how a hardware switch is connected to the rest of the system. But then you go on to claim that they are impossible/difficult to work with. Did you source a routerboard at any point here? Fairly sure that OpenWRT can be built for most routerboards, and the 2011 should be a fairly common device on the second hand market. Maybe this is a better question. Was the goal always to build from scratch? Or did you discard the concept of using someone elses hardware for a particular reason?
- stragies 2y agoThe RouterBoard RB2011 is stuck on an ancient release version of OpenWrt, 19.07, see here: https://openwrt.org/toh/mikrotik/rb2011 https://openwrt.org/toh/mikrotik/rb2011 The problem seems to be somewhat related to the NAND, which is IIRC somehow different from the supported RouterBoards. Somebody proposed a new solution to this problem, but that hasn't landed yet: https://forum.openwrt.org/t/wiki-cleanup-for-mikrotik-rb2011/89858 https://forum.openwrt.org/t/wiki-cleanup-for-mikrotik-rb2011... and other threads.
- MartijnBraam 2y agoI have a stack of them here, they are great to work with for their intended purpose but they are not that great if you want to run custom software. I also grabbed a RB2011 diagram because it's a simple diagram that explains it well I think, the RB1100AHx4 is a better example technically since it's using the same switch chip but it's more confusing because they use the two CPU ports together and claim it's a 2.5Gbps link while it's two 1.25Gbps link and they ignore the encoding overhead. The reason why I build this from scratch is that it that the expense is reasonable and this should end up in the FOSDEM video recording boxes and it has to fix a few issues specific to that design like it needing to expose 4 network ports to the front panel of the case but also be connected to the internal SBC. There's simply not that much space in the case to put passthroughs in the case to a switch to not have an external loop cable for the SBC and with a dumb switch the system can't be monitored anymore. Since quite a few of these boxes are built this becomes a reasonable solution (if you ignore design time, but this is volunteer work)
- shmerl 2y agoFrom what I've read, it's very hard to make a 10 Gbps switch without relying on some blobs since those accelerator chip makers don't have any open drivers support. And you supposedly need those chips since switching it on the CPU is very taxing.
- BertoldVdb 2y agoMarvell 88E6393X works in 'dumb/externally managed' mode without firmware. You can use it with Linux switchdev like the one in this article.
- nicholasbraker 2y agoThe first paragraph reads as if it's taken directly from The Hitchhikers Guide To The Galaxy.