3 ms·
This is a seriously impressive project and it's been fun to follow along. I almost hesitate to say anything, but the constant scope creep (that he readily admi
by Aurornis 2y ago
This is a seriously impressive project and it's been fun to follow along.
I almost hesitate to say anything, but the constant scope creep (that he readily admits in the blog) is starting to feel like a real obstacle to getting anything done. It's fun to see all of the theorizing about 96-port switches with $11K of FPGAs, but it would be fantastic if something like the simple 4-port switch design could be spun out as an open source project that people could iterate on.
I've also been following his solder-in probe projects for years ( https://www.antikernel.net/products.html https://www.antikernel.net/products.html ). There was a small Kickstarter and some reviews, but now the one distributor has 404s on the product pages and there are a lot of "coming soon" labels attached to all the different products and variations. Again, it would be fantastic if just one of those projects was available in distribution semi-reliably.
- azonenberg 2y agoThe plan is 96 ports total for my own use, split across two 48-port 1U switches. And that's something that is now well within reach: I have the power boards assembled and ready to go, I have the FPGAs and PHYs and (not populated) line card boards in hand. (EDIT: clarification, the plan had been 96 ports total since at least 2015, since that's what I had in existing Cisco switching. The major scope creep over that time was deciding that 2x 48 port switches with a bigger FPGA would be a more power efficient, easier to build, and generally superior design to 4x 24 port) All that's left hardware wise is to design the main logic board with the FPGA, processor, and SFPs, plus the mechanical and thermal design of the 1U chassis. I've been focusing on firmware due to the tariff situation and wasn't planning on spinning the production-ish hardware for quite a while. I don't plan any further scope changes now that I've e.g. already put in the production line card PCB order and mostly finalized the fabric architecture. I'm just holding off on the final logic board design until I'm sure about, for example, whether I'll want external packet buffer SRAM like LATENTPINK had or if I'll be fine with the KU5P's internal memory resourecs. As far as the probes go, yes the antikernel.net site is way out of date and I've been too busy to refresh it. * The AKL-PT1 did a kickstarter and I shipped probes to backers, but I stopped selling them because the fixed tip/ground spacing was too awkward and made it difficult to use. * The AKL-PT2 (first gen solder in probe) was for sale for a while but extremely fragile, only 4 GHz bandwidth, and also had fixed signal/ground spacing. * A few in-between designs were total flops and never reached the point of shipping a single unit * The AKL-PT5 (second gen solder in probe) is >16 GHz bandwidth and has a flexible resistor based solder-in tip and a much nicer form factor. R&D is done but I ran into supply chain problems with the specialized resistor it needs (9 month lead time, poor yield, etc). I think I've mostly worked around those issues and I have a PVT run of a hundred units in the pipeline now. Fingers crossed they'll be hitting Digikey in the next 6-12 months.
- jmcguckin 2y agoIf i wanted to do a motherboard that held 16 or 24 raspberry-pi compute modules, could i connect them together using the onboard gig ethernet without using any magnetics? Just by using the gige output from the module going to a commercial switch chip.
- fxtentacle 2y agoIn general, that should work. Here's an example board with 4x CM4 modules and a RTL8367N switching chip: https://github.com/will127534/Miniature-CM4-Cluster https://github.com/will127534/Miniature-CM4-Cluster
- azonenberg 2y agoYou're going to at least want to AC couple through some capacitors to avoid problems with common mode / RX bias issues between the different PHYs. But if it's on-mobo you don't need to have as much of a fault voltage rating compared to standard Ethernet which IIRC specs 1 kV isolation on the transformers.
- 486sx33 2y ago[dead]
- Bluecobra 2y agoIt might help to take a look at Arista/Metamako 10G L1 switches for some inspiration. These are 48x ports and use the FPGA for specific network applications (including switching). You might be able to find some of the older models on the cheap on eBay. I’ve opened up a few and the board itself seems fairly simplistic. I do recall a giant copper heatsink though.
- azonenberg 2y agoThe board-level architecture is going to be super simple as big FPGA designs go: * XCKU5P in the middle * 12x GTYs routed to 4x Samtec ARF6 connector for the line cards * 2x GTYs routed to 2x SFP28 uplinks * RGMII to back panel management PHY * Parallel SRAM bus to STM32H735 management processor * A bunch of Murata MYMGK modules for power conversion off the 12V rail * STM32L431 in QFN48 or more likely BGA100 depending on IO requirements as a PMIC and to manage reset sequencing etc This will be fully FPGA based, no separate switch ASIC, and I want to do all of the hardware design. I'm not sure there is much I can learn from somebody else's FPGA switch design at the board level - it's basically just going to be a bunch of transceivers hooked up to SFPs and some power distribution. All the magic happens inside the FPGA.
- userbinator 2y agobut it would be fantastic if something like the simple 4-port switch design could be spun out as an open source project that people could iterate on. You can just get something like a Realtek single-chip switch IC and put it on a PCB. Low-port-count unmanaged switches are cheap commodity products already.
- superb_dev 2y agoBut not a cheap open source commodity
- azonenberg 2y agoThis is not going to be cheap. That was never the goal. In low volume when you combine several custom multilayer boards, custom powder-coated sheet metal work, etc even if you allow for the practically-free recycled FPGA this is going to probably end up being a $5-10K project. The last custom one-off 1U I did from ProtoCase was like $800ish just for the enclosure, and that was before the recent tariff hikes.
- superb_dev 2y agoOh fair, maybe just “open source commodity”
- ajb 2y agoYou can actually now get realtek rtl838x based ones that run openWRT. That's open firmware not open hardware, but good for most purposes.