9 ms·
Corundum: Open-source, high performance, FPGA-based NIC
- pedrocr 7y agoReally interesting project. Couldn't find the motivation explained. Is this just for research? Is it usable in production running in an FPGA? Are there plans to produce hardware?
- crispyambulance 7y agoIt's from a group at UCSD, so yes, this is research. The applications for these kinds of things range from SDN (software-defined networking) where low-latency is a concern and to applications in network monitoring. One could, for example, put together a system that performs line-rate TLS decryption at 10Gbps. You need an FPGA (a big one) for something like that. There are commercial vendors for this kind stuff (selling closed source IP and hardware). It is not yet in Open Compute networking projects, but I expect that's coming soon. You can now buy "whitebox" switches that run open network linux and put your own applications on them. In the not-too-distant future those "applications" will also extend to stuff that can run on FPGA hardware .
- wbl 7y agoNope! Netflix does 10G tls on commodity hardware in kernel space. CPU can do a lot.
- shaklee3 7y agoI believe they are doing 100Gbps now: https://t.co/cbb7NA9vJf?amp=1 https://t.co/cbb7NA9vJf?amp=1 It's hard for me to see the use case of an FPGA nic. The reasons outlined above don't seem compelling when a commodity nic like mellanox do so much more already.
- aepiepaey 7y agoDirect link: https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf
- jjrh 7y ago> The reasons outlined above don't seem compelling when a commodity nic like mellanox do so much more already. This is could be useful for people doing testing and benchmarking on network appliances.
- shaklee3 7y agoYeah, I suppose that's a valid use case. Things like ixia need to be fpga-based to measure absolute latency without any uncertainty. You cannot currently get that with enough flexibility in commodity cards.
- wmf 7y agoIt looks like the UCSD team are exploring data center TDMA which no commercial NIC supports. http://cseweb.ucsd.edu/~snoeren/papers/tdma-eurosys12.pdf http://cseweb.ucsd.edu/~snoeren/papers/tdma-eurosys12.pdf
- alexforencich 7y agoThe group web page is here: https://circuit-switching.sysnet.ucsd.edu/ https://circuit-switching.sysnet.ucsd.edu/ Corundum was originally geared more towards optical circuit switching applications, but it's certainly not limited to that. Since it's open source, the transmit scheduler can be swapped out for all sorts of NIC and protocol related research.
- alexforencich 7y agoMellanox NICs (and basically all commercial NICs) do not do what we want. Software is not precise enough, and is on the wrong side of the NIC hardware queues. The whole point of Corundum is to get control of the hardware transmit scheduler on the NIC itself.
- pedrocr 7y agoI saw it was from UCSD but that in itself wasn't an answer. There are plenty of things that are usable in production that are built by research groups initially. SDN isn't the answer either unless these FPGAs can be used directly in production and so there's a path for network cards to no longer being built on dedicated hardware. So to clarify my question I could see this being: 1) A pure research effort on network card hardware design. Useful to test things in a lab and publish papers. 2) Something that can be pushed into production by actually shipping an FPGA in the router, perhaps in specialized situations where the fixed hardware isn't flexible enough. 3) A step before actual hardware can be manufactured, and network cards themselves become a whitebox style business where multiple generic vendors show up because the designs are open-source. Either is interesting.
- zamadatix 7y agoA low end consumer CPU can do 10-20 Gbps of AES per core, it certainly doesn't require a big FPGA.
- alexforencich 7y agoOriginal motivation is to support optical switching research for datacenter networking applications. The research group web page is here: https://circuit-switching.sysnet.ucsd.edu/ https://circuit-switching.sysnet.ucsd.edu/ . It is also mentioned in these slides: https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENLITENED_Annual_Meeting.pdf https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENL.... However, the design is very generic and should be interesting to applications outside optical switching. The main point was to get control over the transmit scheduler, coupled with a very large number of hardware transmit queues. There are a number of experimental protocols and similar that could benefit from this vs. implementation in DPDK. It is still in development; not sure if I would trust it yet for production workloads. We will not be producing hardware; the design runs on pretty much any board that has the correct interfaces, including many FPGA dev boards and commercially available FPGA-based NICs such as the Exablaze X10 and X25.
- sebastianconcpt 7y agoCorundum is an open-source, high-performance FPGA-based NIC. Features include a high performance datapath, 10G/25G/100G Ethernet, PCI express gen 3, a custom, high performance, tightly-integrated PCIe DMA engine, many (1000+) transmit, receive, completion, and event queues, MSI interrupts, multiple interfaces, multiple ports per interface, per-port transmit scheduling including high precision TDMA, flow hashing, RSS, checksum offloading, and native IEEE 1588 PTP timestamping. A Linux driver is included that integrates with the Linux networking stack. Development and debugging is facilitated by an extensive simulation framework that covers the entire system from a simulation model of the driver and PCI express interface on one side to the Ethernet interfaces on the other side. Corundum has several unique architectural features. First, transmit, receive, completion, and event queue states are stored efficiently in block RAM or ultra RAM, enabling support for thousands of individually-controllable queues. These queues are associated with interfaces, and each interface can have multiple ports, each with its own independent scheduler. This enables extremely fine-grained control over packet transmission. Coupled with PTP time synchronization, this enables high precision TDMA.
- throwaway15846 7y agoFGPA = field-programmable gate array. NIC = network interface card. PTP = Precision Time Protocol. TDMA = time-division multiple access. I’m trying to understand what it is and what makes it special.
- ncmncm 7y agoWhat makes TDMA special? It seems to be a provisioning scheme to reserve queue slots to enable guaranteed bandwidth for specific connections. What makes Corundum special? Mainly that you can add client code into it and maybe free up an isolated core, or stage packets to send on short notice. Mainly of interest for low-latency finance and for high-performance compute clusters.
- topranks 7y agoTDMA usually means old legacy time-division based communications standards such as E1/T1 (G.704) or SONET/SDH. Emulating such services in a packet-switched network requires very precise timing hence the reference to it in terms of IEEE1588 / Precision Time Protocol.
- jdsnape 7y agoI've used NetFPGA (https://netfpga.org/ https://netfpga.org/ ) before which seems a little more complete, will be interesting to see how this compares.
- alexforencich 7y agoNetFPGA is a toolbox for network-based packet processing. It is not a NIC, and their NIC reference designs leave a lot to be desired. Corundum is specifically a NIC.
- musicale 7y agoIf NetFPGA gives you access to the PCI bus, it should be possible to make it into a NIC. Simple matter of some verilog and a linux driver. ;-) Of course, we also have non-FPGA Smart NICs from the likes of Netronome, etc. which can do things like accelerate EBPF or run P4.
- alexforencich 7y agoWell, we're planning on porting Corundum to the NetFPGA SUME hardware at some point in the near future. Should be relatively straightforward as the PCIe interface on the Virtex 7 is the same as on the Ultrascale parts. NetFPGA does have a NIC reference design, but AFAIK it's just the Xilinx XDMA core connected to a Xilinx 10G MAC. No accessible transmit scheduler, no offloading of any kind, etc. Just about as spartan as you can get, and it's built from completely closed components so you can't really make many modifications to it. For what we're doing, we can't use any existing commercial NICs or smart NICs because they can't provide the precision we need in terms of controlling transmit timing. We don't care about EBPF, P4, etc. We care about PTP synchronized packet transmission with microsecond precision.
- appleflaxen 7y agoI vaguely remember a corundum project related to the ruby (and rust?) languages, but search engines fail because the terms are all related to mineralogy. FWIW: this appears to be different / unrelated to ruby or rust.
- archi42 7y agoThe hobbyist in me is disappointed: Even the ExaNIC X10 seems to come at 350US$ used, and forget about the four figure UltraScale boards :( Of course, the FPGA can do a lot of stuff in addition, so for HPC this might be really nice to offload application-specific stuff really early (because sustained 2*10G traffic is bound to go somewhere, e.g. CPU[s]).
- ncmncm 7y ago$350 is cheap for such a powerful NIC. You could pay $10k for a Napatech, or $2k for a Solarflare, and not get this level of programmability. libexanic is a remarkably clean user-space kernel-bypass library that allows you to do processing on early fragments of the packet while the rest are still being received.
- imtringued 7y agoComparing used prices to new prices is dishonest. It costs $2,961.00 new. [0] https://www.shi.com/Products/ProductDetail.aspx?SHISystemID=ShiCommodity&ProductIdentity=33645693 https://www.shi.com/Products/ProductDetail.aspx?SHISystemID=...
- ncmncm 7y agoI think we get them for much less. But Netronomes are remarkably cheap, and you can "offload" eBPC to run on the card's custom cores.
- alexforencich 7y agoNetronome cards are a gigantic PITA to program, and they are not capable of the precise timing that we need.
- alexforencich 7y agoI don't know where the heck that price came from. The cards are more like $1200 new. https://www.cdw.com/product/exablaze-exanic-x10-network-adapter/3981568?enkwrd=exanic+x10 https://www.cdw.com/product/exablaze-exanic-x10-network-adap...
- MeteOzturk 7y agoMight be silly question but is there a technique to rapidly program FPGA's without interrupting other processes. Say I have multiple soft-CPU's and only want to use my gates to enable ethernet once its needed or the user plugs in?
- hamiltonkibbe 7y ago“Partial reconfiguration” is the term you’re looking for
- aylons 7y agoNot a silly question and actually a very powerful, seldom used feature: partial reconfiguration. There are, however, several limitations to it. Clock cannot be changed, for example, and usually neither can I/O, specially high-speed transceivers. This has been improving (Ultrascale Xilinx allows for reconfiguring I/O), but you still have to reserve area for reconfiguring (meaning literal area, as in a geographic region in the FPGA). However, I/O versatility as you suggested has very few advantages to it. You need the reserved logic for ethernet to be programmed at when you plugin. Why would you leve it unprogrammed? If is simply disabled, it won't use any extra power resources, and your soft-CPU's won't be able to take advantage of these resources while you are working. Maybe you could use the area for new soft-cpus, but then you'll hit the problem of over segmenting your design and allowing for less optimization. This would inevitably impact timing constraints and area usage. Also, FPGA programming may take minutes to finish, and always at least a few seconds. This will be very noticeable by an user and not very efficient if it has to be done frequently. There are, of courses, good uses for that. But there is also a lot of effort on doing it right and you always risk overdoing it.
- aseipp 7y agoIs programming speed really that bad for the ultra high-end devices? Minutes? I don't remember it being that bad for the Amazon F1 when I ported a Xilinx build to use the F1 SDK (I didn't spend lots of time with our prior one, so I wouldn't know.) Of course, their programming strategy is extremely customized, but even for very high-utilization images, it was only ever on the order of seconds. Vivado is absolutely terribly slow though, no matter what you do, or what device you use. (Not to mention if you want to use the ILA support over the internet...) Also, for some designs you can mitigate the reconfiguration time issue by having two regions and draining requests to one of them, before doing an update. Most of the Xilinx tooling for OpenCL does this kind of thing by default (4-6 "opencl kernel" regions.) But of course it's not always an option to give up that much space...
- imtringued 7y agoAlthough this is cool, the downside is that it only works on specific FPGAs. Are hardware independent designs possible so that one can run it on any sufficiently large FPGA?
- deleted 7y ago[deleted]
- wmf 7y agoNo, because every FPGA and even different boards using the same FPGA are wired differently.
- aseipp 7y agoMaking RTL portable, so it can be used with different toolchains, is certainly possible -- and how/why you do this is, like everything, is an engineering tradeoff. Sometimes when you write C code it's easier to just use Linux features and not care about portability! Sometimes it's very easy (or very desirable) to keep portability for your code, which might be easy (use the standard library) or hard (use lots of #ifdef or whatever.) But, unlike portable C code, to run designs like this on real hardware, you need to do things like describe how the physical pins on the FPGA are connected to the board peripherals (for instance, describing which pin might be connected to an LED, vs a UART). This generally requires a small amount of glue, and depending on how the project is structured, some amount of Verilog/VHDL code, as well. It's not like saying "cc -O2 foo.c" with your ported C compiler that has a POSIX standard library. This is just the case if you're using the same base FPGA, but with different board layouts. Using different FPGAs (for example, a current-gen FPGA by vendor XYZ, vs XYZ gen N-1), or especially when porting between vendors -- the details can become vastly more complex very quickly.
- alexforencich 7y agoCorundum should work on any ultrascale or ultrascale plus FPGA with the necessary interfaces. Probably also virtex 7. We're considering porting to Intel/Altera parts at some point as well...mainly the PCIe interface will need work, but the rest should be directly portable.
- lightedman 7y agoI really wish people would try making original names for their products. Not only am I now going to have to sift through Steven Universe stuff when I look up corundum for people, now I'll have to filter this out as well. Very annoying for geology-hobbyists.
- alexforencich 7y agoThere are two hard things in computer science: cache invalidation, naming things, and off-by- one errors.
- LinuxBender 7y agoThis looks very interesting. If this were included in the Linux kernel, would it read and utilize existing sysctl memory and qlen values as well as have it's own sysctl settings, or would all the settings be derived at module load in the modprobe parameters? Why I am asking is that I current disable TOE (TCP offload engine) on all my nic's as they have their own buffer and retry settings and ignore the OS network settings.
- alexforencich 7y agoIt's still in development at the moment. We'll see about the interface. But there are no plans to implement any segmentation offloads or TOE in Corundum, that will be left up to the network stack. However, scatter/gather DMA support is planned so that software GSO will work. Right now, most of the low-level twiddling is done from a user space app that directly accesses device registers. For a research device, that's fine, but that would obviously have to be improved for a commercial product.
- alexforencich 7y agoAuthor of Corundum here--if you have any questions, ask away.
- deleted 7y ago[deleted]
- nullc 7y agoAre you aware of the extraordinary good deal on huge kintex (K420T) nic-like dev boards on aliexpress? There are boards with 4 sfp+, ones with 2 sfp+ and 2 QSFP+, and even one with 4 QSFP28 (and UltraScale+ XCVU9P)... https://www.aliexpress.com/store/group/FPGA-DEV/620372_250309057.html?spm=2114.12010612.pcShopHead_35478622.1_1 https://www.aliexpress.com/store/group/FPGA-DEV/620372_25030... they sound like great targets for your work...
- alexforencich 7y agoYes, I am aware of those. However, the kintex PCIe interface is a bit of a pain as it has a TLP straddling mode that can't be disabled, so it will be some time before it's supported as it will require some significant reworking in the PCIe interface modules. I am planning on supporting straddling eventually as this will improve PCIe link utilization on the ultrascale and ultrascale plus parts. If someone wants to donate a board, I can look in to supporting it.
- hobo_mark 7y agoInteresting, I never heard of straddling, what is it supposed to achieve?
- alexforencich 7y agoStradding is an artifact of very wide interfaces. On the Ultrascale+ parts, the PCIe gen 3 x16 interface comes out as a 512 bit wide interface. Every cycle of the 250 MHz PCIe user clock transfers 64 bytes of data. The issue has to do with how packets are moved over this type of interface. If your packets are all a multiple of 64 bytes, no problem, you get 100% throughput. However, if your packets are NOT a multiple of 64 bytes in length, you have a problem. What byte lane do packets start and end in? The simplest implementation is to always start packets in byte lane 0. The interface logic for this is the simplest - the packets always start in the same place, so the fields always end up in the same place. However, if your packet is 65 bytes long, the utilization is horrible - it doesn't fit in one cycle, so you have to add an extra cycle for every packet, and bus utilization falls to 50% as you have 63 empty byte lanes after every packet. Straddling is an attempt to mitigate this issue. Instead of only staring packets in lane 0, the interface is adjusted to support starting packets in several places. Say, byte lanes 0 and 32. Or 0, 16, 32, and 48. Now, when you have a packet end in byte lane 0, you can start the next packet in the same clock cycle, but in byte lane 16 or 32. This increases the interface utilization. The trade-off is now the logic has to deal with parts of two packets in the same clock cycle, and it has to deal with multiple possible packet offsets. The specific annoyance with PCIe packets is that the max payload size is usually 256 bytes, but every packet has a 12 or 16 byte TLP header attached, which really screws things up when combined with the small max payload size.
- JackRabbitSlim 7y agoTDMA in a data center environment, anyone else have some heavy deja vu?
- alexforencich 7y agoThat's something we joke about quite a bit in the research group - "back to the future!"