5 ms·

I saw the page via wayback machine since the site was down at the time. Seems like a cool academic project. I've been getting into FPGAs personally, with this e

I saw the page via wayback machine since the site was down at the time. Seems like a cool academic project. I've been getting into FPGAs personally, with this exact application in mind. I've been really stymied by how to process packets in hardware. The obvious approach seems to be to run an RTOS or even full-fledged linux if your FPGA has hard IP cores on it. But is there a better way? How much performance would one lose? I'm also a bit confused about how to communicate on PCI-Express (I'm a software guy ... so learning about DMA). I have seen soft IP for TCP/IP stacks but it seems too crazy. I'm doing this as a hobby education project btw. It has been great fun so far! Wish there were meetups on this topic.

chatmasta 10y ago
Look up a paper called "packetshader." IIRC it was the first research to demonstrate packet processing via GPU.
mechagodzilla 10y ago
Generally speaking, FPGAs are for building hardware pipelines with fixed throughput and finite state machines to control things. If your first thought is "Linux", you're almost certainly better off with an off the shelf SoC. If you have a well understood problem (say, filtering packets based on a handful of patterns), you would design a filtering pipeline that would connect between two Ethernet MACs, applying the filtering at full line rate in between. Actually doing tcp/ip involves tracking a bunch of state and allocating very finite resources, but might not be necessary depending on your goal.
- trapperkeeper79 10y ago
  The "connecting two ethernet MACs [with] filtering" seems like an interesting project. I guess a "simple" (and possibly useless) idea could be to do filtering based on MAC addresses in the enthernet frame. Is there open IP for this? I guess I'm asking how a hobbyist gets into this :-) If anyone knows of an existing project, I'd appreciate a pointer.
  - pjc50 10y ago
    OpenCores is one possible source of examples. You could start by modifying e.g. http://opencores.org/project,mac_layer_switch http://opencores.org/project,mac_layer_switch
  - musicnarcoman 10y ago
    It is definitely possible, me and a few other students did it in a university course. Assuming the FPGA at least has a media access controller (translates bits to analogue signals) it is not to hard to encode/decode ethernet frames. Sadly, on the top of my head I do not know of any open IP. We wrote our own, but it is closed.
  - abjectcircle 10y ago
    I don't know about open IP, but it would be fairly straightforward but not trivial. Knowing verilog would be required, though, but it can be learned from a digital design book [1]. From there, you would probably want some experience in using the tools. For example, for Xilinx FPGAs, you could run through the Vivado tutorials. Then to design the IP you would design the state machines and logic to take data from one ethernet controller (for example, the Xilinx ethernet MAC [2]) and send it to another. The actual verilog would not be especially complex, however interfacing with the ethernet controllers and other peripherals would take some time. [1] Such as https://www.amazon.com/Logic-Design-Verification-SystemVerilog-Revised/dp/1523364025 https://www.amazon.com/Logic-Design-Verification-SystemVeril... [2] https://www.xilinx.com/support/documentation/ip_documentation/tri_mode_ethernet_mac/v8_2/pg051-tri-mode-eth-mac.pdf https://www.xilinx.com/support/documentation/ip_documentatio...
  - nickpsecurity 10y ago
    Add MACsec while you're at it to immunize you against attacks involving taps or a subset of compromised Ethernet NIC's. Well, a subset of it maybe as it got complicated like some other standards. https://www.juniper.net/documentation/en_US/junos15.1/topics/concept/macsec.html https://www.juniper.net/documentation/en_US/junos15.1/topics...
  - scott_wilson46 10y ago
    In actual fact for Xilinx based FPGA's this is quite straightforward. An example for 10G Base R: You can get Xilinx's component for their pma/pcs for 10g base-r ethernet for free from vivado and stick one of the macs from open cores on the end of it (probably this: http://opencores.org/project,xge_ll_mac http://opencores.org/project,xge_ll_mac - I used it for prototyping and it seems to work (before creating my own pcs/pma block and mac to cut down the latency) Once you have that, then you would need to deal with the ethernet frames streaming through the FPGA, probably 64-bits at a time at 156MHz for 10G, so you need to pull out the fields you are interested in (like mac addresses, ip addresses, etc). You can buffer the incoming packet into a FIFO whilst waiting for the stuff you want to filter on. Once you have all your fields you can decide whether you want to pass the packet through to the tx side or not (I usually read the packet out of the FIFO either way and just hold the valid low for packets I don't want to send). Hope this makes sense!
mechagodzilla 10y ago
Regarding PCIe - assuming you're using a hard IP MAC/PHY in your fpga, it's a packed-based protocol that basically only has three packets - read requests, write requests and completions. Requests have an address and a length, and target physical memory. Completions have the data and a matching tag from the read request. A dma transaction is just when the endpoint generates the request instead of the host CPU, but otherwise it's basically identical.
digler999 10y ago
Dont be fooled by the appearance of VHDL/Verilog. you are building hardware with them. So writing a TCP/IP stack in VHDL would be almost like physically building a TCP/IP stack with physical logic gate IC's. I looked into this a few years ago. If you can settle for just sending bytes on an ethernet cable, you can relatively easily TX/RX using UDP Multicast. The FPGA can beam bits straight to the PHY adapter and out they go.
- Koshkin 10y ago
  > you are building hardware with them Actually, not quite, not with FPGA. Sure, you can use an HDL to synthesize a custom microchip, in which case you indeed would be "building hardware", but you can also target an FPGA specifically, in which case you would essentially be programming a kind of a computer in a way similar to how it was done on those (early) switchboard-based computers that were configured by manually making wired connections between a fixed set of logical and arithmetical devices. You could say that one would be "building hardware" in such case as well, but since the system could be easily reconfigured at any time so that it could perform a different function, to me it looks more like programming... (I guess, the right way to look at this is as something where the distinction between hardware and software gets blurred to the point that any serious argument about the meaning of words becomes, well, meaningless.)
  - dom0 10y ago
    "encoding of control flow" (program, automaton, turing machines) vs. "encoding of data flow" (PAL configurations, CPLD, FPGA bitstreams, analog computer configuration)
  - RandomOpinion 10y ago
    The early computers (for the most part) were von Neumann architecture machines just like modern computers, with an instruction pointer, registers, branching, and so forth that most people associate with conventional software development. FPGAs are a undifferentiated sea of configurable logic gates with configurable connections between them[0]; it's not quite wiring together bare transistors but it's not that far removed either. None of the elements of a von Neumann architecture are there; if a engineer wants any of those things in their design, they would have to assemble it themselves from the available gates[1]. So, no, building a design for a FPGA has little to do with programming in the sense that most people mean it. [0] With various special function blocks interspersed at regular intervals depending on the manufacturer and model of the FPGA [1] or buy a pre-made soft-IP core
pjc50 10y ago
> run an RTOS or even full-fledged linux if your FPGA has hard IP cores on it Letting the OS touch more than a small percentage of the packets would be a gross waste of resources. If you want to write a program to process packets, use a real processor. (Putting an FPGA in a computer and then running a soft core on the FPGA is even sillier for performance.) The main reason to use an FPGA for this is to build structures that effectively encode your firewall or other layer processing rules in tables in the hardware, so the hardware can make a decision without having to consult the processor. MPLS and VLAN routing is the obvious case, but you can usually keep a short IP routing table in there too. Ideally you'd be able to make this decision early on while recieving a packet so you can start transmitting it while the last bits are still coming in. What you want to avoid is buffering ("bufferbloat").
- deleted 10y ago
  [deleted]
- wtallis 10y ago
  > What you want to avoid is buffering ("bufferbloat"). Excess buffering is a problem, but a little bit is fine unless you're trying to do high frequency trading. If you don't move packets to an external DRAM, you're in no danger of having bloated buffers with just the memory on the FPGA. Or you could implement an active queue management algorithm to keep your buffers from inducing too much unnecessary latency even when they are generously sized. There are a bunch of AQMs out there to pick from, of varying complexity. And it doesn't seem like there's near enough research into hardware implementations suitable for use in switches or network co-processors.
wmf 10y ago
The place to start is NetFPGA since they have open source Verilog, tutorials, and a community. http://www.cl.cam.ac.uk/research/srg/netos/projects/netfpga/workshop/technion-august-2015/ http://www.cl.cam.ac.uk/research/srg/netos/projects/netfpga/...

chatmasta 10y ago

Look up a paper called "packetshader." IIRC it was the first research to demonstrate packet processing via GPU.

mechagodzilla 10y ago

Generally speaking, FPGAs are for building hardware pipelines with fixed throughput and finite state machines to control things. If your first thought is "Linux", you're almost certainly better off with an off the shelf SoC. If you have a well understood problem (say, filtering packets based on a handful of patterns), you would design a filtering pipeline that would connect between two Ethernet MACs, applying the filtering at full line rate in between. Actually doing tcp/ip involves tracking a bunch of state and allocating very finite resources, but might not be necessary depending on your goal.

trapperkeeper79 10y ago
The "connecting two ethernet MACs [with] filtering" seems like an interesting project. I guess a "simple" (and possibly useless) idea could be to do filtering based on MAC addresses in the enthernet frame. Is there open IP for this? I guess I'm asking how a hobbyist gets into this :-) If anyone knows of an existing project, I'd appreciate a pointer.
- pjc50 10y ago
  OpenCores is one possible source of examples. You could start by modifying e.g. http://opencores.org/project,mac_layer_switch http://opencores.org/project,mac_layer_switch
- musicnarcoman 10y ago
  It is definitely possible, me and a few other students did it in a university course. Assuming the FPGA at least has a media access controller (translates bits to analogue signals) it is not to hard to encode/decode ethernet frames. Sadly, on the top of my head I do not know of any open IP. We wrote our own, but it is closed.
- abjectcircle 10y ago
  I don't know about open IP, but it would be fairly straightforward but not trivial. Knowing verilog would be required, though, but it can be learned from a digital design book [1]. From there, you would probably want some experience in using the tools. For example, for Xilinx FPGAs, you could run through the Vivado tutorials. Then to design the IP you would design the state machines and logic to take data from one ethernet controller (for example, the Xilinx ethernet MAC [2]) and send it to another. The actual verilog would not be especially complex, however interfacing with the ethernet controllers and other peripherals would take some time. [1] Such as https://www.amazon.com/Logic-Design-Verification-SystemVerilog-Revised/dp/1523364025 https://www.amazon.com/Logic-Design-Verification-SystemVeril... [2] https://www.xilinx.com/support/documentation/ip_documentation/tri_mode_ethernet_mac/v8_2/pg051-tri-mode-eth-mac.pdf https://www.xilinx.com/support/documentation/ip_documentatio...
- nickpsecurity 10y ago
  Add MACsec while you're at it to immunize you against attacks involving taps or a subset of compromised Ethernet NIC's. Well, a subset of it maybe as it got complicated like some other standards. https://www.juniper.net/documentation/en_US/junos15.1/topics/concept/macsec.html https://www.juniper.net/documentation/en_US/junos15.1/topics...
- scott_wilson46 10y ago
  In actual fact for Xilinx based FPGA's this is quite straightforward. An example for 10G Base R: You can get Xilinx's component for their pma/pcs for 10g base-r ethernet for free from vivado and stick one of the macs from open cores on the end of it (probably this: http://opencores.org/project,xge_ll_mac http://opencores.org/project,xge_ll_mac - I used it for prototyping and it seems to work (before creating my own pcs/pma block and mac to cut down the latency) Once you have that, then you would need to deal with the ethernet frames streaming through the FPGA, probably 64-bits at a time at 156MHz for 10G, so you need to pull out the fields you are interested in (like mac addresses, ip addresses, etc). You can buffer the incoming packet into a FIFO whilst waiting for the stuff you want to filter on. Once you have all your fields you can decide whether you want to pass the packet through to the tx side or not (I usually read the packet out of the FIFO either way and just hold the valid low for packets I don't want to send). Hope this makes sense!

mechagodzilla 10y ago

Regarding PCIe - assuming you're using a hard IP MAC/PHY in your fpga, it's a packed-based protocol that basically only has three packets - read requests, write requests and completions. Requests have an address and a length, and target physical memory. Completions have the data and a matching tag from the read request. A dma transaction is just when the endpoint generates the request instead of the host CPU, but otherwise it's basically identical.

digler999 10y ago

Dont be fooled by the appearance of VHDL/Verilog. you are building hardware with them. So writing a TCP/IP stack in VHDL would be almost like physically building a TCP/IP stack with physical logic gate IC's. I looked into this a few years ago. If you can settle for just sending bytes on an ethernet cable, you can relatively easily TX/RX using UDP Multicast. The FPGA can beam bits straight to the PHY adapter and out they go.

Koshkin 10y ago
> you are building hardware with them Actually, not quite, not with FPGA. Sure, you can use an HDL to synthesize a custom microchip, in which case you indeed would be "building hardware", but you can also target an FPGA specifically, in which case you would essentially be programming a kind of a computer in a way similar to how it was done on those (early) switchboard-based computers that were configured by manually making wired connections between a fixed set of logical and arithmetical devices. You could say that one would be "building hardware" in such case as well, but since the system could be easily reconfigured at any time so that it could perform a different function, to me it looks more like programming... (I guess, the right way to look at this is as something where the distinction between hardware and software gets blurred to the point that any serious argument about the meaning of words becomes, well, meaningless.)
- dom0 10y ago
  "encoding of control flow" (program, automaton, turing machines) vs. "encoding of data flow" (PAL configurations, CPLD, FPGA bitstreams, analog computer configuration)
- RandomOpinion 10y ago
  The early computers (for the most part) were von Neumann architecture machines just like modern computers, with an instruction pointer, registers, branching, and so forth that most people associate with conventional software development. FPGAs are a undifferentiated sea of configurable logic gates with configurable connections between them[0]; it's not quite wiring together bare transistors but it's not that far removed either. None of the elements of a von Neumann architecture are there; if a engineer wants any of those things in their design, they would have to assemble it themselves from the available gates[1]. So, no, building a design for a FPGA has little to do with programming in the sense that most people mean it. [0] With various special function blocks interspersed at regular intervals depending on the manufacturer and model of the FPGA [1] or buy a pre-made soft-IP core

pjc50 10y ago

> run an RTOS or even full-fledged linux if your FPGA has hard IP cores on it Letting the OS touch more than a small percentage of the packets would be a gross waste of resources. If you want to write a program to process packets, use a real processor. (Putting an FPGA in a computer and then running a soft core on the FPGA is even sillier for performance.) The main reason to use an FPGA for this is to build structures that effectively encode your firewall or other layer processing rules in tables in the hardware, so the hardware can make a decision without having to consult the processor. MPLS and VLAN routing is the obvious case, but you can usually keep a short IP routing table in there too. Ideally you'd be able to make this decision early on while recieving a packet so you can start transmitting it while the last bits are still coming in. What you want to avoid is buffering ("bufferbloat").

deleted 10y ago
[deleted]
wtallis 10y ago
> What you want to avoid is buffering ("bufferbloat"). Excess buffering is a problem, but a little bit is fine unless you're trying to do high frequency trading. If you don't move packets to an external DRAM, you're in no danger of having bloated buffers with just the memory on the FPGA. Or you could implement an active queue management algorithm to keep your buffers from inducing too much unnecessary latency even when they are generously sized. There are a bunch of AQMs out there to pick from, of varying complexity. And it doesn't seem like there's near enough research into hardware implementations suitable for use in switches or network co-processors.

wmf 10y ago

The place to start is NetFPGA since they have open source Verilog, tutorials, and a community. http://www.cl.cam.ac.uk/research/srg/netos/projects/netfpga/workshop/technion-august-2015/ http://www.cl.cam.ac.uk/research/srg/netos/projects/netfpga/...