7 ms·

Author of Corundum here--if you have any questions, ask away.

deleted 7y ago
[deleted]
nullc 7y ago
Are you aware of the extraordinary good deal on huge kintex (K420T) nic-like dev boards on aliexpress? There are boards with 4 sfp+, ones with 2 sfp+ and 2 QSFP+, and even one with 4 QSFP28 (and UltraScale+ XCVU9P)... https://www.aliexpress.com/store/group/FPGA-DEV/620372_250309057.html?spm=2114.12010612.pcShopHead_35478622.1_1 https://www.aliexpress.com/store/group/FPGA-DEV/620372_25030... they sound like great targets for your work...
- alexforencich 7y ago
  Yes, I am aware of those. However, the kintex PCIe interface is a bit of a pain as it has a TLP straddling mode that can't be disabled, so it will be some time before it's supported as it will require some significant reworking in the PCIe interface modules. I am planning on supporting straddling eventually as this will improve PCIe link utilization on the ultrascale and ultrascale plus parts. If someone wants to donate a board, I can look in to supporting it.
  - hobo_mark 7y ago
    Interesting, I never heard of straddling, what is it supposed to achieve?
    - alexforencich 7y ago
      Stradding is an artifact of very wide interfaces. On the Ultrascale+ parts, the PCIe gen 3 x16 interface comes out as a 512 bit wide interface. Every cycle of the 250 MHz PCIe user clock transfers 64 bytes of data. The issue has to do with how packets are moved over this type of interface. If your packets are all a multiple of 64 bytes, no problem, you get 100% throughput. However, if your packets are NOT a multiple of 64 bytes in length, you have a problem. What byte lane do packets start and end in? The simplest implementation is to always start packets in byte lane 0. The interface logic for this is the simplest - the packets always start in the same place, so the fields always end up in the same place. However, if your packet is 65 bytes long, the utilization is horrible - it doesn't fit in one cycle, so you have to add an extra cycle for every packet, and bus utilization falls to 50% as you have 63 empty byte lanes after every packet. Straddling is an attempt to mitigate this issue. Instead of only staring packets in lane 0, the interface is adjusted to support starting packets in several places. Say, byte lanes 0 and 32. Or 0, 16, 32, and 48. Now, when you have a packet end in byte lane 0, you can start the next packet in the same clock cycle, but in byte lane 16 or 32. This increases the interface utilization. The trade-off is now the logic has to deal with parts of two packets in the same clock cycle, and it has to deal with multiple possible packet offsets. The specific annoyance with PCIe packets is that the max payload size is usually 256 bytes, but every packet has a 12 or 16 byte TLP header attached, which really screws things up when combined with the small max payload size.
      - hobo_mark 7y ago
        Fantastic explanation, thanks.
nullc 7y ago
No interest in 40GB phy? Right now, 40GB is the sweet spot in lower cost surplus hardware: E.g. you can get Arista DCS-7050QX-32 for about $500 shipped on ebay all day long. 100GB/25GB switches are still really expensive.
- alexforencich 7y ago
  No free 40G MAC/PHY. Unfortunately, the Xilinx CMAC is 100G only, and the Xilinx soft 40G MAC/PHY is $$$$. I have looked in to building a 40G/100G switchable MAC/PHY, but it's going to be a serious pain in the rear. Funny you mention that switch, we bought one of those off of eBay for our testbed as it supports PTP. Also, for optical switching applications, one of the most important factors is how long it takes to bring up the link after switching. Because of this, we have no interest in spending time on 40G and 100G interfaces because interlace deskew takes hundreds of microseconds, and 100G also requires FEC which takes hundreds of microseconds to lock. So we're focused on 10G and 25G and running multiple links in parallel, which also provides more architectural flexibility. I added 100G support for three main reasons: the CMAC license is free, so why not?; supporting 100G makes the project a whole lot more interesting than only 10G or 25G, and it provides a simple way of testing the core NIC datapath.
  - nullc 7y ago
    oh... now I understand the purpose of the TDMA. Using actual optical switching to interconnect. Very interesting! Got any pointers to the sort of optical switching components you're using? [I've been out of the networking business professionally for almost a decade now, so I'm a bit out of touch with the state of the art in optical stuff--- I was somewhat surprised recently to learn of the existence and low cost of LR4 40gb optics. :P]
    - alexforencich 7y ago
      That's part of the research we're doing! Take a look at: https://circuit-switching.sysnet.ucsd.edu/ https://circuit-switching.sysnet.ucsd.edu/ And: https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENLITENED_Annual_Meeting.pdf https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENL... The current generation of switches that we're working on uses diffraction gratings patterned onto glass hard drive platters, installed in a modified hard drive, spun by a custom motor controller that's synchronized to the NICs via PTP.
rjsw 7y ago
Did you do a patent search ?
- alexforencich 7y ago
  We have not done anything like that. Presumably that would be an important thing to do if a commercial product is produced at some point.

deleted 7y ago

[deleted]

Are you aware of the extraordinary good deal on huge kintex (K420T) nic-like dev boards on aliexpress? There are boards with 4 sfp+, ones with 2 sfp+ and 2 QSFP+, and even one with 4 QSFP28 (and UltraScale+ XCVU9P)... https://www.aliexpress.com/store/group/FPGA-DEV/620372_250309057.html?spm=2114.12010612.pcShopHead_35478622.1_1 https://www.aliexpress.com/store/group/FPGA-DEV/620372_25030... they sound like great targets for your work...

alexforencich 7y ago
Yes, I am aware of those. However, the kintex PCIe interface is a bit of a pain as it has a TLP straddling mode that can't be disabled, so it will be some time before it's supported as it will require some significant reworking in the PCIe interface modules. I am planning on supporting straddling eventually as this will improve PCIe link utilization on the ultrascale and ultrascale plus parts. If someone wants to donate a board, I can look in to supporting it.
- hobo_mark 7y ago
  Interesting, I never heard of straddling, what is it supposed to achieve?
  - alexforencich 7y ago
    Stradding is an artifact of very wide interfaces. On the Ultrascale+ parts, the PCIe gen 3 x16 interface comes out as a 512 bit wide interface. Every cycle of the 250 MHz PCIe user clock transfers 64 bytes of data. The issue has to do with how packets are moved over this type of interface. If your packets are all a multiple of 64 bytes, no problem, you get 100% throughput. However, if your packets are NOT a multiple of 64 bytes in length, you have a problem. What byte lane do packets start and end in? The simplest implementation is to always start packets in byte lane 0. The interface logic for this is the simplest - the packets always start in the same place, so the fields always end up in the same place. However, if your packet is 65 bytes long, the utilization is horrible - it doesn't fit in one cycle, so you have to add an extra cycle for every packet, and bus utilization falls to 50% as you have 63 empty byte lanes after every packet. Straddling is an attempt to mitigate this issue. Instead of only staring packets in lane 0, the interface is adjusted to support starting packets in several places. Say, byte lanes 0 and 32. Or 0, 16, 32, and 48. Now, when you have a packet end in byte lane 0, you can start the next packet in the same clock cycle, but in byte lane 16 or 32. This increases the interface utilization. The trade-off is now the logic has to deal with parts of two packets in the same clock cycle, and it has to deal with multiple possible packet offsets. The specific annoyance with PCIe packets is that the max payload size is usually 256 bytes, but every packet has a 12 or 16 byte TLP header attached, which really screws things up when combined with the small max payload size.
    - hobo_mark 7y ago
      Fantastic explanation, thanks.

nullc 7y ago

No interest in 40GB phy? Right now, 40GB is the sweet spot in lower cost surplus hardware: E.g. you can get Arista DCS-7050QX-32 for about $500 shipped on ebay all day long. 100GB/25GB switches are still really expensive.

alexforencich 7y ago
No free 40G MAC/PHY. Unfortunately, the Xilinx CMAC is 100G only, and the Xilinx soft 40G MAC/PHY is $$$$. I have looked in to building a 40G/100G switchable MAC/PHY, but it's going to be a serious pain in the rear. Funny you mention that switch, we bought one of those off of eBay for our testbed as it supports PTP. Also, for optical switching applications, one of the most important factors is how long it takes to bring up the link after switching. Because of this, we have no interest in spending time on 40G and 100G interfaces because interlace deskew takes hundreds of microseconds, and 100G also requires FEC which takes hundreds of microseconds to lock. So we're focused on 10G and 25G and running multiple links in parallel, which also provides more architectural flexibility. I added 100G support for three main reasons: the CMAC license is free, so why not?; supporting 100G makes the project a whole lot more interesting than only 10G or 25G, and it provides a simple way of testing the core NIC datapath.
- nullc 7y ago
  oh... now I understand the purpose of the TDMA. Using actual optical switching to interconnect. Very interesting! Got any pointers to the sort of optical switching components you're using? [I've been out of the networking business professionally for almost a decade now, so I'm a bit out of touch with the state of the art in optical stuff--- I was somewhat surprised recently to learn of the existence and low cost of LR4 40gb optics. :P]
  - alexforencich 7y ago
    That's part of the research we're doing! Take a look at: https://circuit-switching.sysnet.ucsd.edu/ https://circuit-switching.sysnet.ucsd.edu/ And: https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENLITENED_Annual_Meeting.pdf https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENL... The current generation of switches that we're working on uses diffraction gratings patterned onto glass hard drive platters, installed in a modified hard drive, spun by a custom motor controller that's synchronized to the NICs via PTP.

rjsw 7y ago

Did you do a patent search ?

alexforencich 7y ago
We have not done anything like that. Presumably that would be an important thing to do if a commercial product is produced at some point.