2 ms·
There are two ways of doing clusters - one is a message passing paradigm, which you can do over Ethernet (to an extent - I’d still take USB3 for 4x the bandwidt
by qdot_me 7y ago
There are two ways of doing clusters - one is a message passing paradigm, which you can do over Ethernet (to an extent - I’d still take USB3 for 4x the bandwidth) - and the other is direct memory access a’la Cray.
What really motivated me to do this hack is the relative abundance of stuff I can now plug into an FPGA :)
- hvidgaard 7y agoI was thinking Ethernet because, A:It's cheap to buy a switch and cluster 100 RPi, B: You can have a desktop with a faster NIC and keep the RPis busy. But as with everything high performance, it depends entirely on the use case.
- qdot_me 7y agoTrue. And with RPi4 having a 1000baseT, it’s not as painful as it seems. Perhaps even the driver can be coaxed into some form of DMA and MPI that is a bit lower latency than IP stack. With secondary IP layer on 802.11, it might actually work reasonably well.
- nightcracker 7y agoThere's also InfiniBand which can reach much greater bandwidths.
- rowanG077 7y agoSo your plan is to attach an FPGA to the PCIe bus? To allow the FPGA to access peripherals on the Pi side or do you want the FPGA to make the pi a lot more powerful?
- morphle 7y agoBoth. The FPGA can interconnect the 16 RPi4s at 40 Gbps and also interconnect the 16 1G Ethernet at 16 Gbps, even interconnect the 16 HDMI, MPI and GPIO, depending on the FPGA. The FPGA can add 256 GB DDR3 and lots of other IO's like SATA and HDMI. (see my other comment for a $159 FPGA). The FPGA can act like a switch, an IO and memory extender and still have room for up to 300 ARM or Risc-V softcores.
- HeWhoLurksLate 7y agoWhat FPGA are you planning on using?
- morphle 7y agoSee my other posts on this page. You want FPGA's with cheap SERDES links that support PCIe Gen 2 at 5 Ghz. The best fit is Lattice ECP5-5G but that's $5 per link. The MPF300 is $10 per 12,5 Gbps link on the discounted developmentboard (with desoldering). A retail priced Cyclone 10CX105 also $10 per link with a smaller 10CX105 at $14.6. But very potent FPGAs that can be a small GPU in itself. I now plan a crowdfunding for our own FPGA in an ASIC, that would bring $0.25 with a hundred links. This HN pages shows me there will be enough buyers for a $300 364 core RPI4 compatible cluster (100 BCM2711 chips connected to 1 FPGA plus 100GB DDR4) but without the RPi4 boards. Instead of attaching RPi4 or BCM2711, you could have 100 SATA, SSD, 30 HDMI, 10G or a rack full of PCIe servers connected to this interconnect FPGA. You are welcome to help realise the project or the crowdfunding.
- anderspitman 7y agoI have no idea what I'd use this for but it sounds awesome.
- morphle 7y agoI suggest the RoarVM message passing Smalltalk Virtual Machine [1][2]. Erlang/OTP with actors would be my second choice. [1]Ungar, D., & Adams, S. S. (2009). Hosting an object heap on manycore hardware. https://sci-hub.tw/10.1145/1640134.1640149 https://sci-hub.tw/10.1145/1640134.1640149 [2]RoarVM demo https://www.youtube.com/watch?v=GeDFcC6ps88 https://www.youtube.com/watch?v=GeDFcC6ps88