4 ms·

I was recently wondering if there's any kind of single board computer that would be suitable for just shoveling data on the network at close to 10gbps. If you h

I was recently wondering if there's any kind of single board computer that would be suitable for just shoveling data on the network at close to 10gbps. If you have something that basically just sendfile()s data, an average desktop PC from 5 years won't pull a sweat saturating 10gbits, barely reaching two digit CPU%. Are there are any SBCs that would have suitable IO for this? The pi4 has USB 3 which internally is connected via PCIe, but the question is whether it could handle eg SATA and Ethernet on it. Not that I actually need this for anything, it would just be cool to have a small case with 1TB SSD that you can plug into a switch and deliver hundreds of MB/s. With 8GB RAM you could even make good use of the fs cache, depending on usage pattern.

jrockway 6y ago
My experience is that these small computers don't even do well at 1Gbps. When I worked on Google Fiber, we really wanted to implement a speed test between our routers and some trusted server, but the chipset we used couldn't generate data fast enough. (We could route at 1Gbps because it took a hardware fast path, but the software-only path wasn't good enough to provide useful speed tests.) I have subsequently tested RPi4s at another ISP, and again, we couldn't get a gigabit out of them. In fact, I have never gotten 10Gbps out of any software-only path. Routers can do it, but it's hardware accelerated and won't generate packets at that speed. I did recently get 7Gbps out of normal Internet usage, though, with a consumer-grade 10Gbps NIC on an Intel system. Pretty decent, but not 10Gbps. Much optimization is necessary to make 10Gbps a thing. It's not a focus for $35 single-board computers. Those things are just obsolete phone chipsets in a desktop form factor; don't expect much out of them.
- iforgotpassword 6y ago
  But it should be about the buses these boards expose shouldn't it? Even the pi4 still has Ethernet over USB which is unnecessary overhead. Getting access to the PCIe lanes directly would be interesting for this. I just researched a bit and it looks like there are boards with SATA port from hardkernel that completely saturate the 1gbit port, so the question is how much faster they could go, and whether it uses a USB bridge too. In general, saturating 10gbit on any x86 from the past couple years isn't a problem. Doing it with a single TCP connection is a bit tricky and requires some tweaking, and jumbo frames help too, but a simple threaded server pushing out data via sendfile to multiple clients easily gets the job done. But yes, I agree that this is not a useful use case for SBCs, one of these things I'd just want to build because I can, not because I should.
- subhobroto 6y ago
  > Routers can do it, but it's hardware accelerated and won't generate packets at that speed Yeah, Are there any SBC with an off-CPU networking fibre/module? What I am building are LTE enabled ethernet bridges. The closest fits that I have found are router boards but most of them don't come with mini PCIe ports that I can connect LTE modems to. There are a few proprietary router boards but I was thinking there could be other options - you would likely know!
discordance 6y ago
I boot my raspi4 off a USB SATA SSD. Runs much faster than the SD Card. Follow these steps: https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb
subhobroto 6y ago
> single board computer that would be suitable for just shoveling data on the network at close to 10gbps I am also interested so here's what I have found: We want a SBC with an off-CPU networking module. You don't want to tie up the CPU shuffling data bits for networking - there are now cheap and dedicated modules for that. Most of what I have found are 1gbps Broadcom modules. My issue is that the closest fits that I have found are router boards. Please reach out if you can, would love to solve this myself.
detaro 6y ago
https://www.solid-run.com/marvell-armada-family/macchiatobin/ https://www.solid-run.com/marvell-armada-family/macchiatobin... (and a few other solid-run offerings)

jrockway 6y ago

My experience is that these small computers don't even do well at 1Gbps. When I worked on Google Fiber, we really wanted to implement a speed test between our routers and some trusted server, but the chipset we used couldn't generate data fast enough. (We could route at 1Gbps because it took a hardware fast path, but the software-only path wasn't good enough to provide useful speed tests.) I have subsequently tested RPi4s at another ISP, and again, we couldn't get a gigabit out of them. In fact, I have never gotten 10Gbps out of any software-only path. Routers can do it, but it's hardware accelerated and won't generate packets at that speed. I did recently get 7Gbps out of normal Internet usage, though, with a consumer-grade 10Gbps NIC on an Intel system. Pretty decent, but not 10Gbps. Much optimization is necessary to make 10Gbps a thing. It's not a focus for $35 single-board computers. Those things are just obsolete phone chipsets in a desktop form factor; don't expect much out of them.

iforgotpassword 6y ago
But it should be about the buses these boards expose shouldn't it? Even the pi4 still has Ethernet over USB which is unnecessary overhead. Getting access to the PCIe lanes directly would be interesting for this. I just researched a bit and it looks like there are boards with SATA port from hardkernel that completely saturate the 1gbit port, so the question is how much faster they could go, and whether it uses a USB bridge too. In general, saturating 10gbit on any x86 from the past couple years isn't a problem. Doing it with a single TCP connection is a bit tricky and requires some tweaking, and jumbo frames help too, but a simple threaded server pushing out data via sendfile to multiple clients easily gets the job done. But yes, I agree that this is not a useful use case for SBCs, one of these things I'd just want to build because I can, not because I should.
subhobroto 6y ago
> Routers can do it, but it's hardware accelerated and won't generate packets at that speed Yeah, Are there any SBC with an off-CPU networking fibre/module? What I am building are LTE enabled ethernet bridges. The closest fits that I have found are router boards but most of them don't come with mini PCIe ports that I can connect LTE modems to. There are a few proprietary router boards but I was thinking there could be other options - you would likely know!

discordance 6y ago

I boot my raspi4 off a USB SATA SSD. Runs much faster than the SD Card. Follow these steps: https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb

subhobroto 6y ago

> single board computer that would be suitable for just shoveling data on the network at close to 10gbps I am also interested so here's what I have found: We want a SBC with an off-CPU networking module. You don't want to tie up the CPU shuffling data bits for networking - there are now cheap and dedicated modules for that. Most of what I have found are 1gbps Broadcom modules. My issue is that the closest fits that I have found are router boards. Please reach out if you can, would love to solve this myself.

detaro 6y ago

https://www.solid-run.com/marvell-armada-family/macchiatobin/ https://www.solid-run.com/marvell-armada-family/macchiatobin... (and a few other solid-run offerings)