4 ms·

Looking at the specs, it seems almost dishonest to promote this for kubernetes. > The nodes interconnected with the onboard 1 Gbps switch. However, each node i

Looking at the specs, it seems almost dishonest to promote this for kubernetes. > The nodes interconnected with the onboard 1 Gbps switch. However, each node is limited with 100 Mbps USB speed. Not only that but Compute Module 3+ are limited to 1GB RAM, is it really expected someone could run a realistic workload? How stable is the control plane node with such limited resources? It seems like picking up 3 raspi4s (4GB RAM each) and powering via PoE would be a must better result.

afterwalk 7y ago
I randomly came across a similar offering and thought about buying for fun: https://www.mininodes.com/product/5-node-raspberry-pi-3-com-carrier-board/ https://www.mininodes.com/product/5-node-raspberry-pi-3-com-... Any idea how the specs compare?
- mbreese 7y ago
  This is still based on the compute modules, so you're going to have the same tradeoffs. These are interesting ideas, but until the compute modules start having more on board RAM, you'd be much better off working with a few RPi 4's. They will be faster (1Gbps ethernet, more RAM), and cheaper since you only need a gigabit switch to connect them together, not a custom carrier board. If the compute modules start to get more powerful, then having to deal with only one power supply and ethernet uplink would be nice. It's a very appealing idea. But, you're going to almost always have a better experience with a bunch of standard RPis.
  - jaxn 7y ago
    This is what I did. Throw PoE hats on the pis and with a PoE switch it's a real clean setup. 3D printed trays for rockmount fiber cassette blanks and set it in a network rack.
detaro 7y ago
If you insist on using Pis (IMHO you're quickly at the point where a (potentially used) NUC or small office PC is the better choice), why would you go for extra expense and effort for PoE? Just connect a 5V PSU to the power pins? I guess it gives you individual power switching without any DIY, but other than that?
- moondev 7y ago
  Simply for not having to deal with all the usb-c adaptors and cables. It's also pretty cool being able to turn the pis on or off by cycling the switch port from your network control plane (Unifi in my case) You could even do it dynamically via the api for a "bare-metal autoscaling" type workflow.
  - detaro 7y ago
    > Simply for not having to deal with all the usb-c adaptors and cables. You can get one large 5V PSU and go directly to the pins on the Pi. No need to bother with individual USB adaptors...
    - deleted 7y ago
      [deleted]
wedn3sday 7y ago
I agree, using the USB bus for inter node communication seems like a poor design choice. Anyone got any insight on why they wouldnt use the much faster ethernet connection?
- cconstantine 7y ago
  My understanding is that the ethernet module on the raspberry pi is a usb device, not a pci device.
- oakwhiz 7y ago
  The SoC used in the Pi 3 does not have an Ethernet MAC. It's always provided via USB. The CM3 does not have a USB Ethernet device onboard, it's provided by the carrier board.
Floegipoky 7y ago
I built a cluster of 6 pi 3b+ duct taped to a USB hub for power and a switch and router a few years ago for the explicit purpose of experimenting with clustering technologies, including kubernetes. I wasn't running it for long periods of time but it was surprisingly stable. But to your point, 1GB RAM doesn't get you very far with many of the popular distributed systems these days.
kylek 7y ago
I'd hope people wouldn't be using a setup like this for any other reason than learning k8s or tinkering/fun. edit-before-actually-posting (sorry I'm a bad person for typing a reply before clicking the link): Wait, they're selling these things? Ok, then I'm stumped. What would you do with 8 rpi's that you couldn't do with one?
gclawes 7y ago
I'm holding out to see if a CM 4 with 4GB ram comes out before I commit to something like this.

afterwalk 7y ago

I randomly came across a similar offering and thought about buying for fun: https://www.mininodes.com/product/5-node-raspberry-pi-3-com-carrier-board/ https://www.mininodes.com/product/5-node-raspberry-pi-3-com-... Any idea how the specs compare?

mbreese 7y ago
This is still based on the compute modules, so you're going to have the same tradeoffs. These are interesting ideas, but until the compute modules start having more on board RAM, you'd be much better off working with a few RPi 4's. They will be faster (1Gbps ethernet, more RAM), and cheaper since you only need a gigabit switch to connect them together, not a custom carrier board. If the compute modules start to get more powerful, then having to deal with only one power supply and ethernet uplink would be nice. It's a very appealing idea. But, you're going to almost always have a better experience with a bunch of standard RPis.
- jaxn 7y ago
  This is what I did. Throw PoE hats on the pis and with a PoE switch it's a real clean setup. 3D printed trays for rockmount fiber cassette blanks and set it in a network rack.

detaro 7y ago

If you insist on using Pis (IMHO you're quickly at the point where a (potentially used) NUC or small office PC is the better choice), why would you go for extra expense and effort for PoE? Just connect a 5V PSU to the power pins? I guess it gives you individual power switching without any DIY, but other than that?

moondev 7y ago
Simply for not having to deal with all the usb-c adaptors and cables. It's also pretty cool being able to turn the pis on or off by cycling the switch port from your network control plane (Unifi in my case) You could even do it dynamically via the api for a "bare-metal autoscaling" type workflow.
- detaro 7y ago
  > Simply for not having to deal with all the usb-c adaptors and cables. You can get one large 5V PSU and go directly to the pins on the Pi. No need to bother with individual USB adaptors...
  - deleted 7y ago
    [deleted]

wedn3sday 7y ago

I agree, using the USB bus for inter node communication seems like a poor design choice. Anyone got any insight on why they wouldnt use the much faster ethernet connection?

cconstantine 7y ago
My understanding is that the ethernet module on the raspberry pi is a usb device, not a pci device.
oakwhiz 7y ago
The SoC used in the Pi 3 does not have an Ethernet MAC. It's always provided via USB. The CM3 does not have a USB Ethernet device onboard, it's provided by the carrier board.

Floegipoky 7y ago

I built a cluster of 6 pi 3b+ duct taped to a USB hub for power and a switch and router a few years ago for the explicit purpose of experimenting with clustering technologies, including kubernetes. I wasn't running it for long periods of time but it was surprisingly stable. But to your point, 1GB RAM doesn't get you very far with many of the popular distributed systems these days.

kylek 7y ago

I'd hope people wouldn't be using a setup like this for any other reason than learning k8s or tinkering/fun. edit-before-actually-posting (sorry I'm a bad person for typing a reply before clicking the link): Wait, they're selling these things? Ok, then I'm stumped. What would you do with 8 rpi's that you couldn't do with one?

gclawes 7y ago

I'm holding out to see if a CM 4 with 4GB ram comes out before I commit to something like this.