5 ms·

Raspberry Pi based Ceph Storage Cluster with 15 nodes and 55 TiB raw capacity

40 points by louwrentius 6y ago

ktpsns 6y ago
Nice, so this setup requires 18 raspberries for 30 USB-3 hard disks, 2TB each. I know this is about hacking and fun, but I wonder if there's no more better, certainly more traditional way of attaching these external disks to some powerful computer. What about a single server and attaching them with USB-3? Power consumption will be comparable with idling processors, but I assume the overall performance will be better with a traditional x86 desktop or server processor.
- Scandiravian 6y ago
  I've considered such a setup myself and decided that I also wanted redundancy at the hardware/node level. If my storage server goes down, then everything is down until it's fixed. I think the person who made this setup had the same considerations
Scandiravian 6y ago
It's an interesting approach and I think it might be reasonable in terms of power. I also really like the amount of redundancy in the setup, but the cost is probably too steep for my preferences An RPI 4 8gb is approximately 90$, so 15 nodes becomes 1350$ I think there's room for a board with a reasonable sized (16gb-32gb) emmc, 2-4 sata ports (with 4-8gb ram per sata connection), and 2.5/10gbe for smaller ceph clusters
- tutfbhuf 6y ago
  You could buy drives with higher capacity, then you need fewer RPI, fewer cables, consume less space/energy and still achieve 55 TiB capacity.
  - Scandiravian 6y ago
    Ceph recommends 1gb ram per tb of storage. Going lower gives massive hits to performance and the 1gbe ethernet connection becomes a bottleneck with fewer nodes, so I don't think it's viable to scale this way
    - tutfbhuf 6y ago
      True, the Pi 4 with 8GB RAM would allow for 8 TB max recommended storage per node. Thus, requires you to have 7 Pis for 56 TB. I think that's still a good setup, even with 1GE. btw, 10GE is the minimum requirement for a non-hobby ceph cluster
dusted 6y ago
Interesting, I don't know about Ceph, and now I must investigate this! Before I built my current storage solution, I had been pondering something like this too, there's a certain allure to not have that "one box with all the drives", and be able to just expand forever by plugging more nodes on the network. I was thinking zfs over nbd, and ultimately decided against it, both because of the complexity, and becaure zfs wants to have ECC memory which the pi does not have (and for some reason I don't trust external enclosures to be as reliable as the PSU/Controller combo in a normal server).

40 points by louwrentius 6y ago

Nice, so this setup requires 18 raspberries for 30 USB-3 hard disks, 2TB each. I know this is about hacking and fun, but I wonder if there's no more better, certainly more traditional way of attaching these external disks to some powerful computer. What about a single server and attaching them with USB-3? Power consumption will be comparable with idling processors, but I assume the overall performance will be better with a traditional x86 desktop or server processor.

Scandiravian 6y ago
I've considered such a setup myself and decided that I also wanted redundancy at the hardware/node level. If my storage server goes down, then everything is down until it's fixed. I think the person who made this setup had the same considerations

Scandiravian 6y ago

It's an interesting approach and I think it might be reasonable in terms of power. I also really like the amount of redundancy in the setup, but the cost is probably too steep for my preferences An RPI 4 8gb is approximately 90$, so 15 nodes becomes 1350$ I think there's room for a board with a reasonable sized (16gb-32gb) emmc, 2-4 sata ports (with 4-8gb ram per sata connection), and 2.5/10gbe for smaller ceph clusters

tutfbhuf 6y ago
You could buy drives with higher capacity, then you need fewer RPI, fewer cables, consume less space/energy and still achieve 55 TiB capacity.
- Scandiravian 6y ago
  Ceph recommends 1gb ram per tb of storage. Going lower gives massive hits to performance and the 1gbe ethernet connection becomes a bottleneck with fewer nodes, so I don't think it's viable to scale this way
  - tutfbhuf 6y ago
    True, the Pi 4 with 8GB RAM would allow for 8 TB max recommended storage per node. Thus, requires you to have 7 Pis for 56 TB. I think that's still a good setup, even with 1GE. btw, 10GE is the minimum requirement for a non-hobby ceph cluster

dusted 6y ago

Interesting, I don't know about Ceph, and now I must investigate this! Before I built my current storage solution, I had been pondering something like this too, there's a certain allure to not have that "one box with all the drives", and be able to just expand forever by plugging more nodes on the network. I was thinking zfs over nbd, and ultimately decided against it, both because of the complexity, and becaure zfs wants to have ECC memory which the pi does not have (and for some reason I don't trust external enclosures to be as reliable as the PSU/Controller combo in a normal server).