Interesting, I don't know about Ceph, and now I must investigate this! Before I built my current storage solution, I had been pondering something like this too

Interesting, I don't know about Ceph, and now I must investigate this! Before I built my current storage solution, I had been pondering something like this too, there's a certain allure to not have that "one box with all the drives", and be able to just expand forever by plugging more nodes on the network. I was thinking zfs over nbd, and ultimately decided against it, both because of the complexity, and becaure zfs wants to have ECC memory which the pi does not have (and for some reason I don't trust external enclosures to be as reliable as the PSU/Controller combo in a normal server).