4 ms·

Wasn't aware of HomelabOS, looks pretty interesting! How is your Longhorn performing? I tried setting it up on my nodes but with Gigabit networking and possibl

Wasn't aware of HomelabOS, looks pretty interesting! How is your Longhorn performing? I tried setting it up on my nodes but with Gigabit networking and possibly the Pis pretty average CPUs I would get pretty awful performance both in distributed volumes (with replicas etc) but also on strict-local ones (for reasons I haven't yet figured out). I am now considering using something like https://github.com/rancher/local-path-provisioner https://github.com/rancher/local-path-provisioner, since I mainly intend to use Longhorn for DBs that handle fault tolerance / backups etc on their own.

NickBusey 2y ago
I haven't gotten to performance testing just yet, that should happen pretty soon. Thanks for the local-path-provisioner recommendation, I'll definitely give it a go!
- s3rg4fts 2y ago
  Good luck! This is pretty useful for benchmarks https://github.com/longhorn/kbench?tab=readme-ov-file https://github.com/longhorn/kbench?tab=readme-ov-file
psini 2y ago
My experience with longhorn, or rook/ceph, or any other StorageClass solution that enables read write many is that eventually, despite all the "self-healing" and "highly available" claims, it all explodes at some point. Volumes become unmountable, pools stay yellow forever, desynchronization happens... Perhaps (most probably even) I just don't know how to configure them properly, but maybe that is telling of the difficulty to operate/maintain these solutions. What I took away from this is to try my most to never deploy anything that requires shared persistent volumes. If I need something stateful, it needs to speak to a database or S3 backend, or something that handles redundancy at some other level than filesystem. If I really really need to have a local volume, I'll use local-path-provisoner like you said, which means pinning the pod to a single node, but really that is a concession I am willing to make to not deal with ceph/rook. Great writeup, it's like I'm reading about my own journey managing kubernetes clusters! Best of luck

NickBusey 2y ago

I haven't gotten to performance testing just yet, that should happen pretty soon. Thanks for the local-path-provisioner recommendation, I'll definitely give it a go!

s3rg4fts 2y ago
Good luck! This is pretty useful for benchmarks https://github.com/longhorn/kbench?tab=readme-ov-file https://github.com/longhorn/kbench?tab=readme-ov-file

psini 2y ago

My experience with longhorn, or rook/ceph, or any other StorageClass solution that enables read write many is that eventually, despite all the "self-healing" and "highly available" claims, it all explodes at some point. Volumes become unmountable, pools stay yellow forever, desynchronization happens... Perhaps (most probably even) I just don't know how to configure them properly, but maybe that is telling of the difficulty to operate/maintain these solutions. What I took away from this is to try my most to never deploy anything that requires shared persistent volumes. If I need something stateful, it needs to speak to a database or S3 backend, or something that handles redundancy at some other level than filesystem. If I really really need to have a local volume, I'll use local-path-provisoner like you said, which means pinning the pod to a single node, but really that is a concession I am willing to make to not deal with ceph/rook. Great writeup, it's like I'm reading about my own journey managing kubernetes clusters! Best of luck