4 ms·

I've been trying to switch my home cluster from Debian + K3s to Talos but keep running into issues. What does your persistent storage layer look like on Talos?

I've been trying to switch my home cluster from Debian + K3s to Talos but keep running into issues. What does your persistent storage layer look like on Talos? How have you found it's hardware stability over the long term?

esseph 1y ago
Talos is the Linux kernel at heart, so.. just fine.
redrove 1y ago
>What does your persistent storage layer look like on Talos? Well, for its own storage: it's an immutable OS that you can configure via a single YAML file, it automatically provisions appropriate partitions for you, or you can even install the ZFS extension and have it use ZFS (no zfs on root though). For application/data storage there's a myriad of options to choose from[0]; after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs and I've been using it for many years without any issues. If you don't have 10gig networking you can even do iSCSI from another host (or nvmeof via democratic-csi but that's quite esoteric). >How have you found it's hardware stability over the long term? It's Linux so pretty good! No complaints and everything just works. If something is down it's always me misconfiguring or a hardware failure. [0] https://www.talos.dev/v1.11/kubernetes-guides/configuration/storage/#storage-clusters https://www.talos.dev/v1.11/kubernetes-guides/configuration/...
- avtar 1y ago
  > after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs Curious to know what issues you ran into with Longhorn.
  - redrove 1y ago
    It was years ago but I recall high CPU usage being an issue in particular. In general it's just not as battle tested as ceph and I needed something more bulletproof. However I will say this: I'm sure that issue with the CPU usage was fixed (I was watching the GitHub issue) and you might not need your distributed FS to be CERN ready for your lab; AND the UI and built-in backups Longhorn offers are great for beginners so I'd suggest giving it a try if you don't already know you want ceph or OpenEBS Mayastor for the performance and so on.
    - avtar 1y ago
      Thanks! I haven’t had to implement replicated storage (still using EFS) but I was curious about Longhorn.

esseph 1y ago

Talos is the Linux kernel at heart, so.. just fine.

redrove 1y ago

>What does your persistent storage layer look like on Talos? Well, for its own storage: it's an immutable OS that you can configure via a single YAML file, it automatically provisions appropriate partitions for you, or you can even install the ZFS extension and have it use ZFS (no zfs on root though). For application/data storage there's a myriad of options to choose from[0]; after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs and I've been using it for many years without any issues. If you don't have 10gig networking you can even do iSCSI from another host (or nvmeof via democratic-csi but that's quite esoteric). >How have you found it's hardware stability over the long term? It's Linux so pretty good! No complaints and everything just works. If something is down it's always me misconfiguring or a hardware failure. [0] https://www.talos.dev/v1.11/kubernetes-guides/configuration/storage/#storage-clusters https://www.talos.dev/v1.11/kubernetes-guides/configuration/...

avtar 1y ago
> after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs Curious to know what issues you ran into with Longhorn.
- redrove 1y ago
  It was years ago but I recall high CPU usage being an issue in particular. In general it's just not as battle tested as ceph and I needed something more bulletproof. However I will say this: I'm sure that issue with the CPU usage was fixed (I was watching the GitHub issue) and you might not need your distributed FS to be CERN ready for your lab; AND the UI and built-in backups Longhorn offers are great for beginners so I'd suggest giving it a try if you don't already know you want ceph or OpenEBS Mayastor for the performance and so on.
  - avtar 1y ago
    Thanks! I haven’t had to implement replicated storage (still using EFS) but I was curious about Longhorn.