17 ms·

A big problem is local storage provisioner for kubernetes is still not GA. I want to use an SSD attached to one master, and use that as storage for the other RP

A big problem is local storage provisioner for kubernetes is still not GA. I want to use an SSD attached to one master, and use that as storage for the other RPIs. Doing this is still undocumented/the Wild West.

mosselman 7y ago
I am not a kubernetes expert so I have no idea. Sounds like a fun challenge. I remember seeing something about network mounts for linux in general. Maybe go around kubernetes? If you solve it I'd be interested in how you did it, so an update would be greatly appreciated.
- zimmertr 7y ago
  Kubernetes supports pretty much any type of storage through API Extensions so network mounts are supported. People often use the NFS-Client & NFS-Provisioner projects for this.
vetinari 7y ago
Disclaimer: k8s beginner here. As far as I understand, local storage provisioner is for the node local storage; the storage doesn't follow the workload to whatever node is the pod scheduled, but schedules the pod to the node containing the storage device. It doesn't allow for pods to access local storage outside the node. So for worker nodes using storage on the master node, isn't it better to use either iSCSI or NFS?
- escardin 7y ago
  This is exactly right. Local storage is conceptually the persistent volume equivalent to an emptydir volume mount. If you want to have a 'storage node' in a simple way, the NFS storage provider is the way to go. You install the nfs client libs on each node, setup an NFS share and configure and run the provisioner[1]. My experience with iSCSI is to stay the heck away from it. It is not what you want. iSCSI is really meant for people who already have iSCSI SANs and not people who have a disk they want to share. The more I learned about it, the more I learned that I should have picked something else for every use. It's not that it's bad, it's that it solves a much different problem than I expected given the networked nature of it. [1] https://github.com/kubernetes-incubator/external-storage/tree/master/nfs https://github.com/kubernetes-incubator/external-storage/tre... (I think this is the right one, been a while).
  - zimmertr 7y ago
    NFS-Provisioner and NFS-Client have been around for a while. They're about to be promoted from Incubator actually.
    - yebyen 7y ago
      That is fantastic, I remember when they were new ideas without any implementation but just a few people trying some things on GitHub issues. https://engineering.bitnami.com/articles/scaling-wordpress-in-kubernetes.html https://engineering.bitnami.com/articles/scaling-wordpress-i... This article does the NFS approach justice I think, I was pleased to find it has been a working strategy for a while!
zimmertr 7y ago
Have you used OpenEBS?
yebyen 7y ago
Local Persistent Volumes though has been GA since 1.14 and I think entered beta in 1.10 or 1.12, it should be usable now I was just about to try to start using this, have you seen this article: https://vocon-it.com/2018/12/20/kubernetes-local-persistent-volumes/ https://vocon-it.com/2018/12/20/kubernetes-local-persistent-... The strategy seems to be to create a storage class per app and make sure each persistent volume claim binds a distinct storage class. It sounds like a lot of heavy lifting but it's just a few YAML files... the alternative seems to be https://github.com/rancher/local-path-provisioner https://github.com/rancher/local-path-provisioner which uses the same strategy under the hood to fulfill PVCs as they come online using the Local Persistent Volume strategy, but does not require the storageclass arrangement I understood from the tutorial linked before. If you want to use an SSD attached to only one special/master node, I don't think that's the wild wild west, I think that's actually closer to what you might see in a traditional network storage architecture, possible to use something like Portworx or Rook to make that work if they are supported on RPi. That's not the same as local storage provisioner though, If you know more about it than I do I'm happy to hear more about where you got stuck, since I'll be trying to implement this strategy for myself on a non-RPi cluster with a stable set of nodes soon (stable as in, pets not cattle).
boon 7y ago
Another frustration with the RPi cluster is the arm requirement. If you're only building/running your own stuff, that's fine, but finding containers and helm charts that support arm can be frustratingly difficult. This is speaking from experience. I love my k3s RPi cluster, it's fantastic once you get things working. I just had to augment with some x86 nodes in order to _also_ run some software that just wouldn't run anywhere else.

mosselman 7y ago

I am not a kubernetes expert so I have no idea. Sounds like a fun challenge. I remember seeing something about network mounts for linux in general. Maybe go around kubernetes? If you solve it I'd be interested in how you did it, so an update would be greatly appreciated.

zimmertr 7y ago
Kubernetes supports pretty much any type of storage through API Extensions so network mounts are supported. People often use the NFS-Client & NFS-Provisioner projects for this.

vetinari 7y ago

Disclaimer: k8s beginner here. As far as I understand, local storage provisioner is for the node local storage; the storage doesn't follow the workload to whatever node is the pod scheduled, but schedules the pod to the node containing the storage device. It doesn't allow for pods to access local storage outside the node. So for worker nodes using storage on the master node, isn't it better to use either iSCSI or NFS?

escardin 7y ago
This is exactly right. Local storage is conceptually the persistent volume equivalent to an emptydir volume mount. If you want to have a 'storage node' in a simple way, the NFS storage provider is the way to go. You install the nfs client libs on each node, setup an NFS share and configure and run the provisioner[1]. My experience with iSCSI is to stay the heck away from it. It is not what you want. iSCSI is really meant for people who already have iSCSI SANs and not people who have a disk they want to share. The more I learned about it, the more I learned that I should have picked something else for every use. It's not that it's bad, it's that it solves a much different problem than I expected given the networked nature of it. [1] https://github.com/kubernetes-incubator/external-storage/tree/master/nfs https://github.com/kubernetes-incubator/external-storage/tre... (I think this is the right one, been a while).
- zimmertr 7y ago
  NFS-Provisioner and NFS-Client have been around for a while. They're about to be promoted from Incubator actually.
  - yebyen 7y ago
    That is fantastic, I remember when they were new ideas without any implementation but just a few people trying some things on GitHub issues. https://engineering.bitnami.com/articles/scaling-wordpress-in-kubernetes.html https://engineering.bitnami.com/articles/scaling-wordpress-i... This article does the NFS approach justice I think, I was pleased to find it has been a working strategy for a while!

zimmertr 7y ago

Have you used OpenEBS?

yebyen 7y ago

Local Persistent Volumes though has been GA since 1.14 and I think entered beta in 1.10 or 1.12, it should be usable now I was just about to try to start using this, have you seen this article: https://vocon-it.com/2018/12/20/kubernetes-local-persistent-volumes/ https://vocon-it.com/2018/12/20/kubernetes-local-persistent-... The strategy seems to be to create a storage class per app and make sure each persistent volume claim binds a distinct storage class. It sounds like a lot of heavy lifting but it's just a few YAML files... the alternative seems to be https://github.com/rancher/local-path-provisioner https://github.com/rancher/local-path-provisioner which uses the same strategy under the hood to fulfill PVCs as they come online using the Local Persistent Volume strategy, but does not require the storageclass arrangement I understood from the tutorial linked before. If you want to use an SSD attached to only one special/master node, I don't think that's the wild wild west, I think that's actually closer to what you might see in a traditional network storage architecture, possible to use something like Portworx or Rook to make that work if they are supported on RPi. That's not the same as local storage provisioner though, If you know more about it than I do I'm happy to hear more about where you got stuck, since I'll be trying to implement this strategy for myself on a non-RPi cluster with a stable set of nodes soon (stable as in, pets not cattle).

boon 7y ago

Another frustration with the RPi cluster is the arm requirement. If you're only building/running your own stuff, that's fine, but finding containers and helm charts that support arm can be frustratingly difficult. This is speaking from experience. I love my k3s RPi cluster, it's fantastic once you get things working. I just had to augment with some x86 nodes in order to _also_ run some software that just wouldn't run anywhere else.