6 ms·

Cannot find much on mounting storage to the container. It exactly that --storage-- that has been recognized as a difficult problem when moving an infrastructure

by cies 10y ago

Cannot find much on mounting storage to the container. It exactly that --storage-- that has been recognized as a difficult problem when moving an infrastructure to containers. Given the title "The Hard Way", I kind of expected it would dive deeper into this topic.

jordanthoms 10y ago
Of course - following the first law of technical documentation, the amount written about it is inversely proportional to the complexity of the problem. Seriously though, it is getting a bit silly how almost every single kubernetes demo involving persistent data just uses emptyDir or hostPath and we are all supposed to pretend that those are a production-ready solution that won't lose data. AFAIK the PersistentVolumeClaim and StorageClass stuff will be the solution for this?
- cies 10y ago
  To me it seems that there are many approaches. A storage cluster seems to be a necessity for anything serious. But then the questions is: which one? Open source solutions are not too many (Gluster, Ceph, FreeNAS come to mind), and from the closed source camp ScaleIO looks really promising. But non of these are part of your K8s tutorial :)
  - creshal 10y ago
    And gluster at least has horrible performance and stability issues. And with containers, kernel panics kill an entire host…
    - strikerz 10y ago
      Do you have more details on performance and stability issues? The recent releases of gluster have been working very well. Gluster is completely in userspace. How would stability issues, if any, in userspace cause kernel panics? Can you comment more about that?
      - creshal 10y ago
        > Do you have more details on performance and stability issues? Tried using it for a while (last November until ~April) on Debian Jessie, with latest versions. Lots of bugs with SSL mode, lots of desynchronization issues, crippling performance issues, … And no matter what I ran into, the bugs were already known and filed for months at that point, with zero developer reaction. > Gluster is completely in userspace. Unless you try using its NFS server to get not entirely disastrous performance.
        strikerz 10y ago
        What workloads were you using Gluster for? How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. > Unless you try using its NFS server to get not entirely disastrous performance. Wrong again, Gluster's NFS server is in userspace too. Not sure how that can cause a kernel panic.
        creshal 10y ago
        > What workloads were you using Gluster for? Sharing config files. Wordpress sites. Django sites. The performance was too shitty for everything. > How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. IRC. I ended up finding all my errors unresolved by it in the mailing list archives with zero developer attention, and since I didn't have the time to help RedHat make their product actually usable, I dropped Gluster. > Not sure how that can cause a kernel panic. I guess it's technically not a kernel panic if I/O just stalls so hard that you can neither mount nor unmount nor otherwise access any of your NFS targets… but you have to hard reset the whole machine anyway.
    - lazylizard 10y ago
      we used to run 3 gluster nodes, with a few hundred gb of shared storage..mostly used to keep logs and as war file directory for jboss/tomcat. from what i can tell gluster itself seems ok..but someone also enabled ctdb..which seems to hang every half a year or so..
  - tw04 10y ago
    Probably depends on the scale/scope of your deployment. I know NetApp has a docker plugin for their storage platforms, I would imagine other vendors do as well: https://github.com/NetApp/netappdvp https://github.com/NetApp/netappdvp
  - user5994461 10y ago
    These storage systems are meant to be SAN devices. Meaning either Amazon EBS volume or Google volumes, on their respective clouds. I recall reading that EBS is being [partially] implemented and tested. (Read: The documentation is full of "known bug: ..." and "please don't try that in production").
- e12e 10y ago
  Lol, this is how I've been feeling about every virutalization and container/jail solution since the dawn of this century. The early jail-type solutions were somewhat excused, because the answer was either "local filesytem", "nfs" (later nfs and/or iSCSI) - or for the adventurous some kind of cluster filesytem. On that note, are anyone using encrypted NFSv4 in production? On paper it ticks most of the right boxes, but I'm not convinced it actually works...
  - zwischenzug 10y ago
    It's mandated by our security team, but no-one has ever got it working...
- deleted 10y ago
  [deleted]
ThePhysicist 10y ago
Check out the "persistent volumes" section of the manual: http://kubernetes.io/docs/user-guide/persistent-volumes/#parameters http://kubernetes.io/docs/user-guide/persistent-volumes/#par... as well as the "volume management" section: http://kubernetes.io/docs/user-guide/volumes/#types-of-volumes http://kubernetes.io/docs/user-guide/volumes/#types-of-volum... Those two should give a good overview of the available options. Basically, you can use local paths, NFS shares, vendor-specific storage (e.g. AWS EBS) and various other things.
- creshal 10y ago
  > Those two should give a good overview of the available options. Basically, you can use local paths, NFS shares, vendor-specific storage (e.g. AWS EBS) and various other things. Yeah, but which of them actually work?
  - ThePhysicist 10y ago
    It depends on the environment: If you're on AWS, you can use EBS, if you have an NFS share you can use that. In the simplest case, you can just a `hostPath` volume to mount a local directory from the pod's host.
  - cies 10y ago
    > Yeah, but which of them actually work? And on top of that, which of them work when scaling the app out. In other words: how does each option deal with having multiple containers mount it. edit: From this link -- http://kubernetes.io/docs/user-guide/volumes/#types-of-volumes http://kubernetes.io/docs/user-guide/volumes/#types-of-volum... -- I found that the following volumes allow mounting by multiple writers: nfs, glusterfs and cephfs. But which one of these works well under (production) load?
    - db48x 10y ago
      There is no single "production load"; everyone's load is different, so you'll have to try it and measure. This is always the case with any storage system.
objectivefs 10y ago
One way to handle persistent container storage is to use a shared filesystem such as glusterfs or objectivefs. It's easy to set up and can be very helpful when moving your infrastructure to containers. Once created you can mount your shared filesystem either on the host and share it with the container or directly in the container, see e.g. https://objectivefs.com/howto/how-to-use-objectivefs-with-docker https://objectivefs.com/howto/how-to-use-objectivefs-with-do...
- cies 10y ago
  If I understand correctly you are suggesting a block storage solution. But that would then prohibit a scale out, right? (I'm reading on the ObjectiveFS site that it does allow the concurrent access of shares... Hmmm.) The other solution is object storage, which is the real way forward (IMHO). This route make a app more adherent to the 12-factor principles.
  - objectivefs 10y ago
    Both GlusterFS and ObjectiveFS are shared filesystems instead of block storage, so they work with POSIX files/directories (like NFS) making them easier to scale out. If your application can work directly with an object store that is of course also a great way to go.
thinkersilver 10y ago
I can understand the frustration. Been watching this space to see a plausible story on having storage volumes available to containers from any host. Histpath and emptydir aren't really acceptable as solutions since they add complexity to your cluster set up. There are some cluster storage technoloies out there but there are no tutorials, or overviews detailing, the pros and cons, and their performance limitations in a k8s environment. Networking was the hot topic this time last year but storage is the topic no-one is willing to talk about. Tectonic is attempting to build a solution designed for the container use-case[1]. But that's a long way off. If you are building cloud-native applications on-prem and you need state you're on your own for now. [1] https://coreos.com/blog/torus-distributed-storage-by-coreos.html https://coreos.com/blog/torus-distributed-storage-by-coreos....
- cies 10y ago
  > If you are building cloud-native applications on-prem and you need state you're on your own for now. Same conclusion I've come to. I guess K8s is just one part of the solution, it needs to be paired with a storage solution to be really able to replace existing infra. When looking at open source solutions the landscape is pretty empty with Gluster, Ceph and FreeNAS. Where only Gluster and Ceph provide some level of HA.
kozikow 10y ago
Petsets (http://kubernetes.io/docs/user-guide/petset/#what-is-a-pet-set http://kubernetes.io/docs/user-guide/petset/#what-is-a-pet-s...) are adressing the problem of storage persistently linked to the container.
kordless 10y ago
Check out Quobyte: https://www.quobyte.com/containers https://www.quobyte.com/containers

by cies 10y ago

jordanthoms 10y ago

Of course - following the first law of technical documentation, the amount written about it is inversely proportional to the complexity of the problem. Seriously though, it is getting a bit silly how almost every single kubernetes demo involving persistent data just uses emptyDir or hostPath and we are all supposed to pretend that those are a production-ready solution that won't lose data. AFAIK the PersistentVolumeClaim and StorageClass stuff will be the solution for this?

cies 10y ago
To me it seems that there are many approaches. A storage cluster seems to be a necessity for anything serious. But then the questions is: which one? Open source solutions are not too many (Gluster, Ceph, FreeNAS come to mind), and from the closed source camp ScaleIO looks really promising. But non of these are part of your K8s tutorial :)
- creshal 10y ago
  And gluster at least has horrible performance and stability issues. And with containers, kernel panics kill an entire host…
  - strikerz 10y ago
    Do you have more details on performance and stability issues? The recent releases of gluster have been working very well. Gluster is completely in userspace. How would stability issues, if any, in userspace cause kernel panics? Can you comment more about that?
    - creshal 10y ago
      > Do you have more details on performance and stability issues? Tried using it for a while (last November until ~April) on Debian Jessie, with latest versions. Lots of bugs with SSL mode, lots of desynchronization issues, crippling performance issues, … And no matter what I ran into, the bugs were already known and filed for months at that point, with zero developer reaction. > Gluster is completely in userspace. Unless you try using its NFS server to get not entirely disastrous performance.
      - strikerz 10y ago
        What workloads were you using Gluster for? How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. > Unless you try using its NFS server to get not entirely disastrous performance. Wrong again, Gluster's NFS server is in userspace too. Not sure how that can cause a kernel panic.
        creshal 10y ago
        > What workloads were you using Gluster for? Sharing config files. Wordpress sites. Django sites. The performance was too shitty for everything. > How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. IRC. I ended up finding all my errors unresolved by it in the mailing list archives with zero developer attention, and since I didn't have the time to help RedHat make their product actually usable, I dropped Gluster. > Not sure how that can cause a kernel panic. I guess it's technically not a kernel panic if I/O just stalls so hard that you can neither mount nor unmount nor otherwise access any of your NFS targets… but you have to hard reset the whole machine anyway.
  - lazylizard 10y ago
    we used to run 3 gluster nodes, with a few hundred gb of shared storage..mostly used to keep logs and as war file directory for jboss/tomcat. from what i can tell gluster itself seems ok..but someone also enabled ctdb..which seems to hang every half a year or so..
- tw04 10y ago
  Probably depends on the scale/scope of your deployment. I know NetApp has a docker plugin for their storage platforms, I would imagine other vendors do as well: https://github.com/NetApp/netappdvp https://github.com/NetApp/netappdvp
- user5994461 10y ago
  These storage systems are meant to be SAN devices. Meaning either Amazon EBS volume or Google volumes, on their respective clouds. I recall reading that EBS is being [partially] implemented and tested. (Read: The documentation is full of "known bug: ..." and "please don't try that in production").
e12e 10y ago
Lol, this is how I've been feeling about every virutalization and container/jail solution since the dawn of this century. The early jail-type solutions were somewhat excused, because the answer was either "local filesytem", "nfs" (later nfs and/or iSCSI) - or for the adventurous some kind of cluster filesytem. On that note, are anyone using encrypted NFSv4 in production? On paper it ticks most of the right boxes, but I'm not convinced it actually works...
- zwischenzug 10y ago
  It's mandated by our security team, but no-one has ever got it working...
deleted 10y ago
[deleted]

ThePhysicist 10y ago

Check out the "persistent volumes" section of the manual: http://kubernetes.io/docs/user-guide/persistent-volumes/#parameters http://kubernetes.io/docs/user-guide/persistent-volumes/#par... as well as the "volume management" section: http://kubernetes.io/docs/user-guide/volumes/#types-of-volumes http://kubernetes.io/docs/user-guide/volumes/#types-of-volum... Those two should give a good overview of the available options. Basically, you can use local paths, NFS shares, vendor-specific storage (e.g. AWS EBS) and various other things.

creshal 10y ago
> Those two should give a good overview of the available options. Basically, you can use local paths, NFS shares, vendor-specific storage (e.g. AWS EBS) and various other things. Yeah, but which of them actually work?
- ThePhysicist 10y ago
  It depends on the environment: If you're on AWS, you can use EBS, if you have an NFS share you can use that. In the simplest case, you can just a `hostPath` volume to mount a local directory from the pod's host.
- cies 10y ago
  > Yeah, but which of them actually work? And on top of that, which of them work when scaling the app out. In other words: how does each option deal with having multiple containers mount it. edit: From this link -- http://kubernetes.io/docs/user-guide/volumes/#types-of-volumes http://kubernetes.io/docs/user-guide/volumes/#types-of-volum... -- I found that the following volumes allow mounting by multiple writers: nfs, glusterfs and cephfs. But which one of these works well under (production) load?
  - db48x 10y ago
    There is no single "production load"; everyone's load is different, so you'll have to try it and measure. This is always the case with any storage system.

objectivefs 10y ago

One way to handle persistent container storage is to use a shared filesystem such as glusterfs or objectivefs. It's easy to set up and can be very helpful when moving your infrastructure to containers. Once created you can mount your shared filesystem either on the host and share it with the container or directly in the container, see e.g. https://objectivefs.com/howto/how-to-use-objectivefs-with-docker https://objectivefs.com/howto/how-to-use-objectivefs-with-do...

cies 10y ago
If I understand correctly you are suggesting a block storage solution. But that would then prohibit a scale out, right? (I'm reading on the ObjectiveFS site that it does allow the concurrent access of shares... Hmmm.) The other solution is object storage, which is the real way forward (IMHO). This route make a app more adherent to the 12-factor principles.
- objectivefs 10y ago
  Both GlusterFS and ObjectiveFS are shared filesystems instead of block storage, so they work with POSIX files/directories (like NFS) making them easier to scale out. If your application can work directly with an object store that is of course also a great way to go.

thinkersilver 10y ago

I can understand the frustration. Been watching this space to see a plausible story on having storage volumes available to containers from any host. Histpath and emptydir aren't really acceptable as solutions since they add complexity to your cluster set up. There are some cluster storage technoloies out there but there are no tutorials, or overviews detailing, the pros and cons, and their performance limitations in a k8s environment. Networking was the hot topic this time last year but storage is the topic no-one is willing to talk about. Tectonic is attempting to build a solution designed for the container use-case[1]. But that's a long way off. If you are building cloud-native applications on-prem and you need state you're on your own for now. [1] https://coreos.com/blog/torus-distributed-storage-by-coreos.html https://coreos.com/blog/torus-distributed-storage-by-coreos....

cies 10y ago
> If you are building cloud-native applications on-prem and you need state you're on your own for now. Same conclusion I've come to. I guess K8s is just one part of the solution, it needs to be paired with a storage solution to be really able to replace existing infra. When looking at open source solutions the landscape is pretty empty with Gluster, Ceph and FreeNAS. Where only Gluster and Ceph provide some level of HA.

kozikow 10y ago

Petsets (http://kubernetes.io/docs/user-guide/petset/#what-is-a-pet-set http://kubernetes.io/docs/user-guide/petset/#what-is-a-pet-s...) are adressing the problem of storage persistently linked to the container.

kordless 10y ago

Check out Quobyte: https://www.quobyte.com/containers https://www.quobyte.com/containers