5 ms·

To me it seems that there are many approaches. A storage cluster seems to be a necessity for anything serious. But then the questions is: which one? Open source

by cies 10y ago

To me it seems that there are many approaches. A storage cluster seems to be a necessity for anything serious. But then the questions is: which one? Open source solutions are not too many (Gluster, Ceph, FreeNAS come to mind), and from the closed source camp ScaleIO looks really promising. But non of these are part of your K8s tutorial :)

creshal 10y ago
And gluster at least has horrible performance and stability issues. And with containers, kernel panics kill an entire host…
- strikerz 10y ago
  Do you have more details on performance and stability issues? The recent releases of gluster have been working very well. Gluster is completely in userspace. How would stability issues, if any, in userspace cause kernel panics? Can you comment more about that?
  - creshal 10y ago
    > Do you have more details on performance and stability issues? Tried using it for a while (last November until ~April) on Debian Jessie, with latest versions. Lots of bugs with SSL mode, lots of desynchronization issues, crippling performance issues, … And no matter what I ran into, the bugs were already known and filed for months at that point, with zero developer reaction. > Gluster is completely in userspace. Unless you try using its NFS server to get not entirely disastrous performance.
    - strikerz 10y ago
      What workloads were you using Gluster for? How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. > Unless you try using its NFS server to get not entirely disastrous performance. Wrong again, Gluster's NFS server is in userspace too. Not sure how that can cause a kernel panic.
      - creshal 10y ago
        > What workloads were you using Gluster for? Sharing config files. Wordpress sites. Django sites. The performance was too shitty for everything. > How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. IRC. I ended up finding all my errors unresolved by it in the mailing list archives with zero developer attention, and since I didn't have the time to help RedHat make their product actually usable, I dropped Gluster. > Not sure how that can cause a kernel panic. I guess it's technically not a kernel panic if I/O just stalls so hard that you can neither mount nor unmount nor otherwise access any of your NFS targets… but you have to hard reset the whole machine anyway.
- lazylizard 10y ago
  we used to run 3 gluster nodes, with a few hundred gb of shared storage..mostly used to keep logs and as war file directory for jboss/tomcat. from what i can tell gluster itself seems ok..but someone also enabled ctdb..which seems to hang every half a year or so..
tw04 10y ago
Probably depends on the scale/scope of your deployment. I know NetApp has a docker plugin for their storage platforms, I would imagine other vendors do as well: https://github.com/NetApp/netappdvp https://github.com/NetApp/netappdvp
user5994461 10y ago
These storage systems are meant to be SAN devices. Meaning either Amazon EBS volume or Google volumes, on their respective clouds. I recall reading that EBS is being [partially] implemented and tested. (Read: The documentation is full of "known bug: ..." and "please don't try that in production").

by cies 10y ago

creshal 10y ago

And gluster at least has horrible performance and stability issues. And with containers, kernel panics kill an entire host…

strikerz 10y ago
Do you have more details on performance and stability issues? The recent releases of gluster have been working very well. Gluster is completely in userspace. How would stability issues, if any, in userspace cause kernel panics? Can you comment more about that?
- creshal 10y ago
  > Do you have more details on performance and stability issues? Tried using it for a while (last November until ~April) on Debian Jessie, with latest versions. Lots of bugs with SSL mode, lots of desynchronization issues, crippling performance issues, … And no matter what I ran into, the bugs were already known and filed for months at that point, with zero developer reaction. > Gluster is completely in userspace. Unless you try using its NFS server to get not entirely disastrous performance.
  - strikerz 10y ago
    What workloads were you using Gluster for? How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. > Unless you try using its NFS server to get not entirely disastrous performance. Wrong again, Gluster's NFS server is in userspace too. Not sure how that can cause a kernel panic.
    - creshal 10y ago
      > What workloads were you using Gluster for? Sharing config files. Wordpress sites. Django sites. The performance was too shitty for everything. > How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed. IRC. I ended up finding all my errors unresolved by it in the mailing list archives with zero developer attention, and since I didn't have the time to help RedHat make their product actually usable, I dropped Gluster. > Not sure how that can cause a kernel panic. I guess it's technically not a kernel panic if I/O just stalls so hard that you can neither mount nor unmount nor otherwise access any of your NFS targets… but you have to hard reset the whole machine anyway.
lazylizard 10y ago
we used to run 3 gluster nodes, with a few hundred gb of shared storage..mostly used to keep logs and as war file directory for jboss/tomcat. from what i can tell gluster itself seems ok..but someone also enabled ctdb..which seems to hang every half a year or so..

tw04 10y ago

Probably depends on the scale/scope of your deployment. I know NetApp has a docker plugin for their storage platforms, I would imagine other vendors do as well: https://github.com/NetApp/netappdvp https://github.com/NetApp/netappdvp

user5994461 10y ago

These storage systems are meant to be SAN devices. Meaning either Amazon EBS volume or Google volumes, on their respective clouds. I recall reading that EBS is being [partially] implemented and tested. (Read: The documentation is full of "known bug: ..." and "please don't try that in production").