4 ms·

I am curious to see how they recommend handling storage on bare metal k8s.

lima 7y ago
In many environments, you don't actually need distributed storage: - Application servers are stateless. - All state lives in stateful databases. - Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.
- turk73 7y ago
  But StatefulSets require storage...someplace. We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).
  - lima 7y ago
    The point is that with native HA, it doesn't have to be distributed storage. You can use locally attached SSDs or NVMe drives, which are straight forward to manage and fast.
    - acdha 7y ago
      Importantly, they also have independent failure modes. I’ve never known a SAN to be within an order of magnitude of the advertised uptime — and some interesting failures have been caused by HA features. Local storage avoids whole classes of problem and has much greater aggregate performance if you can make it work for your app.
q3k 7y ago
I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/ https://rook.io/
- the-rc 7y ago
  I worked on Colossus at Google and Ceph is the closest thing out there. Gregory Farnum gave a great talk about it at the Open Source Summit 2017. I heard from someone at a large company, though, that it's not getting a lot of love from Red Hat nowadays, even if you're a large paying customer. Now I'm curious.
  - kevin_nisbet 7y ago
    I'm curious also. I don't have any special insight into RedHat, but rook went 1.0 just last week: https://blog.rook.io/rook-v1-0-a-major-milestone-689ca4c75508 https://blog.rook.io/rook-v1-0-a-major-milestone-689ca4c7550... That announcement gave me the impression the project is making active progress.
  - noahdesu 7y ago
    I'm working on Rook+Ceph at Red Hat. Rook 1.0 was just released last week which added support for the very latest Nautilus release of Ceph.
    - the-rc 7y ago
      Great to hear. I couldn't get more details from them, but it was something about both rebalancing and data recovery. They picked SwiftStack, which is at least in part open source. Maybe by now you have figured who that company is. My team is still very interested in Ceph and Rook, for the record.
- vallard 7y ago
  Big fan of rook myself. We took several nodes loaded with 16 disk drives and used Rook to make a Ceph and Minio cluster out of it. Great storage solution.
  - the-rc 7y ago
    Have you done any disaster recovery exercises? We've started to play with it, but haven't gone that far yet.
elsonrodriguez 7y ago
Me too. These days I usually recommend that any data an app wants to persist be written to an object store or database. And that any app that needs to write critical data to a disk should be provisioned with config management on bare metal (or pods statically assigned to machines with attached storage if K8s is a must). This is mostly due to how tough the problem of storage is on K8s, and how much is at risk when it goes south.
nikore 7y ago
I have been using openebs (https://www.openebs.io/ https://www.openebs.io/) at my work and its been pretty stable thus far.

lima 7y ago

In many environments, you don't actually need distributed storage: - Application servers are stateless. - All state lives in stateful databases. - Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.

turk73 7y ago
But StatefulSets require storage...someplace. We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).
- lima 7y ago
  The point is that with native HA, it doesn't have to be distributed storage. You can use locally attached SSDs or NVMe drives, which are straight forward to manage and fast.
  - acdha 7y ago
    Importantly, they also have independent failure modes. I’ve never known a SAN to be within an order of magnitude of the advertised uptime — and some interesting failures have been caused by HA features. Local storage avoids whole classes of problem and has much greater aggregate performance if you can make it work for your app.

q3k 7y ago

I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/ https://rook.io/

the-rc 7y ago
I worked on Colossus at Google and Ceph is the closest thing out there. Gregory Farnum gave a great talk about it at the Open Source Summit 2017. I heard from someone at a large company, though, that it's not getting a lot of love from Red Hat nowadays, even if you're a large paying customer. Now I'm curious.
- kevin_nisbet 7y ago
  I'm curious also. I don't have any special insight into RedHat, but rook went 1.0 just last week: https://blog.rook.io/rook-v1-0-a-major-milestone-689ca4c75508 https://blog.rook.io/rook-v1-0-a-major-milestone-689ca4c7550... That announcement gave me the impression the project is making active progress.
- noahdesu 7y ago
  I'm working on Rook+Ceph at Red Hat. Rook 1.0 was just released last week which added support for the very latest Nautilus release of Ceph.
  - the-rc 7y ago
    Great to hear. I couldn't get more details from them, but it was something about both rebalancing and data recovery. They picked SwiftStack, which is at least in part open source. Maybe by now you have figured who that company is. My team is still very interested in Ceph and Rook, for the record.
vallard 7y ago
Big fan of rook myself. We took several nodes loaded with 16 disk drives and used Rook to make a Ceph and Minio cluster out of it. Great storage solution.
- the-rc 7y ago
  Have you done any disaster recovery exercises? We've started to play with it, but haven't gone that far yet.

elsonrodriguez 7y ago

Me too. These days I usually recommend that any data an app wants to persist be written to an object store or database. And that any app that needs to write critical data to a disk should be provisioned with config management on bare metal (or pods statically assigned to machines with attached storage if K8s is a must). This is mostly due to how tough the problem of storage is on K8s, and how much is at risk when it goes south.

nikore 7y ago

I have been using openebs (https://www.openebs.io/ https://www.openebs.io/) at my work and its been pretty stable thus far.