19 ms·

We've been running Kubernetes (500+ containers) in production for over a year now. I believe (and hope) that 2017 will be the year that persistent data storage

We've been running Kubernetes (500+ containers) in production for over a year now. I believe (and hope) that 2017 will be the year that persistent data storage will be solved. We are ready to move our data out of OpenStack and have our data services (Elasticsearch, Cassandra, MySQL, MongoDB) join the rest of our apps on Kube-orchestrated infrastructure. But, we're not there yet. The options just aren't good enough. Look at the list of PV types for Kube [1]. You have technologies like Fibre Channel that are simply too expensive when compared with local storage on a Linux server. There's iSCSI, which is mostly the same story. Ceph is great for object storage but not performant enough for busy databases. GCE and AWS volumes are not applicable to our private cloud [2]. Cinder, to me, has the stench of OpenStack. Maybe it's better now? NFS? No way. Not performant. I'm looking forward to seeing what shakes out in the next few months. It's just really hard to beat local storage right now. [1] http://kubernetes.io/docs/user-guide/persistent-volumes/#types-of-persistent-volumes http://kubernetes.io/docs/user-guide/persistent-volumes/#typ... [2] Beyond a certain size, it becomes more cost-effective to host your own Kubernetes cluster on managed or colocated hardware.

jat850 10y ago
Curious if you've tried gluster at all. Using Kubernetes also and about to cross the threshold of stateful data - performance is important but not critical since we're using it in a fairly low volume, low throughput way, but it will grow over time and we want to future-proof it a bit. gluster seemed like the best-case fit for us but have done no empirical testing yet.
sandGorgon 10y ago
curious to know what you are doing right now ? I'm planning to deploy a redis cluster on a private cloud and am wondering what to use. do you use hostpath right now ?
- chrissnell 10y ago
  We don't put any persistent data into Kube. Everything goes into OpenStack instances (Ubuntu), orchestrated by Chef. We hate it. OpenStack SDN has been flaky, Chef is a pain and doesn't support the latest Ubuntu releases well, none of the devs or technical ops engineers like it. It's my #1 goal for 2017: figure out persistent volumes for Kube.
  - logronoide 10y ago
    Change Chef for Ansible and I have the same architecture. And you know what? I have decided to put on hold K8S until persistence is properly managed. Meanwhile, we have decided to give a chance to serverless architectures with AWS lambda.
    - markbnj 10y ago
      I agree that persistence in k8s is tricky, especially at scale, but at least in our case that doesn't drive us way from the platform. Kube is awesome for services, and if we have to keep a few things on gcloud instances bolted to reliable storage for the moment that's at least less heterogeneous than what we had before kube came along. In other words I don't think you have to kube all the things to see a lot of benefit.
  - thickice 10y ago
    Are you running Openstack yourself or a vendor backed version ? My understanding is deploying OpenStack in production is a nightmare. Would you mind sharing your experience (cluster size, upgrade, support etc.) ?
  - parasubvert 10y ago
    Disclaimer I work at Pivotal but - I'd suggest taking a look at http://bosh.io http://bosh.io which can handle full stack automation ; there already are quality releases for MySQL, elastic, mongo (from anynines), and Cassandra, with commercially supported releases in some cases with fully orchestrated vm and volume management on Openstack. These are for cloud foundry nominally but (yay actually collaborating foundations) the open service broker API ( https://www.openservicebrokerapi.org/ https://www.openservicebrokerapi.org/ ) allows you to hook these into Kube and (the best part) standardize how your CI/CD pipelines manage the lifecycle of these services independently of whether they're backed by Openstack today or Kubernetes tomorrow. Persistent volumes for these sorts of services on Kube will require the new PetSet primitive to mature.
jtroyer 10y ago
Along with Docker's efforts, there are others working on container-based storage. This landscape[1] lists Ceph, Gluster, but also Portworx, Minio, Diamanti, Dell EMC's Rex-Ray, and SolidFire. I think also folks like StorageOS and Supergiant and frankly the whole storage & platform industry players are running in this direction. [1] https://github.com/cncf/landscape https://github.com/cncf/landscape
basch 10y ago
does VsphereVolume mean you can use data virtualization slash software defined storage like vSAN? https://en.wikipedia.org/wiki/Data_virtualization https://en.wikipedia.org/wiki/Data_virtualization https://en.wikipedia.org/wiki/Software-defined_storage https://en.wikipedia.org/wiki/Software-defined_storage
geertj 10y ago
> The options just aren't good enough. I'd be curious to understand your storage requirements as a production Kubernetes user. What would you like to see for performance, cost and RPO/RTO?
- bogomipz 10y ago
  local NAND flash?
colanderman 10y ago
Check out the startup I work for, ClearSky Data [1]. We provide cloud-hosted block storage with SAN-level performance to your private datacenter and enterprise-grade durability and availability at a competitive price. I'd be glad to answer any questions you have (I'm an engineer) or point you to someone who can. [1] http://www.clearskydata.com/ http://www.clearskydata.com/
- chrissnell 10y ago
  We don't use public cloud. We're hosted on dedicated hardware for a number of reasons. While we're not opposed to commercial solutions, we strongly prefer open source solutions for obvious reasons (like, this ClusterHQ situation).
  - colanderman 10y ago
    You don't have to. We provide iSCSI or FC access points in your private datacenter.
    - mst 10y ago
      Is the hardware storing the data in-datacentre too? I think that was the key thing.
      - colanderman 10y ago
        We (ClearSky) store your data off-site (except a small cache), but provide performance as if the data were on-site.
        cryptarch 10y ago
        So it's stored on a public cloud?
        colanderman 10y ago
parasubvert 10y ago
I'd suggest looking at ScaleIO from EMC which is free for unsupported use I believe. It is blazing fast, runs on bog standard Linux, and supports K8S and Cinder. It's the most impressive block storage product for high performance I've seen. http://cloudscaling.com/blog/cloud-computing/killing-the-storage-unicorn-purpose-built-scaleio-spanks-multi-purpose-ceph-on-performance/ http://cloudscaling.com/blog/cloud-computing/killing-the-sto...
- pas 10y ago
  Any chance to see a comparison on recent Ceph version(s)? (Maybe with BlueStore as Ceph's backend?)
tw04 10y ago
>NFS? No way. Not performant. Really? https://www.spec.org/sfs2008/results/res2011q4/sfs2008-20111003-00198.html https://www.spec.org/sfs2008/results/res2011q4/sfs2008-20111... And those are nothing compared to modern systems.
- seanp2k2 10y ago
  I've had many problems with NFS, but they were not performance related. Mostly FS hangs with some program waiting for IO that never comes.
  - cookiecaper 10y ago
    This is the biggest issue I've had with NFS. If the mount goes stale, programs can get stuck in an uninterruptable state that takes a reboot to clear out.
    - otterley 10y ago
      Just mount the file system with "soft,intr" options and you won't have that problem. Otherwise NFS is (perhaps unrealistically) optimistic that the server will come back.
- kureikain 10y ago
  I have lots of issue to get NFS works on some particular db as well. Such as RethinkDB and InfluxDB...
  - user5994461 10y ago
    Last I checked, databases came with numerous warning in the docs "don't run me on a network drive". Is there any reason why you are trying this at all?
    - brazzledazzle 10y ago
      I think that's a good rule of thumb but you can certainly do it if you know what you're doing (which includes having the right hardware).
a-robinson 10y ago
Yeah, I'm really looking forward to a true local storage option. I'd recommend watching https://github.com/kubernetes/kubernetes/issues/7562 https://github.com/kubernetes/kubernetes/issues/7562 and https://github.com/kubernetes/kubernetes/pull/30044 https://github.com/kubernetes/kubernetes/pull/30044 if you want to keep up with how things develop.
X-Istence 10y ago
Ceph RBD is plenty fast for having a database on... was able to get better performance than fibre channel or NFS to a NetApp using Ceph, ran some nice large Oracle instances on VM's on top of OpenStack backed by Ceph RBD.
- chrissnell 10y ago
  My testing showed otherwise but I'd love to see what you've done. What sort of equipment did you use, what kind of network, and what how many IOPS did you see?
  - X-Istence 10y ago
    SuperMicro has their IOPS optimized Ceph storage SKU's, that is what was used. Looks like they have updated since we purchased: https://www.supermicro.com/solutions/storage_ceph.cfm https://www.supermicro.com/solutions/storage_ceph.cfm We went for upgraded network capacity though, 20 Gbit/sec cluster backend, 20 Gbit/sec cluster frontend... 12 * 8 TB drives, with 800 GB NVMe for the Ceph journal. Fast, large Ceph journal was key. Total installation was about 3 PB raw, that is 1 PB useable with replication size 3. 33 Ceph OSD nodes, 3 Ceph monitor nodes and Juniper low latency switching using the QFX5100. Full IPv6 network on both frontend/backend. 11 nodes per rack, each rack being it's own /64 routed domain. 3 racks. I'm no longer doing contract work for the company, but last I heard they were expanding it out to 6 racks with an additional 3 PB raw capacity added on because of growing datasets. It's an OpenStack cluster that is connected to this Ceph cluster, 40 Gbit/sec storage backend network, with 40 Gbit/sec front-end that VM's have all their traffic on. So storage and standard traffic don't mix. The performance and IOPS even virtualized were enough that the entire company is moving their bare metal databases to VM's. I am unable to disclose IOPS or Oracle database performance due to contractual obligations unfortunately.
contingencies 10y ago
Look at the list of PV types for Kube What I see is a lot of complex network filesystems, vendor-specific solutions and gateway protocols to expensive SAN solutions, which are already chalk and cheese in terms of features and performance. Arguably one of the best features of unix-style systems is support for arbitrary mount points, filesystem drivers and (network or local) blockstores. Storage is, essentially, a well-solved problem at the OS level. The fact that this option is marked "single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster" raises eyebrows. By choosing to expose individual remote storage model semantics as Kube-level PV drivers instead of just leaving this to the OS, what I would argue we essentially see here is the legacy of a cluster orchestration system that came out of Google... a system optimized for large, homogenous, dynamic workloads to provide organization-internal IaaS, and not reduced feature-set systems with simpler architectural properties (eg. no multi-client network-aware filesystem locking). I would argue that, in fact, what many people actually want is simpler, and the current pressure to use 'one size fits all' cluster orchestration systems with a high minimum bar of functionality and nodecount (read: minimum hardware investment) is misplaced. At the very least, there's some legitimacy to this line of thinking.
- cookiecaper 10y ago
  Yes. k8s is cool but it is vastly overcomplicated for the needs of the non-Googles. We've been porting my company's production infrastructure to it over the last year and while it's been fun, I don't think it's been the correct thing for us. Since suggesting your company is not in the same class of the companies that see literally billions of unique users every day, and thus may not need such overcomplicated solutions, is sure to make your boss irate, it's a good idea to familiarize with whatever new hotness has Facebook or Google's name attached to it. Your clueless colleagues will race each other to announce the latest Google/FB engineering blog post in Slack so they can look the smartest and then convince your boss that since your Google-dom will be upon you tomorrow, you must adopt HotNewStuff today. This impulse is behind the proliferation of Hadoop and "Big Data", containers and orchestration, and MongoDB and NoSQL. All of these are useful tools that are valuable and good as necessary, but widely abused because people who don't really know what they're doing think this will give them an out. You'll be stuck maintaining something interesting but really not mature or production-ready like k8s for years, just about long enough for it to become smooth and stable, at which time something else will come along to repeat the cycle. :)
luka-birsa 10y ago
Same here. Running ~100 app servers in K8S and the rest (databases & legacy apps) as regular GCE instances with PD drives. But long term going K8S-only is instrumental for us to prevent vendor lock-in. I really hope that storage for K8S happens this year in a form that is simpler than Gluster/Ceph/etc and preferably integrated. Right now we're using NFS and it's ok for the simple applications but I right now I wouldn't dare deploy my databases in K8S and feel good about it. What gives me hope is the speed the K8S is moving forward and the superb experience we had thus far.
- bogomipz 10y ago
  Any idea how does GCE handles PD drives underneath your K8? Is it network-attacched storage? Are those NVMe drives I wonder?
  - bogomipz 10y ago
    I thought this was a valid question as I am curious how you you able to run databases using K8 on GCE. Why would somebody downvote asking a question?
- TheIronYuppie 10y ago
  What would it look like to be integrated? Could you just use your existing node solutions and hostpath and be set? Disclosure: I work at Google on Kubernetes
  - adictator 10y ago
    I think NFS without its issues (performance & security) would be ideal as an integrated solution.
    - brazzledazzle 10y ago
      If you can solve that elegantly and safely I suggest you build a startup around it.
- 10y ago
objectivefs 10y ago
If you have access to an object store, you can get persistent file system storage for your containers using ObjectiveFS[0]. You can run it either on the hosts or inside the containers depending on your goal. [0]: https://objectivefs.com https://objectivefs.com
bogomipz 10y ago
I'm in a similar boat. Does Kubernetes have a persistent local storage API story on the horizon? The one thing I have found on Mesos that I really liked are persisted volume resources: http://mesos.apache.org/documentation/latest/persistent-volume/ http://mesos.apache.org/documentation/latest/persistent-volu... I was hoping K8 has something similar but it didn't the last time that I looked. Edit: I just looked at your 1st link. I see the PV docs, none of those fit my use case(local) unfortunately.
dankohn1 10y ago
Hi, I'm the executive director of CNCF (which hosts Kubernetes) and the co-author of our landscape document, which has a section on storage [0]. I'll just state the obvious that you're very much correct about stateful storage still being the most immature aspect of the Kubernetes and cloud native story, but there are definitely tons of folks working on it. And in the meantime, people are succeeding in production using last-generation solutions on bare metal or provider offerings in the cloud. I'll make a quick pitch that if you are a cloud native end user interested in engaging with the community, please email me about joining CNCF's end user board (my email is in my profile). We just dropped the price from $50 K to $4.5 K, and it now includes 5 tickets to CloudNativeCon/KubeCon ($1.7 K and 2 tickets for startups). [0] https://github.com/cncf/landscape https://github.com/cncf/landscape
user5994461 10y ago
I don't want to break your hopes but stateful containers will only ever run on GCE and AWS. The entire existence of stateful containers depends on having network storage at hands. The only good network storages are GCE and AWS volumes, which are proprietary trade-secret technologies only available there. If you want to play it old-school. You can run virtual machines with VmWare on bare metal servers with SAN disks (iSCSI/Fibber Channel). The Vm can be hot migrated from an host to another. It works and it's been battle tested for almost a decade. (IMO: Docker is not only new but a toy in comparison to that).
- benley 10y ago
  If virtual machines with live migration is what you're after, vmware is not your only option. You can get largely the same effect without involving any iscsi/FC/SAN tech with another open source project, which happens to originate from the same place as Kubernetes: http://www.ganeti.org/ http://www.ganeti.org/ Ganeti has been battle-tested for about a decade too, supports an assortment of storage backends, including some clustered ones like Ceph, can do live migration between hypervisor nodes, and it's a nicely maintained Python app with some Haskell parts in it. ganeti came out of my team at google in the mid 2000s, fwiw. I did not work on it, but I've certainly used it. It's pretty nice.
  - WestCoastJustin 10y ago
    Thank you. I've used Ganeti extensively and it help wrangle our Xen/DRBD rats nest very quickly. It's an awesome project.
- davidopp__ 10y ago
  > I don't want to break your hopes but stateful containers will only ever run on GCE and AWS. Actually Kubernetes is starting to work on support for persistent local volumes; we know the lack of this feature is a significant barrier for running some stateful applications on Kubernetes, particularly on bare metal. The concrete proposal for how we are thinking to do it is at https://github.com/kubernetes/kubernetes/pull/30044 https://github.com/kubernetes/kubernetes/pull/30044 The high-level feature requests are at https://github.com/kubernetes/features/issues/121 https://github.com/kubernetes/features/issues/121 and https://github.com/kubernetes/kubernetes/issues/7562 https://github.com/kubernetes/kubernetes/issues/7562 (Disclosure: I work on Kubernetes at Google.)
blorgle 10y ago
This doesn't really make much sense to me. If your systems support software level replication (Elasticsearch, Cassandra, MySQL, MongoDB all do) then why do you need persistent storage? You just need container scheduling anti-affinity and enough replicas. You only need persistent storage for systems which don't support that replication. Ceph can certainly be deployed as performant for DB workloads. You say "Cinder has the stench of OpenStack" but Cinder is just a Python based webapp which povides an API to arbitrary storage backends (Ceph RBD, iSCSI, NetApp ONTAP, whatever). How can it be "better now"? It doesn't provide storage on its own. If your ops team was using the default "proof of concept" LVM backend then I could see how you might get a bad impression but that just means your ops team doesn't know much about OpenStack. Am I missing something obvious?
- chrissnell 10y ago
  Yes, you're missing a few things. Maybe not obvious, though. First off, the replication thing. It is true that ES, C*, and Mongo replicate within their cluster mostly automatically. However, this is not without cost. It takes non-trivial amounts of network capacity, disk I/O, and CPU cycles to migrate shards from a failed (or downed) node to a newly stood-up node. Often, many GBs must be moved and for something like ES, where shard replicas reside on many different nodes, that means much of your cluster feels the impact of this. The cluster can heal, but healing isn't easy. Why would a cluster node go down? It's not always hardware failure. CoreOS regularly self-updates and reboots itself without intervention. In a Kubernetes cluster, this is a non-event because pods are simply rescheduled elsewhere the the degradation is momentary. If we were talking about 300 GB of persistent data, though, that's a serious amount of data that will get reshuffled every time there is a node reboot, especially when you consider that an Elasticsearch cluster may span dozens of physical nodes and experience dozens of node reboots in the course of a normal day. Maybe we could hack something that would disable shard reallocation in ES (there's a setting for this) when scheduled reboots happen but that's pretty hacky. Besides, ES is just one of a number of different datastores in use at my workplace. As for Cinder, it's reliant on OpenStack APIs which (at least as of Juno) are reliant on things like RabbitMQ. We've seen a number of OpenStack failures due to RabbitMQ partitioning and split-brained scenarios. We're also back to the disk-on-network problem again: SCSI backplane ---ethernet---> client will never be as fast as local disk.
jondubois 10y ago
Yes I think that Kubernetes is too stubborn with regards to how it wants to decouple Pods/Containers from specific Nodes. Many new databases (especially NoSQL ones) already support clustering and rely on being tightly coupled to specific nodes/machines in order to work. I think K8s currently makes this a bit too difficult - They need to improve support/documentation for using host storage for these kinds of clustered DBs. NFS doesn't make sense for storing structured data because it doesn't know what the best way to partition/search your data is going to be (the directory tree structure isn't always what we need) - I think that this can only be solved at the DB layer unfortunately.
thockingoog 10y ago
I get that local storage is good for perf, and everyone has some, but you HAVE to understand how disastrously bad it is for availability (unless you do replication yourself, a la Cassandra). That said, we hear you, we're contemplating options for local-disk volumes.

jat850 10y ago

Curious if you've tried gluster at all. Using Kubernetes also and about to cross the threshold of stateful data - performance is important but not critical since we're using it in a fairly low volume, low throughput way, but it will grow over time and we want to future-proof it a bit. gluster seemed like the best-case fit for us but have done no empirical testing yet.

sandGorgon 10y ago

curious to know what you are doing right now ? I'm planning to deploy a redis cluster on a private cloud and am wondering what to use. do you use hostpath right now ?

chrissnell 10y ago
We don't put any persistent data into Kube. Everything goes into OpenStack instances (Ubuntu), orchestrated by Chef. We hate it. OpenStack SDN has been flaky, Chef is a pain and doesn't support the latest Ubuntu releases well, none of the devs or technical ops engineers like it. It's my #1 goal for 2017: figure out persistent volumes for Kube.
- logronoide 10y ago
  Change Chef for Ansible and I have the same architecture. And you know what? I have decided to put on hold K8S until persistence is properly managed. Meanwhile, we have decided to give a chance to serverless architectures with AWS lambda.
  - markbnj 10y ago
    I agree that persistence in k8s is tricky, especially at scale, but at least in our case that doesn't drive us way from the platform. Kube is awesome for services, and if we have to keep a few things on gcloud instances bolted to reliable storage for the moment that's at least less heterogeneous than what we had before kube came along. In other words I don't think you have to kube all the things to see a lot of benefit.
- thickice 10y ago
  Are you running Openstack yourself or a vendor backed version ? My understanding is deploying OpenStack in production is a nightmare. Would you mind sharing your experience (cluster size, upgrade, support etc.) ?
- parasubvert 10y ago
  Disclaimer I work at Pivotal but - I'd suggest taking a look at http://bosh.io http://bosh.io which can handle full stack automation ; there already are quality releases for MySQL, elastic, mongo (from anynines), and Cassandra, with commercially supported releases in some cases with fully orchestrated vm and volume management on Openstack. These are for cloud foundry nominally but (yay actually collaborating foundations) the open service broker API ( https://www.openservicebrokerapi.org/ https://www.openservicebrokerapi.org/ ) allows you to hook these into Kube and (the best part) standardize how your CI/CD pipelines manage the lifecycle of these services independently of whether they're backed by Openstack today or Kubernetes tomorrow. Persistent volumes for these sorts of services on Kube will require the new PetSet primitive to mature.

jtroyer 10y ago

Along with Docker's efforts, there are others working on container-based storage. This landscape[1] lists Ceph, Gluster, but also Portworx, Minio, Diamanti, Dell EMC's Rex-Ray, and SolidFire. I think also folks like StorageOS and Supergiant and frankly the whole storage & platform industry players are running in this direction. [1] https://github.com/cncf/landscape https://github.com/cncf/landscape

basch 10y ago

does VsphereVolume mean you can use data virtualization slash software defined storage like vSAN? https://en.wikipedia.org/wiki/Data_virtualization https://en.wikipedia.org/wiki/Data_virtualization https://en.wikipedia.org/wiki/Software-defined_storage https://en.wikipedia.org/wiki/Software-defined_storage

geertj 10y ago

> The options just aren't good enough. I'd be curious to understand your storage requirements as a production Kubernetes user. What would you like to see for performance, cost and RPO/RTO?

bogomipz 10y ago
local NAND flash?

colanderman 10y ago

Check out the startup I work for, ClearSky Data [1]. We provide cloud-hosted block storage with SAN-level performance to your private datacenter and enterprise-grade durability and availability at a competitive price. I'd be glad to answer any questions you have (I'm an engineer) or point you to someone who can. [1] http://www.clearskydata.com/ http://www.clearskydata.com/

chrissnell 10y ago
We don't use public cloud. We're hosted on dedicated hardware for a number of reasons. While we're not opposed to commercial solutions, we strongly prefer open source solutions for obvious reasons (like, this ClusterHQ situation).
- colanderman 10y ago
  You don't have to. We provide iSCSI or FC access points in your private datacenter.
  - mst 10y ago
    Is the hardware storing the data in-datacentre too? I think that was the key thing.
    - colanderman 10y ago
      We (ClearSky) store your data off-site (except a small cache), but provide performance as if the data were on-site.
      - cryptarch 10y ago
        So it's stored on a public cloud?
        colanderman 10y ago

parasubvert 10y ago

I'd suggest looking at ScaleIO from EMC which is free for unsupported use I believe. It is blazing fast, runs on bog standard Linux, and supports K8S and Cinder. It's the most impressive block storage product for high performance I've seen. http://cloudscaling.com/blog/cloud-computing/killing-the-storage-unicorn-purpose-built-scaleio-spanks-multi-purpose-ceph-on-performance/ http://cloudscaling.com/blog/cloud-computing/killing-the-sto...

pas 10y ago
Any chance to see a comparison on recent Ceph version(s)? (Maybe with BlueStore as Ceph's backend?)

tw04 10y ago

>NFS? No way. Not performant. Really? https://www.spec.org/sfs2008/results/res2011q4/sfs2008-20111003-00198.html https://www.spec.org/sfs2008/results/res2011q4/sfs2008-20111... And those are nothing compared to modern systems.

seanp2k2 10y ago
I've had many problems with NFS, but they were not performance related. Mostly FS hangs with some program waiting for IO that never comes.
- cookiecaper 10y ago
  This is the biggest issue I've had with NFS. If the mount goes stale, programs can get stuck in an uninterruptable state that takes a reboot to clear out.
  - otterley 10y ago
    Just mount the file system with "soft,intr" options and you won't have that problem. Otherwise NFS is (perhaps unrealistically) optimistic that the server will come back.
kureikain 10y ago
I have lots of issue to get NFS works on some particular db as well. Such as RethinkDB and InfluxDB...
- user5994461 10y ago
  Last I checked, databases came with numerous warning in the docs "don't run me on a network drive". Is there any reason why you are trying this at all?
  - brazzledazzle 10y ago
    I think that's a good rule of thumb but you can certainly do it if you know what you're doing (which includes having the right hardware).

a-robinson 10y ago

Yeah, I'm really looking forward to a true local storage option. I'd recommend watching https://github.com/kubernetes/kubernetes/issues/7562 https://github.com/kubernetes/kubernetes/issues/7562 and https://github.com/kubernetes/kubernetes/pull/30044 https://github.com/kubernetes/kubernetes/pull/30044 if you want to keep up with how things develop.

X-Istence 10y ago

Ceph RBD is plenty fast for having a database on... was able to get better performance than fibre channel or NFS to a NetApp using Ceph, ran some nice large Oracle instances on VM's on top of OpenStack backed by Ceph RBD.

chrissnell 10y ago
My testing showed otherwise but I'd love to see what you've done. What sort of equipment did you use, what kind of network, and what how many IOPS did you see?
- X-Istence 10y ago
  SuperMicro has their IOPS optimized Ceph storage SKU's, that is what was used. Looks like they have updated since we purchased: https://www.supermicro.com/solutions/storage_ceph.cfm https://www.supermicro.com/solutions/storage_ceph.cfm We went for upgraded network capacity though, 20 Gbit/sec cluster backend, 20 Gbit/sec cluster frontend... 12 * 8 TB drives, with 800 GB NVMe for the Ceph journal. Fast, large Ceph journal was key. Total installation was about 3 PB raw, that is 1 PB useable with replication size 3. 33 Ceph OSD nodes, 3 Ceph monitor nodes and Juniper low latency switching using the QFX5100. Full IPv6 network on both frontend/backend. 11 nodes per rack, each rack being it's own /64 routed domain. 3 racks. I'm no longer doing contract work for the company, but last I heard they were expanding it out to 6 racks with an additional 3 PB raw capacity added on because of growing datasets. It's an OpenStack cluster that is connected to this Ceph cluster, 40 Gbit/sec storage backend network, with 40 Gbit/sec front-end that VM's have all their traffic on. So storage and standard traffic don't mix. The performance and IOPS even virtualized were enough that the entire company is moving their bare metal databases to VM's. I am unable to disclose IOPS or Oracle database performance due to contractual obligations unfortunately.

contingencies 10y ago

Look at the list of PV types for Kube What I see is a lot of complex network filesystems, vendor-specific solutions and gateway protocols to expensive SAN solutions, which are already chalk and cheese in terms of features and performance. Arguably one of the best features of unix-style systems is support for arbitrary mount points, filesystem drivers and (network or local) blockstores. Storage is, essentially, a well-solved problem at the OS level. The fact that this option is marked "single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster" raises eyebrows. By choosing to expose individual remote storage model semantics as Kube-level PV drivers instead of just leaving this to the OS, what I would argue we essentially see here is the legacy of a cluster orchestration system that came out of Google... a system optimized for large, homogenous, dynamic workloads to provide organization-internal IaaS, and not reduced feature-set systems with simpler architectural properties (eg. no multi-client network-aware filesystem locking). I would argue that, in fact, what many people actually want is simpler, and the current pressure to use 'one size fits all' cluster orchestration systems with a high minimum bar of functionality and nodecount (read: minimum hardware investment) is misplaced. At the very least, there's some legitimacy to this line of thinking.

cookiecaper 10y ago
Yes. k8s is cool but it is vastly overcomplicated for the needs of the non-Googles. We've been porting my company's production infrastructure to it over the last year and while it's been fun, I don't think it's been the correct thing for us. Since suggesting your company is not in the same class of the companies that see literally billions of unique users every day, and thus may not need such overcomplicated solutions, is sure to make your boss irate, it's a good idea to familiarize with whatever new hotness has Facebook or Google's name attached to it. Your clueless colleagues will race each other to announce the latest Google/FB engineering blog post in Slack so they can look the smartest and then convince your boss that since your Google-dom will be upon you tomorrow, you must adopt HotNewStuff today. This impulse is behind the proliferation of Hadoop and "Big Data", containers and orchestration, and MongoDB and NoSQL. All of these are useful tools that are valuable and good as necessary, but widely abused because people who don't really know what they're doing think this will give them an out. You'll be stuck maintaining something interesting but really not mature or production-ready like k8s for years, just about long enough for it to become smooth and stable, at which time something else will come along to repeat the cycle. :)

luka-birsa 10y ago

Same here. Running ~100 app servers in K8S and the rest (databases & legacy apps) as regular GCE instances with PD drives. But long term going K8S-only is instrumental for us to prevent vendor lock-in. I really hope that storage for K8S happens this year in a form that is simpler than Gluster/Ceph/etc and preferably integrated. Right now we're using NFS and it's ok for the simple applications but I right now I wouldn't dare deploy my databases in K8S and feel good about it. What gives me hope is the speed the K8S is moving forward and the superb experience we had thus far.

bogomipz 10y ago
Any idea how does GCE handles PD drives underneath your K8? Is it network-attacched storage? Are those NVMe drives I wonder?
- bogomipz 10y ago
  I thought this was a valid question as I am curious how you you able to run databases using K8 on GCE. Why would somebody downvote asking a question?
TheIronYuppie 10y ago
What would it look like to be integrated? Could you just use your existing node solutions and hostpath and be set? Disclosure: I work at Google on Kubernetes
- adictator 10y ago
  I think NFS without its issues (performance & security) would be ideal as an integrated solution.
  - brazzledazzle 10y ago
    If you can solve that elegantly and safely I suggest you build a startup around it.
10y ago

objectivefs 10y ago

If you have access to an object store, you can get persistent file system storage for your containers using ObjectiveFS[0]. You can run it either on the hosts or inside the containers depending on your goal. [0]: https://objectivefs.com https://objectivefs.com

bogomipz 10y ago

I'm in a similar boat. Does Kubernetes have a persistent local storage API story on the horizon? The one thing I have found on Mesos that I really liked are persisted volume resources: http://mesos.apache.org/documentation/latest/persistent-volume/ http://mesos.apache.org/documentation/latest/persistent-volu... I was hoping K8 has something similar but it didn't the last time that I looked. Edit: I just looked at your 1st link. I see the PV docs, none of those fit my use case(local) unfortunately.

dankohn1 10y ago

Hi, I'm the executive director of CNCF (which hosts Kubernetes) and the co-author of our landscape document, which has a section on storage [0]. I'll just state the obvious that you're very much correct about stateful storage still being the most immature aspect of the Kubernetes and cloud native story, but there are definitely tons of folks working on it. And in the meantime, people are succeeding in production using last-generation solutions on bare metal or provider offerings in the cloud. I'll make a quick pitch that if you are a cloud native end user interested in engaging with the community, please email me about joining CNCF's end user board (my email is in my profile). We just dropped the price from $50 K to $4.5 K, and it now includes 5 tickets to CloudNativeCon/KubeCon ($1.7 K and 2 tickets for startups). [0] https://github.com/cncf/landscape https://github.com/cncf/landscape

user5994461 10y ago

I don't want to break your hopes but stateful containers will only ever run on GCE and AWS. The entire existence of stateful containers depends on having network storage at hands. The only good network storages are GCE and AWS volumes, which are proprietary trade-secret technologies only available there. If you want to play it old-school. You can run virtual machines with VmWare on bare metal servers with SAN disks (iSCSI/Fibber Channel). The Vm can be hot migrated from an host to another. It works and it's been battle tested for almost a decade. (IMO: Docker is not only new but a toy in comparison to that).

benley 10y ago
If virtual machines with live migration is what you're after, vmware is not your only option. You can get largely the same effect without involving any iscsi/FC/SAN tech with another open source project, which happens to originate from the same place as Kubernetes: http://www.ganeti.org/ http://www.ganeti.org/ Ganeti has been battle-tested for about a decade too, supports an assortment of storage backends, including some clustered ones like Ceph, can do live migration between hypervisor nodes, and it's a nicely maintained Python app with some Haskell parts in it. ganeti came out of my team at google in the mid 2000s, fwiw. I did not work on it, but I've certainly used it. It's pretty nice.
- WestCoastJustin 10y ago
  Thank you. I've used Ganeti extensively and it help wrangle our Xen/DRBD rats nest very quickly. It's an awesome project.
davidopp__ 10y ago
> I don't want to break your hopes but stateful containers will only ever run on GCE and AWS. Actually Kubernetes is starting to work on support for persistent local volumes; we know the lack of this feature is a significant barrier for running some stateful applications on Kubernetes, particularly on bare metal. The concrete proposal for how we are thinking to do it is at https://github.com/kubernetes/kubernetes/pull/30044 https://github.com/kubernetes/kubernetes/pull/30044 The high-level feature requests are at https://github.com/kubernetes/features/issues/121 https://github.com/kubernetes/features/issues/121 and https://github.com/kubernetes/kubernetes/issues/7562 https://github.com/kubernetes/kubernetes/issues/7562 (Disclosure: I work on Kubernetes at Google.)

blorgle 10y ago

This doesn't really make much sense to me. If your systems support software level replication (Elasticsearch, Cassandra, MySQL, MongoDB all do) then why do you need persistent storage? You just need container scheduling anti-affinity and enough replicas. You only need persistent storage for systems which don't support that replication. Ceph can certainly be deployed as performant for DB workloads. You say "Cinder has the stench of OpenStack" but Cinder is just a Python based webapp which povides an API to arbitrary storage backends (Ceph RBD, iSCSI, NetApp ONTAP, whatever). How can it be "better now"? It doesn't provide storage on its own. If your ops team was using the default "proof of concept" LVM backend then I could see how you might get a bad impression but that just means your ops team doesn't know much about OpenStack. Am I missing something obvious?

chrissnell 10y ago
Yes, you're missing a few things. Maybe not obvious, though. First off, the replication thing. It is true that ES, C*, and Mongo replicate within their cluster mostly automatically. However, this is not without cost. It takes non-trivial amounts of network capacity, disk I/O, and CPU cycles to migrate shards from a failed (or downed) node to a newly stood-up node. Often, many GBs must be moved and for something like ES, where shard replicas reside on many different nodes, that means much of your cluster feels the impact of this. The cluster can heal, but healing isn't easy. Why would a cluster node go down? It's not always hardware failure. CoreOS regularly self-updates and reboots itself without intervention. In a Kubernetes cluster, this is a non-event because pods are simply rescheduled elsewhere the the degradation is momentary. If we were talking about 300 GB of persistent data, though, that's a serious amount of data that will get reshuffled every time there is a node reboot, especially when you consider that an Elasticsearch cluster may span dozens of physical nodes and experience dozens of node reboots in the course of a normal day. Maybe we could hack something that would disable shard reallocation in ES (there's a setting for this) when scheduled reboots happen but that's pretty hacky. Besides, ES is just one of a number of different datastores in use at my workplace. As for Cinder, it's reliant on OpenStack APIs which (at least as of Juno) are reliant on things like RabbitMQ. We've seen a number of OpenStack failures due to RabbitMQ partitioning and split-brained scenarios. We're also back to the disk-on-network problem again: SCSI backplane ---ethernet---> client will never be as fast as local disk.

jondubois 10y ago

Yes I think that Kubernetes is too stubborn with regards to how it wants to decouple Pods/Containers from specific Nodes. Many new databases (especially NoSQL ones) already support clustering and rely on being tightly coupled to specific nodes/machines in order to work. I think K8s currently makes this a bit too difficult - They need to improve support/documentation for using host storage for these kinds of clustered DBs. NFS doesn't make sense for storing structured data because it doesn't know what the best way to partition/search your data is going to be (the directory tree structure isn't always what we need) - I think that this can only be solved at the DB layer unfortunately.

thockingoog 10y ago

I get that local storage is good for perf, and everyone has some, but you HAVE to understand how disastrously bad it is for availability (unless you do replication yourself, a la Cassandra). That said, we hear you, we're contemplating options for local-disk volumes.