3 ms·

Yeah. Would be really interesting to see a container technology that people were eager to use for DB management. Persistent storage is the thing that hasn't cli

Yeah. Would be really interesting to see a container technology that people were eager to use for DB management. Persistent storage is the thing that hasn't clicked yet. My guess is this technology will be some kind of cluster management. If there were a 'platform' for horizontally scaling data that managed sharding, rebalancing & backups plus gave me user-space hooks to override policies, databases might slowly & cautiously build on top of it. Most DBs have their own of these (pgpool, mongos, redis sentinel, all of cassandra & hadoop) but would be cool to centralize the common reqs and collaborate.

erulabs 10y ago
> would be cool to centralize the common reqs and collaborate Oh baby the desire to pitch my WIP SaaS product. This is something I'm extremely keen on and I couldn't agree more!
regularfry 10y ago
The problem is that the constraints of wanting your data and CPU on the same node, needing data synced to more than one node, and avoiding being too much of a performance bottleneck, all combine to make it very hard to do better than something which has privileged knowledge of how the database works under the hood. Performant consistency is hard.
william20111 10y ago
smartOS gives you persistent storage btw. And its lovely ZFS storage..
- justinsaccount 10y ago
  saying smartos gives you persistent storage is like saying docker gives you persistent storage because you can use docker run .... -v /storage:/storage myapp That's not what people are talking about. If you run a container on smartos and the physical machine it is running on blows up, what happens to your persistent storage?
  - awinter-py 10y ago
    technically it still persists, just in the form of hawking radiation instead of bits & bytes. Some people think this is what makes amazon glacier so cheap.
roland-s 10y ago
I'm not too knowledgeable on the subject but I know there are a lot of groups working on persistent storage for Docker. I think most of the time this means a docker volume that lives on some network attached storage (this could be redundant across the compute cluster or on dedicated storage servers), which is then linked to a container via the network. If you remove that container or destroy the host, the volume persists and you can recreate the container on a new host and re-attach it to the associated volume (automatically). As far as databases and horizontal scaling, this could probably be done currently within each Docker container at the database layer as you mentioned (CockroachDB also comes to mind). So let's say you add a new docker container with its own persistent+redundant storage volumes, and that container is running a cockroachDB node. You can add more containers as cockroachDB nodes, each with their own persistent+redundant storage, and you get all the DB sharding etc across nodes from cockroachDB while the redundancy+persistence are handled by Docker (plus storage solution) as far as restarting, migrating, reconnecting, etc. This probably has all sorts of problems I'm blind to but that's my rudimentary understanding. Handling horizontal scaling, sharding etc at the storage layer directly through Docker sounds much much harder, and I don't even know what I've been talking about so far so I'll leave it at that Here's a list of some people doing persistent Docker storage : Flocker Portworx https://portworx.com/ StorageOS https://storageos.com/ Rancher Longhorn https://github.com/rancher/longhorn Quobyte https://www.quobyte.com/containers Nexenta https://nexenta.com/solutions/containers Infinit https://infinit.sh/docker EMC REX-ray https://github.com/codedellemc/rexray Ceph Gluster Mesos https://mesosphere.github.io/marathon/docs/external-volumes.html Kubernetes https://kubernetes.io/docs/concepts/storage/persistent-volumes/ vSAN Nutanix http://next.nutanix.com/t5/Nutanix-Connect-Blog/Containers-Enter-the-Acropolis-Data-Fabric/ba-p/11122 Netapp (Docker volume plugins for SolidFire+ONTAP+E-series) AWS EBS/EFS
ferrantim 10y ago
I'm biased but checkout Portworx. It is designed to be that and if you don't think it is, I'd be curious why is missing. https://portworx.com https://portworx.com
justinsaccount 10y ago
https://github.com/CrunchyData/postgres-operator https://github.com/CrunchyData/postgres-operator is the most promising thing I've seen lately.

erulabs 10y ago

> would be cool to centralize the common reqs and collaborate Oh baby the desire to pitch my WIP SaaS product. This is something I'm extremely keen on and I couldn't agree more!

regularfry 10y ago

The problem is that the constraints of wanting your data and CPU on the same node, needing data synced to more than one node, and avoiding being too much of a performance bottleneck, all combine to make it very hard to do better than something which has privileged knowledge of how the database works under the hood. Performant consistency is hard.

william20111 10y ago

smartOS gives you persistent storage btw. And its lovely ZFS storage..

justinsaccount 10y ago
saying smartos gives you persistent storage is like saying docker gives you persistent storage because you can use docker run .... -v /storage:/storage myapp That's not what people are talking about. If you run a container on smartos and the physical machine it is running on blows up, what happens to your persistent storage?
- awinter-py 10y ago
  technically it still persists, just in the form of hawking radiation instead of bits & bytes. Some people think this is what makes amazon glacier so cheap.

roland-s 10y ago

I'm not too knowledgeable on the subject but I know there are a lot of groups working on persistent storage for Docker. I think most of the time this means a docker volume that lives on some network attached storage (this could be redundant across the compute cluster or on dedicated storage servers), which is then linked to a container via the network. If you remove that container or destroy the host, the volume persists and you can recreate the container on a new host and re-attach it to the associated volume (automatically). As far as databases and horizontal scaling, this could probably be done currently within each Docker container at the database layer as you mentioned (CockroachDB also comes to mind). So let's say you add a new docker container with its own persistent+redundant storage volumes, and that container is running a cockroachDB node. You can add more containers as cockroachDB nodes, each with their own persistent+redundant storage, and you get all the DB sharding etc across nodes from cockroachDB while the redundancy+persistence are handled by Docker (plus storage solution) as far as restarting, migrating, reconnecting, etc. This probably has all sorts of problems I'm blind to but that's my rudimentary understanding. Handling horizontal scaling, sharding etc at the storage layer directly through Docker sounds much much harder, and I don't even know what I've been talking about so far so I'll leave it at that Here's a list of some people doing persistent Docker storage : Flocker Portworx https://portworx.com/ StorageOS https://storageos.com/ Rancher Longhorn https://github.com/rancher/longhorn Quobyte https://www.quobyte.com/containers Nexenta https://nexenta.com/solutions/containers Infinit https://infinit.sh/docker EMC REX-ray https://github.com/codedellemc/rexray Ceph Gluster Mesos https://mesosphere.github.io/marathon/docs/external-volumes.html Kubernetes https://kubernetes.io/docs/concepts/storage/persistent-volumes/ vSAN Nutanix http://next.nutanix.com/t5/Nutanix-Connect-Blog/Containers-Enter-the-Acropolis-Data-Fabric/ba-p/11122 Netapp (Docker volume plugins for SolidFire+ONTAP+E-series) AWS EBS/EFS

ferrantim 10y ago

I'm biased but checkout Portworx. It is designed to be that and if you don't think it is, I'd be curious why is missing. https://portworx.com https://portworx.com

justinsaccount 10y ago

https://github.com/CrunchyData/postgres-operator https://github.com/CrunchyData/postgres-operator is the most promising thing I've seen lately.