7 ms·

The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless y

The author is correct. We should be running stateful containers, but until the the software is ready the default advice quite rightly should be not to unless you understand the implications. For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's definitely good enough for running the likes of Jira or Gitlab have single points of failure pretty much whatever way you scale them (unless you build a HA nfs server first). Before that, we were running test clusters with just fleet, using custom bash and etcd for service discovery. Works fine, but you're probably better off with frameworks. Now where things can go horribly wrong is you decide to use a clustered database, run it in kubernetes, thinking it's fine since you can't mount external storage for replica's but you have 3 nodes over 3 zones and you have backups. Kubernetes will quite happily schedule all of your replica's on the same host, meaning a single reboot and you're dead. There are hacks around this such as defining a hostPort, but of course if someone updates the RC you could happily be restoring to backup. You could run two multiple replica controllers to get around this and that'll allow you to mount storage again but that takes a way a lot of the elegance of the framework. Point is, you can do it, but you'll need a bunch of uglyness and you'll need to be careful. If that doesn't sound great, way until the software properly supports it. :)

rco8786 11y ago
Does kubernetes not have a mechanism for defining host/rack diversity constraints?
- kalmar 11y ago
  I'm not sure of the current status, but I believe the mechanism would be to label nodes with rack information. You'd then be able to specify anti-affinity constraints based on those labels that would be taken into account by the scheduler. Edit: browsing issues briefly, I don't think this functionality is available just yet. Discussion of node selectors: https://github.com/kubernetes/kubernetes/issues/341#issuecomment-154663349 https://github.com/kubernetes/kubernetes/issues/341#issuecom... -
- cpitman 11y ago
  I know that OpenShift 3, built on kubernetes, supports both affinity and anti-affinity policies (https://access.redhat.com/documentation/en/openshift-enterprise/version-3.0/openshift-enterprise-30-administrator-guide/chapter-9-scheduler https://access.redhat.com/documentation/en/openshift-enterpr...). I don't know if these are part of kubernetes or part of the extensions custom to OpenShift.
  - davidopp_ 11y ago
    All of those scheduling policies are also available in native Kubernetes. The Kubernetes version of that documentation is here: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler_algorithm.md https://github.com/kubernetes/kubernetes/blob/master/docs/de...
- ownagefool 11y ago
  The scheduler is just best effort at the moment there's some ways around it. You could indeed create seperate replication-controllers against labels, such as your aws-zone, and run them seperatly. This flies in the face of having replication-controllers and sucks in my opinion. The other option is a single replication-controller and have your containers use a resouce that can't be reused, thus ensuring you're scheduled elsewhere. This is hacky. I suspect it'll be another year before kubernetes is all singing and dancing, but I do like using it.
Lennie 11y ago
There are still a whole lots of pieces of the puzzle missing. I think it's all about creating ready to use containers which are running on different hosts which can replicate their data between themselves and handle failovers (they need to connect to etcd or zookeeper or consul for that). For PostgreSQL it seem to have already solved (?) 2 times by different people: https://github.com/zalando/patroni https://github.com/zalando/patroni https://flynn.io/ https://flynn.io/ But we don't only need it for PostgreSQL. You need ready to use data-replicating Docker images for storing/indexing log files. And for statistics. Probably ready to use containers for replicated/clustered Redis. We need something for a replicated S3-like service for storing static files. Which can be used for different things like: - let's say you are running a website with Wordpress it can be configured to put it's files in a 'CDN'. So now whenever you you deploy Wordpress you point reverse proxy with caching (like Varnish) at your S3-like service. A S3-like service could also be used to hold your own Docker images. You could put a Docker registry in front of it to push and pull to. We need something, maybe also S3-type service, for storing the files of your local git repositories (which holds the sources of your Docker-images). As far as I know in the ecosystem there is still a scheduler missing which can deploy these containers on the right hosts. And there is also no standard API for starting new containers or whole machines when the other containers (or some monitoring tool) in a cluster noticed one is missing. These things take time. Lots of time. :-(
- Titanous 11y ago
  Flynn co-founder here. We plan to expand to support many more popular open source databases next year (and expose a framework for others to do the same). The goal is that a supported database should "just work" on the platform and be sanely configured out of the box with high availability, automatic failover/recovery, encrypted streaming backups, app credential provisioning, etc. Our Postgres appliance is unique in that it is explicitly designed to not lose data in the face of failure. Simply wrapping up a database in a container with leader election and replication is not enough, as there are many pitfalls that can cause data to be lost during failures. Of course we are limited by the guarantees the database can provide, so some datastore appliances we build may have clearly defined caveats. Storing binary blobs is another tricky matter, and we're exploring what we can do with little to no configuration. We currently store app images and git repos in Postgres, which works but will only scale so far. We are also improving our scheduler so that it is capable of providing the constraints necessary to place stateful services properly.
josephjacks 11y ago
Take a look at the new DaemonSet abstraction Kubernetes. Great for running things like HDFS and other distributed data workloads/frameworks: https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/design/daemon.md https://github.com/kubernetes/kubernetes/blob/release-1.1/do...
- Lennie 11y ago
  Looks like Kubernetes is improving. Hadn't seen this yet. Sometimes it gets hard to follow everything in the Docker ecosystem.
- ownagefool 11y ago
  Thanks, I wasn't aware of that.
josephjacks 11y ago
Take a look at the new DaemonSet abstraction Kubernetes. Great for running things like HDFS and other distributed data workloads/frameworks: https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/design/daemon.md https://github.com/kubernetes/kubernetes/blob/release-1.1/do...
TheIronYuppie 11y ago
Disclaimer: I work at Google on Kubernetes Yep - and the ability to run just one per node is coming "soon" (ideally by next milestone). It's something we wanted to get in, but just didn't have the time by 1.1. My #1 recommendation is to do whatever you do today. If you mount an NFS mount into your VM to store your MySQL data, that's what you should do again. The speed at restarting a pod is very low - but you do need to do the (small) hack today if you want to be absolutely sure you don't have multiple pods with on the same node. To be clear, the ugly-ness is one line in your config file: nodePort: <some arbitrary number >1024 & <65536>

rco8786 11y ago

Does kubernetes not have a mechanism for defining host/rack diversity constraints?

kalmar 11y ago
I'm not sure of the current status, but I believe the mechanism would be to label nodes with rack information. You'd then be able to specify anti-affinity constraints based on those labels that would be taken into account by the scheduler. Edit: browsing issues briefly, I don't think this functionality is available just yet. Discussion of node selectors: https://github.com/kubernetes/kubernetes/issues/341#issuecomment-154663349 https://github.com/kubernetes/kubernetes/issues/341#issuecom... -
cpitman 11y ago
I know that OpenShift 3, built on kubernetes, supports both affinity and anti-affinity policies (https://access.redhat.com/documentation/en/openshift-enterprise/version-3.0/openshift-enterprise-30-administrator-guide/chapter-9-scheduler https://access.redhat.com/documentation/en/openshift-enterpr...). I don't know if these are part of kubernetes or part of the extensions custom to OpenShift.
- davidopp_ 11y ago
  All of those scheduling policies are also available in native Kubernetes. The Kubernetes version of that documentation is here: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler_algorithm.md https://github.com/kubernetes/kubernetes/blob/master/docs/de...
ownagefool 11y ago
The scheduler is just best effort at the moment there's some ways around it. You could indeed create seperate replication-controllers against labels, such as your aws-zone, and run them seperatly. This flies in the face of having replication-controllers and sucks in my opinion. The other option is a single replication-controller and have your containers use a resouce that can't be reused, thus ensuring you're scheduled elsewhere. This is hacky. I suspect it'll be another year before kubernetes is all singing and dancing, but I do like using it.

Lennie 11y ago

There are still a whole lots of pieces of the puzzle missing. I think it's all about creating ready to use containers which are running on different hosts which can replicate their data between themselves and handle failovers (they need to connect to etcd or zookeeper or consul for that). For PostgreSQL it seem to have already solved (?) 2 times by different people: https://github.com/zalando/patroni https://github.com/zalando/patroni https://flynn.io/ https://flynn.io/ But we don't only need it for PostgreSQL. You need ready to use data-replicating Docker images for storing/indexing log files. And for statistics. Probably ready to use containers for replicated/clustered Redis. We need something for a replicated S3-like service for storing static files. Which can be used for different things like: - let's say you are running a website with Wordpress it can be configured to put it's files in a 'CDN'. So now whenever you you deploy Wordpress you point reverse proxy with caching (like Varnish) at your S3-like service. A S3-like service could also be used to hold your own Docker images. You could put a Docker registry in front of it to push and pull to. We need something, maybe also S3-type service, for storing the files of your local git repositories (which holds the sources of your Docker-images). As far as I know in the ecosystem there is still a scheduler missing which can deploy these containers on the right hosts. And there is also no standard API for starting new containers or whole machines when the other containers (or some monitoring tool) in a cluster noticed one is missing. These things take time. Lots of time. :-(

Titanous 11y ago
Flynn co-founder here. We plan to expand to support many more popular open source databases next year (and expose a framework for others to do the same). The goal is that a supported database should "just work" on the platform and be sanely configured out of the box with high availability, automatic failover/recovery, encrypted streaming backups, app credential provisioning, etc. Our Postgres appliance is unique in that it is explicitly designed to not lose data in the face of failure. Simply wrapping up a database in a container with leader election and replication is not enough, as there are many pitfalls that can cause data to be lost during failures. Of course we are limited by the guarantees the database can provide, so some datastore appliances we build may have clearly defined caveats. Storing binary blobs is another tricky matter, and we're exploring what we can do with little to no configuration. We currently store app images and git repos in Postgres, which works but will only scale so far. We are also improving our scheduler so that it is capable of providing the constraints necessary to place stateful services properly.

josephjacks 11y ago

Take a look at the new DaemonSet abstraction Kubernetes. Great for running things like HDFS and other distributed data workloads/frameworks: https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/design/daemon.md https://github.com/kubernetes/kubernetes/blob/release-1.1/do...

Lennie 11y ago
Looks like Kubernetes is improving. Hadn't seen this yet. Sometimes it gets hard to follow everything in the Docker ecosystem.
ownagefool 11y ago
Thanks, I wasn't aware of that.

josephjacks 11y ago

TheIronYuppie 11y ago

Disclaimer: I work at Google on Kubernetes Yep - and the ability to run just one per node is coming "soon" (ideally by next milestone). It's something we wanted to get in, but just didn't have the time by 1.1. My #1 recommendation is to do whatever you do today. If you mount an NFS mount into your VM to store your MySQL data, that's what you should do again. The speed at restarting a pod is very low - but you do need to do the (small) hack today if you want to be absolutely sure you don't have multiple pods with on the same node. To be clear, the ugly-ness is one line in your config file: nodePort: <some arbitrary number >1024 & <65536>