6 ms·

People are working on things on-top of Kubernetes, such as https://github.com/CrunchyData/postgres-operator https://github.com/CrunchyData/postgres-operator

by philips 10y ago

cookiecaper 10y ago
I hate this. Kubernetes is designed specifically for stateless applications; everything about running stateful applications on it now is experimental. Running k8s/Docker in prod is hard enough for actually stateless apps, running a database in it is an utterly horrifying idea. Please limit the extent of your lemming-ness to at least not attempting to containerize PostgreSQL or other old-school database systems. They could not be less suited to this type of application.
- elvinyung 10y ago
  > Kubernetes is designed specifically for stateless applications Is that actually true? The Borg paper[1] mentions that Google uses it to run stateful things like Bigtable and GFS nodes, and as far as I can tell, Kubernetes has the exact same goals as Borg.[2] I would love to see a source for your claim. [1] https://pdos.csail.mit.edu/6.824/papers/borg.pdf https://pdos.csail.mit.edu/6.824/papers/borg.pdf [2] http://queue.acm.org/detail.cfm?id=2898444 http://queue.acm.org/detail.cfm?id=2898444
  - threeseed 10y ago
    "Omega, an offspring of Borg stored the state of the cluster in a centralized Paxos-based transaction-oriented store that was accessed by the different parts of the cluster control plane (such as schedulers), using optimistic concurrency control to handle the occasional conflicts" Seems like Google never stored state within the containers.
    - elvinyung 10y ago
      I think that's a description of how the cluster metadata is managed, and not a description of the actual cluster tasks.
    - deleted 10y ago
      [deleted]
    - Xorlev 10y ago
      Omega isn't what's used AFAIK. In any case, the actual scheduler data is not the state in question.
  - cookiecaper 10y ago
    I don't think there's a direct quote of k8s devs saying "this is designed for stateless applications" (although there very well may be), but it's pretty evident if you're following. Blog post on the official site, about 9 months old, whose title expresses surprise that k8s may be reasonable for stateful applications. The title is "Stateful Applications in Containers!? Kubernetes 1.3 Says “Yes!”" [0] k8s's "StatefulSets", which emerged in 1.5 (4 months ago, Dec 2016) and were transitioned from the previous feature "PetSets", are still marked a "beta" feature in the newest release (1.6, made a couple of weeks ago). [1] In Nov. 2016, CoreOS issued a blog post with the subheading "Stateless is Easy, Stateful is Hard" while introducing a new pattern called "operators". Operators are intended to help Kubernetes better handle stateful applications (since the habit of k8s admins is to terminate pods casually). These are complicated to implement and haven't been widely adopted; basically, you register a third-party resource type, tell your operator to listen in for pod deletions, and hijacks these commands to ensure that everything is shut down in the correct order and with ample opportunity to save. [2] ----- This is all just the k8s level -- let's not forget that the container engine underlying k8s will delete all changes made to the image by default, and has similar weird issues with stateful applications. Accidentally stop the wrong container on the docker cli? You have quite the problem now. And on top of ALL THAT, let's also not forget that these databases are designed to sit on whole boxes and consume almost all of the memory, have kernel-level parameters tweaked to their liking, and so forth. Though databases can co-habitate if forced, production deployments almost always have these guys with dedicated hardware, for various reasons related to performance, reliability, and data safety. Oh, just use Rook, yet another distributed filesystem, you say? They have a nice k8s operator? Please no. What's the reason to risk what is presumably important data this way? Just so you can say "Hello LITERALLY EVERYTHING runs in k8s now, can I have an award?" You wanna do it for dev data or something that doesn't matter, go nuts. But please do not do this for something important. The thought that there are people out there doing this to themselves right now, all to gratify their own vanity by being part of the fad despite their obligations to their colleagues and customers, is so depressing. [0] http://blog.kubernetes.io/2016/07/stateful-applications-in-containers-kubernetes.html http://blog.kubernetes.io/2016/07/stateful-applications-in-c... [1] https://github.com/kubernetes/features/issues/137#issuecomment-277368629 https://github.com/kubernetes/features/issues/137#issuecomme... [2] https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html
- Zikes 10y ago
  Kubernetes is designed specifically to abstract the clustered nature of multiple machines into something that's singly addressable. The upcoming addition of federation is intended to add another abstraction on top of that, allowing the addressing of multiple clusters as a single cluster. The primitives currently included in Kubernetes are mostly geared towards scheduling containerized services onto multiple machines in order to achieve a desired state, but it's only that containerization that has the stateless/ephemeral connotation. Kubernetes is perfectly capable of handling stateful storage and it's as simple as mounting a storage volume: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes https://kubernetes.io/docs/concepts/storage/persistent-volum...
  - cookiecaper 10y ago
    Although not immediately destroying data on exit is important, stateful applications have other important requirements. StatefulSets are the Kubernetes feature intended to address some of them, but it is not yet marked stable.
- linkmotif 10y ago
  Huh? Why would you say something like this? What about StatefulSets and all the related resources?
  - cookiecaper 10y ago
    These are marked beta, and they don't address all of the needs of stateful applications. This blog post [0] from the official k8s blog covers many of the caveats well, and basically describes databases as the anti-use case with statements like: >What is the benefit you hope to gain by running your application in a StatefulSet? [...] If you have a few instances of your storage application, and they are successfully meeting the demands of your organization, and those demands are not rapidly increasing, you’re already at a local optimum. [0] http://blog.kubernetes.io/2016/12/statefulset-run-scale-stateful-applications-in-kubernetes.html http://blog.kubernetes.io/2016/12/statefulset-run-scale-stat...
    - linkmotif 10y ago
      They've been out of beta a few months now and are well supported in GCE. You can't have an application without state, and the k8s team recognizes this. Anyway, OP is about MySQL I guess :)
      - cookiecaper 10y ago
        They're not out of beta. See this comment [0] on the GitHub issue titled "StatefulSets", which was made on February 3, 2017, and says: >This is beta. May have minor changes in 1.6 but not going GA. The latest release of k8s is 1.6, made on March 28, 2017, about two weeks ago. [1] Furthermore, the issue still bears the "stage/beta" tag. [0] https://github.com/kubernetes/features/issues/137#issuecomment-277368629 https://github.com/kubernetes/features/issues/137#issuecomme... [1] https://github.com/kubernetes/kubernetes/releases/tag/v1.6.0 https://github.com/kubernetes/kubernetes/releases/tag/v1.6.0
        linkmotif 10y ago
        Oh my bad, then. Thanks for the links.
- gtirloni 10y ago
  If you allow me to deviate from the main topic, I'd be curious to understand why you chose to use Kubernetes.

People are working on things on-top of Kubernetes, such as https://github.com/CrunchyData/postgres-operator https://github.com/CrunchyData/postgres-operator

by philips 10y ago

People are working on things on-top of Kubernetes, such as https://github.com/CrunchyData/postgres-operator https://github.com/CrunchyData/postgres-operator

cookiecaper 10y ago

I hate this. Kubernetes is designed specifically for stateless applications; everything about running stateful applications on it now is experimental. Running k8s/Docker in prod is hard enough for actually stateless apps, running a database in it is an utterly horrifying idea. Please limit the extent of your lemming-ness to at least not attempting to containerize PostgreSQL or other old-school database systems. They could not be less suited to this type of application.

elvinyung 10y ago
> Kubernetes is designed specifically for stateless applications Is that actually true? The Borg paper[1] mentions that Google uses it to run stateful things like Bigtable and GFS nodes, and as far as I can tell, Kubernetes has the exact same goals as Borg.[2] I would love to see a source for your claim. [1] https://pdos.csail.mit.edu/6.824/papers/borg.pdf https://pdos.csail.mit.edu/6.824/papers/borg.pdf [2] http://queue.acm.org/detail.cfm?id=2898444 http://queue.acm.org/detail.cfm?id=2898444
- threeseed 10y ago
  "Omega, an offspring of Borg stored the state of the cluster in a centralized Paxos-based transaction-oriented store that was accessed by the different parts of the cluster control plane (such as schedulers), using optimistic concurrency control to handle the occasional conflicts" Seems like Google never stored state within the containers.
  - elvinyung 10y ago
    I think that's a description of how the cluster metadata is managed, and not a description of the actual cluster tasks.
  - deleted 10y ago
    [deleted]
  - Xorlev 10y ago
    Omega isn't what's used AFAIK. In any case, the actual scheduler data is not the state in question.
- cookiecaper 10y ago
  I don't think there's a direct quote of k8s devs saying "this is designed for stateless applications" (although there very well may be), but it's pretty evident if you're following. Blog post on the official site, about 9 months old, whose title expresses surprise that k8s may be reasonable for stateful applications. The title is "Stateful Applications in Containers!? Kubernetes 1.3 Says “Yes!”" [0] k8s's "StatefulSets", which emerged in 1.5 (4 months ago, Dec 2016) and were transitioned from the previous feature "PetSets", are still marked a "beta" feature in the newest release (1.6, made a couple of weeks ago). [1] In Nov. 2016, CoreOS issued a blog post with the subheading "Stateless is Easy, Stateful is Hard" while introducing a new pattern called "operators". Operators are intended to help Kubernetes better handle stateful applications (since the habit of k8s admins is to terminate pods casually). These are complicated to implement and haven't been widely adopted; basically, you register a third-party resource type, tell your operator to listen in for pod deletions, and hijacks these commands to ensure that everything is shut down in the correct order and with ample opportunity to save. [2] ----- This is all just the k8s level -- let's not forget that the container engine underlying k8s will delete all changes made to the image by default, and has similar weird issues with stateful applications. Accidentally stop the wrong container on the docker cli? You have quite the problem now. And on top of ALL THAT, let's also not forget that these databases are designed to sit on whole boxes and consume almost all of the memory, have kernel-level parameters tweaked to their liking, and so forth. Though databases can co-habitate if forced, production deployments almost always have these guys with dedicated hardware, for various reasons related to performance, reliability, and data safety. Oh, just use Rook, yet another distributed filesystem, you say? They have a nice k8s operator? Please no. What's the reason to risk what is presumably important data this way? Just so you can say "Hello LITERALLY EVERYTHING runs in k8s now, can I have an award?" You wanna do it for dev data or something that doesn't matter, go nuts. But please do not do this for something important. The thought that there are people out there doing this to themselves right now, all to gratify their own vanity by being part of the fad despite their obligations to their colleagues and customers, is so depressing. [0] http://blog.kubernetes.io/2016/07/stateful-applications-in-containers-kubernetes.html http://blog.kubernetes.io/2016/07/stateful-applications-in-c... [1] https://github.com/kubernetes/features/issues/137#issuecomment-277368629 https://github.com/kubernetes/features/issues/137#issuecomme... [2] https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html
Zikes 10y ago
Kubernetes is designed specifically to abstract the clustered nature of multiple machines into something that's singly addressable. The upcoming addition of federation is intended to add another abstraction on top of that, allowing the addressing of multiple clusters as a single cluster. The primitives currently included in Kubernetes are mostly geared towards scheduling containerized services onto multiple machines in order to achieve a desired state, but it's only that containerization that has the stateless/ephemeral connotation. Kubernetes is perfectly capable of handling stateful storage and it's as simple as mounting a storage volume: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes https://kubernetes.io/docs/concepts/storage/persistent-volum...
- cookiecaper 10y ago
  Although not immediately destroying data on exit is important, stateful applications have other important requirements. StatefulSets are the Kubernetes feature intended to address some of them, but it is not yet marked stable.
linkmotif 10y ago
Huh? Why would you say something like this? What about StatefulSets and all the related resources?
- cookiecaper 10y ago
  These are marked beta, and they don't address all of the needs of stateful applications. This blog post [0] from the official k8s blog covers many of the caveats well, and basically describes databases as the anti-use case with statements like: >What is the benefit you hope to gain by running your application in a StatefulSet? [...] If you have a few instances of your storage application, and they are successfully meeting the demands of your organization, and those demands are not rapidly increasing, you’re already at a local optimum. [0] http://blog.kubernetes.io/2016/12/statefulset-run-scale-stateful-applications-in-kubernetes.html http://blog.kubernetes.io/2016/12/statefulset-run-scale-stat...
  - linkmotif 10y ago
    They've been out of beta a few months now and are well supported in GCE. You can't have an application without state, and the k8s team recognizes this. Anyway, OP is about MySQL I guess :)
    - cookiecaper 10y ago
      They're not out of beta. See this comment [0] on the GitHub issue titled "StatefulSets", which was made on February 3, 2017, and says: >This is beta. May have minor changes in 1.6 but not going GA. The latest release of k8s is 1.6, made on March 28, 2017, about two weeks ago. [1] Furthermore, the issue still bears the "stage/beta" tag. [0] https://github.com/kubernetes/features/issues/137#issuecomment-277368629 https://github.com/kubernetes/features/issues/137#issuecomme... [1] https://github.com/kubernetes/kubernetes/releases/tag/v1.6.0 https://github.com/kubernetes/kubernetes/releases/tag/v1.6.0
      - linkmotif 10y ago
        Oh my bad, then. Thanks for the links.
gtirloni 10y ago
If you allow me to deviate from the main topic, I'd be curious to understand why you chose to use Kubernetes.