3 ms·

Open source ones? Or did you mean internally within Google? Kubernetes is growing in popularity. Among older systems, Mesos (with Marathon, usually) seems to b

Open source ones? Or did you mean internally within Google? Kubernetes is growing in popularity. Among older systems, Mesos (with Marathon, usually) seems to be quite popular.

Cyph0n 10y ago
No, in general. Is there a solution that is actually well tested in production? I can go through the marketing spiel claiming that the solution in question is "scalable", but I'd rather see that in practice.
- lobster_johnson 10y ago
  Kubernetes is definitely being used at scale in production. Spotify uses Helios [1] to run their stuff. We evaluated it briefly and decided it was too limited and immature compared to Kubernetes. Mesos + a framework such as Marathon or Aurora was the most needy choice before Kubernetes came on the scene. Mesos probably scales farther than Kubernetes in terms of pure cluster sizes, but it also depends on what framework you use on top (Mesos itself is just a scheduler). I don't know if any of them are as flexible as Kubernetes in terms of things like volume management, config/secret management and security. It's also worth pointing out that Kubernetes can run on Mesos. [1] https://github.com/spotify/helios https://github.com/spotify/helios
  - bogomipz 10y ago
    >"Mesos + a framework such as Marathon or Aurora was the most needy choice before Kubernetes came on the scene." How is Mesos "needy"? Can you elaborate? Needy has a negative connotation and it's not word I would necessarily associate with the Apache Mesos project. I've run a number of clusters in production now for just over a year with Marathon and it pretty much "just works." I have done 5 or 6 rolling upgrades now without issues. I haven't found it to be needy at all, quite the opposite, its been rock solid and the management overhead has been nominal. >" I don't know if any of them are as flexible as Kubernetes in terms of things like volume management, config/secret management and security." I think that Mesos is actually more flexible as it allows you to cherry pick the non-scheduler specific components to fit your use case. As an example for secrets management you can use something like Consul Vault or integrate Keywhiz or completely roll your own. I feel like with Kubenetes you buy into the "whole thing". Using the example of secret management - you have one choice for secret management and the last time I checked they were stored in etcd in clear text. So if that doesn't fit your security requirements it seemed like you were out of luck. Mesos also has a nice for for persistent volumes: See: http://mesos.apache.org/documentation/latest/persistent-volume/ http://mesos.apache.org/documentation/latest/persistent-volu... and: http://schd.ws/hosted_files/mesosconeu2016/08/MesosConEurope2016PPVv1.0.pdf http://schd.ws/hosted_files/mesosconeu2016/08/MesosConEurope...
    - lobster_johnson 10y ago
      > How is Mesos "needy" Sorry, that was the damn autocorrect — I typed "beefy". As in large and powerful. I'm not surprised if Mesos has solutions for what you describe. When I said "I don't know", I meant it literally! :-) That said, Kubernetes provides nice built-ins, but doesn't force you to use them. You can use Vault for secrets management, for example.
- alecthomas 10y ago
  Mesos was partially developed at Twitter, and was well on the way to powering most of their infrastructure when I left 3 years ago.
- twic 10y ago
  The container scheduling bit of Cloud Foundry (Diego aka Elastic Runtime) is used to run Pivotal Web Services, which is a public cloud thing that is reasonably big, although I don't know the actual numbers. Baidu run some big instances in-house, but again, I don't know how big.

Open source ones? Or did you mean internally within Google? Kubernetes is growing in popularity. Among older systems, Mesos (with Marathon, usually) seems to be quite popular.

Cyph0n 10y ago

No, in general. Is there a solution that is actually well tested in production? I can go through the marketing spiel claiming that the solution in question is "scalable", but I'd rather see that in practice.

lobster_johnson 10y ago
Kubernetes is definitely being used at scale in production. Spotify uses Helios [1] to run their stuff. We evaluated it briefly and decided it was too limited and immature compared to Kubernetes. Mesos + a framework such as Marathon or Aurora was the most needy choice before Kubernetes came on the scene. Mesos probably scales farther than Kubernetes in terms of pure cluster sizes, but it also depends on what framework you use on top (Mesos itself is just a scheduler). I don't know if any of them are as flexible as Kubernetes in terms of things like volume management, config/secret management and security. It's also worth pointing out that Kubernetes can run on Mesos. [1] https://github.com/spotify/helios https://github.com/spotify/helios
- bogomipz 10y ago
  >"Mesos + a framework such as Marathon or Aurora was the most needy choice before Kubernetes came on the scene." How is Mesos "needy"? Can you elaborate? Needy has a negative connotation and it's not word I would necessarily associate with the Apache Mesos project. I've run a number of clusters in production now for just over a year with Marathon and it pretty much "just works." I have done 5 or 6 rolling upgrades now without issues. I haven't found it to be needy at all, quite the opposite, its been rock solid and the management overhead has been nominal. >" I don't know if any of them are as flexible as Kubernetes in terms of things like volume management, config/secret management and security." I think that Mesos is actually more flexible as it allows you to cherry pick the non-scheduler specific components to fit your use case. As an example for secrets management you can use something like Consul Vault or integrate Keywhiz or completely roll your own. I feel like with Kubenetes you buy into the "whole thing". Using the example of secret management - you have one choice for secret management and the last time I checked they were stored in etcd in clear text. So if that doesn't fit your security requirements it seemed like you were out of luck. Mesos also has a nice for for persistent volumes: See: http://mesos.apache.org/documentation/latest/persistent-volume/ http://mesos.apache.org/documentation/latest/persistent-volu... and: http://schd.ws/hosted_files/mesosconeu2016/08/MesosConEurope2016PPVv1.0.pdf http://schd.ws/hosted_files/mesosconeu2016/08/MesosConEurope...
  - lobster_johnson 10y ago
    > How is Mesos "needy" Sorry, that was the damn autocorrect — I typed "beefy". As in large and powerful. I'm not surprised if Mesos has solutions for what you describe. When I said "I don't know", I meant it literally! :-) That said, Kubernetes provides nice built-ins, but doesn't force you to use them. You can use Vault for secrets management, for example.
alecthomas 10y ago
Mesos was partially developed at Twitter, and was well on the way to powering most of their infrastructure when I left 3 years ago.
twic 10y ago
The container scheduling bit of Cloud Foundry (Diego aka Elastic Runtime) is used to run Pivotal Web Services, which is a public cloud thing that is reasonably big, although I don't know the actual numbers. Baidu run some big instances in-house, but again, I don't know how big.