5 ms·

It can be called: "How we built 'x' with Kubernetes". Really the only thing that is specific to a bank (as I see it) is that they use separate linkerd in order

It can be called: "How we built 'x' with Kubernetes". Really the only thing that is specific to a bank (as I see it) is that they use separate linkerd in order to do the secure stuff. Which is essentially what banks have been doing for ages. I commented before on how Kube has just taken over and beat mesos/marathon stack. This talk is an example to that. You can see how many people jumped on the Kube stack and running successful deployments on it.

graffitici 10y ago
But what about "Big Data" workloads? Running Spark or Cassandra clusters say? My understanding is that having a custom scheduler makes mesos more attractive for those tasks? Has your experience been different?
- avitzurel 10y ago
  Personally I always ran big data on Amazon stack so never really needed that ability
- agibsonccc 10y ago
  On prem aka "non aws" it does tend to matter yes.
agibsonccc 10y ago
Disclosure: I mainly use DC/OS mesos myself. I've evaluated k8s for our use case and didn't find it was quite what we were looking for. Our customers and stack are mainly JVM based. We do on prem deployments not cloud where GCE is already doing pretty well. We also mainly work with the microsoft side of things (azure,enterprise stuff) Not convinced of this. Direct mesos and yarn integration with spark (not to mention a lot of the software already being built on top) is going to keep mesos/marathon relevant for a long time. It's definitely better for big data workloads. I think a lot of startups will jump to this for sure. Many startups don't actually have big data stacks and prefer to use go based stuff (mainly because it's simpler). In that case k8s makes sense for that. A lot of these companies will likely prefer DC/OS and mesos/marathon because they have in house zookeeper expertise already. ZK is a dependency for much of the big data ecosystem as well a kafka and mesos. The synergy is a lot better. That being said: Many here will disagree. I definitely think k8s is winning developer mindshare overall, but I don't think it will have 100% of the market.
- davidopp__ 10y ago
  We're working on directly-integrated Spark-on-Kubernetes right now and would love to get input from folks who are interested. The Github issue where we're discussing it is here: https://github.com/kubernetes/kubernetes/issues/34377 https://github.com/kubernetes/kubernetes/issues/34377 Comcast has a prototype of YARN on Kubernetes here: https://github.com/Comcast/kube-yarn https://github.com/Comcast/kube-yarn (Disclosure: I work on the Kubernetes project at Google.)
  - agibsonccc 10y ago
    The YARN stuff is interesting. I would be curious to see something akin to: https://spark.apache.org/docs/1.6.1/running-on-mesos.html https://spark.apache.org/docs/1.6.1/running-on-mesos.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn... My main area of interest with this is: We have a lot of Java native interface code we run. I don't want folks to have to worry about configuring library paths and the like. My support loads get messy quick the second "c code" comes in the picture. The ability to hook in to k8s to spin up executors would be pretty neat. The Mesos containerizer has that beat right now. 1 of the main reasons I heavily prefer mesos is also dc/os. dcos package install thing is head and shoulders above just having a docker runtime. As for yarn: A lot of my customers use yarn and don't know what k8s is. OR if they do have a cluster that uses docker, they call it a "docker cluster" and it's often separate. It's not as hard of a sell for me for the YARN folks to just say: "install my docker daemon as an RPM or cloudera parcel so you can run executors" vs: "Install k8s in your docker cluster with all this extra stuff in there" K8s for me is in this limbo of: Not integrated enough or too complex for embedding in commercial use. You guys have a great story going. Especially on the UX side of things. I know a lot of folks in the k8s ecosystem, but many of them aren't focused on big data or the JVM. Mesos on the other hand was spark's parent project. The integration levels show there. I'll be keeping an eye on it :). My inherent problem with a lot of this is it's still prototyping. A lot of this support is still very green field and I'm stoked you guys are working on it. If anything because more competition is always good.
    - 10y ago

graffitici 10y ago

But what about "Big Data" workloads? Running Spark or Cassandra clusters say? My understanding is that having a custom scheduler makes mesos more attractive for those tasks? Has your experience been different?

avitzurel 10y ago
Personally I always ran big data on Amazon stack so never really needed that ability
agibsonccc 10y ago
On prem aka "non aws" it does tend to matter yes.

agibsonccc 10y ago

Disclosure: I mainly use DC/OS mesos myself. I've evaluated k8s for our use case and didn't find it was quite what we were looking for. Our customers and stack are mainly JVM based. We do on prem deployments not cloud where GCE is already doing pretty well. We also mainly work with the microsoft side of things (azure,enterprise stuff) Not convinced of this. Direct mesos and yarn integration with spark (not to mention a lot of the software already being built on top) is going to keep mesos/marathon relevant for a long time. It's definitely better for big data workloads. I think a lot of startups will jump to this for sure. Many startups don't actually have big data stacks and prefer to use go based stuff (mainly because it's simpler). In that case k8s makes sense for that. A lot of these companies will likely prefer DC/OS and mesos/marathon because they have in house zookeeper expertise already. ZK is a dependency for much of the big data ecosystem as well a kafka and mesos. The synergy is a lot better. That being said: Many here will disagree. I definitely think k8s is winning developer mindshare overall, but I don't think it will have 100% of the market.

davidopp__ 10y ago
We're working on directly-integrated Spark-on-Kubernetes right now and would love to get input from folks who are interested. The Github issue where we're discussing it is here: https://github.com/kubernetes/kubernetes/issues/34377 https://github.com/kubernetes/kubernetes/issues/34377 Comcast has a prototype of YARN on Kubernetes here: https://github.com/Comcast/kube-yarn https://github.com/Comcast/kube-yarn (Disclosure: I work on the Kubernetes project at Google.)
- agibsonccc 10y ago
  The YARN stuff is interesting. I would be curious to see something akin to: https://spark.apache.org/docs/1.6.1/running-on-mesos.html https://spark.apache.org/docs/1.6.1/running-on-mesos.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn... My main area of interest with this is: We have a lot of Java native interface code we run. I don't want folks to have to worry about configuring library paths and the like. My support loads get messy quick the second "c code" comes in the picture. The ability to hook in to k8s to spin up executors would be pretty neat. The Mesos containerizer has that beat right now. 1 of the main reasons I heavily prefer mesos is also dc/os. dcos package install thing is head and shoulders above just having a docker runtime. As for yarn: A lot of my customers use yarn and don't know what k8s is. OR if they do have a cluster that uses docker, they call it a "docker cluster" and it's often separate. It's not as hard of a sell for me for the YARN folks to just say: "install my docker daemon as an RPM or cloudera parcel so you can run executors" vs: "Install k8s in your docker cluster with all this extra stuff in there" K8s for me is in this limbo of: Not integrated enough or too complex for embedding in commercial use. You guys have a great story going. Especially on the UX side of things. I know a lot of folks in the k8s ecosystem, but many of them aren't focused on big data or the JVM. Mesos on the other hand was spark's parent project. The integration levels show there. I'll be keeping an eye on it :). My inherent problem with a lot of this is it's still prototyping. A lot of this support is still very green field and I'm stoked you guys are working on it. If anything because more competition is always good.
  - 10y ago