7 ms·
Kubeflow – Machine Learning Toolkit for Kubernetes
- smooc 9y agoThis looks pretty cool. Is it dependent on Google’s kubernetes or can it be run on Openshift or DC/OS as well?
- ericand 9y agoIt says it runs in "in any environment in which Kubernetes runs." So as long as you are asking if it runs on Openshift's Kubernetes, than yes.
- TheIronYuppie 9y agoAbsolutely! Redhat are already contributing :) Disclosure: I work at Google on Kubeflow
- Voloskaya 9y agoThe controller part is a Custom Resource Definition. It can run on any cloud. However, to benefit from GPUs you need to configure the controller correctly. Default configuration are there for GCP and Azure, but you would need to do that manually for other cloud (not that it is very hard)
- TheIronYuppie 9y agoCorrect, anytime you bleed through hardware, it requires some setup, sadly Disclosure: I work at Google on Kubeflow
- TheIronYuppie 9y agoYep, anywhere k8s runs! (Or that's the idea anyway). If it doesn't, please file a bug! Disclosure: I work at Google on Kubeflow
- sub-mod 9y agoWith Openshift you can also exploit the S2I feature(This is not based on CRD.)You can refer this Blog on some ways which this can be done for TF. https://blog.openshift.com/openshift-commons-briefing-110-containerizing-tensorflow-applications-on-openshift-with-subin-modeel-red-hat/ https://blog.openshift.com/openshift-commons-briefing-110-co...
- ericand 9y agoLooks like different components could be added in the future, but not clear how. The following are included: - A JupyterHub to create & manage interactive Jupyter notebooks. - A Tensorflow Training Controller that can be configured to use CPUs or GPUs, and adjusted to the size of a cluster with a single setting. - A TF Serving container.
- TheIronYuppie 9y agoCorrect! We're currently thinking a lot about orchestration of the various components but for now, our goal is to use the native loose coupling between services available in K8s. So if you wanted Spark for data processing, for example, you could start a service, and the deployment, and feed that into the TF CRD. Disclosure: I work at Google
- mountainerd 9y agoPretty interesting. I'm guessing this is something that Google uses internally for their Kubernetes workflows.
- nikolay 9y agoI am at KubeCon 2017 in Austin, TX and, yeah, based on the presentation, it looks like an internal tool they just opened to the public with some bold goals.
- TheIronYuppie 9y agoHi! It was actually designed from the start to be an extension of GitHub.com/tensor flow/k8s and then it took on larger goals = Making an entire ML stack (of any ML framework) easy to use, portable and composable on k8s. Disclosure: I work at Google on Kubeflow
- TheIronYuppie 9y agoIts very close to how we think about ML internally, but not what we use. Your best bet to read that is look at the TFX paper[1] which describes our internal thoughts in great detail. (Though Kubeflow is not designed to be an externalization of TFX, we're very much working in collaboration with that team) [1] http://www.kdd.org/kdd2017/papers/view/tfx-a-tensorflow-based-production-scale-machine-learning-platform http://www.kdd.org/kdd2017/papers/view/tfx-a-tensorflow-base... Disclosure: I work at Google on Kubeflow
- quadrature 9y agoDoes google intend on open sourcing TFX ?. I only ask because we're building a lot of the same infrastructure.
- TheIronYuppie 9y agoWe're absolutely looking at it! Please join our discussion, we'd love to talk about what you're building and if we can help and/or what you'd like us to OSS. Disclosure: I work at Google on Kubeflow
- TheIronYuppie 9y agoHi! I’m David Aronchick, PM on Kubeflow, I’m happy to answer any questions! I was one of the early PMs on Kubernetes, and we very much want to make this a community project, so please join us in thinking about what’s next! - GH: https://GitHub.com/Google/Kubeflow https://GitHub.com/Google/Kubeflow - kubeflow-Discuss: https://groups.google.com/forum/m/#!forum/kubeflow-discuss https://groups.google.com/forum/m/#!forum/kubeflow-discuss NOTE: The name "flow" does not refer directly to TensorFlow; if anything it's a nod at all the river themes that pop up in the ML community (eg FBLearner Flow) Disclosure: I work at Google on Kubeflow
- pacala 9y agoGreat to hear this not tied to TensorFlow! How would one use a different DL platform, say PyTorch or DyNet?
- TheIronYuppie 9y agoThe steps would basically be: - Containerize the DL platform - Create a k8s manifest (similar to our CRD if necessary) - Create a service endpoint - integrate all that into the JH deployment This is less hard than it sounds, but we'd love help! We only started with TF because that's what we know. Disclosure: I work at Google on Kubeflow
- minerals 9y agoInteresting, though I don't see how it is better than a plain docker image over kubernetes? Not much of a hassle now too. And how is it different from what DL4J is already doing with Zeppelin and supporting both Keras, TF, MXNet and PyTorch on the way?
- bonesss 9y ago> ...how [is it] better than a plain docker image over kubernetes? Scalability for people with existing on-premise (or cloud based), kubernetes workflows, especially once it comes to training or heavy crunching. That's not to say that Docker Machine/Swarm/Compose couldn't handle the same, but it's an extra step for kubernetes users and pushes people onto a slightly different toolchain than minikube->K8s.
- humanfromearth 9y agoAny chance to make this a helm chart?
- TheIronYuppie 9y agoWe're looking at a bunch of deployment packaging solutions, helm is probably one of the top ones! Disclosure: I work at Google on Kubeflow
- yuvipanda 9y agoChime in at https://github.com/google/kubeflow/issues/23 https://github.com/google/kubeflow/issues/23 :)