3 ms·

Congrats to the Kubeflow team. This is a really exciting new project for everyone trying to do ML on Kubeflow. I'm excited to see what this will mean for the Ku

Congrats to the Kubeflow team. This is a really exciting new project for everyone trying to do ML on Kubeflow. I'm excited to see what this will mean for the Kubernetes in general given the close contact between the Kubeflow team and the Kubernetes team. I'd love to see some more data centric primitives coming out in new K8s releases.

TheIronYuppie 9y ago
Interesting point! Anything in particular? Specifically, we love the extensibility of the Kubernetes CRD and that has given us a lot of what we were looking for, and Hardware Acceleration as a schedulable resource is due to land very soon. But do tell! Disclosure: I work at Google on Kubernetes
- jdoliner 9y ago
  I have a lot of ideas about this. I'm one of the creators of Pachyderm, which does data pipelining on Kubernetes. The first concrete thing that I think K8s could have would be some sort of a notion of a DataSet. Right now the closest thing I think is a gitRepo volume. GitRepo volumes are different from most other k8s volumes in that they're not meant for persisting state, they're meant as a way to inject data (which might actually be code) into a k8s environment. I'd like to see that idea expanded, I could easily imagine a similar thing that allowed you to mount data from Object stores like s3 and GCS, that would do a lot to make K8s immediately useful for data workflows. Being able to output to volumes like this and pass them between various pods would be the next logical step.
  - TheIronYuppie 9y ago
    This is an amazing idea! Have you thought about exploring to see if we can overload CSI to accomplish something like this? https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md https://github.com/kubernetes/community/blob/master/contribu...

3 ms·

Congrats to the Kubeflow team. This is a really exciting new project for everyone trying to do ML on Kubeflow. I'm excited to see what this will mean for the Ku

by jdoliner 9y ago

TheIronYuppie 9y ago
Interesting point! Anything in particular? Specifically, we love the extensibility of the Kubernetes CRD and that has given us a lot of what we were looking for, and Hardware Acceleration as a schedulable resource is due to land very soon. But do tell! Disclosure: I work at Google on Kubernetes
- jdoliner 9y ago
  I have a lot of ideas about this. I'm one of the creators of Pachyderm, which does data pipelining on Kubernetes. The first concrete thing that I think K8s could have would be some sort of a notion of a DataSet. Right now the closest thing I think is a gitRepo volume. GitRepo volumes are different from most other k8s volumes in that they're not meant for persisting state, they're meant as a way to inject data (which might actually be code) into a k8s environment. I'd like to see that idea expanded, I could easily imagine a similar thing that allowed you to mount data from Object stores like s3 and GCS, that would do a lot to make K8s immediately useful for data workflows. Being able to output to volumes like this and pass them between various pods would be the next logical step.
  - TheIronYuppie 9y ago
    This is an amazing idea! Have you thought about exploring to see if we can overload CSI to accomplish something like this? https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md https://github.com/kubernetes/community/blob/master/contribu...