2 ms·
2 others, and yes, everyone has access to the central git repo and to our shared private Docker repository. And no, we use TensorFlow and Chainer, but both are
by fxtentacle 6y ago
2 others, and yes, everyone has access to the central git repo and to our shared private Docker repository.
And no, we use TensorFlow and Chainer, but both are python frameworks. Plus Numpy, of course.
I like that KubeFlow has an introduction video, but I find it quite odd, too. They talk a lot about how they will make things simpler, but then I learn that I'll need to run Kubernetes on my laptop, my servers, and potentially the cloud.
Plus they use irritating marketing phrases like "you can just focus on your model" or "let KubeFlow handle the abstraction of running on X". Most deep learning models nowadays are memory-limited, so improving the model may well mean optimizing GPU memory usage. And after trying to port a working model from Ubuntu + 1080 TI to Google Cloud + V100 and/or Google Cloud TPU, I distrust anyone who would treat those significant hardware differences as "just an abstraction".
So at the very least, I'll need to enforce consistent GPUs and Operating Systems among all servers, just to make things run OK everywhere.
- streetcat1 6y agoSo ML ops is just part of a solution, which I think is the reason why it is a hard to sell. What I think might work is auto ml combined with models ops, or rather auto model ops. Or even better: auto data managmenet -> auto pre-processes -> auto ml -> auto ops.
- fxtentacle 6y agoI think a service that rents out GPU-equipped bare metal Ubuntu + Docker servers by the hour would be what I'd want to use. If they also offer a private Docker repo, fast S3-compatible storage and some pre-built images with Jupyter preinstalled, that might be the entire ML pipeline that I need.