3 ms·

Your comment is worded in a slightly mean way, but I don't disagree. I work on autonomous 3D navigation. Clicking on their "Autonomous Vehicles" solution I see:

Your comment is worded in a slightly mean way, but I don't disagree. I work on autonomous 3D navigation. Clicking on their "Autonomous Vehicles" solution I see: - model management solutions - enormous quantities of data Well, I have docker containers for my data, and docker containers for the underlying TensorFlow versions, and docker containers for model source code that went into large-scale training. All of that is coordinated using a git repo which has some shell scripts to execute the correct model with matching data on a compatible TF image. So if someone says "I need you to re-run last week's model", I check out the script from that time and run it on an empty server. I honestly don't get what MLOps is or why I would need it.

antonvs 6y ago
How many other ML people do you work with? Do they all use the same system? Are you only working with Tensorflow and no other tools? Generally things like MLOps are targeted at providing systems for people that either don't know how, or don't have the time, to set something up themselves, especially when the data science team is more than just one or two people, and when they're using a range of tools that need to interoperate in various ways. Something like Kubeflow, for example, does a lot more than what you describe.
- fxtentacle 6y ago
  2 others, and yes, everyone has access to the central git repo and to our shared private Docker repository. And no, we use TensorFlow and Chainer, but both are python frameworks. Plus Numpy, of course. I like that KubeFlow has an introduction video, but I find it quite odd, too. They talk a lot about how they will make things simpler, but then I learn that I'll need to run Kubernetes on my laptop, my servers, and potentially the cloud. Plus they use irritating marketing phrases like "you can just focus on your model" or "let KubeFlow handle the abstraction of running on X". Most deep learning models nowadays are memory-limited, so improving the model may well mean optimizing GPU memory usage. And after trying to port a working model from Ubuntu + 1080 TI to Google Cloud + V100 and/or Google Cloud TPU, I distrust anyone who would treat those significant hardware differences as "just an abstraction". So at the very least, I'll need to enforce consistent GPUs and Operating Systems among all servers, just to make things run OK everywhere.
  - streetcat1 6y ago
    So ML ops is just part of a solution, which I think is the reason why it is a hard to sell. What I think might work is auto ml combined with models ops, or rather auto model ops. Or even better: auto data managmenet -> auto pre-processes -> auto ml -> auto ops.
    - fxtentacle 6y ago
      I think a service that rents out GPU-equipped bare metal Ubuntu + Docker servers by the hour would be what I'd want to use. If they also offer a private Docker repo, fast S3-compatible storage and some pre-built images with Jupyter preinstalled, that might be the entire ML pipeline that I need.
carlosf 6y ago
MLOps is just a buzzword, but maintaining ML projects in large organizations has several major pains, e.g.: http://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf http://papers.nips.cc/paper/5656-hidden-technical-debt-in-ma... I use MLFlow at my company for solving two specific problems: tracking experiments and model performance. But if you're small enough, you probably don't need MLOps, Devops, etc...
cdavid 6y ago
I would say that you don't need it if you don't feel hindered in your work. But it is solving a real need. Problems arise when you start to have larger teams who interact with models, when the models are applied in production at a regular cadence. Having your model in docker generated from git is a good first step, but does not solve the most important issues when you are at scale: reproducible training, tracking of experiments including data, ML-specific observability for your models in prod, etc. See e.g. https://martinfowler.com/articles/cd4ml.html https://martinfowler.com/articles/cd4ml.html. More concretely, since the above easily sounds like a buzzword soup: 1. Experiment-wise, you don't want just want to track your ML model definition, but also track the data and everything else used to build it, not just use it. A typical thing I have seen at every company I worked at: we have this model but we don't know how to reproduce it because the lost the data, or there was some magic numbers in training that may be on an internal wiki if you're lucky. 2. For some important use cases, you want to iteratively work on improving the model in production. That often means work on the data side, feature engineering, tracking skew prod vs training, etc. the model is not often changed. In almost every case, a useful model is a model that sees 100s if not more iterations in production. You need 1. to do 2. 3. Some of those tools are useful to enforce invariant or detect data issues, which is again very common when you run models for a long time. See e.g. tfdev. But to go back to your point: MLOps is a 2nd order kind of thing. The first order is of course that most ML-related projects are useless, poorly conceived, or even lack any kind of quantitative analysis on the business and/or product. Companies should invest there before MLops IMO.

antonvs 6y ago

How many other ML people do you work with? Do they all use the same system? Are you only working with Tensorflow and no other tools? Generally things like MLOps are targeted at providing systems for people that either don't know how, or don't have the time, to set something up themselves, especially when the data science team is more than just one or two people, and when they're using a range of tools that need to interoperate in various ways. Something like Kubeflow, for example, does a lot more than what you describe.

fxtentacle 6y ago
2 others, and yes, everyone has access to the central git repo and to our shared private Docker repository. And no, we use TensorFlow and Chainer, but both are python frameworks. Plus Numpy, of course. I like that KubeFlow has an introduction video, but I find it quite odd, too. They talk a lot about how they will make things simpler, but then I learn that I'll need to run Kubernetes on my laptop, my servers, and potentially the cloud. Plus they use irritating marketing phrases like "you can just focus on your model" or "let KubeFlow handle the abstraction of running on X". Most deep learning models nowadays are memory-limited, so improving the model may well mean optimizing GPU memory usage. And after trying to port a working model from Ubuntu + 1080 TI to Google Cloud + V100 and/or Google Cloud TPU, I distrust anyone who would treat those significant hardware differences as "just an abstraction". So at the very least, I'll need to enforce consistent GPUs and Operating Systems among all servers, just to make things run OK everywhere.
- streetcat1 6y ago
  So ML ops is just part of a solution, which I think is the reason why it is a hard to sell. What I think might work is auto ml combined with models ops, or rather auto model ops. Or even better: auto data managmenet -> auto pre-processes -> auto ml -> auto ops.
  - fxtentacle 6y ago
    I think a service that rents out GPU-equipped bare metal Ubuntu + Docker servers by the hour would be what I'd want to use. If they also offer a private Docker repo, fast S3-compatible storage and some pre-built images with Jupyter preinstalled, that might be the entire ML pipeline that I need.

carlosf 6y ago

MLOps is just a buzzword, but maintaining ML projects in large organizations has several major pains, e.g.: http://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf http://papers.nips.cc/paper/5656-hidden-technical-debt-in-ma... I use MLFlow at my company for solving two specific problems: tracking experiments and model performance. But if you're small enough, you probably don't need MLOps, Devops, etc...

cdavid 6y ago

I would say that you don't need it if you don't feel hindered in your work. But it is solving a real need. Problems arise when you start to have larger teams who interact with models, when the models are applied in production at a regular cadence. Having your model in docker generated from git is a good first step, but does not solve the most important issues when you are at scale: reproducible training, tracking of experiments including data, ML-specific observability for your models in prod, etc. See e.g. https://martinfowler.com/articles/cd4ml.html https://martinfowler.com/articles/cd4ml.html. More concretely, since the above easily sounds like a buzzword soup: 1. Experiment-wise, you don't want just want to track your ML model definition, but also track the data and everything else used to build it, not just use it. A typical thing I have seen at every company I worked at: we have this model but we don't know how to reproduce it because the lost the data, or there was some magic numbers in training that may be on an internal wiki if you're lucky. 2. For some important use cases, you want to iteratively work on improving the model in production. That often means work on the data side, feature engineering, tracking skew prod vs training, etc. the model is not often changed. In almost every case, a useful model is a model that sees 100s if not more iterations in production. You need 1. to do 2. 3. Some of those tools are useful to enforce invariant or detect data issues, which is again very common when you run models for a long time. See e.g. tfdev. But to go back to your point: MLOps is a 2nd order kind of thing. The first order is of course that most ML-related projects are useless, poorly conceived, or even lack any kind of quantitative analysis on the business and/or product. Companies should invest there before MLops IMO.