4 ms·

Maybe I’m a bit naive, but I’m convinced any great traditional software engineer or devops engineer in combination with a data scientist/ML person should be abl

Maybe I’m a bit naive, but I’m convinced any great traditional software engineer or devops engineer in combination with a data scientist/ML person should be able to setup the ops pipeline for a ML project. The details and algorithms themselves may look new and exciting, but operationalizing algorithms isn’t a totally new thing.

ramoz 5y ago
In my experience, you’re right. A team of software engineers & platform engineers are doing their normal job deploying production grade models at scale. Caveat, this requires deeper or more advanced computer engineering in practice. Devops roles are generally breaching into data engineering more and more. Again, in my case, the devops/platform team creates all the automation to move a lot of data around daily. In terms of the model training, we essentially treat it like a supply chain where the software team is closely engaged with the model team who aren’t even really deep data scientists themselves. Just software engineers focused on building ai/ml models.
indigochill 5y ago
Dunno if this technically counts since it's managed, but I'm a semi devops-y person (had all the buzzwords foisted on me and just learned on the job by making all the mistakes) and guided an ML intern through setting up their ML pipeline in AWS. A few more mistakes from unfamiliarity with AWS's ML-specific options at the time notwithstanding, it was certainly not the most traumatic devops experience I've had. IIRC the only stuff to learn there was (a) PyTorch (I think that was the right ML bit) and (b) the AWS black magic, and the intern ended up picking up both once I got them pointed in the right general direction on the devops end.
rlayton2 5y ago
You are generally right. The data is a big difference. Managing it, it's too big for git etc is a massive problem. ML also has a distinction between the system for training a model and the system that uses it, but that's just two projects to manage
kajecounterhack 5y ago
ML models are unlike traditional algorithms in that most need to be retrained frequently to deal with distribution shifts, and monitoring to ensure your model is still accurately modeling the real world distribution is a key component. If you are someone relying on ML to do a good job / needing to continuously improve your ML models, you end up caring a lot about how much it costs to retrain, how ongoing labeling operations are going & how quickly it goes from newly-labeled data --> retrained model --> evaluated model --> deployed model. Furthermore, ML orgs often have to make labeling and training efficient by sharing resources & strategically triaging only the most impactful experiments / picking the most impactful things to label. There's also the pace of progress in the field, and often organizations will have dedicated data science / ML folks who want to run experiments to improve the models upon each retraining. The infra needed to unlock rapid prototyping is quite a lot. If you have a lot of models and you're re-building the end-to-end stack each time, you end up with a ton of wasted work. A lot of this stuff is also pretty specialized. It's a bit like asking all your engineers to set up their own web server, proxy, monitoring / alerting, maybe a load balancer, etc etc. Plenty of people know how to do it, and for small companies doing 1 or 2 ML models maybe that's fine. But for corporations at scale it makes no sense to do it like that. And when you're at scale, a small team probably works on the load balancers vs the base infra vs the software platform vs the features. ML works the same way. Some real world examples where you can imagine how much infra you need within ML-heavy organizations: - Say you want to build a _motorcycle detector for a self-driving car_. You need to build a data extraction pipeline that processes images, gets the segmented objects, sends each object for labeling, then when you have all your labeled data, you need to split it into test/train/validation (and make sure you use the same splits as everyone else building the car's software), then you need to have these piles of images integrated with additional information needed for training (e.g. how fast the object was moving, what time of day), you need to upsample/downsample certain cases (e.g maybe you need to upsample examples of motorcycles at night), then train a net (locally, or in the cloud, or maybe as part of N experiments to tune, on shared infrastructure that M people are using where jobs need to be prioritized), then evaluate (do you need to build infra to harvest important metrics, like how well your model performs when the car is driving on a slope? in fog?), then optimize for onboard (are you going to run it on CPU? GPU? Accelerator? Do you optimize it using TensorRT? Your own quantization infra? Distillation?), then deploy (and monitor -- is the model eating too much memory during inference? Being run too many times? Not doing the right thing?) Okay your model works -- you sure it will keep working when motorcycles look different 10 years or even just 1 year from now? - Say you want to build a _spam detector for your social media website_. You do everything above, build and deploy your model to the cloud, and suddenly you realize it's not working, a new spam campaign has occurred that your model can't account for. You need to add more labeled data, but how much and where are you going to get it? After adding it, what's your overall data look like? Adding it didn't help your net as much as you expected, why? The model-level eval looked improved, but combined with a rule-system together, it got worse. Crap, how to debug? Okay finally working, how stale is the data in your model after 1 year? Did we regress on something when we solved the spam campaign? You have a computational budget for how big the net can get, because it's used real-time to judge spamminess of posts on a major website -- maybe you care about what hardware your model is running on and how to best optimize for that hardware. Maybe you use cloud TPUs, where large batch sizes help you to scale. Maybe you use graphcore or something that thrives on small batch size. What if you started on one, moved to the other, and suddenly your net isn't working as well? What if you upgrade from a gtx2080ti -> gtx3080ti and see that your net has a prediction regression? Do you have infra to detect these regressions? Over time, when your data got an order of magnitude bigger, you noticed that your net's hyperparams were no longer optimal. You needed to increase your learning rate, or decrease it. Did you notice this issue, and do you have the infra to do that tuning quickly? You notice your labeling budget is too small to label everything flagged as spam. How do you decide which things are most worthwhile to label? You have to build infra for all of this. MLOps are needed every step of the way. It's not that different from needing SREs and cloud infra engineers to run your cloud services & organizations. Labeling infra alone is a big enough market for companies like Hive and Scale to build billion-dollar businesses.

ramoz 5y ago

In my experience, you’re right. A team of software engineers & platform engineers are doing their normal job deploying production grade models at scale. Caveat, this requires deeper or more advanced computer engineering in practice. Devops roles are generally breaching into data engineering more and more. Again, in my case, the devops/platform team creates all the automation to move a lot of data around daily. In terms of the model training, we essentially treat it like a supply chain where the software team is closely engaged with the model team who aren’t even really deep data scientists themselves. Just software engineers focused on building ai/ml models.

indigochill 5y ago

Dunno if this technically counts since it's managed, but I'm a semi devops-y person (had all the buzzwords foisted on me and just learned on the job by making all the mistakes) and guided an ML intern through setting up their ML pipeline in AWS. A few more mistakes from unfamiliarity with AWS's ML-specific options at the time notwithstanding, it was certainly not the most traumatic devops experience I've had. IIRC the only stuff to learn there was (a) PyTorch (I think that was the right ML bit) and (b) the AWS black magic, and the intern ended up picking up both once I got them pointed in the right general direction on the devops end.

rlayton2 5y ago

You are generally right. The data is a big difference. Managing it, it's too big for git etc is a massive problem. ML also has a distinction between the system for training a model and the system that uses it, but that's just two projects to manage

kajecounterhack 5y ago

ML models are unlike traditional algorithms in that most need to be retrained frequently to deal with distribution shifts, and monitoring to ensure your model is still accurately modeling the real world distribution is a key component. If you are someone relying on ML to do a good job / needing to continuously improve your ML models, you end up caring a lot about how much it costs to retrain, how ongoing labeling operations are going & how quickly it goes from newly-labeled data --> retrained model --> evaluated model --> deployed model. Furthermore, ML orgs often have to make labeling and training efficient by sharing resources & strategically triaging only the most impactful experiments / picking the most impactful things to label. There's also the pace of progress in the field, and often organizations will have dedicated data science / ML folks who want to run experiments to improve the models upon each retraining. The infra needed to unlock rapid prototyping is quite a lot. If you have a lot of models and you're re-building the end-to-end stack each time, you end up with a ton of wasted work. A lot of this stuff is also pretty specialized. It's a bit like asking all your engineers to set up their own web server, proxy, monitoring / alerting, maybe a load balancer, etc etc. Plenty of people know how to do it, and for small companies doing 1 or 2 ML models maybe that's fine. But for corporations at scale it makes no sense to do it like that. And when you're at scale, a small team probably works on the load balancers vs the base infra vs the software platform vs the features. ML works the same way. Some real world examples where you can imagine how much infra you need within ML-heavy organizations: - Say you want to build a _motorcycle detector for a self-driving car_. You need to build a data extraction pipeline that processes images, gets the segmented objects, sends each object for labeling, then when you have all your labeled data, you need to split it into test/train/validation (and make sure you use the same splits as everyone else building the car's software), then you need to have these piles of images integrated with additional information needed for training (e.g. how fast the object was moving, what time of day), you need to upsample/downsample certain cases (e.g maybe you need to upsample examples of motorcycles at night), then train a net (locally, or in the cloud, or maybe as part of N experiments to tune, on shared infrastructure that M people are using where jobs need to be prioritized), then evaluate (do you need to build infra to harvest important metrics, like how well your model performs when the car is driving on a slope? in fog?), then optimize for onboard (are you going to run it on CPU? GPU? Accelerator? Do you optimize it using TensorRT? Your own quantization infra? Distillation?), then deploy (and monitor -- is the model eating too much memory during inference? Being run too many times? Not doing the right thing?) Okay your model works -- you sure it will keep working when motorcycles look different 10 years or even just 1 year from now? - Say you want to build a _spam detector for your social media website_. You do everything above, build and deploy your model to the cloud, and suddenly you realize it's not working, a new spam campaign has occurred that your model can't account for. You need to add more labeled data, but how much and where are you going to get it? After adding it, what's your overall data look like? Adding it didn't help your net as much as you expected, why? The model-level eval looked improved, but combined with a rule-system together, it got worse. Crap, how to debug? Okay finally working, how stale is the data in your model after 1 year? Did we regress on something when we solved the spam campaign? You have a computational budget for how big the net can get, because it's used real-time to judge spamminess of posts on a major website -- maybe you care about what hardware your model is running on and how to best optimize for that hardware. Maybe you use cloud TPUs, where large batch sizes help you to scale. Maybe you use graphcore or something that thrives on small batch size. What if you started on one, moved to the other, and suddenly your net isn't working as well? What if you upgrade from a gtx2080ti -> gtx3080ti and see that your net has a prediction regression? Do you have infra to detect these regressions? Over time, when your data got an order of magnitude bigger, you noticed that your net's hyperparams were no longer optimal. You needed to increase your learning rate, or decrease it. Did you notice this issue, and do you have the infra to do that tuning quickly? You notice your labeling budget is too small to label everything flagged as spam. How do you decide which things are most worthwhile to label? You have to build infra for all of this. MLOps are needed every step of the way. It's not that different from needing SREs and cloud infra engineers to run your cloud services & organizations. Labeling infra alone is a big enough market for companies like Hive and Scale to build billion-dollar businesses.