3 ms·

has anyone done a comparison of ML pipelines from a devops centric perspective ? For example, Metaflow doesnt support kubernetes today - https://github.com/Net

has anyone done a comparison of ML pipelines from a devops centric perspective ? For example, Metaflow doesnt support kubernetes today - https://github.com/Netflix/metaflow/issues/16 https://github.com/Netflix/metaflow/issues/16 so ultimately the scale up story in most of these management tools is iffy. I previously asked about kubeflow here - https://news.ycombinator.com/item?id=24808090 https://news.ycombinator.com/item?id=24808090 . Seems people think its pretty "horrendous". It seems most of these tools assume a very specialised devops team who will work around the ml tool...rather than the ml tool making this easy.

vtuulos 6y ago
As of today, Metaflow supports AWS Batch for scaling out and up. While supporting K8S is convenient in the operations point of view (assuming you have a K8S cluster already), it doesn't make the scalability story any better.
- sandGorgon 6y ago
  sure - im not talking about scalability from a science perspective, but really from practical applicability perspective. Most people already are running k8s of some kind. I beginning to see k8s increasingly as an invariant. Plus not all of us are on AWS. Secondly, AWS Batch is only applicable for metaflow for ml training.
nicktorba 6y ago
This would be super useful. Based on this thread, the comparison should include * metaflow (model training on AWS Batch) * polyaxon (model training on kubernetes) * pachyderm (experimentation) * hopsworks (model training/serving/ and more, mostly on kubernetes) * cortex (model serving on kubernetes) * seldon-core (model serving and monitoring on kubernetes) and likely more that I missed. I can see why it would be so hard to put together this comparison. Even with all these tools, there is still a lot of manual work for data scientists or DevOps engineers the data scientists pass their models off to. It also seems there is yet to be a fully open source DevOps stack. Most companies still build custom software to glue together manual processes (like integrations between different tools for training, deploying, monitoring, etc). This could be one factor why more comparisons of these tools and stack discussions have not been more popular - they can't share them yet.

vtuulos 6y ago

As of today, Metaflow supports AWS Batch for scaling out and up. While supporting K8S is convenient in the operations point of view (assuming you have a K8S cluster already), it doesn't make the scalability story any better.

sandGorgon 6y ago
sure - im not talking about scalability from a science perspective, but really from practical applicability perspective. Most people already are running k8s of some kind. I beginning to see k8s increasingly as an invariant. Plus not all of us are on AWS. Secondly, AWS Batch is only applicable for metaflow for ml training.

nicktorba 6y ago

This would be super useful. Based on this thread, the comparison should include * metaflow (model training on AWS Batch) * polyaxon (model training on kubernetes) * pachyderm (experimentation) * hopsworks (model training/serving/ and more, mostly on kubernetes) * cortex (model serving on kubernetes) * seldon-core (model serving and monitoring on kubernetes) and likely more that I missed. I can see why it would be so hard to put together this comparison. Even with all these tools, there is still a lot of manual work for data scientists or DevOps engineers the data scientists pass their models off to. It also seems there is yet to be a fully open source DevOps stack. Most companies still build custom software to glue together manual processes (like integrations between different tools for training, deploying, monitoring, etc). This could be one factor why more comparisons of these tools and stack discussions have not been more popular - they can't share them yet.