4 ms·

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a s

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experiments for parameter tuning * serving/deploying for production * a permission system so researchers and developers can only access what they are supposed to * software heritage, probably via Docker images or Nix packages, combined with source code references * (cherry on top: some kind of integrated labeling system and UI) Right now you have to cobble this together from different tools that are all pretty suboptimal. The big players can set up sophisticated systems, but I'm curious to hear how other startups are currently solving this.

mmq 6y ago
Hi @the_duke, disclaimer I am one of the authors of an open-source solution (https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon) that specializes in the experimentation and automation phase of the data-science lifecycle. Our tool provides exactly the kind of abstraction you mentioned: * Training, data operations, and interactive workspaces (https://polyaxon.com/docs/experimentation/ https://polyaxon.com/docs/experimentation/) * A scalable history and comparison table (https://polyaxon.com/docs/management/runs-dashboard/comparison/ https://polyaxon.com/docs/management/runs-dashboard/comparis...) * Currently pipelines and concurrency management is on the commercial version (https://polyaxon.com/docs/automation/ https://polyaxon.com/docs/automation/) but several companies use Polyaxon with other tools like Kubeflow (https://medium.com/mercari-engineering/continuous-delivery-and-automation-pipelines-in-machine-learning-with-polyaxon-and-kubeflow-d6a3668715de https://medium.com/mercari-engineering/continuous-delivery-a...) or it can be used with MetaFlow for the pipelines part. I would really like to hear your thoughts and feedback.
- manojlds 6y ago
  > but several companies use Polyaxon with other tools like Kubeflow or it can be used with MetaFlow for the pipelines part. Isn't that what the parent you are replying to is talking about with "Right now you have to cobble this together from different tools that are all pretty suboptimal."
  - mmq 6y ago
    Our tool provides several solutions, however, we do not force users to use all of these abstractions. It's very important for us that our product is interoperable with the rest of the ecosystem. If a company is already using a pipelining tool, a visualization tool, or a data management tool, Polyaxon will work and integrate with those tools seamlessly. That being said, and I fully understand where the OP is coming from, there are several companies not interested in managing several solutions and all the complexity that comes with the infrastructure, deployment, maintenance, upgrades, user facing clients, authn/authz, permissions... Polyaxon provides the right abstractions for covering the experimentation and the automation phase.
solumos 6y ago
Same - I'm a SWE embedded in a small (but growing) ML team. We have all of the same problems. It seems that the "all-in" platforms are too "rigid", and all of the point solutions for the things you mentioned aren't proven enough.
- thecellardoor 6y ago
  I think that by definition this is a tradeoff. Most times you talk to data scientists that want a fully automated end-2-end solution, that doesn't require they change anything about their current workflow, and that any future modifications to their workflow would be supported as well. That is magical thinking. I prefer best of breed solutions that integrate nicely with other best of breed solutions every day. That way if a tool doesn't suit you tomorrow, you can relatively easily swap it out for something better
savin-goyal 6y ago
Hi! Metaflow ships with a CloudFormation template for AWS that automates the set-up of a blob store (S3), compute environment (Batch), metadata tracking service (RDS), orchestrator (Step-Functions) notebooks (Sagemaker) and all the necessary IAM permissions to ensure data integrity. Using Metaflow, you can then write your workflows in Python/R and Metaflow will take care of managing your ML dev/prod lifecycle. https://github.com/Netflix/metaflow-tools/tree/master/aws/cloudformation https://github.com/Netflix/metaflow-tools/tree/master/aws/cl...
edameme 6y ago
I'd love to hear what you think about https://dagshub.com/ https://dagshub.com/. We're building it with community collaboration in mind. It doesn't cover all the bases you mention, but we do: * data and model storage * experiment tracking * pipeline management * access control * data, model, code, pipeline versioning We're also strictly based on Git and other Open Source formats and tools so connecting with other tools you use like Colab for IDEs or Jenkins/Kubeflow for training is super straightforward (we have examples for some)
jamesblonde 6y ago
You should checkout Hopsworks (disclaimer: work on it). It does all of the above (including a Feature Store, notebooks as jobs, airflow for ML pipelines, model serving (TensorFlowServing, Flask, and soon KFserving), experiments, a project-based multi-tenancy model that supports sensitive data on a shared cluster, and a UI. It does not have a labelling system - but you can pip install libraries. You don't need to learn Docker - each project has a conda environment, and we compile Docker images for projects transparently, so jobs are tied to Docker images, but you don't need to write a Dockerfile (this is a huge win for data scientists). You can run Python jobs (connect to a k8s cluster) or Spark/Flink jobs (on Hopsworks itself). Open-source: * https://github.com/logicalclocks/hopsworks https://github.com/logicalclocks/hopsworks Managed platform on AWS/Azure (with elastic compute/storage, integration with managed K8s, LDAP/AD): * https://hopsworks.ai https://hopsworks.ai
bfirsh 6y ago
I think the solution to this is a bunch of pluggable tools that integrate well. "AI Platforms" do everything, but they do each thing not very well and force you into a particular way of working. (There is a reason we don't use "software platforms" any longer.) But unfortunately, as you say, most of the pluggable tools are not very good and/or not mature enough. Here's our attempt at model storage, experiment tracking, and software heritage: https://replicate.ai/ https://replicate.ai/ For interactive dev environments, Colab, Deepnote, and Streamlit are all great. For deploying to production, Cortex mentioned in the post is great. All are a work in progress, but I think we'll soon have a really powerful ecosystem of tools.
jaz46 6y ago
There are lots of great platforms and tools in this space that are trying to solve these problems. They all have their tradeoffs and of course the list of needs/goals above is pretty diverse. You are very likely going to be using a handful of tools that cover to full gamut of needs. This is heavily discussed in the blog post about an MLOps Canonical Stack and many of the tools being suggested below are included. https://towardsdatascience.com/rise-of-the-canonical-stack-in-machine-learning-724e7d2faa75 https://towardsdatascience.com/rise-of-the-canonical-stack-i...
ricklamers 6y ago
Others have mentioned some cool projects in this space, but you mentioned self hosted specifically so I’ll share what we’re working on since it might match what you’re looking for. As a new project we are still figuring out some of major topics you described. In short, we built a data science pipeline tool that should fit well with existing workflows in machine learning and data science. We chose to embrace and integrate open source projects to create a simple and seamless experience with best in breed solutions for various tasks. We are particularly happy with our deep integration of JupyterLab building on the excellent Jupyter Enterpise Gateway project from IBM (Codait) for connecting kernels directly to your pipelines. For scheduling we build on top of Celery combined with containerization primitives. For stable and well defined dependency management we built a small environment abstraction on top of Docker. It works really well in our experience! Feel free to check out the project on https://github.com/orchest/orchest https://github.com/orchest/orchest Self hosting should be as easy as running about two lines of code.
zaroth 6y ago
Just wanted to let you know I favorited this comment for the responses it garnered. Great question, great efforts at providing answers. This could be its own “Ask HN” thread.
ruksi 6y ago
we at Valohai (https://valohai.com/ https://valohai.com/) do check all the mentioned feature boxes and do serve a lot of startups * data stores: automatic download/upload to/from AWS S3, Azure Blob, Google Cloud Storage, OpenStack Swift or stores that implements S3-like interface * interactive environments: we do have notebook hosting with automatic orchestration * training: history, comparisons, parameters, hyperparameter tuning with Optuna, Hyperopt or custom optimizer (https://github.com/valohai/optimo); https://github.com/valohai/optimo); additionally visualizations about training progress and hardware resource monitoring * serving for production: our deployments allow you to build, push, manage and monitor HTTP/S based services on Kubernetes clusters (https://docs.valohai.com/core-concepts/deployments/ https://docs.valohai.com/core-concepts/deployments/) but you can just as easily download your model and deploy it yourself as your use-case requires * a permission system: we have organization management with teams and such, but your mileage may vary depending how fine grained control you need * software heritage: all runs are containerized and how they was ran is recorded so everything is reproducible if the base image and data exist at the original source, we also keep track of data heritage (what files X were used to produce these files Y https://valohai.com/patch-notes/2019-09-03/ https://valohai.com/patch-notes/2019-09-03/) * labeling system/UI: full web UI, command line client and a REST API (https://docs.valohai.com/valohai-api/ https://docs.valohai.com/valohai-api/) but no labeling tools though Essentially your whole machine learning pipeline under one roof; from data preprocessing and training to deployment and monitoring. Also, we are technically agnostic, you can just as easily run Python/Julia/C++ or Unity engine to generate synthetic datasets (https://www.youtube.com/watch?v=QxMuWuk_W10 https://www.youtube.com/watch?v=QxMuWuk_W10) not self-service or free though; our technical support team handles all the setup and maintenance let me know if you have questions about Valohai or MLOps (https://valohai.com/mlops/ https://valohai.com/mlops/) in general, I've seen quite a lot of projects and pipelines as I work at Valohai as an ML engineer helping our customers to setup end-to-end ML pipelines

mmq 6y ago

Hi @the_duke, disclaimer I am one of the authors of an open-source solution (https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon) that specializes in the experimentation and automation phase of the data-science lifecycle. Our tool provides exactly the kind of abstraction you mentioned: * Training, data operations, and interactive workspaces (https://polyaxon.com/docs/experimentation/ https://polyaxon.com/docs/experimentation/) * A scalable history and comparison table (https://polyaxon.com/docs/management/runs-dashboard/comparison/ https://polyaxon.com/docs/management/runs-dashboard/comparis...) * Currently pipelines and concurrency management is on the commercial version (https://polyaxon.com/docs/automation/ https://polyaxon.com/docs/automation/) but several companies use Polyaxon with other tools like Kubeflow (https://medium.com/mercari-engineering/continuous-delivery-and-automation-pipelines-in-machine-learning-with-polyaxon-and-kubeflow-d6a3668715de https://medium.com/mercari-engineering/continuous-delivery-a...) or it can be used with MetaFlow for the pipelines part. I would really like to hear your thoughts and feedback.

manojlds 6y ago
> but several companies use Polyaxon with other tools like Kubeflow or it can be used with MetaFlow for the pipelines part. Isn't that what the parent you are replying to is talking about with "Right now you have to cobble this together from different tools that are all pretty suboptimal."
- mmq 6y ago
  Our tool provides several solutions, however, we do not force users to use all of these abstractions. It's very important for us that our product is interoperable with the rest of the ecosystem. If a company is already using a pipelining tool, a visualization tool, or a data management tool, Polyaxon will work and integrate with those tools seamlessly. That being said, and I fully understand where the OP is coming from, there are several companies not interested in managing several solutions and all the complexity that comes with the infrastructure, deployment, maintenance, upgrades, user facing clients, authn/authz, permissions... Polyaxon provides the right abstractions for covering the experimentation and the automation phase.

solumos 6y ago

Same - I'm a SWE embedded in a small (but growing) ML team. We have all of the same problems. It seems that the "all-in" platforms are too "rigid", and all of the point solutions for the things you mentioned aren't proven enough.

thecellardoor 6y ago
I think that by definition this is a tradeoff. Most times you talk to data scientists that want a fully automated end-2-end solution, that doesn't require they change anything about their current workflow, and that any future modifications to their workflow would be supported as well. That is magical thinking. I prefer best of breed solutions that integrate nicely with other best of breed solutions every day. That way if a tool doesn't suit you tomorrow, you can relatively easily swap it out for something better

savin-goyal 6y ago

Hi! Metaflow ships with a CloudFormation template for AWS that automates the set-up of a blob store (S3), compute environment (Batch), metadata tracking service (RDS), orchestrator (Step-Functions) notebooks (Sagemaker) and all the necessary IAM permissions to ensure data integrity. Using Metaflow, you can then write your workflows in Python/R and Metaflow will take care of managing your ML dev/prod lifecycle. https://github.com/Netflix/metaflow-tools/tree/master/aws/cloudformation https://github.com/Netflix/metaflow-tools/tree/master/aws/cl...

edameme 6y ago

I'd love to hear what you think about https://dagshub.com/ https://dagshub.com/. We're building it with community collaboration in mind. It doesn't cover all the bases you mention, but we do: * data and model storage * experiment tracking * pipeline management * access control * data, model, code, pipeline versioning We're also strictly based on Git and other Open Source formats and tools so connecting with other tools you use like Colab for IDEs or Jenkins/Kubeflow for training is super straightforward (we have examples for some)

jamesblonde 6y ago

You should checkout Hopsworks (disclaimer: work on it). It does all of the above (including a Feature Store, notebooks as jobs, airflow for ML pipelines, model serving (TensorFlowServing, Flask, and soon KFserving), experiments, a project-based multi-tenancy model that supports sensitive data on a shared cluster, and a UI. It does not have a labelling system - but you can pip install libraries. You don't need to learn Docker - each project has a conda environment, and we compile Docker images for projects transparently, so jobs are tied to Docker images, but you don't need to write a Dockerfile (this is a huge win for data scientists). You can run Python jobs (connect to a k8s cluster) or Spark/Flink jobs (on Hopsworks itself). Open-source: * https://github.com/logicalclocks/hopsworks https://github.com/logicalclocks/hopsworks Managed platform on AWS/Azure (with elastic compute/storage, integration with managed K8s, LDAP/AD): * https://hopsworks.ai https://hopsworks.ai

bfirsh 6y ago

I think the solution to this is a bunch of pluggable tools that integrate well. "AI Platforms" do everything, but they do each thing not very well and force you into a particular way of working. (There is a reason we don't use "software platforms" any longer.) But unfortunately, as you say, most of the pluggable tools are not very good and/or not mature enough. Here's our attempt at model storage, experiment tracking, and software heritage: https://replicate.ai/ https://replicate.ai/ For interactive dev environments, Colab, Deepnote, and Streamlit are all great. For deploying to production, Cortex mentioned in the post is great. All are a work in progress, but I think we'll soon have a really powerful ecosystem of tools.

jaz46 6y ago

There are lots of great platforms and tools in this space that are trying to solve these problems. They all have their tradeoffs and of course the list of needs/goals above is pretty diverse. You are very likely going to be using a handful of tools that cover to full gamut of needs. This is heavily discussed in the blog post about an MLOps Canonical Stack and many of the tools being suggested below are included. https://towardsdatascience.com/rise-of-the-canonical-stack-in-machine-learning-724e7d2faa75 https://towardsdatascience.com/rise-of-the-canonical-stack-i...

ricklamers 6y ago

Others have mentioned some cool projects in this space, but you mentioned self hosted specifically so I’ll share what we’re working on since it might match what you’re looking for. As a new project we are still figuring out some of major topics you described. In short, we built a data science pipeline tool that should fit well with existing workflows in machine learning and data science. We chose to embrace and integrate open source projects to create a simple and seamless experience with best in breed solutions for various tasks. We are particularly happy with our deep integration of JupyterLab building on the excellent Jupyter Enterpise Gateway project from IBM (Codait) for connecting kernels directly to your pipelines. For scheduling we build on top of Celery combined with containerization primitives. For stable and well defined dependency management we built a small environment abstraction on top of Docker. It works really well in our experience! Feel free to check out the project on https://github.com/orchest/orchest https://github.com/orchest/orchest Self hosting should be as easy as running about two lines of code.

zaroth 6y ago

Just wanted to let you know I favorited this comment for the responses it garnered. Great question, great efforts at providing answers. This could be its own “Ask HN” thread.

ruksi 6y ago

we at Valohai (https://valohai.com/ https://valohai.com/) do check all the mentioned feature boxes and do serve a lot of startups * data stores: automatic download/upload to/from AWS S3, Azure Blob, Google Cloud Storage, OpenStack Swift or stores that implements S3-like interface * interactive environments: we do have notebook hosting with automatic orchestration * training: history, comparisons, parameters, hyperparameter tuning with Optuna, Hyperopt or custom optimizer (https://github.com/valohai/optimo); https://github.com/valohai/optimo); additionally visualizations about training progress and hardware resource monitoring * serving for production: our deployments allow you to build, push, manage and monitor HTTP/S based services on Kubernetes clusters (https://docs.valohai.com/core-concepts/deployments/ https://docs.valohai.com/core-concepts/deployments/) but you can just as easily download your model and deploy it yourself as your use-case requires * a permission system: we have organization management with teams and such, but your mileage may vary depending how fine grained control you need * software heritage: all runs are containerized and how they was ran is recorded so everything is reproducible if the base image and data exist at the original source, we also keep track of data heritage (what files X were used to produce these files Y https://valohai.com/patch-notes/2019-09-03/ https://valohai.com/patch-notes/2019-09-03/) * labeling system/UI: full web UI, command line client and a REST API (https://docs.valohai.com/valohai-api/ https://docs.valohai.com/valohai-api/) but no labeling tools though Essentially your whole machine learning pipeline under one roof; from data preprocessing and training to deployment and monitoring. Also, we are technically agnostic, you can just as easily run Python/Julia/C++ or Unity engine to generate synthetic datasets (https://www.youtube.com/watch?v=QxMuWuk_W10 https://www.youtube.com/watch?v=QxMuWuk_W10) not self-service or free though; our technical support team handles all the setup and maintenance let me know if you have questions about Valohai or MLOps (https://valohai.com/mlops/ https://valohai.com/mlops/) in general, I've seen quite a lot of projects and pipelines as I work at Valohai as an ML engineer helping our customers to setup end-to-end ML pipelines