4 ms·

Congrats on the launch! This looks interesting, however I feel like this space is quite crowded. You mentioned that your most important feature is the fact that

Congrats on the launch! This looks interesting, however I feel like this space is quite crowded. You mentioned that your most important feature is the fact that you are open-source, but off the top of my head I can think of several projects: * Kubeflow: https://github.com/kubeflow/kubeflow https://github.com/kubeflow/kubeflow * MLFlow: https://github.com/mlflow/mlflow https://github.com/mlflow/mlflow * Pachyderm: https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm * DVC: https://github.com/iterative/dvc https://github.com/iterative/dvc * Polyaxon: https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon * Sacred: https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred * pytorch-lightning + grid: https://github.com/PyTorchLightning/pytorch-lightning https://github.com/PyTorchLightning/pytorch-lightning * DeterminedAI: https://github.com/determined-ai/determined https://github.com/determined-ai/determined * Metaflow: https://github.com/Netflix/metaflow https://github.com/Netflix/metaflow * Aim: https://github.com/aimhubio/aim https://github.com/aimhubio/aim * And so many more... In addition to this list, several other hosted platform offer experiments tracking and model management. How do you compare to all of these tools, and why do you think users should move from one of them to use replicate, thank you.

bfirsh 6y ago
Yeah, I agree this space is crowded. But we’ve found so few ML researchers/engineers are actually using these tools. This could either be that people aren’t aware of them yet, or that they’re not good enough. I think it’s a mix of both, honestly, but we’re betting that there’s more of the latter in the mix. :) I could do comparisons of each of these tools, and some of them are solving quite different problems, but the overarching difference is we’re just trying to do less. These systems might make sense if you’re setting up a company’s ML pipeline, but we found lots of individuals struggling to keep track of their work and store their models. They balked at the idea of setting up things like Kubeflow or MLflow.
- mlthoughts2018 6y ago
  Can you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment tracking system and model training diagnostic / metric system (a la Tensorboard). It is exceedingly easy to DIY, I’ve been doing that stuff DIY with teams of less than 8 engineers for many years. What I’ve seen is that most vendor solutions aren’t flexible enough. Many firms have their own on-prem and data privacy restrictions that make hosting model training artifacts on a vendor’s servers impossible, and it’s way easier to sell building in house. I’m very surprised you are hearing that nobody is using MLflow and Kubeflow (though Kubeflow has a lot of genuine usability and bug problems). I am hearing the exact opposite. Everybody, from tiny 5 person startups to giant ecommerce firms is just spinning up simple stuff for model tracking, mostly around MLflow.
  - tzsdata 6y ago
    I am in a similar role and have the exact same opinion. Most vendors lack flexibility, can have significant costs even for small teams, and, at the end of the day, an in-house built solution isn't that difficult. I think it's going to be a tough road to build a sustainable business in this space.
  - bfirsh 6y ago
    Sorry yeah -- not saying nobody's using MLflow. We see teams and organizations using MLflow. What we don't see though is individual researchers/engineers/data scientists pick it up and use it. From the people we have talked to who use MLflow, we hear it gets the job done, but individual contributors don't love it. We really believe that widespread adoption comes from making something individuals love and use every day. That's the reason Docker was so successful, for example. The lack of flexibility really resonates. That's the reason we're trying to be small and not too opinionated. We're something we can drop into your in-house system as a component, kinda like lots of deployment systems are built around Docker.

bfirsh 6y ago

Yeah, I agree this space is crowded. But we’ve found so few ML researchers/engineers are actually using these tools. This could either be that people aren’t aware of them yet, or that they’re not good enough. I think it’s a mix of both, honestly, but we’re betting that there’s more of the latter in the mix. :) I could do comparisons of each of these tools, and some of them are solving quite different problems, but the overarching difference is we’re just trying to do less. These systems might make sense if you’re setting up a company’s ML pipeline, but we found lots of individuals struggling to keep track of their work and store their models. They balked at the idea of setting up things like Kubeflow or MLflow.

mlthoughts2018 6y ago
Can you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment tracking system and model training diagnostic / metric system (a la Tensorboard). It is exceedingly easy to DIY, I’ve been doing that stuff DIY with teams of less than 8 engineers for many years. What I’ve seen is that most vendor solutions aren’t flexible enough. Many firms have their own on-prem and data privacy restrictions that make hosting model training artifacts on a vendor’s servers impossible, and it’s way easier to sell building in house. I’m very surprised you are hearing that nobody is using MLflow and Kubeflow (though Kubeflow has a lot of genuine usability and bug problems). I am hearing the exact opposite. Everybody, from tiny 5 person startups to giant ecommerce firms is just spinning up simple stuff for model tracking, mostly around MLflow.
- tzsdata 6y ago
  I am in a similar role and have the exact same opinion. Most vendors lack flexibility, can have significant costs even for small teams, and, at the end of the day, an in-house built solution isn't that difficult. I think it's going to be a tough road to build a sustainable business in this space.
- bfirsh 6y ago
  Sorry yeah -- not saying nobody's using MLflow. We see teams and organizations using MLflow. What we don't see though is individual researchers/engineers/data scientists pick it up and use it. From the people we have talked to who use MLflow, we hear it gets the job done, but individual contributors don't love it. We really believe that widespread adoption comes from making something individuals love and use every day. That's the reason Docker was so successful, for example. The lack of flexibility really resonates. That's the reason we're trying to be small and not too opinionated. We're something we can drop into your in-house system as a component, kinda like lots of deployment systems are built around Docker.