3 ms·
Yeah, I agree this space is crowded. But we’ve found so few ML researchers/engineers are actually using these tools. This could either be that people aren’t awa
by bfirsh 6y ago
Yeah, I agree this space is crowded. But we’ve found so few ML researchers/engineers are actually using these tools. This could either be that people aren’t aware of them yet, or that they’re not good enough.
I think it’s a mix of both, honestly, but we’re betting that there’s more of the latter in the mix. :)
I could do comparisons of each of these tools, and some of them are solving quite different problems, but the overarching difference is we’re just trying to do less. These systems might make sense if you’re setting up a company’s ML pipeline, but we found lots of individuals struggling to keep track of their work and store their models. They balked at the idea of setting up things like Kubeflow or MLflow.
- mlthoughts2018 6y agoCan you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment tracking system and model training diagnostic / metric system (a la Tensorboard). It is exceedingly easy to DIY, I’ve been doing that stuff DIY with teams of less than 8 engineers for many years. What I’ve seen is that most vendor solutions aren’t flexible enough. Many firms have their own on-prem and data privacy restrictions that make hosting model training artifacts on a vendor’s servers impossible, and it’s way easier to sell building in house. I’m very surprised you are hearing that nobody is using MLflow and Kubeflow (though Kubeflow has a lot of genuine usability and bug problems). I am hearing the exact opposite. Everybody, from tiny 5 person startups to giant ecommerce firms is just spinning up simple stuff for model tracking, mostly around MLflow.
- tzsdata 6y agoI am in a similar role and have the exact same opinion. Most vendors lack flexibility, can have significant costs even for small teams, and, at the end of the day, an in-house built solution isn't that difficult. I think it's going to be a tough road to build a sustainable business in this space.
- bfirsh 6y agoSorry yeah -- not saying nobody's using MLflow. We see teams and organizations using MLflow. What we don't see though is individual researchers/engineers/data scientists pick it up and use it. From the people we have talked to who use MLflow, we hear it gets the job done, but individual contributors don't love it. We really believe that widespread adoption comes from making something individuals love and use every day. That's the reason Docker was so successful, for example. The lack of flexibility really resonates. That's the reason we're trying to be small and not too opinionated. We're something we can drop into your in-house system as a component, kinda like lots of deployment systems are built around Docker.