4 ms·

Can you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment trackin

by mlthoughts2018 6y ago

Can you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment tracking system and model training diagnostic / metric system (a la Tensorboard). It is exceedingly easy to DIY, I’ve been doing that stuff DIY with teams of less than 8 engineers for many years. What I’ve seen is that most vendor solutions aren’t flexible enough. Many firms have their own on-prem and data privacy restrictions that make hosting model training artifacts on a vendor’s servers impossible, and it’s way easier to sell building in house. I’m very surprised you are hearing that nobody is using MLflow and Kubeflow (though Kubeflow has a lot of genuine usability and bug problems). I am hearing the exact opposite. Everybody, from tiny 5 person startups to giant ecommerce firms is just spinning up simple stuff for model tracking, mostly around MLflow.

tzsdata 6y ago
I am in a similar role and have the exact same opinion. Most vendors lack flexibility, can have significant costs even for small teams, and, at the end of the day, an in-house built solution isn't that difficult. I think it's going to be a tough road to build a sustainable business in this space.
bfirsh 6y ago
Sorry yeah -- not saying nobody's using MLflow. We see teams and organizations using MLflow. What we don't see though is individual researchers/engineers/data scientists pick it up and use it. From the people we have talked to who use MLflow, we hear it gets the job done, but individual contributors don't love it. We really believe that widespread adoption comes from making something individuals love and use every day. That's the reason Docker was so successful, for example. The lack of flexibility really resonates. That's the reason we're trying to be small and not too opinionated. We're something we can drop into your in-house system as a component, kinda like lots of deployment systems are built around Docker.

tzsdata 6y ago

I am in a similar role and have the exact same opinion. Most vendors lack flexibility, can have significant costs even for small teams, and, at the end of the day, an in-house built solution isn't that difficult. I think it's going to be a tough road to build a sustainable business in this space.

bfirsh 6y ago

Sorry yeah -- not saying nobody's using MLflow. We see teams and organizations using MLflow. What we don't see though is individual researchers/engineers/data scientists pick it up and use it. From the people we have talked to who use MLflow, we hear it gets the job done, but individual contributors don't love it. We really believe that widespread adoption comes from making something individuals love and use every day. That's the reason Docker was so successful, for example. The lack of flexibility really resonates. That's the reason we're trying to be small and not too opinionated. We're something we can drop into your in-house system as a component, kinda like lots of deployment systems are built around Docker.