3 ms·

Congratulations with the launch. We've used https://github.com/iterative/dvc https://github.com/iterative/dvc for a long time and quite happy. What's the main

by m0sth8 6y ago

Congratulations with the launch. We've used https://github.com/iterative/dvc https://github.com/iterative/dvc for a long time and quite happy. What's the main difference between replicate.ai and dvc?

mwnivek 6y ago
I'd be curious about comparison with https://github.com/mlflow/mlflow https://github.com/mlflow/mlflow
- bfirsh 6y ago
  We talked to a bunch of MLflow users, and the general impression we got is that it is heavyweight and hard to set up. MLflow is an all-encompassing "ML platform". Which is fine if you need that, but we're trying to just do one thing well. (Imagine if Git called itself a "software platform".) In terms of features, Replicate points directly at an S3 bucket (so you don't have to run a server and Postgres DB), it saves your training code (for reproducibility and to commit to Git after the fact), and it has a nice API for reading and analyzing your experiments in a notebook.
  - Jugurtha 6y ago
    Congrats on the launch! >MLflow is an all-encompassing "ML platform" Not really. We're trying to use MLflow with our "ML platform"[0]. Namely, it can save a model that expects high dimensional inputs, which is most models I've seen that aren't trivial, and can "deploy" the model but with an expectation of two dimensional DataFrame inputs. Apparently, they're working on that. There are also many ambiguities concerning Keras and Tensorflow stemming from "What is a Keras model? Is it a Tensorflow model now they're integrated? Why are Keras models logged with the tensorflow model logger when you use the autolog functionality?". These are shared ambiguities, as there are several ways to save and load models with Tensorflow, and we're looking into the Keras/Tensorflow integration closely. MLflow uses `cloudpickle` and unpickling expects not only the same 'protocol', but the same Python version. Had to dig deeper than necessary. One other problem is when a model relies on ancillary functions, which you must be able to ship somehow. You end up tinkering with its guts, too. Could you shed some light on how do you deal with these matters. Namely, high dimensional inputs for models, pre-processing/post-processing functions, serialization brittleness, and Keras/Tensorflow "duality". We have to inherit that complexity to spare our users from having to mentally think of saving their experiments (we do that automatically to save models, metrics, params). The workflow is data --> collaborative notebooks with scheduling features and job --> (generate appbooks) --> automatically tracked models/params/metrics --> one click deployment --> 'REST' API or form to invoke model. Aaaaaand again, congrats on the launch! - [0]: https://iko.ai https://iko.ai
bfirsh 6y ago
Thanks! DVC is closely tied to Git. We've heard people find that quite heavyweight when you're running experiments. We think we can build a much better experience if we detach ourselves from Git. With Replicate, you just run your training script as usual, and it automatically tracks everything from within Python. You don't have to run any additional commands to track things. DVC is really good for storing data sets though, and we see potential for integration there: https://github.com/replicate/replicate/issues/359 https://github.com/replicate/replicate/issues/359
- ishcheklein 6y ago
  Hey, one of the DVC maintainers here! TL;DR: I think it should be compared with the upcoming DVC feature - https://github.com/iterative/dvc/wiki/Experiments https://github.com/iterative/dvc/wiki/Experiments . Stay tuned - it'll be released very soon but you can try it now in beta. First of all, congrats on the launch! I do really like the aesthetics of the website, and the overall approach. It resonates with our vision and philosophy! Good feedback on experiments feeling heavyweight! We've been focused on doing great foundation to manage data and pipelines in the previous DVC versions and were aware about this problem (https://github.com/iterative/dvc/issues/2799 https://github.com/iterative/dvc/issues/2799). As I mentioned - Experiments feature is already there in beta testing. It means that users don't have to do commits anymore until they are ready, still can share experiments (it's a long topic and we'll write a blog post at some point since I really excited about the way it'll be implemented using custom Git refs), support for DL workflow (auto-checkpoints), and more. Would love to discuss and share any details, it would be great to compare the approaches.
  - bfirsh 6y ago
    Would love to chat -- I'll shoot you an email. :)
    - toto15151 6y ago
      ghj
- 6y ago
edolev 6y ago
Congrats on the launch! This looks exciting. My company has been using Comet.ml and they cover a few use cases that are missing here. Specifically things like real time visualizations and sharing experiments which is key when working in a team. Are you planning on adding those?
- fagerhult 6y ago
  Thank you! We have an issue on the roadmap for adding a web GUI: https://github.com/replicate/replicate/issues/295 https://github.com/replicate/replicate/issues/295 We haven't thought about it in great detail yet, so I'd be curious to hear your thoughts and ideas if you'd like to add a comment to that issue!

by m0sth8 6y ago

Congratulations with the launch. We've used https://github.com/iterative/dvc https://github.com/iterative/dvc for a long time and quite happy. What's the main difference between replicate.ai and dvc?

mwnivek 6y ago

I'd be curious about comparison with https://github.com/mlflow/mlflow https://github.com/mlflow/mlflow

bfirsh 6y ago
We talked to a bunch of MLflow users, and the general impression we got is that it is heavyweight and hard to set up. MLflow is an all-encompassing "ML platform". Which is fine if you need that, but we're trying to just do one thing well. (Imagine if Git called itself a "software platform".) In terms of features, Replicate points directly at an S3 bucket (so you don't have to run a server and Postgres DB), it saves your training code (for reproducibility and to commit to Git after the fact), and it has a nice API for reading and analyzing your experiments in a notebook.
- Jugurtha 6y ago
  Congrats on the launch! >MLflow is an all-encompassing "ML platform" Not really. We're trying to use MLflow with our "ML platform"[0]. Namely, it can save a model that expects high dimensional inputs, which is most models I've seen that aren't trivial, and can "deploy" the model but with an expectation of two dimensional DataFrame inputs. Apparently, they're working on that. There are also many ambiguities concerning Keras and Tensorflow stemming from "What is a Keras model? Is it a Tensorflow model now they're integrated? Why are Keras models logged with the tensorflow model logger when you use the autolog functionality?". These are shared ambiguities, as there are several ways to save and load models with Tensorflow, and we're looking into the Keras/Tensorflow integration closely. MLflow uses `cloudpickle` and unpickling expects not only the same 'protocol', but the same Python version. Had to dig deeper than necessary. One other problem is when a model relies on ancillary functions, which you must be able to ship somehow. You end up tinkering with its guts, too. Could you shed some light on how do you deal with these matters. Namely, high dimensional inputs for models, pre-processing/post-processing functions, serialization brittleness, and Keras/Tensorflow "duality". We have to inherit that complexity to spare our users from having to mentally think of saving their experiments (we do that automatically to save models, metrics, params). The workflow is data --> collaborative notebooks with scheduling features and job --> (generate appbooks) --> automatically tracked models/params/metrics --> one click deployment --> 'REST' API or form to invoke model. Aaaaaand again, congrats on the launch! - [0]: https://iko.ai https://iko.ai

bfirsh 6y ago

Thanks! DVC is closely tied to Git. We've heard people find that quite heavyweight when you're running experiments. We think we can build a much better experience if we detach ourselves from Git. With Replicate, you just run your training script as usual, and it automatically tracks everything from within Python. You don't have to run any additional commands to track things. DVC is really good for storing data sets though, and we see potential for integration there: https://github.com/replicate/replicate/issues/359 https://github.com/replicate/replicate/issues/359

ishcheklein 6y ago
Hey, one of the DVC maintainers here! TL;DR: I think it should be compared with the upcoming DVC feature - https://github.com/iterative/dvc/wiki/Experiments https://github.com/iterative/dvc/wiki/Experiments . Stay tuned - it'll be released very soon but you can try it now in beta. First of all, congrats on the launch! I do really like the aesthetics of the website, and the overall approach. It resonates with our vision and philosophy! Good feedback on experiments feeling heavyweight! We've been focused on doing great foundation to manage data and pipelines in the previous DVC versions and were aware about this problem (https://github.com/iterative/dvc/issues/2799 https://github.com/iterative/dvc/issues/2799). As I mentioned - Experiments feature is already there in beta testing. It means that users don't have to do commits anymore until they are ready, still can share experiments (it's a long topic and we'll write a blog post at some point since I really excited about the way it'll be implemented using custom Git refs), support for DL workflow (auto-checkpoints), and more. Would love to discuss and share any details, it would be great to compare the approaches.
- bfirsh 6y ago
  Would love to chat -- I'll shoot you an email. :)
  - toto15151 6y ago
    ghj
6y ago

edolev 6y ago

Congrats on the launch! This looks exciting. My company has been using Comet.ml and they cover a few use cases that are missing here. Specifically things like real time visualizations and sharing experiments which is key when working in a team. Are you planning on adding those?

fagerhult 6y ago
Thank you! We have an issue on the roadmap for adding a web GUI: https://github.com/replicate/replicate/issues/295 https://github.com/replicate/replicate/issues/295 We haven't thought about it in great detail yet, so I'd be curious to hear your thoughts and ideas if you'd like to add a comment to that issue!