16 ms·

Launch HN: Replicate (YC W20) – Version control for machine learning

205 points by bfirsh 6y ago

bfirsh 6y ago
Hello HN! We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: https://replicate.ai/ https://replicate.ai/ Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose. We built https://www.arxiv-vanity.com/ https://www.arxiv-vanity.com/ together for fun, which led to us teaming up to build more tools for ML. We spent a year talking to lots of people in the ML community and building all sorts of prototypes, but we kept on coming back to a foundational problem: not many people in machine learning use version control. This causes all sorts of problems: people are manually keeping track of things in spreadsheets, model weights are scattered on S3, and results can’t be reproduced. So why isn’t everyone using Git? Git doesn’t work well with machine learning. It can’t store trained machine learning models, it can’t handle key/value metadata, and it’s not designed to record information automatically from a training script. There are some solutions for these things, but they feel like band-aids. We came to the conclusion that we need a native version control system for ML. It’s sufficiently different to normal software that we can’t just put band-aids on Git. We believe the tool should be small, easy to use, and extensible. We found people struggling to migrate to “AI Platforms”. A tool should do one thing well and combine with other tools to produce the system you need. Finally, we also believe it should be open source. There are a number of proprietary solutions, but something so foundational needs to be built by and for the ML community. Replicate is a first cut at something we think is useful: It is a Python library that uploads your files and metadata (like hyperparameters) to Amazon S3 or Google Cloud Storage. You can get back to any point in time using the command-line interface, analyze your results inside a notebook using the Python API, and load your models in production systems. We’d love to hear your feedback, and hear your stories about how you’ve done this before. Also – building a version control system is rather complex, and to make this a reality we need your help. Join us in Discord if you want to be involved in the early design and help build it: https://discord.gg/QmzJApGjyE https://discord.gg/QmzJApGjyE
- mritchie712 6y ago
  > it can’t handle key/value metadata What do you mean by that? Is a JSON no good? I guess you mean the diffs will be unordered?
  - bfirsh 6y ago
    Yep, and we can do lots of other nice things. We can produce nice tables with the key/value data, filter it ("show me all experiments with an accuracy greater than 0.9"), produce well-formatted diffs across an arbitrary number of things, give you a nice Python API for analyzing the data in a notebook, and so on. There are some examples of these things on the home page, all of which would be very fiddly to do with JSON files in Git: https://replicate.ai/#features https://replicate.ai/#features
- nemoniac 6y ago
  Wait? So you can upload to Amazon or Google but nowhere else? Like to your own servers, for example?
  - bfirsh 6y ago
    You can save data to a path on the filesystem, so one way to do this is with a network mount. Lots of academic departments have their own GPU clusters, and they tend to have a shared network filesystem. We want to have more ways to do this though. We were close to adding SFTP support, but didn't get round to it. Another method could be to implement our own server, but we're trying to keep it simple for now. I'd be curious to hear your feedback here: https://github.com/replicate/replicate/issues/366 https://github.com/replicate/replicate/issues/366
    - IanCal 6y ago
      If you support S3 compatible storage providers that could mean people can use things like minio, that could be a shorter path to supporting lots of backends.
- salty_biscuits 6y ago
m0sth8 6y ago
Congratulations with the launch. We've used https://github.com/iterative/dvc https://github.com/iterative/dvc for a long time and quite happy. What's the main difference between replicate.ai and dvc?
- mwnivek 6y ago
  I'd be curious about comparison with https://github.com/mlflow/mlflow https://github.com/mlflow/mlflow
  - bfirsh 6y ago
    We talked to a bunch of MLflow users, and the general impression we got is that it is heavyweight and hard to set up. MLflow is an all-encompassing "ML platform". Which is fine if you need that, but we're trying to just do one thing well. (Imagine if Git called itself a "software platform".) In terms of features, Replicate points directly at an S3 bucket (so you don't have to run a server and Postgres DB), it saves your training code (for reproducibility and to commit to Git after the fact), and it has a nice API for reading and analyzing your experiments in a notebook.
    - Jugurtha 6y ago
      Congrats on the launch! >MLflow is an all-encompassing "ML platform" Not really. We're trying to use MLflow with our "ML platform"[0]. Namely, it can save a model that expects high dimensional inputs, which is most models I've seen that aren't trivial, and can "deploy" the model but with an expectation of two dimensional DataFrame inputs. Apparently, they're working on that. There are also many ambiguities concerning Keras and Tensorflow stemming from "What is a Keras model? Is it a Tensorflow model now they're integrated? Why are Keras models logged with the tensorflow model logger when you use the autolog functionality?". These are shared ambiguities, as there are several ways to save and load models with Tensorflow, and we're looking into the Keras/Tensorflow integration closely. MLflow uses `cloudpickle` and unpickling expects not only the same 'protocol', but the same Python version. Had to dig deeper than necessary. One other problem is when a model relies on ancillary functions, which you must be able to ship somehow. You end up tinkering with its guts, too. Could you shed some light on how do you deal with these matters. Namely, high dimensional inputs for models, pre-processing/post-processing functions, serialization brittleness, and Keras/Tensorflow "duality". We have to inherit that complexity to spare our users from having to mentally think of saving their experiments (we do that automatically to save models, metrics, params). The workflow is data --> collaborative notebooks with scheduling features and job --> (generate appbooks) --> automatically tracked models/params/metrics --> one click deployment --> 'REST' API or form to invoke model. Aaaaaand again, congrats on the launch! - [0]: https://iko.ai https://iko.ai
mfDjB 6y ago
As someone who tried to use git to do this for large sets of data, I'm very glad this exists. Will be trying this out in the future.
- kevlar1818 6y ago
  You may also be interested in a simple tool I'm building that works in concert with source control to store, version, and reproduce large data: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud My project is in its infancy (open-sourced less than a month ago), but I'm pleased with its UX thus far. There's lots to add in terms of documentation, but Dud currently uses Rclone[1] for remote syncing. [1]: https://rclone.org/ https://rclone.org/
- konda 6y ago
  I'd checkout dolthub.com! Dolt is built for git workflows (branches and merges) on top of large datasets.
  - timsehn 6y ago
    Thanks for the shout out here. CEO of DoltHub speaking :-)
    - deleted 6y ago
      [deleted]
mindhash 6y ago
Congrats on the launch. I have built an open source tool (called hyperML) for similar problem sometime back. I think the problem is not just storing in version control but being able to quickly retrieve those in live /test systems or containers. Mounting and loading datasets and models is painful. It kind of what makes local training a better option. If only the weights were version controlled by libraries (tf, pytorch or scikit) this whole problem will be much easier to solve.
breck 6y ago
nit: "Throw away your spreadsheet" scares me a little. I love spreadsheets, and think there are 100x+ more users of spreadsheets than notebooks (though the overlap of notebook users and ML users is probably close to 1, so I see your point). I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools).
- fagerhult 6y ago
  Hi, Andreas here. Yes spreadsheets are great, and better than notebooks in many cases. But I always felt like I was doing something wrong when I used spreadsheets and markdown files to manually record metrics and hyperparameters for my experiments. It's error prone and easy to forget to update the spreadsheet with new experiments. So we're trying to automate recording this metadata, but then give you that metadata in various ways for you to inspect it. One of those ways is actually spreadsheets: https://github.com/replicate/replicate/issues/289 https://github.com/replicate/replicate/issues/289
- Jugurtha 6y ago
  Do you not think that using spreadsheets for ML experiment tracking is a symptom of broken tooling? I'm asking because one of the reasons we're building our platform[0] with automatic experiment tracking, collaborative notebooks, and a bunch of things, is because experiment tracking was inconsistent between team members. Differences in tools used: (spreadsheets, flat files, logs, pen and paper, human memory). Forgetting to do it. Snippets to do it flying around. Different locations (laptop, group workstation, git repository, cloud sheet). Dissociated from the notebook that produced the model. Tighter tracking should answer questions like: what notebook ran on which data and produced which model with which parameters and which scores? Then questions like: give me all notebooks that ran on this dataset which produced a model with scores that are [condition]. Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself. I think it's good there are tools with this granularity that can be composed. - [0]: https://iko.ai https://iko.ai
  - breck 6y ago
    > Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself. Yes, and sorry because rereading my comment I wasn't clear, but this is what I meant. I view Spreadsheets as the view (but also an editor of the view), but I don't mean people should be working with XLS files (TSVs seem to work the best— I personally hate tabs but may have lost that battle). I do think though of Spreadsheets as the primary view, and so always design my data structures with the understanding of "how will this interoperate with spreadsheets". JSON is the pits. IMO 2-D DSLs with Spreadsheets as the primary view/editor paradigm are the future. I think Spreadsheets are 1, if not 2 OOM better than notebooks for doing actual work (notebooks can be good for presenting results in a narrative): - non-linear for both humans and machines (allows for really creative and fast out-of-order parsing techniques on the machine side) - concise signal with high information density - unlimited cursors - fantastically easier for version control and multi-player experiences I maintain a list of all data science tools to try and stay on top of best practices, and when I see a tool is notebook first, I think "good, less work for me to track this one because they aren't getting the core things right yet." (Though often times notebooks will have really innovative orthogonal features).
lcap 6y ago
How does this compare to tools like neptune.ai, weights and biases and so on? I can see the advantage of having control of one's data, whereas these tools use their own servers. However what I love about them is the amazing UI that allows me to compare experiments.
- bfirsh 6y ago
  This came out of a practical problem: at Spotify, Andreas couldn't let any data leave their network. He wasn't going to go through procurement to buy an enterprise version of one of those products, so his only option left was open source software. But it's also out of principle: we think such a foundational thing needs to be open source. There is a reason most people use Git and not Perforce. Replicate can work alongside visualization tools -- your data is safe in your own S3 bucket, but you can use the hosted visualization tool to complement that. You could also imagine visualization tools built on top of Replicate. One thing we've been thinking about is doing visualization inside notebooks. It's like a programmable Tensorboard: https://colab.research.google.com/drive/18sVRE4Zi484G2rBeOYjobE3zek2gDBvy https://colab.research.google.com/drive/18sVRE4Zi484G2rBeOYj... I'd be curious to hear your thoughts about that. It's pretty primitive so far, but we've got to start somewhere I suppose. :)
  - lcap 6y ago
    The idea looks very good! I have to say however that I am a bit wary of notebooks. I agree how great they are to visualize things, but they also tend to have a negative effect on the coding standards of a team. This is again probably a personal bias and they look to be suited for this use case. Great launch anyway!
rq1 6y ago
I was looking for something similar today. I just adopted it. :) Thank you for your amazing work! Do you have the intention to integrate it with PT Lightning or as a PT Lightning Logger? It would be nice to have it maintained there. It’s used all over huggingface-Transformers examples.
- fagerhult 6y ago
  Fantastic, thank you for those kind words! And great idea to integrate with PT Lightning. I just opened an issue: https://github.com/replicate/replicate/issues/367 https://github.com/replicate/replicate/issues/367, feel free to add more detail and comments! -andreas
mkuklik 6y ago
Congrats on the launch. Have you looked at https://comet.ml https://comet.ml If so, how do you compare to them?
- bfirsh 6y ago
  Thanks! It's open source and you're in control of your own data. See https://news.ycombinator.com/item?id=25151741 https://news.ycombinator.com/item?id=25151741
flc-anonym 6y ago
Congrats on the launch! This looks interesting, however I feel like this space is quite crowded. You mentioned that your most important feature is the fact that you are open-source, but off the top of my head I can think of several projects: * Kubeflow: https://github.com/kubeflow/kubeflow https://github.com/kubeflow/kubeflow * MLFlow: https://github.com/mlflow/mlflow https://github.com/mlflow/mlflow * Pachyderm: https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm * DVC: https://github.com/iterative/dvc https://github.com/iterative/dvc * Polyaxon: https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon * Sacred: https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred * pytorch-lightning + grid: https://github.com/PyTorchLightning/pytorch-lightning https://github.com/PyTorchLightning/pytorch-lightning * DeterminedAI: https://github.com/determined-ai/determined https://github.com/determined-ai/determined * Metaflow: https://github.com/Netflix/metaflow https://github.com/Netflix/metaflow * Aim: https://github.com/aimhubio/aim https://github.com/aimhubio/aim * And so many more... In addition to this list, several other hosted platform offer experiments tracking and model management. How do you compare to all of these tools, and why do you think users should move from one of them to use replicate, thank you.
- bfirsh 6y ago
  Yeah, I agree this space is crowded. But we’ve found so few ML researchers/engineers are actually using these tools. This could either be that people aren’t aware of them yet, or that they’re not good enough. I think it’s a mix of both, honestly, but we’re betting that there’s more of the latter in the mix. :) I could do comparisons of each of these tools, and some of them are solving quite different problems, but the overarching difference is we’re just trying to do less. These systems might make sense if you’re setting up a company’s ML pipeline, but we found lots of individuals struggling to keep track of their work and store their models. They balked at the idea of setting up things like Kubeflow or MLflow.
  - mlthoughts2018 6y ago
    Can you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment tracking system and model training diagnostic / metric system (a la Tensorboard). It is exceedingly easy to DIY, I’ve been doing that stuff DIY with teams of less than 8 engineers for many years. What I’ve seen is that most vendor solutions aren’t flexible enough. Many firms have their own on-prem and data privacy restrictions that make hosting model training artifacts on a vendor’s servers impossible, and it’s way easier to sell building in house. I’m very surprised you are hearing that nobody is using MLflow and Kubeflow (though Kubeflow has a lot of genuine usability and bug problems). I am hearing the exact opposite. Everybody, from tiny 5 person startups to giant ecommerce firms is just spinning up simple stuff for model tracking, mostly around MLflow.
gitgud 6y ago
Congrats on the launch! What's the business concept behind the tool? Can't find anything on the homepage, or in the post. I didn't know YC funded opensource tools like this, it's kind of refreshing.
- bfirsh 6y ago
  Yeah, YC is funding lots of open source projects. PostHog[0] was in our batch. GitLab, Docker, Mattermost, and CoreOS come to mind as other open source YC companies. There are a number of businesses we could build around the project. A cloud service or enterprise products/support are the obvious ones. Right now, we're focused on community building, because a potential open source business can't be successful with a healthy open source project. [0] https://news.ycombinator.com/item?id=22376732 https://news.ycombinator.com/item?id=22376732
  - gitgud 6y ago
    > There are a number of businesses we could build around the project. So you got funding from YCombinator without a concrete plan to make a business? That's pretty interesting, I always thought they wanted profitable businesses, and turned down ideas they didn't think would work. Great to hear they're betting on open-source more!
    - bfirsh 6y ago
      Funnily we applied with a different thing. We tried a number of different ideas before we settled on this lower-level thing, as I describe in the main comment. Even still, I think most people apply to YC without a concrete plan of how to make a business. It's normally so early stage, that the plan is yet to be validated and will probably change. A "plausible" plan is perhaps a better way to put it. ;)
deleted 6y ago
[deleted]
thecellardoor 6y ago
Congrats on the launch. To me this feels like DVC but with a slightly more convenient python API, and without the pipeline which might be really great. How are you organizing experiments though? Without Git it seems too easy to have experiments become unusable. And filtering according to accuracy is not really a solution..
edameme 6y ago
Congrats on the launch! Very exciting tool. I'm one of the creators of https://DAGsHub.com https://DAGsHub.com which lets you host and collaborate on data science projects – think data science pull requests and data merging. We're integrated with DVC for data versioning and experiments, but what you're building is definitely super interesting.
visarga 6y ago
What is the best way to do dataset versioning when using Replicate? I get that Replicate saves the dataset version but not the data itself. Is DVC a good fit? Second question - can you tag or attach a comment to a group of experiments?
- fagerhult 6y ago
  Great questions! At the moment we recommend passing dataset URIs as params to replicate.init(): https://replicate.ai/docs/guides/training-data https://replicate.ai/docs/guides/training-data, but of course this assumes immutable and stable URIs. DVC would definitely be a good fit, and we have a ticket on our roadmap to integrate Replicate with DVC, Tecton, etc. https://github.com/replicate/replicate/issues/294 https://github.com/replicate/replicate/issues/294 We also have a roadmap ticket for grouping experiments: https://github.com/replicate/replicate/issues/297 https://github.com/replicate/replicate/issues/297, but for now we're recommending params for tags as well. If you have ideas for the design of these features, we really appreciate feedback and comments on these Github issues!
otsaloma 6y ago
Great to see this! I've been looking at alternatives for experiment tracking, but most of what I've come across have looked way too heavy (and expensive) for a small team. Is R support planned? Or might this just work already with reticulate?
foodbaby 6y ago
This looks similar in philosophy and approach to https://guild.ai https://guild.ai although guild ai can parse stdout to extract metrics to track, and is integrated with tensorboard.

205 points by bfirsh 6y ago

Hello HN! We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: https://replicate.ai/ https://replicate.ai/ Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose. We built https://www.arxiv-vanity.com/ https://www.arxiv-vanity.com/ together for fun, which led to us teaming up to build more tools for ML. We spent a year talking to lots of people in the ML community and building all sorts of prototypes, but we kept on coming back to a foundational problem: not many people in machine learning use version control. This causes all sorts of problems: people are manually keeping track of things in spreadsheets, model weights are scattered on S3, and results can’t be reproduced. So why isn’t everyone using Git? Git doesn’t work well with machine learning. It can’t store trained machine learning models, it can’t handle key/value metadata, and it’s not designed to record information automatically from a training script. There are some solutions for these things, but they feel like band-aids. We came to the conclusion that we need a native version control system for ML. It’s sufficiently different to normal software that we can’t just put band-aids on Git. We believe the tool should be small, easy to use, and extensible. We found people struggling to migrate to “AI Platforms”. A tool should do one thing well and combine with other tools to produce the system you need. Finally, we also believe it should be open source. There are a number of proprietary solutions, but something so foundational needs to be built by and for the ML community. Replicate is a first cut at something we think is useful: It is a Python library that uploads your files and metadata (like hyperparameters) to Amazon S3 or Google Cloud Storage. You can get back to any point in time using the command-line interface, analyze your results inside a notebook using the Python API, and load your models in production systems. We’d love to hear your feedback, and hear your stories about how you’ve done this before. Also – building a version control system is rather complex, and to make this a reality we need your help. Join us in Discord if you want to be involved in the early design and help build it: https://discord.gg/QmzJApGjyE https://discord.gg/QmzJApGjyE

mritchie712 6y ago
> it can’t handle key/value metadata What do you mean by that? Is a JSON no good? I guess you mean the diffs will be unordered?
- bfirsh 6y ago
  Yep, and we can do lots of other nice things. We can produce nice tables with the key/value data, filter it ("show me all experiments with an accuracy greater than 0.9"), produce well-formatted diffs across an arbitrary number of things, give you a nice Python API for analyzing the data in a notebook, and so on. There are some examples of these things on the home page, all of which would be very fiddly to do with JSON files in Git: https://replicate.ai/#features https://replicate.ai/#features
nemoniac 6y ago
Wait? So you can upload to Amazon or Google but nowhere else? Like to your own servers, for example?
- bfirsh 6y ago
  You can save data to a path on the filesystem, so one way to do this is with a network mount. Lots of academic departments have their own GPU clusters, and they tend to have a shared network filesystem. We want to have more ways to do this though. We were close to adding SFTP support, but didn't get round to it. Another method could be to implement our own server, but we're trying to keep it simple for now. I'd be curious to hear your feedback here: https://github.com/replicate/replicate/issues/366 https://github.com/replicate/replicate/issues/366
  - IanCal 6y ago
    If you support S3 compatible storage providers that could mean people can use things like minio, that could be a shorter path to supporting lots of backends.
salty_biscuits 6y ago

m0sth8 6y ago

Congratulations with the launch. We've used https://github.com/iterative/dvc https://github.com/iterative/dvc for a long time and quite happy. What's the main difference between replicate.ai and dvc?

mwnivek 6y ago
I'd be curious about comparison with https://github.com/mlflow/mlflow https://github.com/mlflow/mlflow
- bfirsh 6y ago
  We talked to a bunch of MLflow users, and the general impression we got is that it is heavyweight and hard to set up. MLflow is an all-encompassing "ML platform". Which is fine if you need that, but we're trying to just do one thing well. (Imagine if Git called itself a "software platform".) In terms of features, Replicate points directly at an S3 bucket (so you don't have to run a server and Postgres DB), it saves your training code (for reproducibility and to commit to Git after the fact), and it has a nice API for reading and analyzing your experiments in a notebook.
  - Jugurtha 6y ago
    Congrats on the launch! >MLflow is an all-encompassing "ML platform" Not really. We're trying to use MLflow with our "ML platform"[0]. Namely, it can save a model that expects high dimensional inputs, which is most models I've seen that aren't trivial, and can "deploy" the model but with an expectation of two dimensional DataFrame inputs. Apparently, they're working on that. There are also many ambiguities concerning Keras and Tensorflow stemming from "What is a Keras model? Is it a Tensorflow model now they're integrated? Why are Keras models logged with the tensorflow model logger when you use the autolog functionality?". These are shared ambiguities, as there are several ways to save and load models with Tensorflow, and we're looking into the Keras/Tensorflow integration closely. MLflow uses `cloudpickle` and unpickling expects not only the same 'protocol', but the same Python version. Had to dig deeper than necessary. One other problem is when a model relies on ancillary functions, which you must be able to ship somehow. You end up tinkering with its guts, too. Could you shed some light on how do you deal with these matters. Namely, high dimensional inputs for models, pre-processing/post-processing functions, serialization brittleness, and Keras/Tensorflow "duality". We have to inherit that complexity to spare our users from having to mentally think of saving their experiments (we do that automatically to save models, metrics, params). The workflow is data --> collaborative notebooks with scheduling features and job --> (generate appbooks) --> automatically tracked models/params/metrics --> one click deployment --> 'REST' API or form to invoke model. Aaaaaand again, congrats on the launch! - [0]: https://iko.ai https://iko.ai

mfDjB 6y ago

As someone who tried to use git to do this for large sets of data, I'm very glad this exists. Will be trying this out in the future.

kevlar1818 6y ago
You may also be interested in a simple tool I'm building that works in concert with source control to store, version, and reproduce large data: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud My project is in its infancy (open-sourced less than a month ago), but I'm pleased with its UX thus far. There's lots to add in terms of documentation, but Dud currently uses Rclone[1] for remote syncing. [1]: https://rclone.org/ https://rclone.org/
konda 6y ago
I'd checkout dolthub.com! Dolt is built for git workflows (branches and merges) on top of large datasets.
- timsehn 6y ago
  Thanks for the shout out here. CEO of DoltHub speaking :-)
  - deleted 6y ago
    [deleted]

mindhash 6y ago

Congrats on the launch. I have built an open source tool (called hyperML) for similar problem sometime back. I think the problem is not just storing in version control but being able to quickly retrieve those in live /test systems or containers. Mounting and loading datasets and models is painful. It kind of what makes local training a better option. If only the weights were version controlled by libraries (tf, pytorch or scikit) this whole problem will be much easier to solve.

breck 6y ago

nit: "Throw away your spreadsheet" scares me a little. I love spreadsheets, and think there are 100x+ more users of spreadsheets than notebooks (though the overlap of notebook users and ML users is probably close to 1, so I see your point). I would always save my experiment results so they were ready to analyze in spreadsheets (and other vis tools).

fagerhult 6y ago
Hi, Andreas here. Yes spreadsheets are great, and better than notebooks in many cases. But I always felt like I was doing something wrong when I used spreadsheets and markdown files to manually record metrics and hyperparameters for my experiments. It's error prone and easy to forget to update the spreadsheet with new experiments. So we're trying to automate recording this metadata, but then give you that metadata in various ways for you to inspect it. One of those ways is actually spreadsheets: https://github.com/replicate/replicate/issues/289 https://github.com/replicate/replicate/issues/289
Jugurtha 6y ago
Do you not think that using spreadsheets for ML experiment tracking is a symptom of broken tooling? I'm asking because one of the reasons we're building our platform[0] with automatic experiment tracking, collaborative notebooks, and a bunch of things, is because experiment tracking was inconsistent between team members. Differences in tools used: (spreadsheets, flat files, logs, pen and paper, human memory). Forgetting to do it. Snippets to do it flying around. Different locations (laptop, group workstation, git repository, cloud sheet). Dissociated from the notebook that produced the model. Tighter tracking should answer questions like: what notebook ran on which data and produced which model with which parameters and which scores? Then questions like: give me all notebooks that ran on this dataset which produced a model with scores that are [condition]. Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself. I think it's good there are tools with this granularity that can be composed. - [0]: https://iko.ai https://iko.ai
- breck 6y ago
  > Once you do that, the "spreadsheet" can just be a "view" of the underlying data. Something you can export as, but not the thing itself. Yes, and sorry because rereading my comment I wasn't clear, but this is what I meant. I view Spreadsheets as the view (but also an editor of the view), but I don't mean people should be working with XLS files (TSVs seem to work the best— I personally hate tabs but may have lost that battle). I do think though of Spreadsheets as the primary view, and so always design my data structures with the understanding of "how will this interoperate with spreadsheets". JSON is the pits. IMO 2-D DSLs with Spreadsheets as the primary view/editor paradigm are the future. I think Spreadsheets are 1, if not 2 OOM better than notebooks for doing actual work (notebooks can be good for presenting results in a narrative): - non-linear for both humans and machines (allows for really creative and fast out-of-order parsing techniques on the machine side) - concise signal with high information density - unlimited cursors - fantastically easier for version control and multi-player experiences I maintain a list of all data science tools to try and stay on top of best practices, and when I see a tool is notebook first, I think "good, less work for me to track this one because they aren't getting the core things right yet." (Though often times notebooks will have really innovative orthogonal features).

lcap 6y ago

How does this compare to tools like neptune.ai, weights and biases and so on? I can see the advantage of having control of one's data, whereas these tools use their own servers. However what I love about them is the amazing UI that allows me to compare experiments.

bfirsh 6y ago
This came out of a practical problem: at Spotify, Andreas couldn't let any data leave their network. He wasn't going to go through procurement to buy an enterprise version of one of those products, so his only option left was open source software. But it's also out of principle: we think such a foundational thing needs to be open source. There is a reason most people use Git and not Perforce. Replicate can work alongside visualization tools -- your data is safe in your own S3 bucket, but you can use the hosted visualization tool to complement that. You could also imagine visualization tools built on top of Replicate. One thing we've been thinking about is doing visualization inside notebooks. It's like a programmable Tensorboard: https://colab.research.google.com/drive/18sVRE4Zi484G2rBeOYjobE3zek2gDBvy https://colab.research.google.com/drive/18sVRE4Zi484G2rBeOYj... I'd be curious to hear your thoughts about that. It's pretty primitive so far, but we've got to start somewhere I suppose. :)
- lcap 6y ago
  The idea looks very good! I have to say however that I am a bit wary of notebooks. I agree how great they are to visualize things, but they also tend to have a negative effect on the coding standards of a team. This is again probably a personal bias and they look to be suited for this use case. Great launch anyway!

rq1 6y ago

I was looking for something similar today. I just adopted it. :) Thank you for your amazing work! Do you have the intention to integrate it with PT Lightning or as a PT Lightning Logger? It would be nice to have it maintained there. It’s used all over huggingface-Transformers examples.

fagerhult 6y ago
Fantastic, thank you for those kind words! And great idea to integrate with PT Lightning. I just opened an issue: https://github.com/replicate/replicate/issues/367 https://github.com/replicate/replicate/issues/367, feel free to add more detail and comments! -andreas

mkuklik 6y ago

Congrats on the launch. Have you looked at https://comet.ml https://comet.ml If so, how do you compare to them?

bfirsh 6y ago
Thanks! It's open source and you're in control of your own data. See https://news.ycombinator.com/item?id=25151741 https://news.ycombinator.com/item?id=25151741

flc-anonym 6y ago

Congrats on the launch! This looks interesting, however I feel like this space is quite crowded. You mentioned that your most important feature is the fact that you are open-source, but off the top of my head I can think of several projects: * Kubeflow: https://github.com/kubeflow/kubeflow https://github.com/kubeflow/kubeflow * MLFlow: https://github.com/mlflow/mlflow https://github.com/mlflow/mlflow * Pachyderm: https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm * DVC: https://github.com/iterative/dvc https://github.com/iterative/dvc * Polyaxon: https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon * Sacred: https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred * pytorch-lightning + grid: https://github.com/PyTorchLightning/pytorch-lightning https://github.com/PyTorchLightning/pytorch-lightning * DeterminedAI: https://github.com/determined-ai/determined https://github.com/determined-ai/determined * Metaflow: https://github.com/Netflix/metaflow https://github.com/Netflix/metaflow * Aim: https://github.com/aimhubio/aim https://github.com/aimhubio/aim * And so many more... In addition to this list, several other hosted platform offer experiments tracking and model management. How do you compare to all of these tools, and why do you think users should move from one of them to use replicate, thank you.

bfirsh 6y ago
Yeah, I agree this space is crowded. But we’ve found so few ML researchers/engineers are actually using these tools. This could either be that people aren’t aware of them yet, or that they’re not good enough. I think it’s a mix of both, honestly, but we’re betting that there’s more of the latter in the mix. :) I could do comparisons of each of these tools, and some of them are solving quite different problems, but the overarching difference is we’re just trying to do less. These systems might make sense if you’re setting up a company’s ML pipeline, but we found lots of individuals struggling to keep track of their work and store their models. They balked at the idea of setting up things like Kubeflow or MLflow.
- mlthoughts2018 6y ago
  Can you share any of the data or market research on this? I am an ML manager in a large ecommerce firm and we stand up our own feature store, experiment tracking system and model training diagnostic / metric system (a la Tensorboard). It is exceedingly easy to DIY, I’ve been doing that stuff DIY with teams of less than 8 engineers for many years. What I’ve seen is that most vendor solutions aren’t flexible enough. Many firms have their own on-prem and data privacy restrictions that make hosting model training artifacts on a vendor’s servers impossible, and it’s way easier to sell building in house. I’m very surprised you are hearing that nobody is using MLflow and Kubeflow (though Kubeflow has a lot of genuine usability and bug problems). I am hearing the exact opposite. Everybody, from tiny 5 person startups to giant ecommerce firms is just spinning up simple stuff for model tracking, mostly around MLflow.

gitgud 6y ago

Congrats on the launch! What's the business concept behind the tool? Can't find anything on the homepage, or in the post. I didn't know YC funded opensource tools like this, it's kind of refreshing.

bfirsh 6y ago
Yeah, YC is funding lots of open source projects. PostHog[0] was in our batch. GitLab, Docker, Mattermost, and CoreOS come to mind as other open source YC companies. There are a number of businesses we could build around the project. A cloud service or enterprise products/support are the obvious ones. Right now, we're focused on community building, because a potential open source business can't be successful with a healthy open source project. [0] https://news.ycombinator.com/item?id=22376732 https://news.ycombinator.com/item?id=22376732
- gitgud 6y ago
  > There are a number of businesses we could build around the project. So you got funding from YCombinator without a concrete plan to make a business? That's pretty interesting, I always thought they wanted profitable businesses, and turned down ideas they didn't think would work. Great to hear they're betting on open-source more!
  - bfirsh 6y ago
    Funnily we applied with a different thing. We tried a number of different ideas before we settled on this lower-level thing, as I describe in the main comment. Even still, I think most people apply to YC without a concrete plan of how to make a business. It's normally so early stage, that the plan is yet to be validated and will probably change. A "plausible" plan is perhaps a better way to put it. ;)

deleted 6y ago

[deleted]

thecellardoor 6y ago

Congrats on the launch. To me this feels like DVC but with a slightly more convenient python API, and without the pipeline which might be really great. How are you organizing experiments though? Without Git it seems too easy to have experiments become unusable. And filtering according to accuracy is not really a solution..

edameme 6y ago

Congrats on the launch! Very exciting tool. I'm one of the creators of https://DAGsHub.com https://DAGsHub.com which lets you host and collaborate on data science projects – think data science pull requests and data merging. We're integrated with DVC for data versioning and experiments, but what you're building is definitely super interesting.

visarga 6y ago

What is the best way to do dataset versioning when using Replicate? I get that Replicate saves the dataset version but not the data itself. Is DVC a good fit? Second question - can you tag or attach a comment to a group of experiments?

fagerhult 6y ago
Great questions! At the moment we recommend passing dataset URIs as params to replicate.init(): https://replicate.ai/docs/guides/training-data https://replicate.ai/docs/guides/training-data, but of course this assumes immutable and stable URIs. DVC would definitely be a good fit, and we have a ticket on our roadmap to integrate Replicate with DVC, Tecton, etc. https://github.com/replicate/replicate/issues/294 https://github.com/replicate/replicate/issues/294 We also have a roadmap ticket for grouping experiments: https://github.com/replicate/replicate/issues/297 https://github.com/replicate/replicate/issues/297, but for now we're recommending params for tags as well. If you have ideas for the design of these features, we really appreciate feedback and comments on these Github issues!

otsaloma 6y ago

Great to see this! I've been looking at alternatives for experiment tracking, but most of what I've come across have looked way too heavy (and expensive) for a small team. Is R support planned? Or might this just work already with reticulate?

foodbaby 6y ago

This looks similar in philosophy and approach to https://guild.ai https://guild.ai although guild ai can parse stdout to extract metrics to track, and is integrated with tensorboard.