4 ms·

Hi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hype

by gidim 9y ago

Hi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hyperparameters and results. We think that reproducibility is really important so we’re also giving free access to students, academics and open source projects. Feedback is welcome. Ask me anything.

deleted 9y ago
[deleted]
ah- 9y ago
Are you planning to open source it? A lot of your competitors have, like http://pipeline.ai/ http://pipeline.ai/, https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm and recently https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon.
- mmq 9y ago
  @ah- thanks for mentioning Polyaxon and congrats to the CometMl team for building this nice tool, it's good to see that many projects are trying to solve problems related to reproducibility in ML/DL, many people had to build an internal tool for the companies they work for to solve this issue, and many got frustrated after joining a new team and were not able to reproduce any results. I would like to outline a couple of differences between CometML and Polyaxon, as mentioned before, we are also trying to solve issues related to technical debt in ML, but not only, Polyaxon tries also to simplify training and scheduling parallel and distributed learning. there are also a couple of differences, I see CometML as dashboard, Polyaxon does not have an extensive dashboard as CometML, but it leverages Tensorboard for most of the visualisations. We use the CLI or the API for programatic access to the platform. Most importantly, Polyaxon aims to be an open source and to be installed on premise or in the cloud, it solves the issue related to code tracking based on an internal git and a docker registry, and as someone else mentioned that resources for running an experiment could be an issue for future reproducibility, Polyaxon restarts the experiments with the same resources and dockerfiles, it also tracks hyper params as part of the configuration. For hyper params tuning and suggestion, Polyaxon can also do hyper params search based on a couple of algorithms, and for the next release, it will include also a service similar to vizier for suggesting more experiments/group of experiments based on a given search space. Disclaimer: I am the author of Polyaxon
jchung 9y ago
Looks awesome. I'd love to give it a try with my team. Would you be open to extending the free teams plan to high tech nonprofits in addition to the access you provide to students, academics, and open source projects? We tend to work on distributed projects frequently with industry experts doing pro bono work for us, and something like comet could simplify our collaboration. The size of these pro bono project teams tend to ebb and flow, much like an open source project, so effective collaboration tools are critical for ramping up new folks as well as retaining learning when folks cycle off.
- gidim 9y ago
  Sure. Shoot us an email and we'll get you started! mail@comet.ml
shackenberg 9y ago
Hi, looks great! Love the feature of tracking code changes. But how does this work? Does it upload the code to the servers every time I launch a training?
- gidim 9y ago
  It depends if it's a git project but pretty much yes.
  - shackenberg 9y ago
    So you would considered the latest committed version as the 'code of the current experiment'? Maybe to rephrase: I start a training with my local code and then I change one variable in the code or comment some processing and start the next training. Would I need to do something for comet to know how the code has changed?
    - gidim 9y ago
      No. You'll be able to see both runs with the code diffs.
ReverseCold 9y ago
Does "did what GitHub did for code" sound like a negative thing to anyone else? It turned a decentralized platform (git) into basically the only place individuals store code.

by gidim 9y ago

deleted 9y ago

[deleted]

ah- 9y ago

Are you planning to open source it? A lot of your competitors have, like http://pipeline.ai/ http://pipeline.ai/, https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm and recently https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon.

mmq 9y ago
@ah- thanks for mentioning Polyaxon and congrats to the CometMl team for building this nice tool, it's good to see that many projects are trying to solve problems related to reproducibility in ML/DL, many people had to build an internal tool for the companies they work for to solve this issue, and many got frustrated after joining a new team and were not able to reproduce any results. I would like to outline a couple of differences between CometML and Polyaxon, as mentioned before, we are also trying to solve issues related to technical debt in ML, but not only, Polyaxon tries also to simplify training and scheduling parallel and distributed learning. there are also a couple of differences, I see CometML as dashboard, Polyaxon does not have an extensive dashboard as CometML, but it leverages Tensorboard for most of the visualisations. We use the CLI or the API for programatic access to the platform. Most importantly, Polyaxon aims to be an open source and to be installed on premise or in the cloud, it solves the issue related to code tracking based on an internal git and a docker registry, and as someone else mentioned that resources for running an experiment could be an issue for future reproducibility, Polyaxon restarts the experiments with the same resources and dockerfiles, it also tracks hyper params as part of the configuration. For hyper params tuning and suggestion, Polyaxon can also do hyper params search based on a couple of algorithms, and for the next release, it will include also a service similar to vizier for suggesting more experiments/group of experiments based on a given search space. Disclaimer: I am the author of Polyaxon

jchung 9y ago

Looks awesome. I'd love to give it a try with my team. Would you be open to extending the free teams plan to high tech nonprofits in addition to the access you provide to students, academics, and open source projects? We tend to work on distributed projects frequently with industry experts doing pro bono work for us, and something like comet could simplify our collaboration. The size of these pro bono project teams tend to ebb and flow, much like an open source project, so effective collaboration tools are critical for ramping up new folks as well as retaining learning when folks cycle off.

gidim 9y ago
Sure. Shoot us an email and we'll get you started! mail@comet.ml

shackenberg 9y ago

Hi, looks great! Love the feature of tracking code changes. But how does this work? Does it upload the code to the servers every time I launch a training?

gidim 9y ago
It depends if it's a git project but pretty much yes.
- shackenberg 9y ago
  So you would considered the latest committed version as the 'code of the current experiment'? Maybe to rephrase: I start a training with my local code and then I change one variable in the code or comment some processing and start the next training. Would I need to do something for comet to know how the code has changed?
  - gidim 9y ago
    No. You'll be able to see both runs with the code diffs.

ReverseCold 9y ago

Does "did what GitHub did for code" sound like a negative thing to anyone else? It turned a decentralized platform (git) into basically the only place individuals store code.