9 ms·

Aim: Record, search and compare ML training runs

45 points by polm23 6y ago

marcinzm 6y ago
Honestly, it's really hard for me to tell from the documentations if this is something that is self-hosted, is run locally or is a SaaS product. The phrase "open source" shows up nowhere I can see for example. As best I can tell it's an open source version of comet or wandb that you run locally (or I guess host somewhere). Is that right?
- ellisv 6y ago
  From what I can glean from the docs, you run the UI locally which reads from experiments from `.aim/` The experiments results could be populated by running the training job remotely and committing the `.aim/` directory afterwards. It's kinda like a UI for DVC
  - jszymborski 6y ago
    yes, it looks like it's meant to be run in a similar way to Tensorboard
    - gevorg_s 6y ago
      yep.
  - gevorg_s 6y ago
    Yea, spot on. Right now UI is ran locally and it reads the data from `.aim`. We have seen users deploy it for the team and have seen standalone in the local environment too. Once training is ran, the logs are saved in the `.aim`. We are using a format that is ~50% more memory efficient than the tb logs - and searchable (see the aimrecords repo).
- gevorg_s 6y ago
  Hi, Aim is fully open source and self-hosted (at the moment - just like tensorboard). Open source version of wandb would be one way to put it. But we are building a new way of interacting with the ML training runs that lets the researchers compare lots of them (1000s) in really short period of time while having full access to the context of the experiments. This is a super early version. And lots more work needs to be done. We have implemented a pythonic search to search through the experiments that is easy to use. Hopefully this sheds more light to the work we are doing.
frakt0x90 6y ago
I've seen a lot these projects recently. I have used MLFlow for a while and enjoyed it. How does this compare?
- jszymborski 6y ago
  or how either compares with Tensorboard and its HPARAMS tab? (PSA: PyTorch Lightning makes it even easier to use).
  - gevorg_s 6y ago
    Pls see my answers above.
  - gevorg_s 6y ago
    the big diff and the advantage is that on tb you can't group by hyperparams or easily divide into subplots while having all the research info in fro nt of you. Aim does that and also aggregates groups of runs to reduce the dimension and make it easy to compare. It has a proper search by hyperparsams (and everything else tracked/collected really). All in one panel where you can compare 100s of experiments at a time. Loading many runs on TB, with very long names makes it super slow to analyze the runs really. Tbh that has been a motivation for building this open source tool - to have something efficient and beautiful :)
- williamsmj 6y ago
  Yes, I was looking for the comparison to local MLFlow and Tensorboard (and Losswise, and Weights & Biases, and all the remote options in this ecosystem). It looks nice, but without that, it's difficult to know whether it's worth spending more time looking into it.
  - gevorg_s 6y ago
    We are working on a new paradigm on interacting with Ml training runs. A lot of the effort is now focused on very efficient experiment comparison capabilities - talking about 1000s of them. Lots of challenges on the UI and the backend. When loading TB or any other tool really with lots of experiments it's super slow and becomes useless. Also no way to do effective comparison of runs by hyperparams or other metadata on the tensorboard or MLFlow. Quite basic capabilities.
davidbuniat 6y ago
Oh nice, it's like open-source version of W&B, and seems much better than Tensorboard. How many experiments I can compare at the same time without crashing chrome? :)
- gevorg_s 6y ago
  Aim easily handles 100s of ML training runs. There are folks who have 1000s of experiments too. We haven't oficially benchmarked it yet though.
gevorg_s 6y ago
Hi all I am one of the co-authors to this project, and will try to answer all the Qs here. Was just forwarded this link - one of the community members must have posted it.
- pchal 6y ago
  Interesting effort. Does it snapshot the state of source code at the time an experiment is run? Does it do it without requiring a git commit? I believe the Replicate experiment tracking tool does this.
  - gevorg_s 6y ago
    We have got similar requests couple of times and its in the pipeline. Currently focused on the comparison of 1000s of metrics/training runs. It's a serious challenge both on the Ui and on the storage end. Inviting you to the Aim [slack channel](https://slack.aimstack.io/ https://slack.aimstack.io/). We would love to learn more about such use cases and why they are important.
karankanwar 6y ago
This is great stuff :))
mari_lee__ 6y ago
Awesome! Will look forward to the project's further development.
- gevorg_s 6y ago
  Thanks :)
lusinem 6y ago
Wow! Amazing project!

45 points by polm23 6y ago

Honestly, it's really hard for me to tell from the documentations if this is something that is self-hosted, is run locally or is a SaaS product. The phrase "open source" shows up nowhere I can see for example. As best I can tell it's an open source version of comet or wandb that you run locally (or I guess host somewhere). Is that right?

ellisv 6y ago
From what I can glean from the docs, you run the UI locally which reads from experiments from `.aim/` The experiments results could be populated by running the training job remotely and committing the `.aim/` directory afterwards. It's kinda like a UI for DVC
- jszymborski 6y ago
  yes, it looks like it's meant to be run in a similar way to Tensorboard
  - gevorg_s 6y ago
    yep.
- gevorg_s 6y ago
  Yea, spot on. Right now UI is ran locally and it reads the data from `.aim`. We have seen users deploy it for the team and have seen standalone in the local environment too. Once training is ran, the logs are saved in the `.aim`. We are using a format that is ~50% more memory efficient than the tb logs - and searchable (see the aimrecords repo).
gevorg_s 6y ago
Hi, Aim is fully open source and self-hosted (at the moment - just like tensorboard). Open source version of wandb would be one way to put it. But we are building a new way of interacting with the ML training runs that lets the researchers compare lots of them (1000s) in really short period of time while having full access to the context of the experiments. This is a super early version. And lots more work needs to be done. We have implemented a pythonic search to search through the experiments that is easy to use. Hopefully this sheds more light to the work we are doing.

frakt0x90 6y ago

I've seen a lot these projects recently. I have used MLFlow for a while and enjoyed it. How does this compare?

jszymborski 6y ago
or how either compares with Tensorboard and its HPARAMS tab? (PSA: PyTorch Lightning makes it even easier to use).
- gevorg_s 6y ago
  Pls see my answers above.
- gevorg_s 6y ago
  the big diff and the advantage is that on tb you can't group by hyperparams or easily divide into subplots while having all the research info in fro nt of you. Aim does that and also aggregates groups of runs to reduce the dimension and make it easy to compare. It has a proper search by hyperparsams (and everything else tracked/collected really). All in one panel where you can compare 100s of experiments at a time. Loading many runs on TB, with very long names makes it super slow to analyze the runs really. Tbh that has been a motivation for building this open source tool - to have something efficient and beautiful :)
williamsmj 6y ago
Yes, I was looking for the comparison to local MLFlow and Tensorboard (and Losswise, and Weights & Biases, and all the remote options in this ecosystem). It looks nice, but without that, it's difficult to know whether it's worth spending more time looking into it.
- gevorg_s 6y ago
  We are working on a new paradigm on interacting with Ml training runs. A lot of the effort is now focused on very efficient experiment comparison capabilities - talking about 1000s of them. Lots of challenges on the UI and the backend. When loading TB or any other tool really with lots of experiments it's super slow and becomes useless. Also no way to do effective comparison of runs by hyperparams or other metadata on the tensorboard or MLFlow. Quite basic capabilities.

davidbuniat 6y ago

Oh nice, it's like open-source version of W&B, and seems much better than Tensorboard. How many experiments I can compare at the same time without crashing chrome? :)

gevorg_s 6y ago
Aim easily handles 100s of ML training runs. There are folks who have 1000s of experiments too. We haven't oficially benchmarked it yet though.

gevorg_s 6y ago

Hi all I am one of the co-authors to this project, and will try to answer all the Qs here. Was just forwarded this link - one of the community members must have posted it.

pchal 6y ago
Interesting effort. Does it snapshot the state of source code at the time an experiment is run? Does it do it without requiring a git commit? I believe the Replicate experiment tracking tool does this.
- gevorg_s 6y ago
  We have got similar requests couple of times and its in the pipeline. Currently focused on the comparison of 1000s of metrics/training runs. It's a serious challenge both on the Ui and on the storage end. Inviting you to the Aim [slack channel](https://slack.aimstack.io/ https://slack.aimstack.io/). We would love to learn more about such use cases and why they are important.

karankanwar 6y ago

This is great stuff :))

mari_lee__ 6y ago

Awesome! Will look forward to the project's further development.

gevorg_s 6y ago
Thanks :)

lusinem 6y ago

Wow! Amazing project!