3 ms·
I have extensively tested Optuna and Weights and Biases (WandB) for hyperparameter tuning on multiple task-specific transformer models back in 2020. Optuna los
by gillesjacobs 3y ago
I have extensively tested Optuna and Weights and Biases (WandB) for hyperparameter tuning on multiple task-specific transformer models back in 2020.
Optuna lost out by a long mile back then in feature parity and dashboarding. Optuna did not have hyperband optimization which was and still is one of the best search algos for hyperopt. It looks like it is possible to implement hyperband yourself now, but in the loosely coupled architecture between Sampler and Pruner it's a bit baroque [1].
Anyway back then it was clear WandB was the far superior choice for features, ease of use, experiment tracking and dashboarding. We went with WandB for our lab.
Could be Optuna caught up, but WandB has seen significant development too. Looking at their dashboard docs, it looks meagre compared to what you can do with WandB.
1. https://tech.preferred.jp/en/blog/how-we-implement-hyperband-in-optuna/ https://tech.preferred.jp/en/blog/how-we-implement-hyperband...
- zwaps 3y agosadly, w&B means you have to upload it to the cloud, which is not possible in every case :(
- michaelmior 3y agoWhat is the "it" you're referring to? You don't need to upload your model or the weights. You do need to upload the hyperparameters you're optimizing and your target values, but those seem unlikely to be sensitive. (Although I'm sure there are still some legitimate reasons why someone might not want to do so.)
- gillesjacobs 3y agoYou need to upload your time series (loss, performance metrics) to the cloud. Not your weights or models. Reliance on cloud services is a legitimate worry though for privacy, IP, process control, reliability, etc. The comparison between Optuna and WandB was not apples to apples. Optuna is completely self-hosted and local. It also focuses on hyperopt narrowly with flexible design unlike WandB that now assumes to be capture a large part of cloud-based MLOPS workflow. It would be more fair to compare Optuna to Hyperopt. And I think Optuna was the better choice there, but I did simple PoCing and have no strong opinions.
- michaelmior 3y agoYou don't need to upload any time series data. You can just upload your the final metrics at the end of the run. Admittedly though, W&B loses a lot of its value in that case since you can't really get any insight into runs in progress.
- tkellogg 3y agolooks like they have it now https://optuna.readthedocs.io/en/stable/reference/generated/optuna.pruners.HyperbandPruner.html https://optuna.readthedocs.io/en/stable/reference/generated/...
- 3abiton 3y agoWhen did you do this test? I am curious about the current performance gap.
- gillesjacobs 3y agoBack in 2020, so that's 4 years of active development on both tools since then. Both tools have different scopes though (cf my other comment in the thread).
- bionhoward 3y agoI didn’t even know WandB did hyperparameter optimization, I figured it was a neural network visualizer based on 2 minute papers. Didn’t seem like many alternatives out there to Optuna with TPE + SQL storage persistence in conditional continuous & discrete spaces. Anyway, it’s doable to make a multi objective decide_to_prune function with Optuna, here’s an example https://github.com/optuna/optuna/issues/3450#issuecomment-1995081291 https://github.com/optuna/optuna/issues/3450#issuecomment-19...
- greensh 3y agoThere is also mlflow in combination with optuna which is great to visualize and dashboard.