Being able to use SQL for later analysis is definitely a good idea. For smaller models SQLite for sure is enough but as soon as you want to scale your HPO acros

Being able to use SQL for later analysis is definitely a good idea. For smaller models SQLite for sure is enough but as soon as you want to scale your HPO across multiple servers or even just processes, you will need something that supports a multi-user database. E.g. Optuna supports PostgreSQL and also defaults to SQLite as far as I know.