Different configuration search strategies will perform differently on different fitness landscapes, and fitness landscapes depend on both data and learning algo

Different configuration search strategies will perform differently on different fitness landscapes, and fitness landscapes depend on both data and learning algorithm. An online bandit-style selection of what seems to be working could be a good idea. Portfolio-management algorithms do this sort of thing. Hyperopt can accommodate this type of algorithm. File a ticket on the issue tracker if you'd like to work on this and I'll help you get started.