This isn't your standard MAB or GP hyperparameter optimization; those typically require you to train each NN to convergence before further exploration is done (

This isn't your standard MAB or GP hyperparameter optimization; those typically require you to train each NN to convergence before further exploration is done (ie each 'round' is training a NN). Skimming the paper, OP is closer to freeze-thaw or reversible backpropagation hyperparameter optimization, or Net2Net meta-RL: the hyperparameter optimization is monitoring the loss curve of each trained NN, switching between them based on promisingess like in freeze-thaw, but also switching hyperparameters on the fly and reusing the trained weights to avoid starting from scratch, Net2Net style. Each NN being trained is periodically updated to either clone & tweak a new hyperparameter set to continue training the current NN's parameters, or clone & tweak the best NN's parameters while keeping the old hyperparameters. (They only clone the full NN, so they can't do architecture search, but there's no reason they couldn't use Net2Net or other recent approaches which similarly recycle the trained weights to avoid the huge computational burden of training from scratch.)