With increasing dataset sizes and more compute/experimentation needed to get state-of-the-art results, hyperparameter tuning is becoming important. However, I

With increasing dataset sizes and more compute/experimentation needed to get state-of-the-art results, hyperparameter tuning is becoming important. However, I think it needs to be supported by a framework for distributed computing. We also think it is better to centralize all hyperparameters is an Optimizer object rather than have to (1) change existing code and (2) have hyperparameters spread out throughout the code (which would have to be rewritten again for distributed training). We did something similar to this, but it is data-parallel and is Python (built on top of PySpark) and supports a resource mgr with GPUs so that you can allocate GPUs to Spark executors. The main challenge we had was actually printing out logs live in the Jupyter notebook, while keeping the code Pythonic. Data Scientists do not like to leave their notebook to read logs - even if they know their are 10s of hundreds of parallel executors generating them. https://twitter.com/jim_dowling/status/1141847097756520453 https://twitter.com/jim_dowling/status/1141847097756520453