It would be interesting to see what would happen if you also tried to tune the ensemble towards a specific task in the same way that you could tune a single mod

It would be interesting to see what would happen if you also tried to tune the ensemble towards a specific task in the same way that you could tune a single model. We've definitely seen that tuning the embedding hyperparameters (along with the others) can have a significant impact on performance. [1] Additionally, whenever you open up the space of tunable parameters to include the embeddings or feature representations themselves you can usually significantly outperform just a well tuned classifier. [2] This model seems like it trades off complexity in tuning for complexity of an ensemble, but I wonder what would happen if you tried to have your cake and eat it too and just tuned everything. [1]: https://aws.amazon.com/blogs/machine-learning/fast-cnn-tuning-with-aws-gpu-instances-and-sigopt/ https://aws.amazon.com/blogs/machine-learning/fast-cnn-tunin... [2]: https://blog.sigopt.com/posts/unsupervised-learning-with-even-less-supervision-using-bayesian-optimization https://blog.sigopt.com/posts/unsupervised-learning-with-eve...