If you use Keras, you might have noticed the dropout_W and dropout_U arguments on RNN layers. These calculate dropout using Gal's recommendation, "variational d

If you use Keras, you might have noticed the dropout_W and dropout_U arguments on RNN layers. These calculate dropout using Gal's recommendation, "variational dropout". With other ways of applying dropout, LSTMs typically fail to converge --- and with no dropout, they often over-fit. Gal's variational dropout therefore brings a significant improvement to many leading models. There are several other nice contributions in the thesis as well, including a recommendation for applying dropout to word embedding matrices that I don't think has been well explored yet.