3 ms·
> The encoder is five-layer bidirectional LSTM (long short-term memory) network. In contrast with neural machine translation, we do not use an attention mechani
by minimaxir 8y ago
> The encoder is five-layer bidirectional LSTM (long short-term memory) network. In contrast with neural machine translation, we do not use an attention mechanism but instead have a 1,024-dimension fixed-size vector to represent the input sentence.
5 layers of 1024-cell bidirectional LSTMs (edit: actually 512-cells x2?)? Can consumer GPUs even fit that (+ the decoder) into RAM?
- isoprophlex 8y agoNotwithstanding the memory complexity, I love that they reach such impressive results with relatively straightforward components: theres no attention mechanism, and encoding into latent space is simply max pooling over the last LSTM layer. I don't fully understand how the training step works though. They train only on translation to english and Spanish? Anyway very cool stuff.
- pmalynin 8y agoIt’s not 1024 cells. The size of the hidden vector is 1024. Which is roughly an order of megabyte per cell (1024x1024 matrix). Here they have a cell per word, which is reasonable.
- minimaxir 8y agoMax-Pooling a 1024-cell output LSTM will result in a 1024-sized vector. Looking at the code for the Encoder (https://github.com/facebookresearch/LASER/blob/fec5c7d63daa2bea9d127e8252b10e87278586c0/source/embed.py#L169 https://github.com/facebookresearch/LASER/blob/fec5c7d63daa2...), each LSTM has the same amount of hidden cells. (although the default parameters of that class don't quite match the ones used in the post; so I assume it's 512x2x5).
- pmalynin 8y agoYes but they're not maxpooling in the last dimension. They're max pooling over the sequence length [0], (the other way doesn't really make sense in this context). The output size is 1024, the hidden vector size is 512 but they're using bidirectional LSTMs which concatenates the outputs of each direction -- so the total is 1024 [1]. [0] https://github.com/facebookresearch/LASER/blob/fec5c7d63daa2bea9d127e8252b10e87278586c0/source/embed.py#L246 https://github.com/facebookresearch/LASER/blob/fec5c7d63daa2... [1] https://pytorch.org/docs/stable/nn.html#lstm https://pytorch.org/docs/stable/nn.html#lstm
- minimaxir 8y agoGotcha, that makes sense. (I'm less familiar with dimension ordering in PyTorch)