4 ms·
Vanishing gradient isn't the same as memory efficiency. The memory mirror option is what allows this extremely efficient memory usage by only being 30% more com
by deepnotderp 10y ago
Vanishing gradient isn't the same as memory efficiency. The memory mirror option is what allows this extremely efficient memory usage by only being 30% more compute intensive.
- bsfjgngdnxy 10y agoYes, but that's not what I asked about.
- alexbeloi 10y agoVanishing gradient is solved using model architectural choices: ReLu activation instead of sigmoid or tanh, using batch-normalization, using LSTMs These are orthogonal to memory management and neural net framework choices.