3 ms·
They don't address the fundamental issue of BPTT in an RNN model. To my mind this is a much larger block to biological plausibility than how to track the error
by orbifold 6y ago
They don't address the fundamental issue of BPTT in an RNN model. To my mind this is a much larger block to biological plausibility than how to track the error locally. Besides this has been addressed in various forms in earlier work:
- Hochreiter and Schmidhuber present a truncated version of BPTT in the original LSTM paper that tracks part of the error locally and forward in time (https://www.bioinf.jku.at/publications/older/2604.pdf https://www.bioinf.jku.at/publications/older/2604.pdf)
- Bellec et al. https://www.nature.com/articles/s41467-020-17236-y https://www.nature.com/articles/s41467-020-17236-y present a similar algorithm that tracks the error with something they call eligibility traces. Despite the title of the paper the method is also applicable to non-spiking RNNs.
- Bohnstigl et al. present another solution: https://arxiv.org/pdf/2007.12723.pdf https://arxiv.org/pdf/2007.12723.pdf