3 ms·
Neither RNNs nor transformers will change their parameters during a forward pass. In production GTP just does a forward pass through the network, but never calc
by Buttons840 4y ago
Neither RNNs nor transformers will change their parameters during a forward pass. In production GTP just does a forward pass through the network, but never calculates the parameter gradients, and this would be true if GTP was built on RNNs as well, and without those gradients the AI will not improve.
- seanhunter 4y agoThat seems an implementation detail that would be entirely solvable with some additional work rather than a fundamental limitation though.
- Der_Einzige 4y agoUnfortunately we have this thing called the curse of dimensionality and specifically we do not understand how to prevent catastrophic forgetting. This is why incrementally learning NNs are still so rare in practice.