3 ms·
Being a RNN there is another trick: caching a long prompt, because RNNs only look back one step while transformers see the whole sequence. So you can load your
by visarga 3y ago
Being a RNN there is another trick: caching a long prompt, because RNNs only look back one step while transformers see the whole sequence. So you can load your long context only once and reuse it many times.
- pico_creator 3y agoYup. This is commonly done in the community for the chat models as well (due to the huge amount of reuse for each reply)