3 ms·
Thankfully, there are methods (like causal transformers, but also some question/text/answer architectures) for not re-computing self-attention between the docu
by namibj 3y ago
Thankfully, there are methods (like causal transformers, but also some question/text/answer architectures) for not re-computing self-attention between the document/knowledge-base tokens across the iterations of autoregressive decoding for generating the response.
- aik 3y agoDo you have an example of where these methods still produce good summaries? Eg if you adjust how re-computation of self-attention in autoregressive decoding / between token generations works to significantly decrease the amount of computation needed?