3 ms·
Tokens are mapped to keys, values and queries. Keys and values for past tokens are cached in modern systems, but the essence of the Transformer architecture is
by tomp 2y ago
Tokens are mapped to keys, values and queries.
Keys and values for past tokens are cached in modern systems, but the essence of the Transformer architecture is that each token can attend to every past token, so more tokens in a system prompt still consumes resources.
- fenomas 2y agoThat makes sense, thanks!