3 ms·
You need to include the prompt in every query, which makes it very expensive
by WithinReason 2y ago
You need to include the prompt in every query, which makes it very expensive
- GaggiX 2y agoThe prompt is kv-cached, it's precomputed.
- WithinReason 2y agoGood point, but it still increases the compute of all subsequent tokens