3 ms·
The model still needs to attend to the prompt when generating the answer. Modern attention techniques help here, but for lots of simple queries most of the comp
by macleginn 1y ago
The model still needs to attend to the prompt when generating the answer. Modern attention techniques help here, but for lots of simple queries most of the compute still goes into taking the system prompt into account, I guess.
- saagarjha 1y agoSure, but without the prompt you will probably have significantly "worse" queries, because you'll be starting from scratch without that context.