3 ms·
Because everybody has the same system prompt, the KV caching will make this a non-issue. The only cost is the reduced max context length.
by mrgaro 2mo ago
Because everybody has the same system prompt, the KV caching will make this a non-issue. The only cost is the reduced max context length.