3 ms·
I wonder if this caches the attention keys and values for the "system prompt" that includes this custom instruction information, so the model can resume without
by QuadrupleA 3y ago
I wonder if this caches the attention keys and values for the "system prompt" that includes this custom instruction information, so the model can resume without inferencing the whole thing every time?