3 ms·
You only need to cache the key/value pairs. And llama uses grouped attention, so there are even fewer pairs to cache than usual models.
by thomasahle 3y ago
You only need to cache the key/value pairs. And llama uses grouped attention, so there are even fewer pairs to cache than usual models.