2 ms·
This can be combined with Grouped Query Attention or Multi-Query Attention for an even further reduction in the size of the KV Cache!
by tripplyons 2y ago
This can be combined with Grouped Query Attention or Multi-Query Attention for an even further reduction in the size of the KV Cache!
- WhitneyLand 2y agoBut it’s not free, it cuts quality significantly. It’s not hard to find ways to speed up transformers if you’re willing to give up quality. You could argue some tradeoffs are worth it and that’s true sometimes but I don’t see that they’ve made the case for it here.
- funicular 2y ago[dead]