3 ms·
Would love to see these numbers reproduced on consumer GPUs, not just A100s.
by danielabinav160 3mo ago
Would love to see these numbers reproduced on consumer GPUs, not just A100s.
- tommica 3mo agoMaybe somaday an 8gb videocard can be used for coding...
- romanusrome 3mo ago[dead]
- wolttam 3mo agoThis is an efficiency improvement that significantly lowers the amount of RAM you have to look at, on average, during decode. It should improve performance on most hardware because most LLMs are memory bandwidth bound during decode.