Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
MyPhiloEngine
searching PlanetScale…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
2 ms
·
1.
▲
by
MyPhiloEngine
2mo ago
With 128 GB of unified memory, large models can fit into memory, but memory bandwidth quickly becomes the limiting factor. With long contexts in particular, the KV cache can consume a lot of memory and bandwidth. Quantizing the KV cache to