4 ms·
No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the
by memhole 2y ago
No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the context window. If anyone has good resources on this, that would be awesome. Presently, it feels very much like a dark art to host these without crashing or being massively over-provisioned.
- htrp 2y agoThe dark art is to massively overprovision hardware.
- cratermoon 2y agoThus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2] 1. https://arxiv.org/abs/2412.19437v1 https://arxiv.org/abs/2412.19437v1 2. Quibble over the exact figure. Far less than Open AI, doing more with less.