3 ms·
This sounds like a game changer. I wonder if they need to do a tonne of specific work per model? If this could be implemented in Ollama, I'd be over the moon.
by 1GZ0 2y ago
This sounds like a game changer. I wonder if they need to do a tonne of specific work per model?
If this could be implemented in Ollama, I'd be over the moon.
- nutrientharvest 2y agoOllama can already run Llama-3 70B with a 4GB GPU, or no GPU at all, it'll just be slow. Considering this says it's "not designed for real-time interactive scenarios" it's probably also really slow
- cpill 2y agoso how much GPU RAM does need to get the 70B going fast (ish)?
- AaronFriel 2y agoA good rule of thumb is that models can be quantized to 6 to 8 bits per weight without significantly degrading quality. This is convenient for the math: 70GB plus some overhead for the attention matrices (ongoing requests). This overhead depends on workload and context lengths, but you should expect about 30% more. So, around 100GB for a server under load.
- programd 2y agollama3:70b using llama.cpp (used under the hood by Ollama) on a 11th Gen Intel i5-11400 @ 2.60GHz - no GPU, CPU inference only. "Write a haiku about Hacker News mentioning AI in the title" Here is a haiku: AI whispers secrets HN threads weave tangled debate Intelligence born eval time = 30363.04 ms / 23 runs ( 1320.13 ms per token, 0.76 tokens per second) total time = 34294.80 ms / 33 tokens
- bityard 2y agoThat really doesn't seem bad. When people talk about responses of self-hosted LLMs without a beefy GPU being unusably slow, I always assumed they meant 15 minutes to hours. I do not mind waiting a few minutes if it will summarize the answer a question that will take me many times longer to research.
- logicallee 2y agohow much disk space did it use?