6 ms·
It's frustrating when developers of ML projects don't state even the most basic requirements. Do I need an Nvidia 4090 or a cluster of H100s to run this?
by janice1999 2y ago
It's frustrating when developers of ML projects don't state even the most basic requirements. Do I need an Nvidia 4090 or a cluster of H100s to run this?
- dmezzetti 2y agoA RTX 3090 is more than enough for 7B LLMs. With 4-bit quantization, you can run inference with an even larger LLM using a 24GB GPU. If you're using remote API services, you might be able to just use a CPU.
- malux85 2y agoIt’s frustrating when people ask for hardware requirements without stating what they are trying to do, do you have 100,000,000 books to index or do you have 5 articles? What are the context lengths you need? What about latency? How can someone tell you what hardware you need when you give literally no information about what you’re trying to do?
- tbrownaw 2y agoThere's a difference between "how many CPU-hours will my task need" and "how much memory does this program use to even start up".
- deleted 2y ago[deleted]
- malux85 2y agoHaving some idea of the task will guide the choice of model, which will be an enormous factor in memory use (I.e. whether it will startup or not) Do you need a 70b param model or a 7b model? Theres thousands and thousands of dollars hardware difference there With no idea of the task, one can’t even ball park it
- tbrownaw 2y agoThis particular tool has a page listing recommended models: https://neuml.github.io/txtai/models/ https://neuml.github.io/txtai/models/
- Der_Einzige 2y agoThe embedding models at the heart of txtai can be small enough to run on intel CPUs from ten years ago. It's extremely frustrating when HN commentators don't do even the most basic research into the product that they are critiquing.