4 ms·
It’s frustrating when people ask for hardware requirements without stating what they are trying to do, do you have 100,000,000 books to index or do you have 5 a
by malux85 2y ago
It’s frustrating when people ask for hardware requirements without stating what they are trying to do, do you have 100,000,000 books to index or do you have 5 articles? What are the context lengths you need? What about latency?
How can someone tell you what hardware you need when you give literally no information about what you’re trying to do?
- tbrownaw 2y agoThere's a difference between "how many CPU-hours will my task need" and "how much memory does this program use to even start up".
- deleted 2y ago[deleted]
- malux85 2y agoHaving some idea of the task will guide the choice of model, which will be an enormous factor in memory use (I.e. whether it will startup or not) Do you need a 70b param model or a 7b model? Theres thousands and thousands of dollars hardware difference there With no idea of the task, one can’t even ball park it
- tbrownaw 2y agoThis particular tool has a page listing recommended models: https://neuml.github.io/txtai/models/ https://neuml.github.io/txtai/models/