3 ms·
It depends on your latency requirements, not every RAG task has a user waiting for an immediate response, for my use case it doesn't matter if an answer takes e
by malux85 2y ago
It depends on your latency requirements, not every RAG task has a user waiting for an immediate response, for my use case it doesn't matter if an answer takes even 10's of minutes to generate
- hirako2000 2y agoGiven the cost of running an LLM for 10 minutes, yes it does matter. That's about 15 dollars. The overall answer better be very good and relevant every time for this to tech to make sense.
- malux85 2y agoOh that’s a good point, I have my own GPU rack and run locally because it’s cheaper to do so, so I hadn’t considered that…