3 ms·
It's the economically viable part that I think is currently hard, but agreed this is the right approach. The basic problem is that scaling up understanding ove
by inimino 2y ago
It's the economically viable part that I think is currently hard, but agreed this is the right approach.
The basic problem is that scaling up understanding over a large dataset requires scaling the application of an LLM and tokens are expensive.
- lysecret 2y agoAnd the minutes of latency you currently get with this context lengths.
- ZeroCool2u 2y agoYeah, this is why I mentioned Gemini with the context caching. It's not out yet, but supposedly launching soon. You pay a lower rate for storing the system prompt or whatever you dump in before the user query, plus you don't have to wait the full minute or so for all your research to be ingested every time. https://ai.google.dev/gemini-api/docs/caching#get-started https://ai.google.dev/gemini-api/docs/caching#get-started