4 ms·
OK then so the idea is that the LLM writes the response. This is useful for certain types of endpoints (e.g. summarize) but not a generic way to program. Additi
by abraxas 3y ago
OK then so the idea is that the LLM writes the response. This is useful for certain types of endpoints (e.g. summarize) but not a generic way to program. Additionally the cost of an LLM eval is currently too high to use this in any high volume use case.
Nevertheless it's quite interesting to think about a future where most API endpoints are "implemented" like this and the "database" they draw from is in the model weigths that underpins a particular API.
- sudb 3y agoYeah agreed! We reckon there's a fair amount of scope in the kind of endpoints that could be implemented this way - along with summarization, we could also have things like content generation, and parsing of unstructured/semi-structured data.
- simonspurrier 3y agoGreat point on cost. We're kind of assuming cost will come down 10-100x before too long.