4 ms·

Or maybe it's that each of these things is pretty simple in itself. Clipping context is one line of code, summarizing could be a couple lines to have an LLM sum

Or maybe it's that each of these things is pretty simple in itself. Clipping context is one line of code, summarizing could be a couple lines to have an LLM summarize it for you, etc. So not substantial enough for a formal library. Whereas the combinations of these techniques is very application dependent, so not reusable enough to warrant separating as an independent library. Or maybe it just hasn't matured yet and we'll see more of it in the future. We'll see.

daxfohl 1y ago
Though in a way, this feels similar to things like garbage collection, disk defragmentation, or even query planning. Yes, you could build libraries that do these sorts of things for you, but in all likelihood the LLM providers will embed custom-built versions of them that have been battle tested and trained thoroughly to interop well with the corresponding LLM. So whole there could still be an ecosystem, it would likely be a fairly niche thing for very specific use cases or home-grown LLMs. Maybe something like the equivalent of AWS Firecracker for whatever the equivalent of AWS Lambda is in the future LLM world.