4 ms·

The first problem, context window size, is going to bite a lot of people. The gotcha is that it’s a search problem. The article mentions embeddings and dot pr

The first problem, context window size, is going to bite a lot of people. The gotcha is that it’s a search problem. The article mentions embeddings and dot product, but that’s the most basic and naive search you can do. Search is information retrieval, and it’s a huge problem space. You need a proper retriever that you tune for relevance. You should use a search engine for this, and have multiple features and do reranking. That’s the only way to crack the context window problem. But the good news, is that once you do, things get much much better! You can then apply your search/retriever skills on all kinds of other problems - because search is really the backbone of all AI.

kristjansson 3y ago
Exactly. LLMs are incredible at information processing, and ok-to-terrible at information retrieval. All LLM applications that rely on accurate information are either infeasible or kick the entire can to the retrieval component.
phillipcarter 3y ago
Yeah, we're definitely learning this. It's actually promising how well a very simple cosine similarity pass on data before sending it to an LLM can do [0]. But as we're learning, each further step towards accuracy is bigger and bigger, and there doesn't appear to be any turnkey solutions you can pay for right now. [0]: https://twitter.com/_cartermp/status/1657037648400117760 https://twitter.com/_cartermp/status/1657037648400117760
darkteflon 3y ago
Such a great comment. The nice thing about this is that if search was a key part of your product pre-LLM, you likely already have something useful in place that requires very little adaptation.
azinman2 3y ago
Context window size will eventually be solved, likely with its own trade offs.
- binarymax 3y ago
  The way to think about it is this: you can scan an entire book for the part you’re looking for (a huge context window), or you can look it up in the index in the back (a good retriever). The latter is a better approach when you’re serving production use cases. It’s faster and less expensive.
pmoriarty 3y ago
Anthropic's Claude[1] already has a 100k token context length. [1] - https://poe.com/Claude-instant-100k https://poe.com/Claude-instant-100k
- omeze 3y ago
  Also mentioned in the post; doesn’t work well in practice

kristjansson 3y ago

Exactly. LLMs are incredible at information processing, and ok-to-terrible at information retrieval. All LLM applications that rely on accurate information are either infeasible or kick the entire can to the retrieval component.

phillipcarter 3y ago

Yeah, we're definitely learning this. It's actually promising how well a very simple cosine similarity pass on data before sending it to an LLM can do [0]. But as we're learning, each further step towards accuracy is bigger and bigger, and there doesn't appear to be any turnkey solutions you can pay for right now. [0]: https://twitter.com/_cartermp/status/1657037648400117760 https://twitter.com/_cartermp/status/1657037648400117760

darkteflon 3y ago

Such a great comment. The nice thing about this is that if search was a key part of your product pre-LLM, you likely already have something useful in place that requires very little adaptation.

azinman2 3y ago

Context window size will eventually be solved, likely with its own trade offs.

binarymax 3y ago
The way to think about it is this: you can scan an entire book for the part you’re looking for (a huge context window), or you can look it up in the index in the back (a good retriever). The latter is a better approach when you’re serving production use cases. It’s faster and less expensive.

pmoriarty 3y ago

Anthropic's Claude[1] already has a 100k token context length. [1] - https://poe.com/Claude-instant-100k https://poe.com/Claude-instant-100k

omeze 3y ago
Also mentioned in the post; doesn’t work well in practice