4 ms·

Seems like a common pattern. State of the art models being well replaced by a information retrieval layer (top 10 results) fed into a much lighter model that do

by naillo 3y ago

Seems like a common pattern. State of the art models being well replaced by a information retrieval layer (top 10 results) fed into a much lighter model that does something with that plus the original input. Cool result!

falcor84 3y ago
Yeah, that actually sounds amazing to me. If we could limit the LLM to somehow only act as a "reasoning" rather than a "knowledge" layer, such that all the non-trivial domain knowledge has to come from the information retrieval layer, in a fully referenced way, that could potentially "solve" the hallucination problem, no? Even more than that, I wonder if we could then apply something like this to power some sort of "fact provenance" for the web as a whole, e.g. by populating Wikidata with referenced facts (preferably with extensive human QA).
- esjeon 3y ago
  Yeah, and, on top of that, I think this can lead to smaller (and snappier) agent models, because we no longer have to encode every single piece of information into models. As we carve out more and more parameters and input data, AI development will get more accessible, and we'll get more novel applications. (I'm certainly dreaming here.)
- spacemanspiff01 3y ago
  [dead]
redox99 3y ago
I don't know. ChatGPT and Bing both dramatically deteriorate if you allow them to search the web. And systems that allow you to "talk" to a PDF via top results of vector search being added to the prompt are also pretty underwhelming.
twic 3y ago
This is definitely my bet on where things are going. And not just this particular example - i believe we will identify many recurring submodules and patterns in neural networks that can be extracted into conventional code, leaving a lightweight neural glue layer orchestrating them. This should be more efficient, faster to train, more interpretable, and more reliable, so better for users. But less mysterious, so worse for VCs.

by naillo 3y ago

falcor84 3y ago

Yeah, that actually sounds amazing to me. If we could limit the LLM to somehow only act as a "reasoning" rather than a "knowledge" layer, such that all the non-trivial domain knowledge has to come from the information retrieval layer, in a fully referenced way, that could potentially "solve" the hallucination problem, no? Even more than that, I wonder if we could then apply something like this to power some sort of "fact provenance" for the web as a whole, e.g. by populating Wikidata with referenced facts (preferably with extensive human QA).

esjeon 3y ago
Yeah, and, on top of that, I think this can lead to smaller (and snappier) agent models, because we no longer have to encode every single piece of information into models. As we carve out more and more parameters and input data, AI development will get more accessible, and we'll get more novel applications. (I'm certainly dreaming here.)
spacemanspiff01 3y ago
[dead]

redox99 3y ago

I don't know. ChatGPT and Bing both dramatically deteriorate if you allow them to search the web. And systems that allow you to "talk" to a PDF via top results of vector search being added to the prompt are also pretty underwhelming.

twic 3y ago

This is definitely my bet on where things are going. And not just this particular example - i believe we will identify many recurring submodules and patterns in neural networks that can be extracted into conventional code, leaving a lightweight neural glue layer orchestrating them. This should be more efficient, faster to train, more interpretable, and more reliable, so better for users. But less mysterious, so worse for VCs.