4 ms·

And what makes you think that a GPT model can be better than a 20 year old search algorithm? I mean seriously, at what point exactly did you convince yourself t

And what makes you think that a GPT model can be better than a 20 year old search algorithm? I mean seriously, at what point exactly did you convince yourself that GPT is actually intelligent and can assess anything? You're missing too many variables and focusing too much on what you want without realizing what is actually possible.

PaulHoule 4y ago
GPT fundamentally "compresses" a bunch of knowledge into neural network weights and then "makes stuff up" which is frequently true. There is a lot of research into similar systems which have an information retrieval aspect. For instance a model like that might crunch a query or a short document into a few hundred vectors and just like with conventional search, a document which is close in vector space to the query is likely to be a good answer. That kind of model computes a vector for each token in the model, a bit like a word vector, except that vector is conditioned by the context of the word so maybe "goal" is encoded differently for different sports or "good" is coded differently if you put "not" in front of it. The document embeddings average together those word+context vectors and frequently people apply further dimensional reduction. I worked on similar systems about 10 years ago that didn't use the transformer and these were highly effective for scenarios like "write a paragraph about an invention and look for patents for a similar invention". I'm skeptical that that kind of system does a great job of capturing patterns that involve multiple words so systems that are based on fine-tuning and classification are also interesting to me, such as the cross-encoder described here https://sbert.net/examples/applications/cross-encoder/README.html https://sbert.net/examples/applications/cross-encoder/README... particularly it ought to be possible to train a model that takes a query and a document and tries to guess if the document is relevant or not. People have notably asked ChatGPT to "Write a review of topic X with citations to 10 papers" and found that it just makes up plausible-looking citations. To really succeed at that you'd think the system would do a conventional search, screen the documents for relevance, then feed the document into the model and tell it to cook the documents down to a review. A system like that has a fighting chance of doing the right thing but it's beyond state of the art. At the very least models like ChatGPT have a limited attention window, 4096 tokens in that case, and there is usually more than one token per word. You'd have to get the attention window up to the 400,000-4,000,000 token range to pull off the above directly and that involves confronting numerous scaling challenges.
quickthrower2 4y ago
I read the suggestion as a confluence of the two technologies, not that GPT replaces Google Search.

PaulHoule 4y ago

GPT fundamentally "compresses" a bunch of knowledge into neural network weights and then "makes stuff up" which is frequently true. There is a lot of research into similar systems which have an information retrieval aspect. For instance a model like that might crunch a query or a short document into a few hundred vectors and just like with conventional search, a document which is close in vector space to the query is likely to be a good answer. That kind of model computes a vector for each token in the model, a bit like a word vector, except that vector is conditioned by the context of the word so maybe "goal" is encoded differently for different sports or "good" is coded differently if you put "not" in front of it. The document embeddings average together those word+context vectors and frequently people apply further dimensional reduction. I worked on similar systems about 10 years ago that didn't use the transformer and these were highly effective for scenarios like "write a paragraph about an invention and look for patents for a similar invention". I'm skeptical that that kind of system does a great job of capturing patterns that involve multiple words so systems that are based on fine-tuning and classification are also interesting to me, such as the cross-encoder described here https://sbert.net/examples/applications/cross-encoder/README.html https://sbert.net/examples/applications/cross-encoder/README... particularly it ought to be possible to train a model that takes a query and a document and tries to guess if the document is relevant or not. People have notably asked ChatGPT to "Write a review of topic X with citations to 10 papers" and found that it just makes up plausible-looking citations. To really succeed at that you'd think the system would do a conventional search, screen the documents for relevance, then feed the document into the model and tell it to cook the documents down to a review. A system like that has a fighting chance of doing the right thing but it's beyond state of the art. At the very least models like ChatGPT have a limited attention window, 4096 tokens in that case, and there is usually more than one token per word. You'd have to get the attention window up to the 400,000-4,000,000 token range to pull off the above directly and that involves confronting numerous scaling challenges.

quickthrower2 4y ago

I read the suggestion as a confluence of the two technologies, not that GPT replaces Google Search.