5 ms·

I’m only barely up to speed on the tech so I don’t know if this is feasible or not, but I wonder if it would be possible to combine something like ChatGPT with

I’m only barely up to speed on the tech so I don’t know if this is feasible or not, but I wonder if it would be possible to combine something like ChatGPT with something like Google. Then maybe suggest hints to the model in real time during inference, based on facts found in another dataset. It seems like if it were trying to complete info about John Prescott (as in the article) that it could do better if it had some separate context about John Prescott to tap into, scraped from Wikipedia or something.

andsoitis 4y ago
> combine something like ChatGPT with something like Google. Rumor has it that Microsoft is working with OpenAI to integrate into Bing. https://www.theverge.com/2023/1/4/23538552/microsoft-bing-chatgpt-search-google-competition https://www.theverge.com/2023/1/4/23538552/microsoft-bing-ch...
eightysixfour 4y ago
Yes, ChatGPT just had web browsing disabled. But GPT can search the web: https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/
david_allison 4y ago
It's feasible (and has been done). But there are still issues (at least source 2 & 3): The model confuses multiple people named John Prescott. Query: `Describe the early life of John Prescott` > John Prescott, Baron Prescott of Kingston-upon-Hull, was born on May 31, 1938 in Prestatyn, Wales [1]. He served as a British politician and stood as the Labour candidate in the 2012 election [1]. Prior to his political career he worked at the New England Aquarium for 22 years where he described how whales communicate with each other [2]. He also has a VetMB PhD from Cambridge University and is an expert on leptospirosis in family dogs from a public health perspective [3]. In addition to his work experience he attended Embry-Riddle Aeronautical University and Boston Public Schools [4][5] before becoming an early career teacher himself and eventually entering politics. > [1]: John Prescott - Wikipedia (https://en.wikipedia.org/wiki/John_Prescott https://en.wikipedia.org/wiki/John_Prescott) > [2]: John Prescott, 63, Ex-Director Of New England Aquarium - The New ... (https://www.nytimes.com/1998/07/06/us/john-prescott-63-ex-director-of-new-england-aquarium.html https://www.nytimes.com/1998/07/06/us/john-prescott-63-ex-di...) > [3]: Leptospirosis in the family dog: a public health perspective - PMC (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2228361/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2228361/) > [4]: Embry-Riddle Aeronautical University (https://erau.edu/ https://erau.edu/) > [5]: Boston Public Schools (https://www.bostonpublicschools.org/ https://www.bostonpublicschools.org/) https://labs.kagi.com/ai/contextai?question=Describe+the+early+life+of+John+Prescott https://labs.kagi.com/ai/contextai?question=Describe+the+ear...
hooande 4y ago
This tech works by predicting the next word based on the previous words. It doesn't have a concept of facts, sentences or even concepts. Just "what token is statistically likely to follow the previous tokens" You can limit the sources of information that it uses to determine next word relationships. But then you give up the predictive power of having the entire internet as training data. If you first do a web search to limit the available sources, you increase the computation overhead (first search and then summarize at the token level). and if you're already doing a web search, what's the point in using gpt? You lose the power of big data synthesis and it's just a sort of summarization / style transfer tool. Individual tokens don't have sources and they can't be fact checked. They're just words or parts of words. Doing any kind of fact checking or citation adds an entirely separate technical process that is distinct from the transformer model.
- merely-unlikely 4y ago
  > It doesn't have a concept of facts, sentences or even concepts How different is that from humans I wonder? For example, when learning math we talk about developing an intuition about a concept to help actually use it in daily life. This is separate from the rigor of the actual theory, but theory is harder to remember or fully internalize. Intuition to me is more of a feeling that something is related without necessarily really understanding what is going on.
  - hooande 4y ago
    I'm sorry, but this point is trite and incorrect. Humans can manipulate concepts and compare them to other ideas. Which is why we're able to fact check and cite sources, where language models cannot. It's true that I'm just putting one word after another right now, like a language model. But I can combine those tokens into higher level abstractions like facts and ideas, which I can then compare with other facts and ideas. If ChatGPT could do this, it wouldn't be plausibly wrong nearly as often. Instead it's just "what word is statistically likely to come next?"
dragonwriter 4y ago
Probably you could do something either modifying the prompt based on a tool that uses Google search or reprompting after the initial response (which you don’t show to the user) incorporating search info to get a “final” result.

andsoitis 4y ago

> combine something like ChatGPT with something like Google. Rumor has it that Microsoft is working with OpenAI to integrate into Bing. https://www.theverge.com/2023/1/4/23538552/microsoft-bing-chatgpt-search-google-competition https://www.theverge.com/2023/1/4/23538552/microsoft-bing-ch...

eightysixfour 4y ago

Yes, ChatGPT just had web browsing disabled. But GPT can search the web: https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/

david_allison 4y ago

It's feasible (and has been done). But there are still issues (at least source 2 & 3): The model confuses multiple people named John Prescott. Query: `Describe the early life of John Prescott` > John Prescott, Baron Prescott of Kingston-upon-Hull, was born on May 31, 1938 in Prestatyn, Wales [1]. He served as a British politician and stood as the Labour candidate in the 2012 election [1]. Prior to his political career he worked at the New England Aquarium for 22 years where he described how whales communicate with each other [2]. He also has a VetMB PhD from Cambridge University and is an expert on leptospirosis in family dogs from a public health perspective [3]. In addition to his work experience he attended Embry-Riddle Aeronautical University and Boston Public Schools [4][5] before becoming an early career teacher himself and eventually entering politics. > [1]: John Prescott - Wikipedia (https://en.wikipedia.org/wiki/John_Prescott https://en.wikipedia.org/wiki/John_Prescott) > [2]: John Prescott, 63, Ex-Director Of New England Aquarium - The New ... (https://www.nytimes.com/1998/07/06/us/john-prescott-63-ex-director-of-new-england-aquarium.html https://www.nytimes.com/1998/07/06/us/john-prescott-63-ex-di...) > [3]: Leptospirosis in the family dog: a public health perspective - PMC (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2228361/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2228361/) > [4]: Embry-Riddle Aeronautical University (https://erau.edu/ https://erau.edu/) > [5]: Boston Public Schools (https://www.bostonpublicschools.org/ https://www.bostonpublicschools.org/) https://labs.kagi.com/ai/contextai?question=Describe+the+early+life+of+John+Prescott https://labs.kagi.com/ai/contextai?question=Describe+the+ear...

hooande 4y ago

This tech works by predicting the next word based on the previous words. It doesn't have a concept of facts, sentences or even concepts. Just "what token is statistically likely to follow the previous tokens" You can limit the sources of information that it uses to determine next word relationships. But then you give up the predictive power of having the entire internet as training data. If you first do a web search to limit the available sources, you increase the computation overhead (first search and then summarize at the token level). and if you're already doing a web search, what's the point in using gpt? You lose the power of big data synthesis and it's just a sort of summarization / style transfer tool. Individual tokens don't have sources and they can't be fact checked. They're just words or parts of words. Doing any kind of fact checking or citation adds an entirely separate technical process that is distinct from the transformer model.

merely-unlikely 4y ago
> It doesn't have a concept of facts, sentences or even concepts How different is that from humans I wonder? For example, when learning math we talk about developing an intuition about a concept to help actually use it in daily life. This is separate from the rigor of the actual theory, but theory is harder to remember or fully internalize. Intuition to me is more of a feeling that something is related without necessarily really understanding what is going on.
- hooande 4y ago
  I'm sorry, but this point is trite and incorrect. Humans can manipulate concepts and compare them to other ideas. Which is why we're able to fact check and cite sources, where language models cannot. It's true that I'm just putting one word after another right now, like a language model. But I can combine those tokens into higher level abstractions like facts and ideas, which I can then compare with other facts and ideas. If ChatGPT could do this, it wouldn't be plausibly wrong nearly as often. Instead it's just "what word is statistically likely to come next?"

dragonwriter 4y ago

Probably you could do something either modifying the prompt based on a tool that uses Google search or reprompting after the initial response (which you don’t show to the user) incorporating search info to get a “final” result.