6 ms·

We Evaluated ChatGPT vs. Google on 500 Search Queries

25 points by amrrs 4y ago

djha-skin 4y ago
Okay, cool, it tells me exactly what I need to know. But part of the reason I use search is that I want to know what other real people have said on the subject. ChatGPT would be perfect if it gave me all of those things and then cited a reference. Documentation, a blog post, or even a StackOverflow answer would do. I don't see it in the output of ChatGPT on this article, so I assume that's not in its repertoire? I suppose including it as part of the prompt might work.
marginalia_nu 4y ago
Seems like it's comparing Google's ability to be ChatGPT more so than comparing ChatGPT's ability to be Google. There's more to search than answering questions. The question/answer-format lends itself to some tasks, but not all. For example, you cannot produce a page-long summary of say Plato's Republic without butchering it completely. The work is incredibly dense and every specific word is well chosen and layered with nuance (as hand-written books tend to be). Even a summary of this length is fairly limited[1], although sections 2-4 do a good job of highlighting the sort of points attempts to summarize the work often get wrong. https://iep.utm.edu/republic/ https://iep.utm.edu/republic/
- MuffinFlavored 4y ago
  Correct me if I'm wrong but at a very high level, the same way Google has scraped/crawled probably "trillions" of web pages across the Internet, hasn't ChatGPT basically been "trained" on almost the exact same data? aka just a bunch of text from the Internet that they index/train a "data model" versus serving it "a la carte" from a search engine index that's been highly optimized to focus on relevancy?
  - marginalia_nu 4y ago
    Yeah, I don't see how that affects anything though. They're fundamentally doing different things with the data.
    - MuffinFlavored 4y ago
      I think you are right. Google is basically telling you "ok, we're just a middleman, here's the source, go leave our site and visit this 3rd party site, read it and try to form your own summary, but keep in mind the results might be inaccurate, but since you are leaving our page, we aren't responsible for delivering the content to you anymore" whereas ChatGPT is doing... "here's a (potentially inaccurate) summary I was able to build from one/potentially multiple sources without telling you what those sources are, but since I'm serving you the summary, I'm basically responsible for potentially serving you inaccurate information that you may draw conclusions from/act on"
      - anileated 4y ago
        It does not attribute no matter whether credit or caution is due. Imagine if Google did this with their index (trained a model and just stopped giving you links), web masters would be up in arms.
        marginalia_nu 4y ago
        Most likely it can't, since as I understand it ChatGPT synthesizes a response from the entire training set, rather than simply summarizing any particular document.
        anileated 4y ago
        I don’t think it’s unattainable, intuitively it feels like a couple of extra layers to reflect on where the information might have come from could do it (as well as obviously capturing attribution along with the data). The cynic in me says they would rather keep it ambiguous and not investigate the possibility though. Attribution means everyone can ask for their fair share of the profits made by running the systems trained on the data they supply. Shareholder value and all that.
sanman811 4y ago
I disagree with the evaluation made on the response for a risotto recipe by ChatGPT. While I agree that a lot modern recipe sites are ad filled garbage, I believe there is value in getting a quick glance on who the recipe writer is. Where culture do they come from? What experience do they have? These insights can help me understand how to adjust the recipe ingredients to my tastes. Versus a generic recipe from ChatGPT that doesn’t provide any context. Plus if I find a recipe I really like then I can save that website and dig into other recipes from that person because now I know I am more likely to trust their tastes.

25 points by amrrs 4y ago

Okay, cool, it tells me exactly what I need to know. But part of the reason I use search is that I want to know what other real people have said on the subject. ChatGPT would be perfect if it gave me all of those things and then cited a reference. Documentation, a blog post, or even a StackOverflow answer would do. I don't see it in the output of ChatGPT on this article, so I assume that's not in its repertoire? I suppose including it as part of the prompt might work.

marginalia_nu 4y ago

Seems like it's comparing Google's ability to be ChatGPT more so than comparing ChatGPT's ability to be Google. There's more to search than answering questions. The question/answer-format lends itself to some tasks, but not all. For example, you cannot produce a page-long summary of say Plato's Republic without butchering it completely. The work is incredibly dense and every specific word is well chosen and layered with nuance (as hand-written books tend to be). Even a summary of this length is fairly limited[1], although sections 2-4 do a good job of highlighting the sort of points attempts to summarize the work often get wrong. https://iep.utm.edu/republic/ https://iep.utm.edu/republic/

MuffinFlavored 4y ago
Correct me if I'm wrong but at a very high level, the same way Google has scraped/crawled probably "trillions" of web pages across the Internet, hasn't ChatGPT basically been "trained" on almost the exact same data? aka just a bunch of text from the Internet that they index/train a "data model" versus serving it "a la carte" from a search engine index that's been highly optimized to focus on relevancy?
- marginalia_nu 4y ago
  Yeah, I don't see how that affects anything though. They're fundamentally doing different things with the data.
  - MuffinFlavored 4y ago
    I think you are right. Google is basically telling you "ok, we're just a middleman, here's the source, go leave our site and visit this 3rd party site, read it and try to form your own summary, but keep in mind the results might be inaccurate, but since you are leaving our page, we aren't responsible for delivering the content to you anymore" whereas ChatGPT is doing... "here's a (potentially inaccurate) summary I was able to build from one/potentially multiple sources without telling you what those sources are, but since I'm serving you the summary, I'm basically responsible for potentially serving you inaccurate information that you may draw conclusions from/act on"
    - anileated 4y ago
      It does not attribute no matter whether credit or caution is due. Imagine if Google did this with their index (trained a model and just stopped giving you links), web masters would be up in arms.
      - marginalia_nu 4y ago
        Most likely it can't, since as I understand it ChatGPT synthesizes a response from the entire training set, rather than simply summarizing any particular document.
        anileated 4y ago
        I don’t think it’s unattainable, intuitively it feels like a couple of extra layers to reflect on where the information might have come from could do it (as well as obviously capturing attribution along with the data). The cynic in me says they would rather keep it ambiguous and not investigate the possibility though. Attribution means everyone can ask for their fair share of the profits made by running the systems trained on the data they supply. Shareholder value and all that.

sanman811 4y ago

I disagree with the evaluation made on the response for a risotto recipe by ChatGPT. While I agree that a lot modern recipe sites are ad filled garbage, I believe there is value in getting a quick glance on who the recipe writer is. Where culture do they come from? What experience do they have? These insights can help me understand how to adjust the recipe ingredients to my tastes. Versus a generic recipe from ChatGPT that doesn’t provide any context. Plus if I find a recipe I really like then I can save that website and dig into other recipes from that person because now I know I am more likely to trust their tastes.