5 ms·

The problem is that you cannot trust the output. It's often wrong, but in subtle nonobvious ways. For precise information you still need to check the sources to

The problem is that you cannot trust the output. It's often wrong, but in subtle nonobvious ways. For precise information you still need to check the sources to make sure what you're getting is correct. You can test it out with a (not-so-mainstream) topic that you're an expert in. You'll see lots of mistakes that are obvious to you, but wouldn't be obvious to non-experts. But it's an incredible tool for brainstorming or generating content. I think that soon a large percentage of all online text content will be GPT-generated, and that comes with a lot of new issues that we're not prepared for. It's going to be really difficult to trust anything online and tell fact from fiction.

ThouYS 4y ago
exactly that. It often seems correct, until you look up the actual answer. I was asking it how to unload models from triton server using the REST api, and the results seemed sensible. However after googling the actual API, turns out ChatGPT's answer, while convincing, was utter rubbish.
agumonkey 4y ago
maybe they should adapt it and bridge onto vetted sources
bontaq 4y ago
The new issues will be interesting. Now that I've seen the quality of the output and played with it a bit I'm already squinting at comments here and there. We're in for an even stranger internet.
dunefox 4y ago
So, it's exactly like Google search but more interactive?
- tazjin 4y ago
  Google doesn't hallucinate completely fictitious results. It will however index hallucinated results generated with GPT and published somewhere, so once we're at that point it really doesn't matter anymore.
  - beezlewax 4y ago
    It often links to out of date or ad-ridden content ripped from legitimate sources though. Some kind of hybrid of this and search would be great.
  - throwaway09223 4y ago
    Google search absolutely does hallucinate completely fictitious results. It's called SEO spam. Google just gives you associations provided by random other people on the internet. It's largely garbage, most often deliberately disingenuous (to make you look at an ad). Ad revenue models for the internet encourage the generation of this type of false material. A better criticism would be that the same thing will happen to something like chatgpt -- and the question is whether the model for analysis can better handle it at scale.
    - tazjin 4y ago
      > Google search absolutely does hallucinate completely fictitious results No, it absolutely does not. Yes, there is SEO spam in the index, but no - it is not Google hallucating it. It really exists on the internet, see also the second point of my comment. > the same thing will happen to something like chatgpt This isn't something that "happens to" GPT, GPT is doing it. There's probably even already GPT -> SEO spam pipelines out there generating websites.
      - Pxtl 4y ago
        This is a distinction without a difference. Both Google and the AI can give you clever but fake results. So at this point the question is which produces fewer fakes?
      - throwaway09223 4y ago
        > "it is not Google hallucating it. It really exists on the internet," The exact same thing is true for chatGPT, or any other computer system. It is providing information and associations based on the input dataset. > "GPT is doing it." And google is "doing it," when google decides there is an association between my query and a bad response. Both systems are analyzing a corpus, drawing associations, and returning parts of that corpus. The output is deterministic based on the input. The types of associations differ in their depth, but there is no fundamental difference in terms of agency or outcome.
- skybrian 4y ago
  Google links to sources that you can read for yourself. ChatGPT doesn't, so you don't know where it got its information, and you'd have to do web searches anyway to get some possible sources.
visarga 4y ago
You can trust it. Look again at the OP. They fed the whole language spec into chatGPT, only after that it became capable of coding in their language. If you ask people to solve tasks without references you will see a similar drop in ability. The trick is to feed relevant contextual information instead of using it closed-book. This can be automated with a search engine, or can be a deliberate manual process. But closed-book mode is not the right way to assess people or AIs. What are your counter arguments?
davrosthedalek 4y ago
What's fascinating to me is that you can often point out the error, and it will correct them.
- zed1726 4y ago
  This means knowing the error in advance so it's not really the same problem being solved by search engines exactly. It's just a method of retrieving things you already know and can reason it into the correct state.
drivers99 4y ago
My go-to topic to test it with is talking about characters in the movie Hackers. I noticed in someone else’s session that it would take their correction but still hang on to the incorrect contradictory belief. In my session it came up with a seeming rationalization. So I tested it just now, trying to provide the correct information and directly contradicting the incorrect information (x is A. x is not B). That helped, but eventually it just choked. It seems to be handwaving and guessing (bluffing, bullshitting) at the most likely and generic answers when it doesn’t know something. I’m thinking ChatGPT is best used for generating ideas, not factual information.

ThouYS 4y ago

exactly that. It often seems correct, until you look up the actual answer. I was asking it how to unload models from triton server using the REST api, and the results seemed sensible. However after googling the actual API, turns out ChatGPT's answer, while convincing, was utter rubbish.

agumonkey 4y ago

maybe they should adapt it and bridge onto vetted sources

bontaq 4y ago

The new issues will be interesting. Now that I've seen the quality of the output and played with it a bit I'm already squinting at comments here and there. We're in for an even stranger internet.

dunefox 4y ago

So, it's exactly like Google search but more interactive?

tazjin 4y ago
Google doesn't hallucinate completely fictitious results. It will however index hallucinated results generated with GPT and published somewhere, so once we're at that point it really doesn't matter anymore.
- beezlewax 4y ago
  It often links to out of date or ad-ridden content ripped from legitimate sources though. Some kind of hybrid of this and search would be great.
- throwaway09223 4y ago
  Google search absolutely does hallucinate completely fictitious results. It's called SEO spam. Google just gives you associations provided by random other people on the internet. It's largely garbage, most often deliberately disingenuous (to make you look at an ad). Ad revenue models for the internet encourage the generation of this type of false material. A better criticism would be that the same thing will happen to something like chatgpt -- and the question is whether the model for analysis can better handle it at scale.
  - tazjin 4y ago
    > Google search absolutely does hallucinate completely fictitious results No, it absolutely does not. Yes, there is SEO spam in the index, but no - it is not Google hallucating it. It really exists on the internet, see also the second point of my comment. > the same thing will happen to something like chatgpt This isn't something that "happens to" GPT, GPT is doing it. There's probably even already GPT -> SEO spam pipelines out there generating websites.
    - Pxtl 4y ago
      This is a distinction without a difference. Both Google and the AI can give you clever but fake results. So at this point the question is which produces fewer fakes?
    - throwaway09223 4y ago
      > "it is not Google hallucating it. It really exists on the internet," The exact same thing is true for chatGPT, or any other computer system. It is providing information and associations based on the input dataset. > "GPT is doing it." And google is "doing it," when google decides there is an association between my query and a bad response. Both systems are analyzing a corpus, drawing associations, and returning parts of that corpus. The output is deterministic based on the input. The types of associations differ in their depth, but there is no fundamental difference in terms of agency or outcome.
skybrian 4y ago
Google links to sources that you can read for yourself. ChatGPT doesn't, so you don't know where it got its information, and you'd have to do web searches anyway to get some possible sources.

visarga 4y ago

You can trust it. Look again at the OP. They fed the whole language spec into chatGPT, only after that it became capable of coding in their language. If you ask people to solve tasks without references you will see a similar drop in ability. The trick is to feed relevant contextual information instead of using it closed-book. This can be automated with a search engine, or can be a deliberate manual process. But closed-book mode is not the right way to assess people or AIs. What are your counter arguments?

davrosthedalek 4y ago

What's fascinating to me is that you can often point out the error, and it will correct them.

zed1726 4y ago
This means knowing the error in advance so it's not really the same problem being solved by search engines exactly. It's just a method of retrieving things you already know and can reason it into the correct state.

drivers99 4y ago

My go-to topic to test it with is talking about characters in the movie Hackers. I noticed in someone else’s session that it would take their correction but still hang on to the incorrect contradictory belief. In my session it came up with a seeming rationalization. So I tested it just now, trying to provide the correct information and directly contradicting the incorrect information (x is A. x is not B). That helped, but eventually it just choked. It seems to be handwaving and guessing (bluffing, bullshitting) at the most likely and generic answers when it doesn’t know something. I’m thinking ChatGPT is best used for generating ideas, not factual information.