4 ms·

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. No.

by pdonis 4y ago

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. No. You've fallen for exactly the con the article is describing. ChatGPT knows nothing. It is very good at generating text that seems plausible if you know nothing about the subject matter, but is not reliable as an actual source of information. If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last.

tester457 4y ago
> If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last. For creating bash scripts and english to code it remains remarkably consistent. No need to be a master of multiple languages syntaxes now.
- kibwen 4y ago
  Of all the languages one could ask ChatGPT to generate, bash would be nearly the bottom of the list. The number of subtle footguns in bash are through the roof.
msm_ 4y ago
I think that's unnecessarily pessimistic view. I have recently started using ChatGPT for mid-hard questions (when I can't get an answer immediately using Google), mostly "how do I" questions. It lies of dreams up fake answers about half of the time, but this still means that 50% of time I get my solution without digging through obscure manuals. > It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. I like to think about it like asking questions to a well read but dishonest student during an oral exam. They know some things, and try really hard to answer every question, even resort to making things up in hope of getting a passing grade.
- jacobmischka 4y ago
  So it's a coinflip whether it's giving you correct information or something completely made up? And now you're not digging through the obscure manuals to actually verify? Seems objectively harmful.
  - stale2002 4y ago
    Actually, that is extremely beneficial. For example, I now use GPT3 for every single code debugging problem that I have. I input the bug/problamatic code into the service, ask it for an answer, and then I take the code and see if it fixes my problem I was having. It doesn't work half the time, but thats fine. Then I just figure out the bug the normal way. But the other half the time, it immediately solves my problem, and then I run my code and it works/ does what I want.
    - jacobmischka 4y ago
      I sincerely hope you work on some random SaaS and not any software that actually matters, because this is how you get subtle dangerous bugs.
- bjhartin 4y ago
  It seems to me the utility of the 'good' 50% depends entirely upon your ability to recognize it. How do you know when to dig through those obscure manuals? It seems to me that it'd be more useful when: a) I don't care whether the text says things which are not true Write me a poem about ... b) I care, and will have to verify it's results, but that cost is worth it Generate some test data ... I think we're at a very dangerous intersection between an apparent decline in American's ability to detect nonsense and an automated way to create convincing nonsense.
Throwaway78389 4y ago
How many hours you spent trying ChatGPT out? I spent at least high tens, maybe even hundreds. You're absolutely wrong. Yes, it hallucinates, yes its wrong about obscure topics - but calling having success with it luck is absolutely wrong. It's very consistently good. Especially about things like programming, physics, math - and now I'm using it as my teaching assistant for my pilot training, it's perfect (and I can very simply verify the answers are good with my FAA Pilot Handbook, don't worry).
- CSMastermind 4y ago
  Perhaps you and the poster are approaching your evaluations from different points of view. I've found that if I set out to break ChatGPT I can very easily do it. If my goal is to look for mistakes or find a failure case it's almost trivial to do so. At the same time if I'm looking for success I normally find it. Essentially if you work cooperatively with the tool then you'll find it useful, if you are antagonistic towards it you can also have success in breaking it.
  - midasuni 4y ago
    I asked it to write some simple code do do a task. It confidently told me to use a library and use some functions in it. Couldn’t get it to work. Couldn’t get any examples out of google of it being sued that way. Eventually looked through the code of the library and found that while some functions worked int he way chatgpt was trying, the functions it had selected didn’t work, didn’t support those arguments, and never had.
- pdonis 4y ago
  > I can very simply verify the answers are good with my FAA Pilot Handbook Thank you for agreeing with my point. If you need to check the answers with your FAA Pilot Handbook, wouldn't it be simpler just to read the FAA Pilot Handbook? That handbook, unlike the text generated by ChatGPT, was written by a process that was aware of the semantic relationship between the text being written and the things in the world that the text was referring to. That is what makes the FAA Pilot Handbook a reliable source of information about flying. ChatGPT is not any such process. That's why it is not a reliable source of information--as you agree, since you need to check what it says about flying with the FAA Pilot Handbook.
kvetching 4y ago
So frustrating to see people complain that it provides wrong information. With this logic it can never know anything. It is not made for querying. It's made to complete the text you provide it, with some minor modifications to help it answer questions. It does know the most common patterns of characters in the internet which implicitly contain knowledge.
- undersuit 4y ago
  Not that I see much evidence of what I'm about to assert in ourselves, but you should be able to correct ChatGPT's knowledge if it knows things and isn't just a fancy parrot.
- pdonis 4y ago
  > With this logic it can never know anything. That's correct: ChatGPT does not know anything. That's not what it's built to do. > It does know the most common patterns of characters in the internet which implicitly contain knowledge. No, it doesn't. It knows patterns of text in its training data, but it knows nothing about the semantics of the text--its relationship to other things in the world, which is what is involved with any text that contains knowledge. That is why ChatGPT is not reliable as a source of information.
  - bglazer 4y ago
    I share your skepticism of LLM’s output but I don’t think it’s fair to say they know nothing about semantics. I think it’s still an open question as to what degree LLM’s encode a coherent world model. Also you can just ask chatgpt about objects and their relationships and it gets the answer right way more often than you’d expect by chance, so it has some understanding of the world. Not good enough for me to trust it though
    - freejazz 4y ago
      The answer is clearly none. It literally knows nothing about the world.
rayval 4y ago
I was surprised to find ChatGPT does a better job of translating than Google translate, at least for one difficult task I needed to get done. I have been a Google Translate user for more than a decade. I use it a dozen or more times a day for multiple languages. It is a very useful tool. I recently had to translate a passage that included this phrase: 六六大顺 Google Translate gave me: Liuliu Dashun (basically no translation, just a transliteration or romanization) Bing Translator: Six Six Dashun (a little bit better because 六 does mean six in some contexts. Baidu: Liu Liu (basically no translation) Naver: Six or Six Dashun (partial translation, plus added an incorrect particle) Only ChatGPT gave the correct answer: "Everything going smoothly" There is a cultural context that is essential to the translation, which apparently only ChatGPT has. It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”. In Chinese culture, there is an idiom called “六六大顺 (liù liù dà shùn)”, which means “Everything comes in a smooth way” or more naturally "Everything going smoothly". BTW, after seeing the ChatGTP results, I found this contextual info using Google search, in a website http://www.ichineselearning.com http://www.ichineselearning.com It is ironic that Google search "knows" (helps you find) the cultural context but Google Translate apparently does not leverage this info.
- LarryMullins 4y ago
  I wonder, does it get the meaning of "10-4" when translating English to other languages?
  - d0mine 4y ago
    It does. GPT Chat: 10-4 is a phrase commonly used in radio communication that means "message received and understood." It is typically used to acknowledge a message or request. Google does too https://en.wikipedia.org/wiki/Ten-code https://en.wikipedia.org/wiki/Ten-code
- autokad 4y ago
  > It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”. isn't that number 8? (not number 6)
lamontcg 4y ago
Yeah, ChatGPT is exploiting how bad humans (or at least Americans) are at detecting sociopathic liars and bullshit artists. It is consistent with how awful our business management class is along with our politics and relationships.
speed_spread 4y ago
There's lots of real knowledge encoded in ChatGPT and it can recombine and regurgitate it in enough ways that I'd be okay saying that it knows stuff. The real problem is that it doesn't know its limits and will make shit up instead of saying "I don't know". How to reliably detect and enforce these limits probably requires a completely different approach to AI.
- HarHarVeryFunny 4y ago
  I think ultimately any truly intelligent entity needs to have agency - to be able to interact with the world and test truth (prediction correctness) for itself. Without this you're really just an expert system - a bunch of canned data/rules. The strength of ChatGPT over a GOFAI expert system like Cyc is how broad it's knowledge base is, but the weaknesses are that is doesn't know whether this "knowledge" (being self-deduced from the training set) is true or not, nor does it have any reliable way to combine facts/sources, since those rules are also self-derived and unverified. I'm sure some of this could be mitigated by more human curation (which seems to be the path OpenAI is going down), but then it seems it is just an expert system (maybe this is all people are looking for anyway).
westoncb 4y ago
> If you happened to get information from it that turned out to be ok, you were lucky. This is just saying that sometimes ChatGPT is right and sometimes it's wrong, with an implicit stance on the relative frequencies. In my own experience it would be more accurate to say you're unlucky if it gets it wrong—but then much of this hinges on having a good sense of what it's going to be good at / useful for. (The other thing I see people miss all the time when assessing its correctness: it is somewhat "on demand"—it may include bits that are vague and only correct-ish, but if you point out an error or request refinement you can generally get it to clear/tighten things up. It's a bit like using a Google Maps-like zooming UI, but in addition to being variably abstract it's always approximating: the potential for error is ever present.)
- pdonis 4y ago
  > This is just saying that sometimes ChatGPT is right and sometimes it's wrong No, it's saying that ChatGPT is not reliable. It's not reliable because getting things right or wrong is not part of its design at all. All it does is generate text that matches a pattern. There is nothing anywhere in it that filters or evaluates text based on any semantic connection with anything else.
  - westoncb 4y ago
    > No, it's saying that ChatGPT is not reliable That's another way of saying sometimes it is right and sometimes it is wrong... > All it does is generate text that matches a pattern This is about as meaningful as saying, "all things are is collections of atoms". We are barely beginning to explore the implications of these pattern-extending structures: their most interesting features emerge from their structure, cannot be simply derived from the fact that they do pattern continuation in the same way you're not gonna derive the concept of friendship from analyzing chemical bond structure.

by pdonis 4y ago

tester457 4y ago

> If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last. For creating bash scripts and english to code it remains remarkably consistent. No need to be a master of multiple languages syntaxes now.

kibwen 4y ago
Of all the languages one could ask ChatGPT to generate, bash would be nearly the bottom of the list. The number of subtle footguns in bash are through the roof.

msm_ 4y ago

I think that's unnecessarily pessimistic view. I have recently started using ChatGPT for mid-hard questions (when I can't get an answer immediately using Google), mostly "how do I" questions. It lies of dreams up fake answers about half of the time, but this still means that 50% of time I get my solution without digging through obscure manuals. > It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. I like to think about it like asking questions to a well read but dishonest student during an oral exam. They know some things, and try really hard to answer every question, even resort to making things up in hope of getting a passing grade.

jacobmischka 4y ago
So it's a coinflip whether it's giving you correct information or something completely made up? And now you're not digging through the obscure manuals to actually verify? Seems objectively harmful.
- stale2002 4y ago
  Actually, that is extremely beneficial. For example, I now use GPT3 for every single code debugging problem that I have. I input the bug/problamatic code into the service, ask it for an answer, and then I take the code and see if it fixes my problem I was having. It doesn't work half the time, but thats fine. Then I just figure out the bug the normal way. But the other half the time, it immediately solves my problem, and then I run my code and it works/ does what I want.
  - jacobmischka 4y ago
    I sincerely hope you work on some random SaaS and not any software that actually matters, because this is how you get subtle dangerous bugs.
bjhartin 4y ago
It seems to me the utility of the 'good' 50% depends entirely upon your ability to recognize it. How do you know when to dig through those obscure manuals? It seems to me that it'd be more useful when: a) I don't care whether the text says things which are not true Write me a poem about ... b) I care, and will have to verify it's results, but that cost is worth it Generate some test data ... I think we're at a very dangerous intersection between an apparent decline in American's ability to detect nonsense and an automated way to create convincing nonsense.

Throwaway78389 4y ago

How many hours you spent trying ChatGPT out? I spent at least high tens, maybe even hundreds. You're absolutely wrong. Yes, it hallucinates, yes its wrong about obscure topics - but calling having success with it luck is absolutely wrong. It's very consistently good. Especially about things like programming, physics, math - and now I'm using it as my teaching assistant for my pilot training, it's perfect (and I can very simply verify the answers are good with my FAA Pilot Handbook, don't worry).

CSMastermind 4y ago
Perhaps you and the poster are approaching your evaluations from different points of view. I've found that if I set out to break ChatGPT I can very easily do it. If my goal is to look for mistakes or find a failure case it's almost trivial to do so. At the same time if I'm looking for success I normally find it. Essentially if you work cooperatively with the tool then you'll find it useful, if you are antagonistic towards it you can also have success in breaking it.
- midasuni 4y ago
  I asked it to write some simple code do do a task. It confidently told me to use a library and use some functions in it. Couldn’t get it to work. Couldn’t get any examples out of google of it being sued that way. Eventually looked through the code of the library and found that while some functions worked int he way chatgpt was trying, the functions it had selected didn’t work, didn’t support those arguments, and never had.
pdonis 4y ago
> I can very simply verify the answers are good with my FAA Pilot Handbook Thank you for agreeing with my point. If you need to check the answers with your FAA Pilot Handbook, wouldn't it be simpler just to read the FAA Pilot Handbook? That handbook, unlike the text generated by ChatGPT, was written by a process that was aware of the semantic relationship between the text being written and the things in the world that the text was referring to. That is what makes the FAA Pilot Handbook a reliable source of information about flying. ChatGPT is not any such process. That's why it is not a reliable source of information--as you agree, since you need to check what it says about flying with the FAA Pilot Handbook.

kvetching 4y ago

So frustrating to see people complain that it provides wrong information. With this logic it can never know anything. It is not made for querying. It's made to complete the text you provide it, with some minor modifications to help it answer questions. It does know the most common patterns of characters in the internet which implicitly contain knowledge.

undersuit 4y ago
Not that I see much evidence of what I'm about to assert in ourselves, but you should be able to correct ChatGPT's knowledge if it knows things and isn't just a fancy parrot.
pdonis 4y ago
> With this logic it can never know anything. That's correct: ChatGPT does not know anything. That's not what it's built to do. > It does know the most common patterns of characters in the internet which implicitly contain knowledge. No, it doesn't. It knows patterns of text in its training data, but it knows nothing about the semantics of the text--its relationship to other things in the world, which is what is involved with any text that contains knowledge. That is why ChatGPT is not reliable as a source of information.
- bglazer 4y ago
  I share your skepticism of LLM’s output but I don’t think it’s fair to say they know nothing about semantics. I think it’s still an open question as to what degree LLM’s encode a coherent world model. Also you can just ask chatgpt about objects and their relationships and it gets the answer right way more often than you’d expect by chance, so it has some understanding of the world. Not good enough for me to trust it though
  - freejazz 4y ago
    The answer is clearly none. It literally knows nothing about the world.

rayval 4y ago

I was surprised to find ChatGPT does a better job of translating than Google translate, at least for one difficult task I needed to get done. I have been a Google Translate user for more than a decade. I use it a dozen or more times a day for multiple languages. It is a very useful tool. I recently had to translate a passage that included this phrase: 六六大顺 Google Translate gave me: Liuliu Dashun (basically no translation, just a transliteration or romanization) Bing Translator: Six Six Dashun (a little bit better because 六 does mean six in some contexts. Baidu: Liu Liu (basically no translation) Naver: Six or Six Dashun (partial translation, plus added an incorrect particle) Only ChatGPT gave the correct answer: "Everything going smoothly" There is a cultural context that is essential to the translation, which apparently only ChatGPT has. It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”. In Chinese culture, there is an idiom called “六六大顺 (liù liù dà shùn)”, which means “Everything comes in a smooth way” or more naturally "Everything going smoothly". BTW, after seeing the ChatGTP results, I found this contextual info using Google search, in a website http://www.ichineselearning.com http://www.ichineselearning.com It is ironic that Google search "knows" (helps you find) the cultural context but Google Translate apparently does not leverage this info.

LarryMullins 4y ago
I wonder, does it get the meaning of "10-4" when translating English to other languages?
- d0mine 4y ago
  It does. GPT Chat: 10-4 is a phrase commonly used in radio communication that means "message received and understood." It is typically used to acknowledge a message or request. Google does too https://en.wikipedia.org/wiki/Ten-code https://en.wikipedia.org/wiki/Ten-code
autokad 4y ago
> It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”. isn't that number 8? (not number 6)

lamontcg 4y ago

Yeah, ChatGPT is exploiting how bad humans (or at least Americans) are at detecting sociopathic liars and bullshit artists. It is consistent with how awful our business management class is along with our politics and relationships.

speed_spread 4y ago

There's lots of real knowledge encoded in ChatGPT and it can recombine and regurgitate it in enough ways that I'd be okay saying that it knows stuff. The real problem is that it doesn't know its limits and will make shit up instead of saying "I don't know". How to reliably detect and enforce these limits probably requires a completely different approach to AI.

HarHarVeryFunny 4y ago
I think ultimately any truly intelligent entity needs to have agency - to be able to interact with the world and test truth (prediction correctness) for itself. Without this you're really just an expert system - a bunch of canned data/rules. The strength of ChatGPT over a GOFAI expert system like Cyc is how broad it's knowledge base is, but the weaknesses are that is doesn't know whether this "knowledge" (being self-deduced from the training set) is true or not, nor does it have any reliable way to combine facts/sources, since those rules are also self-derived and unverified. I'm sure some of this could be mitigated by more human curation (which seems to be the path OpenAI is going down), but then it seems it is just an expert system (maybe this is all people are looking for anyway).

westoncb 4y ago

> If you happened to get information from it that turned out to be ok, you were lucky. This is just saying that sometimes ChatGPT is right and sometimes it's wrong, with an implicit stance on the relative frequencies. In my own experience it would be more accurate to say you're unlucky if it gets it wrong—but then much of this hinges on having a good sense of what it's going to be good at / useful for. (The other thing I see people miss all the time when assessing its correctness: it is somewhat "on demand"—it may include bits that are vague and only correct-ish, but if you point out an error or request refinement you can generally get it to clear/tighten things up. It's a bit like using a Google Maps-like zooming UI, but in addition to being variably abstract it's always approximating: the potential for error is ever present.)

pdonis 4y ago
> This is just saying that sometimes ChatGPT is right and sometimes it's wrong No, it's saying that ChatGPT is not reliable. It's not reliable because getting things right or wrong is not part of its design at all. All it does is generate text that matches a pattern. There is nothing anywhere in it that filters or evaluates text based on any semantic connection with anything else.
- westoncb 4y ago
  > No, it's saying that ChatGPT is not reliable That's another way of saying sometimes it is right and sometimes it is wrong... > All it does is generate text that matches a pattern This is about as meaningful as saying, "all things are is collections of atoms". We are barely beginning to explore the implications of these pattern-extending structures: their most interesting features emerge from their structure, cannot be simply derived from the fact that they do pattern continuation in the same way you're not gonna derive the concept of friendship from analyzing chemical bond structure.