3 ms·

"Fool" is the operative word here. ChatGPT is quite capable of producing very plausible sounding text about biology that is totally incorrect. See, for exampl

"Fool" is the operative word here. ChatGPT is quite capable of producing very plausible sounding text about biology that is totally incorrect. See, for example, the example in my comment at https://www.lesswrong.com/posts/28XBkxauWQAMZeXiF/?commentId=kxoDxqsqiabvriwmk https://www.lesswrong.com/posts/28XBkxauWQAMZeXiF/?commentId...

marcinzm 4y ago
You're basically complaining that a single model doesn't have full knowledge of every single area of all of human knowledge. It's got decent knowledge of most areas including programming with probably better overall knowledge than a high school student. That's downright amazing and probably more knowledge than any single human actually has. The rest is likely a matter of improvement along the same lines versus some radical redesign.
- radford-neal 4y ago
  Well, I agree that it's amazing - it almost always produces grammatical output, for instance. But it's not a reliable way of obtaining knowledge. One should not, in particular, try to learn about biology by asking ChatGPT questions. It often produces made-up stuff that is just wrong. And it's very confidently wrong, with the output often coming across like someone barely concealing their contempt that you might doubt them. It may or may not be fixable without radical redesign. The underlying training objective of mimicking what humans might say may be too at variance with an objective of producing true statements.
  - sarchertech 4y ago
    My wife (a physician) asked it multiple medical questions and the majority of the time they were dangerously wrong, but looked perfectly fine to me. I asked it a series of questions about my area of expertise and they were wrong but looked perfectly fine to my wife. It even confidently “solved” the 2 generals problem with a solution that looks completely plausible if you don’t already know that it won’t work.
    - wespiser_2018 4y ago
      This was my experience when I asked it questions about database or distributed systems. I wouldn't consider my an expert in these fields, but I do have more than a passing interest in them. All the answers looked good, used several of the correct terms, and one even referenced the project I worked on, but they just contained flat out wrong information. You reach a point, where when you ask a ML model to generate text given the internet as a corpus, where there just isn't enough text to make something that is both true and convincing. In niche fields, this is just where we are at.

marcinzm 4y ago

You're basically complaining that a single model doesn't have full knowledge of every single area of all of human knowledge. It's got decent knowledge of most areas including programming with probably better overall knowledge than a high school student. That's downright amazing and probably more knowledge than any single human actually has. The rest is likely a matter of improvement along the same lines versus some radical redesign.

radford-neal 4y ago
Well, I agree that it's amazing - it almost always produces grammatical output, for instance. But it's not a reliable way of obtaining knowledge. One should not, in particular, try to learn about biology by asking ChatGPT questions. It often produces made-up stuff that is just wrong. And it's very confidently wrong, with the output often coming across like someone barely concealing their contempt that you might doubt them. It may or may not be fixable without radical redesign. The underlying training objective of mimicking what humans might say may be too at variance with an objective of producing true statements.
- sarchertech 4y ago
  My wife (a physician) asked it multiple medical questions and the majority of the time they were dangerously wrong, but looked perfectly fine to me. I asked it a series of questions about my area of expertise and they were wrong but looked perfectly fine to my wife. It even confidently “solved” the 2 generals problem with a solution that looks completely plausible if you don’t already know that it won’t work.
  - wespiser_2018 4y ago
    This was my experience when I asked it questions about database or distributed systems. I wouldn't consider my an expert in these fields, but I do have more than a passing interest in them. All the answers looked good, used several of the correct terms, and one even referenced the project I worked on, but they just contained flat out wrong information. You reach a point, where when you ask a ML model to generate text given the internet as a corpus, where there just isn't enough text to make something that is both true and convincing. In niche fields, this is just where we are at.