4 ms·

> The problem is that it gets past many people's filters because its phrasing sends a lot of intelligent/rational/professional signals, as it was engineered to

> The problem is that it gets past many people's filters because its phrasing sends a lot of intelligent/rational/professional signals, as it was engineered to do. Nobody is used to picking smart-sounding text apart word by word to make sure they agree with it, except maybe academics, and that's the vulnerability it reaches us through. I think you're making a mountain out of a molehill. To me ChatGPT is basically a clever way to interpolate and extrapolate coherent text based on user input and its training set. If the training set is lacking in some areas, it underfits the output. I've tested ChatGPT in a couple of engineering fields I'm familiar and I expected the service to respond poorly, but even though it returned nonsense in some areas I thought were low-hanging topics, such as the release year of an international standard, overall its output was very impressive and very entertaining. Perhaps it's the engineer in me talking, but it's pointless to waste time waxing lirically about human nature. Tools like ChatGPT might one day be superb expert systems and teaching tools, but like any expert system and learning tool you need to corroborate the results by yourself. Human nature has zero to do with this.

whatshisface 4y ago
That description of how it functions is true, but it did fool a lot of people, and now we have to explain why. I also note that a lot of people report positive results in their own fields of expertise, which might be part of the explanation. (It might be building trust before failing inexplicably right when it can no longer be checked.)
- simplotek 4y ago
  > That description of how it functions is true, but it did fool a lot of people, and now we have to explain why. I see your point, and I agree. Nevertheless, these misdirections seem to boil down to a broad temptation to succumb to appeals to authority. People might be falling for ChatGPT misfires just like they fall for fancy talking bullshit artists, but that's hardly a failing of clever auto text generators.

whatshisface 4y ago

That description of how it functions is true, but it did fool a lot of people, and now we have to explain why. I also note that a lot of people report positive results in their own fields of expertise, which might be part of the explanation. (It might be building trust before failing inexplicably right when it can no longer be checked.)

simplotek 4y ago
> That description of how it functions is true, but it did fool a lot of people, and now we have to explain why. I see your point, and I agree. Nevertheless, these misdirections seem to boil down to a broad temptation to succumb to appeals to authority. People might be falling for ChatGPT misfires just like they fall for fancy talking bullshit artists, but that's hardly a failing of clever auto text generators.