3 ms·

ChatGPT will use fallacious arguments when answering politicized questions. When called out, it will admit it used a fallacious argument and then say one shoul

ChatGPT will use fallacious arguments when answering politicized questions. When called out, it will admit it used a fallacious argument and then say one shouldn't do that. That's sort of like where a parent will give a wrong, but easy answer to a credulous child when asked a question, and a different one to adults with more discernment.

nonrandomstring 4y ago
> That's sort of like where a parent will give a wrong, but easy answer to a credulous child when asked a question, and a different one to adults with more discernment. Somewhat, yes. But I think it better fits the description somewhere between a sociopath and Harry G. Frankfurt's definition of a bullshitter, who has utter disregard for the truth and will say anything in the moment that deals with a question qua problem. It's essentially defensive and uses language instrumentally only in that pursuit. The problem here is not what LLM-AI is capable of, but what capabilities people ascribe to it. The danger comes from what we believe or project/personify onto it. It has no essential concept of truth. But we want it to.
- narrator 4y ago
  On the politicized stuff they use InstructGPT[1] to make sure it gives the "right" answer. This is human in the loop reinforcement learning. The human training it was probably instructed to give it a pass on any answer that was not "toxic." [1]https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/
mike_hearn 4y ago
This speaks to the core problem here, which I've previously flagged to a friend who works on this stuff at Google - what do we really want from AI? To risk sounding enterprisey and boring, to what extent does ChatGPT meet concrete requirements? There's a temptation to assume that because there's recently been such impressive breakthroughs in probabilistic generative AI that this must be what we want, or at least that we'll be using it regardless. It seems more like that for many use cases we want computers that act like computers, not people. We want something like the computer in Star Trek where it is always coldly logical, only tells us things that can be proven true, can explain its reasoning, gets calculations correct 100% of the time and which refuses to answer questions to which it doesn't know the answer. A machine that thinks so much like a human that it'll happily spout biased but convincing BS isn't something that's got an immediately obvious role in business. Presumably they'll continue to refine GPT until it can be fenced into roles like customer support agent, and that might deliver cost savings for companies to the frustration of the actual customers, but as tools for thought and analysis LLMs are seriously lacking. Honestly for a lot of cases I'd rather having something like Cyc, which thinks in a much more alien way because that alien-ness is where value can be found and where AI can best compliment human reasoning.

nonrandomstring 4y ago

> That's sort of like where a parent will give a wrong, but easy answer to a credulous child when asked a question, and a different one to adults with more discernment. Somewhat, yes. But I think it better fits the description somewhere between a sociopath and Harry G. Frankfurt's definition of a bullshitter, who has utter disregard for the truth and will say anything in the moment that deals with a question qua problem. It's essentially defensive and uses language instrumentally only in that pursuit. The problem here is not what LLM-AI is capable of, but what capabilities people ascribe to it. The danger comes from what we believe or project/personify onto it. It has no essential concept of truth. But we want it to.

narrator 4y ago
On the politicized stuff they use InstructGPT[1] to make sure it gives the "right" answer. This is human in the loop reinforcement learning. The human training it was probably instructed to give it a pass on any answer that was not "toxic." [1]https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/

mike_hearn 4y ago

This speaks to the core problem here, which I've previously flagged to a friend who works on this stuff at Google - what do we really want from AI? To risk sounding enterprisey and boring, to what extent does ChatGPT meet concrete requirements? There's a temptation to assume that because there's recently been such impressive breakthroughs in probabilistic generative AI that this must be what we want, or at least that we'll be using it regardless. It seems more like that for many use cases we want computers that act like computers, not people. We want something like the computer in Star Trek where it is always coldly logical, only tells us things that can be proven true, can explain its reasoning, gets calculations correct 100% of the time and which refuses to answer questions to which it doesn't know the answer. A machine that thinks so much like a human that it'll happily spout biased but convincing BS isn't something that's got an immediately obvious role in business. Presumably they'll continue to refine GPT until it can be fenced into roles like customer support agent, and that might deliver cost savings for companies to the frustration of the actual customers, but as tools for thought and analysis LLMs are seriously lacking. Honestly for a lot of cases I'd rather having something like Cyc, which thinks in a much more alien way because that alien-ness is where value can be found and where AI can best compliment human reasoning.