5 ms·
> This is a serious bug. Isn't describing this as a 'bug' rather than a misuse of a powerful text generation tool, playing into the framing that it's a truth t
by ZeroGravitas 4y ago
> This is a serious bug.
Isn't describing this as a 'bug' rather than a misuse of a powerful text generation tool, playing into the framing that it's a truth telling robot brain?
I saw a quote that said "it's a what text would likely come next machine", if it makes up a url pointing to a fake article with a plausible title by a person who works in that area, that's not a bug. That's it doing what it does, generating plausible text that in this case happens to look like, but not be a real article.
edit: to add a source toot:
https://mastodon.scot/@DrewKadel@social.coop/110154048559455079 https://mastodon.scot/@DrewKadel@social.coop/110154048559455...
> Something that seems fundamental to me about ChatGPT, which gets lost over and over again: When you enter text into it, you're asking "What would a response to this sound like?"
> If you put in a scientific question, and it comes back with a response citing a non-existent paper with a plausible title, using a real journal name and an author name who's written things related to your question, it's not being tricky or telling lies or doing anything at all surprising! This is what a response to that question would sound like! It did the thing!
> But people keep wanting the "say something that sounds like an answer" machine to be doing something else, and believing it is doing something else.
> It's good at generating things that sound like responses to being told it was wrong, so people think that it's engaging in introspection or looking up more information or something, but it's not, it's only, ever, saying something that sounds like the next bit of the conversation.
- wilg 4y agoI agree it’s not a bug. Thought it being better at telling the truth would be a good feature! But also, I’m sure this is an active research area so I’m not worried about it really.
- simonw 4y agoI think it's an entire category of bugs. The thing where you paste in a URL and it says "here is a summary of the content of that page: ..." is very definitely a bug. It's a user experience bug - the system should not confuse people by indicating it can do something that it cannot. The thing where you ask for a biography of a living person and it throws in 80% real facts and 20% wild hallucinations - like saying they worked for a company that they did not work for - is a bug. The thing where you ask it for citations and it invents convincing names for academic papers and made-up links to pages that don't exist? That's another bug.
- bestcoder69 4y agoNot necessarily disagreeing, but I run a Slack bot that pretends to summarize URLs, as a joke feature. It’s kinda fun seeing how much it can get right or not from only a URL. So I really hope OpenAI keeps running the fun models that lie, too.
- creatonez 4y agoA better way to put it: It's a bug of the "GPT-4 + Chatbot interface intended to be useful" combination, rather than a bug of GPT-4.
- travisjungroth 4y agoI like the definition of bug as “unexpected behavior”. So this isn’t a bug when it comes the underlying service. But for ChatGPT, a consumer-facing web app that can “answer followup questions, admit its mistakes, challenge false premises and reject inappropriate requests”, then making stuff up and passing it off as true is unexpected behavior.
- lcnPylGDnU4H9OF 4y agoIt sounds like this is unexpected behavior, even from the perspective of those developing at the lowest level in these models. From the essay: > What I find fascinating about this is that these extremely problematic behaviours are not the system working as intended: they are bugs! And we haven’t yet found a reliable way to fix them. Right below that is this link: https://arxiv.org/abs/2212.09251 https://arxiv.org/abs/2212.09251. From the introduction on that page: > As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Especially with a definition as broad as "unexpected behavior", these "novel behaviors" seem to fit. But even without that: > We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views ... and a greater desire to avoid shut down.
- int_19h 4y agoWhether it's a bug or not depends on the application, basically. You want the model to hallucinate things like that if it's writing a fiction book, for example.
- Falkon1313 4y agoI agree. Statements like this miss the point entirely: >Most concerningly, they hallucinate or confabulate: they make things up! That's exactly what they were designed to do! To generate creative responses to input. To make things up. And they're quite good at it - brainstorming, worldbuilding, inspiration for creative writing, finding ideas to pursue about new topics. Unlike an old-fashioned random generator, you can tailor a prompt and style and tone and hold a conversation with it to change details, dig deeper, etc. Get it to make up things that are more interesting and find relations. >They fail spectacularly when prompted with logic puzzles, or basic arithmetic Well, anyone can use a tool wrong, and if you misuse it wrong enough, you'll have problems. Using a chainsaw to trim your fingernails is likely to fail spectacularly, and a nail trimmer is going to fail spectacularly when trying to use it to chop down a tree. That's not a bug in the tool. We don't need to get all alarmed and tell people it's lying. We just need to tell them it's not a calculator or an encyclopedia, it's a creative text generator.
- zerocrates 4y agoBut look at the way these companies talk about these models they're deploying, how people talk about using them here on HN and elsewhere: so much of it is about just asking questions and getting correct answers. All the people talking about how it will destroy Google, how they're using it to learn or teach some topic, etc., Microsoft integrating one into a search engine, the gloss on them as "assistants." The older, less-capable models were used for things more aligned to being just text-generation: "creativity" stuff. But the newer, bigger models and the human-feedback stuff to prod the models into following instructions and being more accurate have really pushed the conversation into this more "Star Trek computer" space. I'm thinking of it kind of like the uncanny valley. On the lower end of the scale, you don't really trust the machine to do anything, but as it gets more and more capable, the places where it doesn't work well become more and more significant because you are trusting and relying on it.