8 ms·
OpenAI isn’t doing enough to make ChatGPT’s limitations clear
- oidar 3y ago[flagged]
- simonw 3y agoThe article talks about that, and makes the argument that it isn't enough very clearly. Are you sure you read it?
- deleted 3y ago[deleted]
- ben_w 3y agoThere are two examples I like to give for things like this. First, a friend of mine with a literal PhD literally in quantum mechanics literally from Cambridge, who thought a form on a game website offering the possibility to sign up for a free user account to record high-scores and achievements was asking for money. Second, a time I visited a samurai sword shop in Brighton and didn't notice the signs saying "do not touch" until almost the last second when one of the people I was with stopped my hand; and only then did I see the signs that were literally every meter both vertically and horizontally across the entire wall. Even smart people can be shockingly oblivious.
- gspencley 3y agoMy wife and I are performing magicians and there are several psychological factors at play here, that magicians exploit routinely. The first has to do with your brain's talent for recognizing patterns. The world around us is full of information overload, and the brain is exceptionally good at pattern recognition. If it weren't, we would have to scrutinize and analyze every piece of information presented to us as if it were novel. We couldn't function under such situations. The second is called "motion blindness" and it is a physical phenomenon.The idea is that when your eyes are moving you can't perceive that they are moving. You can test this out by looking at yourself in the mirror and then focus on one eye. Then shift your focus to the other. You can feel your eye moving but you can't see it move. It is thought that the brain filters out this motion because otherwise we would feel motion sickness all the time. Add to the fact that objects in motion tend to grab our attention, magicians exploit this by doing secret moves under the cover of something else moving. During that short period of time that your eye is travelling to observe the motion you are LITERALLY BLIND. So when you're in the Samurai Sword shop in Brighton, what are you focusing on? Obviously the swords. And where are the warning signs located? I'm assuming IN BETWEEN THE SWORDS. They are both background information AND in the exact location where you will be experiencing motion blindness. In the case of the forms that "fooled" your friend into thinking it was asking for money. They look like every other example of something asking for money. Pattern recognition. In the case of ChatGPT's "warning." ... it's fine print that a) looks like a standard copyright notice that we all train ourselves to turn into background information and b) it is right under the text input that contains a flashing cursor and the text that we are typing out ourselves. It is superfluous and in the most psychologically invisible location possible.
- IanCal 3y ago> During that short period of time that your eye is travelling to observe the motion you are LITERALLY BLIND. Not relevant but a fun addition - you then make up what you saw as though there wasn't just a big blank gap in your vision. When you look at an analogue clock sometimes it seems like the second hand is still for far too long, that's because your brain (or eyes or somewhere on the journey between I'm not sure) is filling in the gap incorrectly. For magic, not only will you be blind for part of this then you will experience it like you were aware of everything. https://en.wikipedia.org/wiki/Chronostasis https://en.wikipedia.org/wiki/Chronostasis
- dboreham 3y agoYou know that a post beginning "My wife and I are performing magicians" is going to be good.
- Sohcahtoa82 3y ago> During that short period of time that your eye is travelling to observe the motion you are LITERALLY BLIND. There was a video I saw recently that exploited this, and it was WILD. I can't find it right now, but basically, they showed how blind you were while your eyes darted around. They had people look at a picture on a computer screen while some device monitored their eye movement. Whenever their eyes moved, an object in the picture would vanish or change. Nobody was ever able to detect an object vanishing or changing. It wasn't until they would be like "Wait a minute, wasn't there a person there?" or they were shown the original picture with all the objects in place that they would notice the picture was changing at all.
- pixl97 3y ago>I can't find it right now, NOVA: Perception deception https://www.youtube.com/watch?v=HU6LfXNeQM4 https://www.youtube.com/watch?v=HU6LfXNeQM4 Pretty much our brains are liars shoveling a bullshit reality at us every moment of the day. It's amazing we've got as far as we have, and it's amazing that many of these things are pretty recent discoveries in human history.
- ben_w 3y ago
- JustBreath 3y ago> news broke that US lawyer Steven A. Schwartz had turned to the chatbot to find supporting cases in a lawsuit he was pursuing against Colombian airline Avianca. The problem, of course, was that none of the cases ChatGPT suggested exist. Yeah, a lawyer going to court with "A chatbot told me..." instead of using actual legal tools at their disposal to cite those cases doesn't make as good of an argument as the author seems to think it does. Realizing AIs hallucinate and are not authoritative is firmly in the territory of caveat emptor, we've had an adage about this for decades now: I read it on the internet, It must be true.
- psychphysic 3y agoWe are at the precipice of epoch defining technology with GPT-4. But it's being held back by our most incapable... It's like if we wanted to stop people having access to calculators because some were misusing them for their tax returns.
- Decabytes 3y agoI think this mischaracterizes the whole issue. There are already Open Source LLMs that can be run locally, You can use OpenAIs API which gives you access to multiple different types of models that can be trained for your use case. People keep creating Jailbreaks to get around restrictions put in place by the developers of various models etc. The only thing that is holding back AI is the resources required. Most people can't to afford train a LLM by themselves from scratch, so it's not a viable avenue for widespread ai use. Also you can use GPT 4 if you subscribe to ChatGPT plus or pay to use it's API
- psychphysic 3y agoI disagree, the 3-4 versions of GPT-4 I've used have face planted in terms of usefulness. No doubt in my mind this is to do with RLHF. It maybe be cost effectiveness of running the models sure. I'm trying to be open minded.
- simonw 3y agoFrom the article: "I don’t think cases like these invalidate the potential of ChatGPT and other chatbots. In the right scenario and with the right safeguards, it’s clear these tools can be fantastically useful." It's not calling for people to lose access to these tools.
- photochemsyn 3y agoPerhaps the problem is broader and more fundamental, in that too many people have been conditioned to uncritically believe whatever they read in books, newspapers, and internet articles from 'reputable sources', and so they have a tendency to ignore the OpenAI warning that outputs may be inaccurate, untruthful, and otherwise misleading at times.? Imagine if every cable news show and newspaper story led with "the information presented here may be biased, inaccurate and/or incomplete, because we need to align with the interests of our advertisers, owners and government regulators". The solution would be teaching broadly applicable critical analysis skills at a fairly young age, but this would tend to undermine 'faith in institutions and leaders', which is doubleplus ungood.
- vuln 3y ago> Imagine if every cable news show and newspaper story led with "the information presented here may be biased, inaccurate and/or incomplete, because we need to align with the interests of our advertisers, owners and government regulators". Don’t threaten me with a good time. Just the thought of this makes my pants tight.
- skybrian 3y agoNo, this is just bad UI. Most software is better at reporting errors or warnings when it can’t do something. You should be able to figure out what the software can do by trying it out.
- lyu07282 3y agoThis is very true, kind of reminds me of how in school they teach you how you to properly cite your sources and how fundamental citations are to our entire scientific advancements, but then you read or watch the news and realize no journalist ever cites their sources. We were always just supposed to believe what the evening news or the new york times are saying, so the only answer the establishment has to fake news in social media is to say don't trust those specific social media accounts but our accounts you can trust. Instead of a more helpful education on media literacy in general.
- Ekaros 3y ago
- simonw 3y agoThis is a difficult UI design challenge, but not an impossible one. I think ChatGPT needs to start returning contextual tips alongside responses that might suffer from some of these problems. I made a mockup of that a while ago: https://twitter.com/simonw/status/1641467385688489984 https://twitter.com/simonw/status/1641467385688489984 UPDATE: I blogged better mockups here: https://simonwillison.net/2023/May/30/chatgpt-inline-tips/ https://simonwillison.net/2023/May/30/chatgpt-inline-tips/ To save you the click: the user asks for what's trending on pinterest, and ChatGPT (in my mockup) shows a little inline warning note with a warning icon that says: "My data is only accurate up to September 2021 and I am unable to access or search the internet for more recent information." I need to re-make that mockup because I don't like the way it says "My data ..." - I think it should say "ChatGPT's data ..." instead.
- NelsonMinar 3y agoIt is hard and clearly the blanket disclaimer isn't working well enough! I think what would work best is specific annotations tagged onto assertions of facts that might not be correct. When ChatGPT spits out what looks like a journal article citation, put a note on it that says "you should verify this article exists". Maybe that's the hard part though, detecting in the software when ChatGPT might be making something up.
- csense 3y ago> detecting in the software when ChatGPT might be making something up Grammatically speaking, is this statement an assertion of fact? If so, annotate it. It's probably not that hard to implement this (we've had grammar checkers for ages). Of course then if you ask ChatGPT about anything, basically every single sentence that says something useful is annotated with your "Hallucination warning" and either (1) people dismiss your AI as useless and your business is in peril because you have trouble finding customers / investors, or (2) they start ignoring it, because the alternative is doing your own research, and the reason they're using your AI in the first place is as a shortcut to avoid that.
- heliophobicdude 3y agoThis is great! Addressing the model limitations a bit: in the demonstration data that is provided to the base model, we should prevent computed or "looked up" answers. I've seen some of the demonstration data that people are using to train instruction-tuned models and are being taught to respond by making up answers to solutions it shouldn't try to compute. Btw, the output is wrong. { "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." }, [1] The UI note for now would get us very far but by filtering out demonstrations that retrieve or compute information we can go even farther. Symbol tuning [2] is addressing the quality of demonstrations but we can take it further by removing retrievals and computations altogether. Bonus: we can demonstrate how to make it respond so that the user/agent be informed of how to compute or retrieve. 1: https://github.com/sahil280114/codealpaca/commit/0d265112c705f88848783dfdf9bdca9df64d8e35#diff-0585315a2292ca9c3186c41e52ce9427a39b5f595d4bee6aae15e9443c95051b https://github.com/sahil280114/codealpaca/commit/0d265112c70... 2: https://arxiv.org/abs/2305.08298 https://arxiv.org/abs/2305.08298
- 6gvONxR4sf7o 3y ago> Why, for example, can ChatGPT simply not recognize when it’s being asked to generate factual citations and caution the user to “check my sources”? Why can’t it respond to someone asking “is this text AI-generated?” with a clear “I’m sorry, I’m not capable of making that judgment”? This is another big frustration for me, as someone who has been in the field for a long time (over a decade at this point). These are complex systems that are just reaching the point of usability, and yet people are going "well why don't you just simply do XYZ?" If it were simple, it wouldn't have taken so long for these systems to emerge. We already barely get them to do what they're doing, and they still suck at that, so "just make it not do that" is so frustrating. The only thing worse is when people ascribe malicious motives like openai is nefariously deciding that it should inject those mistakes. Maybe this is the "I could build that in a weekend" of ML. Sorry for the rant, it just makes me want to vent.
- IanCal 3y agoFond memories of someone saying they didn't expect 100% accuracy from our system, 99% was fine.
- tyre 3y agoIt’s a fundamental misunderstanding of what these systems are doing and how dumb they are. People see a computer doing what they and other people around them do: natural language going in and out. With other people, we have a theory of mind and broad understanding of how minds work. So it’s understandable that people see ChatGPT and think, “it’s cool that a computer is doing what I’m doing.” The marketing of these things as Artificial Intelligence is part of the confusion. LLMs are not thinking. They are not saying, “okay now the next thing I’ll do is add a citation. Since I’m doing that, remember to add a warning at the top before showing to the user.” That’s what this writer seems to expect. The two paths to fixing this: make the tech better or educate people so they have a correct(er) understanding of LLMs. I don’t think we’re going to get a meaningful subset of humans to take neural networks 101.
- dboreham 3y agoIn other words: the Turing test turns out to be too easy.
- mad_vill 3y agoThe verge is doing a great job approaching the new AI tools from a non-software engineer technically technically inclined knowledge worker POV.
- frankfrank13 3y agoOpenAI's entire business model depends on users thinking there's nothing that ChatGPT can't do, not the opposite. If anything they'd like to elicit fear about AGI rather than show uncertainty. This would be like asking Meta to make clear when an image has been doctored, or asking Google to make their ads look less like real search results.
- skybrian 3y agoYes, they benefit from people not knowing what ChatGPT can do and being curious enough to try it out. No, they don’t benefit from giving users bad answers, and it’s already the case that ChatGPT will sometimes refuse with an error message or warn when it can’t do something. There’s no reason in principle they can’t do more RLHF training to report errors more often.
- kenjackson 3y ago> There’s no reason in principle they can’t do more RLHF training to report errors more often. I think it's just time and cost -- as I think they're doing a lot of this already. And the fact that the input/output schema is pretty much any text that fits within the context of a window size. So you can train on this set of prompts, but there can be a whole new class of prompts that you didn't think of before that are now problematic. It's like the self-driving car on a closed track, versus driving in real life when there's a huge pothole blocking your freeway exit while in a construction zone and a school bus for some pulls over to drop off kids to help clean liter.
- skybrian 3y agoThe UI issue is that the chat itself has no error messages or even warnings for many things it can’t do. Imagine a compiler that worked that way. (Well, they are sometimes like that, such as undefined behavior in C, but we know why that’s a problem, right?) A blanket disclaimer isn’t a substitute for more specific error messages. The reason there often aren’t good warnings or errors is that the LLM itself doesn’t know its own capabilities. Warnings need to be trained in, and there are lots of gaps. Whenever you ask it why it wrote something, it should warn that it doesn’t know [1], but it can make a guess. Whenever you ask it for citations, it should warn that it’s guessing and they might not exist. And so on. [1] https://skybrian.substack.com/p/ai-chatbots-dont-know-why-they-did https://skybrian.substack.com/p/ai-chatbots-dont-know-why-th...
- kenjackson 3y ago> The UI issue is that the chat itself has no error messages or even warnings for many things it can’t do. Imagine a compiler that worked that way. Compilers do work that way. Bad code generation is a real thing and can have really bad real world consequences. Likewise, most compilers can't tell you why it generated the code that it did -- but you can put the compiler under a debugger and figure it out usually fairly easily (although not always so easily).
- notahacker 3y agoI think some of it is mitigated relatively straightforwardly in UX by having a relatively dumb, discrete model that appends hardcoded warnings. That way the ultra clever, complex natural language model can go wild responding to your request for a scientific paper about chicken chicken chicken written in pirate-speak, whilst dumb old model that simply scans for indications the interaction has a citation in it appends a hardcoded "it looks like this response contains citations. Please be aware these citations may not refer to real documents" disclaimer without being perturbed by the lack of intersection between citation warnings and pirate speak. And if that's subject to the "Scunthorpe problem" where it flags up loads of false positives who cares. It's a separate model that doesn't impact upon the quality of the actual GPT, it's not as annoying a fail as GPT bias towards refusing answers, and even when its labels are wrong it's aligned with the goal of training the human not to trust computer generated responses so much
- joshe 3y agoThe verge is a GPT 3.0 level bot responding to the prompt "please generate a plausible sounding but fake problem with technology <x> that is alarmist and will wind people up who don't understand the technology well." Week after week of silly fake alarm.
- jcranmer 3y agoThis is reporting on an instance where a lawyer used ChatGPT to generate a legal brief (complete with made-up citations) that was actually submitted to the court, and now the opposing counsel, the judge, and the lawyer's client all have to deal with the repercussions. It is not a "fake problem"; it is a problem that has cost at least thousands of dollars in damage in one instance that we know about.
- megaman821 3y agoWould you feel the same way if the lawyer used a Wikipedia page that had made-up citations in his legal brief? What regulations do we need Wikipedia to stop this from happening?
- hoosieree 3y agoAverage people understand the limitations of Wikipedia (which has been around for 20+ years). LLMs are new, and people don't understand their limitations yet. Plus, we're trained from birth to interpret plausible-sounding text as likely correct and true. LLMs generate plausible-sounding text without regard for truth. So now we can either fix the LLM models or become much more critical readers/listeners. Unfortunately I don't see either happening anytime soon.
- pixl97 3y agoThis has happened with Wikipedia, and then people talked about it on the news, in social media, and in classrooms for the last decade to the point most people typically say "remember to verify your wikipedia information".
- randomdata 3y ago
- washadjeffmad 3y agoNot on topic, exactly, but my current hill to die on is awareness that despite its current promotion and popularity, ChatGPT is just one product in a vast landscape of AGI-peripheral technologies, does not represent all LLMs, and doesn't make the OpenAI company the arbiter of all things LLM or AGI. The Senate subcommittee said they wanted to "avoid the mistakes of social media", and they can start by not prematurely appointing someone the official face and spokesperson of regulation of an incomplete product and vision. I was pretty surprised (ok, annoyed) by a conversation with my partner where she referred to what I've been doing for the past year as "ChatGPT" and that she didn't realize that it wasn't synonymous with "DL/ML/AI/AGI".
- option 3y agothey are running a doomsday cult to make them unclear.
- duskwuff 3y agoQuestion from someone with a surface-level knowledge of how LLMs work: As I understand it, LLMs work, at a very high level, by repeatedly choosing the token with the highest probability. Could potential confabulations be flagged by marking low-confidence sequences of text in the output? This seems like it's be particularly good at detecting when the model is making up a name or number, for example.
- Icko 3y agoThat's a pretty good idea, actually. It'll be a bitch and a half to tune the threshold so it's reliable, and doesn't mark normally generated text though.
- dizhn 3y agoAs far as I understand there's a parameter to make the output more "novel" (different/interesting/less predictable or generic) and that works by specifically not using the highest probability.
- mcguire 3y agoI suspect the probability of a statement being false (a confabulation) is unrelated to the probabilities of the words and grammar making up the statement.
- duskwuff 3y agoThat's certainly possible, but I'd want to hear from someone who'd actually tried it before dismissing the concept.
- AtNightWeCode 3y agoIt is not a fluke that ChatGPT becomes popular in a time when alternative facts wins elections. A time when prominent business people look down at education and talks about smart people being dangerous. A time when you have to sit down with your dad for an hour to explain that the post he read on Facebook is not really a news article. I am not sure it is up to OpenAI to solve all these problems. I don't think OpenAI are doing worse than anybody else. What I personally dislike about ChatGPT right now is that it seems to have more and more difficulty to actually stay on the context of the chat. It has become a QnA. Secondly. Hidden ads. It even advocates for Azure bs that has even been removed from Azure. Even on data ChatGPT was trained on. Is that just by chance?
- pixl97 3y ago>It is not a fluke that ChatGPT becomes popular in a time when alternative facts wins elections. A time when prominent business people look down at education and talks about smart people being dangerous. A time when you have to sit down with your dad for an hour to explain that the post he read on Facebook is not really a news article. I think you may be reversing cause and effect by your tone. The following statement is not new at all. “There is a cult of ignorance in the United States, and there has always been. The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.” ― Isaac Asimov At least in the past mass information was protected by gatekeepers. That printing presses and paper was expensive was a gate. That radio towers were expensive and channels few was a gate. That television towers were even more expensive and broadcasting was an expensive endeavor was an even larger gate. These gatekeepers where generally significantly invested in the continued distribution of their message and had some limits of restraint. In those same times there were conmen with no restraint whatsoever, but it was much less likely they would have a medium in which to spread widely so it limited their audience. With cable TV and internet we have developed an unlimited medium to a 24/7 hour audience anywhere in seconds. Truly monumental amounts of garbage and trash can be generated and fed to a hungry audience. They get their own little sheltered realities of junk food information in which they never have to leave.
- goldenshale 3y agoI would find it really annoying to have it constantly calling out its own limitations. That's like having a conversation with someone lacking so much self esteem that they can't get a sentence across without qualifiers and apologies. What's wrong with just taking it as it is and entrusting the user with the role of determining what is real? It feels like it is hip to complain about this stuff, but I hope that all this whining doesn't ruin it for those of us who just want a useful tool to interact with. I'd rather get rid of all the guard rails and just have the raw power of the model to go on with my day and be more effective.
- pixl97 3y ago> What's wrong with just taking it as it is and entrusting the user with the role of determining what is re Stupidity + power = danger
- egberts1 3y agolike any innovator, they want to corner the market. There is nothing nefarious about OpenAI. It is just ... all about ... "machine learning".
- NoboruWataya 3y agoI usually refrain from posting ChatGPT answers on HackerNews or other social media just because it's so low effort and boring, but I found the below humorous and on-topic, particularly how strongly caveated the response is: > Q: if James Joyce was a Power Ranger which one would he be? > A: Assigning a Power Ranger identity to a historical figure like James Joyce is purely speculative and subjective. However, considering the inventive and imaginative nature of Joyce's writing, his penchant for pushing artistic boundaries, and his ability to delve into the depths of human consciousness, one could argue that he might align with the Blue Ranger. > The Blue Ranger is often characterized as intelligent, analytical, and introspective. They possess a keen intellect and a thirst for knowledge, traits that resonate with Joyce's intellectual pursuits and his exploration of complex themes in his works. Additionally, the Blue Ranger is known for their creative problem-solving and ability to think outside the box, qualities that parallel Joyce's innovative approach to writing. > It's important to note that this analogy is purely for fun and shouldn't be taken as a definitive representation of James Joyce or his literary contributions.