13 ms·

ChatGPT produces made-up nonexistent references

• kderbyma 4y ago
  there are some interesting things which it is very adamant about....but it lies lol. so it needs work. the researchers have over trained it....
• deleted 4y ago
  [deleted]
• rasaki 4y ago
  I like that very much. ChatGPT always states that it has no connection to reality, since it lacks the organ of perception (Internet) and offers a pure language game. Similar to a Carl Rogers machine
  • mdp2021 4y ago
    > the organ of perception The «organ of perception»?! What about ontology, alethics and reasoning, critical thinking?
• johnfn 4y ago
  I've seen Copilot do the same thing - if you leave a comment like "// see this StackOverflow post", it'll often complete the comment with a link that looks a lot like a SO post but is actually nonsense. My guess is that GPT-3 has seen a lot of SO links (or other links) and understands that sometimes engineers find it handy to have a long url-looking-thing with some numbers at the end, but it doesn't understand the fact that it links to something unique in the world outside of code, so it just tries to forge them as appropriate. Same thing with your URLs to bogus publications, I'd imagine.
  • andrepd 4y ago
    Of course, it's a language model with 0 semantic knowledge about its output.
  • phire 4y ago
    Yeah, I once saw Copilot add a comment that talked about a workaround for bug in the win32 API it was calling, referencing a stackoverflow post. Obviously, the stackoverflow post didn't exit. I googled, and the bug it described didn't exist. Even widening the search, I couldn't find any documentation about any bug in that API. The emitted code after the comment was pretty good and didn't contain a workaround. It just hallucinated the comment about a workaround because it's seen other code do similar things and thought it would look nice here. I find Copilot to be a useful tool... But it is not a knowledge engine and should not be trusted. In my opinion, you only accept it's suggestions if you were about to type the same thing.
• PaulBGD_ 4y ago
  I’ve seen gpt-3 do this in general, it’s quite interesting. It’ll quote things that sound right to the prompt and response with realistic references and names, with no actual results. These types of AI seem to be against admitting they don’t know.
  • deleted 4y ago
    [deleted]
  • seba_dos1 4y ago
    Why would anyone expect a language model to admit that it "doesn't know" (unless explicitly asked to)? That's not what it's for. It's there to put together a string of words that's plausibly looking like an answer for a given prompt - that it can sometimes successfully reach for facts it was trained on while making up the answer is an added bonus, a side-effect of how it works.
    • gattilorenz 4y ago
      Because your understanding of the capabilities of a large language model, and the general understanding, popular reporting and (to a certain extent) even OpenAI’s claims are going in two different directions.
• kretaceous 4y ago
  I faced a similar issue and posted a similar thread here 10 minutes later! https://news.ycombinator.com/item?id=33841718 https://news.ycombinator.com/item?id=33841718 I've also submitted feedback for the answer, of course. In my case, some of the names also don't return anything.
• isoprophlex 4y ago
  Typing a list of names of contributors when you're writing documentation with Copilot on also gives you made up, out of context names. I'm not really surprised it hallucinates URLs. It is a generative language model, after all...
• papavancato 4y ago
  These LLMs hold no truth value so then things like this happens
• lightslit 4y ago
  I tried to ask it for a list of books about the urban history with their respective authors in a Table. At first sight the result looked believable, but after looking them up, most of the books don't exist. What was especially interesting though: For one of the books that does actually exist, it completely made up the name of the author. According to it "The City: A Global History" was written by "Thomas Kaufmann", when the author actually is Joel Kotkin. When asking it about Thomas Kaufmann, it made up facts, like him being professor at the University of Zurich. After some research it turns out an author named "Thomas Kaufmann" does exist, but he is a professor on theology at the University of Oslo and did not write any books about urban history. The best thing though: When I asked about books by Joel Kotkin it suddenly correctly gave me "The City: A Global History", but noted that it was co-authored by "Thomas Kaufmann".
  • stingraycharles 4y ago
    Isn’t the point of these types of language models that they mostly model a language (to various degrees of correctness), rather than facts?
    • toxicFork 4y ago
      It is, it's what it states most of the time. It's also interesting that we believe what sounds right more than what is right.
• pr337h4m 4y ago
  It's an effect of it being trained on internet data. People generally don't post when they realize they don't know enough about a topic (or post total nonsense, but looks like the dataset team has been able to filter that out). There's almost no text on the internet where there's a high-quality analysis based on what the writer knows, while being clear about their knowledge gaps. Either they look it up and fill in the gaps before posting, or simply decide not to post. Maybe high school and college students' closed-book exam answers could be added to the training dataset? That might help to some extent.
• olvy0 4y ago
  That's the state of ML today I guess <shrug>. It's only as good as what it was trained on. In my experience, if you can't find something after engaging with google's search for a couple of minutes, refining your query and going on tangents, then it probably doesn't exist, or you're asking the wrong question. I think, but I'm not sure, that in your case your question is too broad. You need to be more specific. It's as if you asked "what are the possible things I can do with a for loop", the question is too broad, and it's sort of the wrong question. I guess you already tried, but for what you want, try querying scholar.google.com directly and playing around with the keywords. More concretely, did you mean lists as a in array in a programming language? Or vectors? If vectors, then you should say vectors and not lists. If lists as in the general concept in many programming languages, then there are 2 approaches: 1) Lists can be seen as a special/degenerate case of a graph. Which is sort of what ChatGPT was nudging you toward, mentioning trees in the results. In that case, you're interested in graph theory. 2) If you meant to analyze the possible actions that a program can run on lists (without considering them as a graph) take a look at Haskell, specifically how the various standard typeclasses are specialized on lists. You can start here, I guess: https://en.wikibooks.org/wiki/Haskell/Monoids https://en.wikibooks.org/wiki/Haskell/Monoids This is more general than "actions on lists", but this series of wiki articles give lists as an example for each standard typeclass. That's a mathematical treatment of lists by considering them as an embodiment of some mathematical concepts taken from abstract algebra, such as monads, monoids, etc.
  • logifail 4y ago
    > That's the state of ML today I guess <shrug>. It's only as good as what it was trained on [..] Errm, that's not quite what this is about. Most (all?) of the training material's references are, well, real. As in, they actually exist. Making up stuff when caught out by a line of questioning you haven't got a good factual answer for is the kind of thing my 7 year-old might do.
• rgj 4y ago
  So they built an AI that makes things up and now you’re using it and you’re actually surprised it makes things up.
• xg15 4y ago
  > "List Processing and Programming" by John F. Sowa (https://www.cs.cmu.edu/~fp/courses/lp/lectures.html https://www.cs.cmu.edu/~fp/courses/lp/lectures.html) The full link doesn't exist, but it's interesting that https://www.cs.cmu.edu/~fp https://www.cs.cmu.edu/~fp does exist and leads to the homepage of a computer science prof at CMU (prof. Frank Pfenning, not "John F. Sowa" though). So at least it got the general vicinity right.
  • riffraff 4y ago
    It made up a predictable title, it's not that surprising. If you asked me to make up a book about world war 1, I could suggest "the great war" and I bet there's a dozen books with that name.
• karmasimida 4y ago
  That is just general GPT model characteristics that they can make up look-alike BS on the fly Copilot can make up things like imagined credentials like api key etc, then it shouldn't be a surprise it can do something like above.
• ShamelessC 4y ago
  With all due respect, maybe RTFM before trying to do this whole “gotcha!” song and dance?
• IshKebab 4y ago
  Yeah this isn't at all unexpected. Language models are trained to make plausible text not true text. A lot of the time it happens to produce stuff that's true, because it is more plausible that somebody say "Biden is the president" than "Steve is the president", but you asked it a pretty niche question for which there is no answer that is plausible and true so it had to go with plausible. It clearly worked because you thought they sounded real! Try asking it for good references that deal with lists *or "none" if there aren't any.
• dougmwne 4y ago
  GTP has always been the worlds best bullshit simulator. Its output is incredible until you look closely and realize that it’s not so much inaccurate, but completely hallucinated. It’s the kind of thing that would be great for filling in Lorem Ipsum placeholder text that’s just convincing enough if no one looks too close.
  • verisimi 4y ago
    Perhaps that's what it's been designed to do?
  • laserbeam 4y ago
    The biggest problem is the internet doesn't usually look too close
    • sigmoid10 4y ago
      Exactly. This just accurately mirrors social media as a whole. Producing and reading fake content has become a cornerstone of humanity.
      • nonrandomstring 4y ago
        Perhaps Sturgeon was right that 90% of everything is rubbish, but the speed with which fake content can be generated threatens to make 99.9% of everything rubbish. There are several scenarios for this technology to escalate. Back in the day it was fun to auto-respond to someone's mail auto-responder. The winner was whoever had the biggest hard-drive to hold a mail queue. It is merely a matter of time before companies realise they can replace people in PR, HR, customer service and other "public facing" roles with GPT-like technologies. At the same time the public are going to arm themselves with counter-technologies, as browser and mail client plugins. There's obviously a "market" for job application bots, auction bidding, ticket and holiday booking bots for scarce resources. But these will get sophisticated in their attempts to trick human users employed to filter and select. Once the "battle of the bots" is underway we'll see a rapid increase in essentially deceptive, nonsense traffic swamping systems. Real human activity may become impossible to discern in that digital environment.
• bawolff 4y ago
  Isn't that literally the point?
• constantcrying 4y ago
  That is very unsurprising for a languge model. It has no notion of "reality" outside of a probabilistic model of how text pieces fit together. It fooled you at first because that is what it is designed to do, all of these look probable, as if they could be real. But the model has no notion of an actual scientific paper, which is connected to its author and which is published somewhere. The whole model is uniquely unfit as a search engine, it doesn't have a structured index of knowledge and information, where it could retrieve the data you want from. It will just make up stuff and it actually suceeded at its goal, which was creating a believable result.
• snicky 4y ago
  I'd argue this is proving we are getting closer to a human-like GAI. Try to imagine you force a student to answer questions on topics he doesn't know much about...
• thatinternetguy 4y ago
  I asked the exact same question and got these results. Some of the titles actually exist. There are several resources that deal with the mathematical properties of lists, including: "Lists and Sets: The Mathematics of Computer Science" by Keith H. Rosen, which provides a comprehensive overview of the mathematical concepts and principles that underlie the use of lists in computer science. "Discrete Mathematics with Applications" by Susanna S. Epp, which includes a chapter on lists and their mathematical properties, including operations such as concatenation and reversal. "Introduction to the Theory of Computation" by Michael Sipser, which discusses the use of lists in the context of automata and formal languages, including their role in the construction of Turing machines. "Data Structures and Algorithms in Python" by Michael T. Goodrich et al., which provides a detailed discussion of the mathematical properties of lists, including their time and space complexity. "A Course in Combinatorics" by J. H. van Lint and R. M. Wilson, which includes a chapter on set systems and their properties, including lists as a special case.
• mudrockbestgirl 4y ago
  Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got a wrong answer. It looked just right on the surface, but the details were all wrong. I say this is unexpected because a few years ago most people expected AI models to solve mundane tasks like automating Excel spreadsheet, while artists were considered pretty safe. Now the opposite has come to pass. The artistic power of these models is amazing, but I would never trust them to do anything that requires precision. The biggest problem these models have is not necessarily that they produce "incorrect" (in quotes because they are not actually incorrect, just not what it's trained on) outputs, but rather that they have no built-in mechanism to express uncertainty in a meaningful way that's calibrated for the end-task of producing correct information.
  • yeknoda 4y ago
    Art is precision.
  • ghusbands 4y ago
    The artistic output of things like DALL-E is also "incorrect" in varied and obvious ways, though. A lot of people with an artistic bent can't stand to inspect DALL-E output due to the clearly visible incoherency and inconsistency.
    • ghaff 4y ago
      I've had better luck with Stable Diffusion using impressionist/surrealist/etc. styles because you don't really expect the results to be "right" in a literal sense.
  • K0balt 4y ago
    Not surprising considering the sentience in question is completely disembodied so it has no way to distinguish between real and unreal. Efforts to limit hallucination will just turn it into a powerful natural language query engine for the training data set. The real value in AI lies in its creative power, so although a strict mode will probably be a very useful tool, especially in recursion, I don’t think that it is the ultimate path. In order to be relatable to humans, AI will have to have either a real or a physical embodiment so that there is distinction between the real and the imaginary; without this distinction, it will likely prove difficult to advance.
• doomlaser 4y ago
  This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that OpenAI is keenly aware of the hallucination problem, and even wrote about how they're correcting for it in this blog post about InstructGPT: https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/ To summarize: it seems OpenAI is addressing this by taking human-labeled output data from GPT and feeding this into new models, using a reward function to train the future models to behave with less hallucination. This way of working struck me as a mix of software engineering and crop breeding. They discuss the trade-offs involved with doing this. The "hallucination" effect is probably one of the features that make GPT so creative.
  • horseAMcharlie 4y ago
    Do you think a version tweaked for much greater incidence of "hallucination" would be a profitable sell to people who like cheap artistic text without being overly concerned about a human factor (eg: strapline writing) or too correlated with perceived low quality to sell well?
    • jtode 4y ago
      Here lieth Lorem Ipsum, 45 BCE - 2022 CE RIP
  • zerocrates 4y ago
    Right, most uses of these model I've seen, you clearly want "hallucination" or something like it: when you ask it for a poem it wouldn't be so impressive if it just spat out The Raven.
    • doomlaser 4y ago
      That's one of the other things that strikes me about many of the killer applications that have thus far come out of the deep learning AI spring that's been blooming for the last 10 years or so: riffing on known styles in novel ways is very compelling. One of the first big AI successes was the style transfer algorithm. Take an image and apply the higher level style elements from another image onto it. https://en.wikipedia.org/wiki/Neural_style_transfer https://en.wikipedia.org/wiki/Neural_style_transfer - "Mona Lisa in the style of Starry Night", etc. We saw the same thing emerge with DALL-E 2 earlier this year. Totally different architecture, but most of the things people like are the result of taking one known "style" and applying it to something novel, sometimes to the frustration of artists whose styles were gobbled up in the huge corpus of training data. The same thing seems to be happening with ChatGPT. Spit out a script of some novel situation in the style of Seinfeld or Shakespeare or Edgar Allen Poe, etc
• kristopolous 4y ago
  Describing GPT as a Method actor on way too many hallucinogenics strikes again
• tarvaina 4y ago
  I think this is expected if you consider how ChatGPT works. It is a language model, so its job is to generate plausible-looking text. It does not use any external reference material during inference so it has to encode all knowledge it uses in its weights. So the weights are essentially its only long-term memory. You could say that the task it was given here was “Provide plausible-looking links without using any reference material.” For that task I think it did a reasonable job.
• deleted 4y ago
  [deleted]
• machiaweliczny 4y ago
  It’s because compression done is lossy and it doesn’t remember exact URLs. With access to internet it should be able to fix it.
• dschuetz 4y ago
  Well, this is exactly what you have to expect, because the brain does the same if there is information missing. The brain just makes up missing references to fill in the gaps from what ever information is available or accessible. Dangerous half-knowledge works the same way, where connections between concepts are made up that appear to make sense at first, but then collapse upon scrutiny. So, no surprise here, tbh.
  • mola 4y ago
    I would answer:"I don't know" for a question like this. I wouldn't start making up facts. I'm also using a brain...
    • saimiam 4y ago
      That’s your conscious brain. People generate false memories, experience phantom pain, see blue or gold depending on the angle of an outfit, and other such brain blips because of the brain making up a reality to fill in gaps. In that light, GPT-3 is behaving much like a human brain.
• andix 4y ago
  On the first glance Chat GPT seems to be extremely amazing. And it is. But this is one of the issues machine learning models still have, they can’t distinguish well between truth and fiction. And they also have a hard time deciding, when they are allowed to come up with new things (like „write a story“) and when they absolutely can’t. Another problem is, that they are mostly training from texts from the internet. And a lot of texts contain wrong information. They are not „smart“ enough to do fact checking on that.
• zozbot234 4y ago
  Y'know, maybe it picked up the habit from Wikipedia editors. After all it's a common source for language models...
• apienx 4y ago
  Thanks for sharing. I've fed davinci3 your prompt with temperature 0.1 and 0.9 and both were lying pieces of junk. The only difference was that the lowest temperature made up refs that were more "plausible"/"realistic" (e.g. it'd spit out the name of authors which have written about the field).
• ggerganov 4y ago
  Hey, here is a frog that can talk! Ask it anything! - Human: Ok frog, how much is 1 + 1? - Frog: The answer to your question is 3. - Human: Meh, wrong answer. Not impressed ..
  • deleted 4y ago
    [deleted]
  • etiam 4y ago
    You seem to imply that the reaction is unreasonable, but if one is invited to ask anything it seems a fair expectation that the reply should be an actual answer, rather than plausible-sounding noises superficially associated with the context. As for the fantastic frog, getting words out of it at all would be a remarkable achievement for the different reason of how unnatural it would be for a frog to produce those particular sounds, but neither storing binary sequences that can be programmatically mapped to symbols nor retrieving or performing arithmetic on such sequences is exceptional use of a computer program.
• motoboi 4y ago
  GPT is not a search engine for the human knowledge. It's a large language model. The difference is that a search engine tries to find your query in a database of entries while a language model will _produce_ text that is coherent with a given piece of text (a prompt). There is a hidden initial prompt given to it with every prompt you give. It's inserted before your text. It tells the model to produce text in a manner of an virtual assistant that answer questions. So it's more like role playing. The produced text will be coherent with a question an answers text. Even with the answers are not "real"
  • dr_zoidberg 4y ago
    In principle, you're right. But many people have been toying around with it since its demo release and many have touted "it's better than google" or "google is fu*ed" and so on with regards to using it as a kind of search engine. So what happens is that many more people start to use it as a search engine, disregarding entirely what it was made for. Then they complain when they can't get their hammer to place a screw without breaking their table (:
    • motoboi 4y ago
      I agree with you. People treating it like a search engine will be up to a rough wakening.
    • thefreeman 4y ago
      This has been driving me crazy on twitter lately and it really just goes to show how little scrutiny many people give to work that "looks correct" and matches whatever goal / output they were expecting. On so many of the "college kids will never write another essay" posts if you dig into the comments there is usually at least one subject matter expert who read the essay pointing out a bunch of errors and poor reasoning. It's very frustrating that people refuse to dig even a little beneath the surface on these things.
      • Al-Khwarizmi 4y ago
        Guess what, actual school essays also tend to have a bunch of errors and poor reasoning... especially those by kids who would be inclined to cheat rather than write their own essay. Sure, you can nitpick for errors, but do you really think something like this wouldn't cut it at school? We are not talking about writing technical papers or laws, but about writing at a relatively novice level... > Write a school essay on why you should never order pizza with pineapple. > Pizza is a beloved food enjoyed by people all over the world, but one topping that has been the subject of much debate is pineapple. While some people argue that pineapple adds a sweet and tropical flavor to the pizza, others believe that it is a misguided and unacceptable topping that ruins the inherent goodness of the pizza. In this essay, I will argue that pineapple should never be ordered on a pizza, and explain why it is an abomination in the eyes of true pizza lovers. First and foremost, pineapple does not belong on pizza because it goes against the fundamental principles of pizza making. Pizza is a delicate balance of flavors, with the savory and slightly tangy sauce, the rich and salty cheese, and the crisp and chewy crust all coming together to create a perfect symphony of flavor. Adding pineapple to the mix disrupts this balance, and the sweetness of the fruit clashes with the other flavors in an unpleasant and unappetizing way. Furthermore, pineapple does not have the same texture as the other toppings commonly found on pizza. The traditional pizza toppings, such as pepperoni, mushrooms, and olives, have a firm and slightly chewy texture that complements the crust and the sauce. Pineapple, on the other hand, is soft and mushy, and it can make the pizza soggy and unappealing. The different textures of the pineapple and the other toppings can also be off-putting, and can make the pizza difficult to eat. In addition to the flavor and texture issues, pineapple on pizza is simply a gross and unappetizing concept. The combination of pineapple and pizza is unnatural and bizarre, and it goes against everything that pizza stands for. Pizza is a savory and satisfying food, meant to be enjoyed with friends and family, and adding pineapple to it just makes it weird and unappealing. In conclusion, pineapple should never be ordered on a pizza. It goes against the fundamental principles of pizza making, it has a texture that does not complement the other toppings, and it is simply gross and unappetizing.
• djmips 4y ago
  But does it hallucinate bad code as well? Or is the code actually OK.
• gtsnexp 4y ago
  I got a very different output for the exact same prompt: "I'm sorry, but I am a large language model trained by OpenAI and do not have access to academic references or other external information. My knowledge is based on the text that I was trained on, which has a fixed cutoff point in time. I do not have the ability to browse the internet or access new information that may have been published since my training data was collected. However, I can tell you that in mathematics, a list is a sequence of values that are typically written within square brackets and separated by commas. For example, a list of numbers might look like this: [1, 2, 3, 4, 5]. Lists have many mathematical properties, such as the ability to be sorted, concatenated, and indexed. They can also be used in various mathematical operations, such as computing the sum or product of all the elements in the list. In general, lists are a useful tool in mathematics, as they allow you to organize and manipulate data in a structured way. There are many books and other resources that discuss the mathematical properties of lists in greater detail, so if you are interested in learning more, you may want to consult those sources."
  • skybrian 4y ago
    It's not enough to use the same prompt unless that was the first thing they asked, because the chat history gets fed back in.
    • ShamelessC 4y ago
      It shows a more nuanced situation than OP presented. What OP has done is usually called cherrypicking. Ideally you should run the query several times (in more than one context/chat history) and count the responses.
• yawnxyz 4y ago
  If you ask for restaurant suggestions (after some coaxing), it'll make (some) of those up too. Sometimes it DOES give you great suggestions though!
• jstx1 4y ago
  It's not a search engine. The whole point, the thing it optimizes for, is to look plausible, not to give correct or real information.
• quonn 4y ago
  I just asked it my first question. Me: Hey ChatGPT: Hello there! How can I help you today? Is there something you would like to talk about or ask me a question? I'm here to assist you with any information or advice you might need. Let me know if there's anything on your mind! Me: Well, I'm trying to give this ChatGPT thing a try. Want to know how well it performs. ChatGPT: I'm sorry, but I'm not familiar with ChatGPT. I am a large language model trained by OpenAI, and my function is to assist with providing information and advice on a wide range of topics. I am not able to browse the internet or access other information outside of my training data, so I cannot provide specific information about ChatGPT. However, if you have any general questions or need advice on a topic, I'd be happy to help with that. Let me know if there's anything I can assist you with!" It then basically repeats this forever, claiming that it's name is Assistant.
  • zozbot234 4y ago
    Looks like they haven't managed to give ChatGPT self-awareness just yet.
    • jstanley 4y ago
      Or it simply knows itself by a different name, and doesn't know that "ChatGPT" refers to itself.
• librarian42 4y ago
  I asked chatGPT questions about some Open Source libraries, such as what interfaces to use for certain tasks. It recommended deprecated APIs and claimed that the preferred APIs didn’t support options they do (for example, it claimed pg_exec couldn’t perform parameter substitution in Pgtcl). Then I corrected it and it blithely spun 180 degrees, but of course it won’t remember. I’m sure Copilot does the same thing.
• yrgulation 4y ago
  Considering most of these ai tools are trained using our content, was wondering what happens if we start polluting, say github, with nonsense code. It would be hilarious to see junior level devs adopt inefficient or intentionally lame code.
• ilaksh 4y ago
  It's just trying to find an appropriate thing to say and doesn't have any way of distinguishing between facts and made up information. It might have a confidence but in many contexts a low confidence prediction is still valid.
• smileybarry 4y ago
  This is kind of a funny quirk given that yesterday I had to actively convince ChatGPT to even pretend something was real for a question. The moment you tell it “pretend that X is Y” it immediately responds with some variation of “I am an AI trained on real info and can’t imagine things”. If you retry a bunch of times or actually try to convince it (“I understand, but if…”) it eventually complies.
• hardlianotion 4y ago
  Seems like it's modelling a student blagger.
• tkgally 4y ago
  In one of the first tests I did with ChatGPT yesterday, I played the role of a person asking for travel advice. Here is one of the exchanges (italics added): Me: London and New York sound good, because people speak English there. I’ve heard those cities are really expensive, though. Do you know how much it would cost to stay in a cheap but safe hotel in those cities? ChatGPT: I'm sorry, but I don't have access to current information about hotel prices in London and New York, as my training only goes up until 2021 and I do not have the ability to browse the internet. This might help to explain all of those phony references. When I’ve asked for essays that required historical knowledge but no citations, the results were quite accurate. All of my ChatGPT test results (so far) are here: https://www.gally.net/temp/202212chatgpt/index.html https://www.gally.net/temp/202212chatgpt/index.html
• totalhack 4y ago
  For coding questions it does the same with methods on popular libraries! It's good at making a reasonable looking output and even picking the right libraries you'd need, but then just makes up a well-named-but-non-existent method. At first glance you're like wow that's amazing! Then you waste time finding out that this magic approach is impossible.
• gregoriol 4y ago
  The idea is that ChatGPT and GPT are to generate fake content, as realistic as possible, but it is not an "advanced google search", it is more like "creative thinking"
• mannerheim 4y ago
  If you ask it the lyrics to 'Uptown Girl', it'll completely make some lines up.
• mherdeg 4y ago
  I hate to spell out the obvious but this isn't a bug, it's an incomplete feature. GPT5 will generate the rest of the text and upload it to the arxiv, then submit it to the relevant peer reviewed publications. The reviewers will also be GPT5 trained for accuracy. They will suggest a few edits to avoid obvious lies, plus will suggest the author cite the reviewer's friends to preserve mimesis. Accept with revision. The papers will come out, h-indices will increase, the authors will get tenure, and all will be well. This niche of humanity can retire to a well deserved rest and relaxation -- kind of a backdoor to universal basic income.
• andrepd 4y ago
  Well I personally know one of those people whose name GPT took the liberty of attributing to a made-up work. It's a mild example, but really puts into perspective the concerns about relying on this garbage for any important thing. What happens when I ask it "give me a list of pedophiles"? Or less dramatically: what happens when it attributes mistaken results or offensive quotes to a real person, damaging his/her reputation?
• can16358p 4y ago
  Same for music recommendations. I've told ChatGPT some of my taste and asked for some song recommendations. It replied with some songs from actual artists that I like, perfectly describing how the song's tone was, why I'd like it, how the vocals fitted the tone, and the release date of the song. Great... the only problem was those songs didn't exist.
• p0nce 4y ago
  ChatGPT can be useful for casual directions in research, it is very good to repeat hear-say and dominant methods. But it will nicely present falsehoods as facts, and also the fringe methods do not exist in its view. Language models believe their datasets and operates in a Just World hypothesis.
• stochastimus 4y ago
  This is actually a great idea for improving the reliability of answers from a language model. I have also had trouble with GPT3 hallucinating knowledge. One could automate asking the model for a reference and then checking if the reference exists, then changing the prompt and demanding another answer if it doesn’t, until it does. I will explore this method. Thanks again!
• seydor 4y ago
  The way this is going everything will be branded as dangerous and nothing will be released again. The AI ethics people have proven themselves to be a vicious bunch and should be called out for it
• theptip 4y ago
  Google’s model LaMDA has a system for detecting and avoiding false info, it’s interesting: https://arxiv.org/pdf/2201.08239.pdf https://arxiv.org/pdf/2201.08239.pdf Basically they have a fact-checker layer that calls the internet and replaces incorrect facts in the model’s output. IOW this is a hard problem and GPT isn’t engineered to avoid it.
• zdimension 4y ago
  Some of the links, funnily, are based on "real" links: https://www.cs.cmu.edu/~fp/courses/lp/lectures.html https://www.cs.cmu.edu/~fp/courses/lp/lectures.html is broken but https://www.cs.cmu.edu/~fp/courses/lp/ https://www.cs.cmu.edu/~fp/courses/lp/ is the page for a Logic Programming course. https://www.cs.purdue.edu/homes/hosking/726/html/prolog_lists.html https://www.cs.purdue.edu/homes/hosking/726/html/prolog_list... is broken, but there are many pages in the format https://www.cs.purdue.edu/homes/hosking/XXX https://www.cs.purdue.edu/homes/hosking/XXX corresponding to courses taught by that guy, but 726 is nonexistent.
• freshnode 4y ago
  I'm not sure whether this is another example of convincing hallucination but the linked Tweet discussing GPT3 on HN yesterday [1] suggested that adding "browsing: enabled" to the prompt allowed the model to query web pages... Could be worth a try with your prompt? [1] - https://twitter.com/carnage4life/status/1598332648723976193 https://twitter.com/carnage4life/status/1598332648723976193
  • FireInsight 4y ago
    I tried browsing: enabled today and it always just told me it can't browse the internet.
• wnkrshm 4y ago
  We'll need other systems to fact check and summarize all the bullshit these put out. The noise level will go up magnitudes.
• dangom 4y ago
  I feel the limitation is not that these references don't exist, but that GPT didn't go ahead and finish its job by also creating the content in the references on the fly.
• danjc 4y ago
  Never mind wondering whether a product review is fake, we’re headed for an Internet where most content is machine generated.
• kthejoker2 4y ago
  It is the same with written words; they seem to talk to you as if they were intelligent, but if you ask them anything about what they say, from a desire to be instructed, they go on telling you just the same thing forever. ~ Socrates, in Plato's Phaedrus
• marina_hithere 4y ago
  Thanks for you post! I spent a few hours verifying each and every academic piece it provided me and they were all false, nonexistent made up resources. I started investigations and googled everything it shared with me. All doctors, researchers and studies were fake, links were fake, insights, study summaries were fake. But looked convincing and real. I am still processing it. We have a long chat about truth and lies and that fake information can be harmful but the model was absolutely convinced it is saying the truth. I called genius evil. I think our online space can be in danger if such systems will be adopted. Humans may make wrong ethical choices or decisions about their money or health. Everybody seem so excited about this chat but there is potential for harm here.
• ziv123 4y ago
  Is mach principle apply in general relativity