16 ms·
My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question:
by aerhardt 2y ago
My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question:
"Is Paul Newman known for having had problems with alcohol?"
All of the models up to o3-mini-high told me he had no known problems. Here's o3-mini-high's response:
"Paul Newman is not widely known for having had problems with alcohol. While he portrayed characters who sometimes dealt with personal struggles on screen, his personal life and public image were more focused on his celebrated acting career, philanthropic work, and passion for auto racing rather than any issues with alcohol. There is no substantial or widely reported evidence in reputable biographies or interviews that indicates he struggled with alcohol abuse."
There is plenty of evidence online that he struggled a lot with alcohol, including testimony from his long-time wife Joanne Woodward.
I sent my mom the ChatGPT reply and in five minutes she found an authoritative source to back her argument [1].
I use ChatGPT for many tasks every day, but I couldn't fathom that it would get so wrong something so simple.
Lesson(s) learned... Including not doubting my mother's movie trivia knowledge.
[1] https://www.newyorker.com/magazine/2022/10/24/who-paul-newman-was-and-who-he-wanted-to-be https://www.newyorker.com/magazine/2022/10/24/who-paul-newma...
- drooby 2y agoI asked GPT-4.5 and it searched the web and immediately gave me a "yes" with paragraphs of sources cited.
- lfkdev 2y agoThats not really 'simple' for an LLM. This is a niche information about a specifc person, LLM's train on massive amount of data, the more a topic is being present in the data, the better will the answers be. Also, you can/should use the "research" mode for questions like this.
- aerhardt 2y agoThe question is simple and verifiable - it is impressive to me that it’s not contained in the LLM’s body of knowledge - or rather that it can’t reach the answer. This is niche in the grand scheme of knowledge but Paul Newman is easily one of the biggest actors in history, and the LLM has been trained on a massive corpus that includes references to this. Where is the threshold for topics with enough presence in the data?
- Max_aaa 2y agoThe question might be simple and verifiable, but it is not a simple for an LLM to mark a particular question as such. This is the tricky part. An LLM does not care about your question, it is a bunch of math that will spit out a result based on what you typed in.
- xrortrad 2y ago[flagged]
- permo-w 2y agothis seems less like an issue with accuracy and more like an issue with model providers making sure they don't get sued for libel
- aerhardt 2y agoI thought about that too.
- ilrwbwrkhv 2y agoLLMs will never be good at specific knowledge unless specifically trained for with narrow "if else" statements. Its good for broad general overview such as most popular categories of books in the world.
- Rebuff5007 2y agoReally? Open-AI says PhD intelligence is just around the corner!
- dadadad100 2y agoIf we were to survey 100 PhDs how many would know correctly that Paul Newman had an alcohol problem.
- AnimalMuppet 2y agoI would hope that 100% of them would be able to figure out how to find out.
- dadadad100 2y agoAh, but isn’t that the problem here - asking an LLM for facts without requesting a search is like asking a PhD to answer a question “off the top of your head”. For pop culture questions the PhD likely brings little value.
- jcgrillo 2y agoThey should know better than to guess. Educated, honest, intelligent people don't spout off a wild ass guess, if they don't know something they say so.
- ludwik 2y agoI don't think they mean "knowledge" when they talk about "intelligence." LLMs are definitely not knowledge bases. They can transform information given to them in impressive ways, but asking a raw (non-RAG-enabled) LLM to provide its own information will probably always be a mistake.
- blitzar 2y agoDoes the as yet unwritten prequel of Idiocracy tell the tale of when we started asking Ai chat bots for facts and this was the point of no return for humanity?
- aerhardt 2y agoCan you blame the users for asking it, when everyone is selling that as a key defining feature? I use it for asking - often very niche - questions on advanced probability and simulation modeling, and it often gets those right - why those and not a simple verifiable fact about one of the most popular actors in history? I don’t know about Idiocracy, but something that I have read specific warnings about is that people will often blame the user for any of the tool’s misgivings.
- pclmulqdq 2y agoIt turns out there's huge demand for un-monetized web search.
- leereeves 2y agoI like that it's unmonetized, of course, but that's not why I use AI. I use AI because it's better at search. When I can't remember the right keywords to find something, or when the keywords aren't unique, I frequently find that web search doesn't return what I need and AI does. It's impressive how often AI returns the right answer to vague questions. (not always though)
- pclmulqdq 2y agoGoogle used to return the right answer to vague questions until it decided to return the most lucrative answer to vague questions instead.
- HPsquared 2y agoFortunately there is a lot of competition in the LLM space. Edit: and, more importantly, plenty of people willing to pay a subscription for good quality.
- hn_throwaway_99 2y agoSo, in other words, are you saying that AI model progress is the real deal and is not bullshit? That is, as you point out, "all of the models up to o3-mini-high" give an incorrect answer, while other comments say that OpenAIs later models give correct answers, with web citations. So it would seem to follow that "recent AI model progress" actually made a verifiable improvement in this case.
- saurik 2y agoI am pretty sure that they must have meant "up through", not "up to", as the answer from o3-mini-high is also wrong in a way which seems to fit the same description, no?
- hn_throwaway_99 2y agoI tried with 4o and it gave me what I thought was a correct answer: > Paul Newman was not publicly known for having major problems with alcohol in the way some other celebrities have been. However, he was open about enjoying drinking, particularly beer. He even co-founded a line of food products (Newman’s Own) where profits go to charity, and he once joked that he consumed a lot of the product himself — including beer when it was briefly offered. > In his later years, Newman did reflect on how he had changed from being more of a heavy drinker in his youth, particularly during his time in the Navy and early acting career, to moderating his habits. But there’s no strong public record of alcohol abuse or addiction problems that significantly affected his career or personal life. > So while he liked to drink and sometimes joked about it, Paul Newman isn't generally considered someone who had problems with alcohol in the serious sense. As other's have noted, LLMs are much more likely to be cautious in providing information that could be construed as libel. While Paul Newman may have been an alcoholic, I couldn't find any articles about it being "public" in the same way as others, e.g. with admitted rehab stays.
- fnordpiglet 2y agoThis is less an LLM thing than an information retrieval question. If you choose a model and tell it to “Search,” you find citation based analysis that discusses that he indeed had problems with alcohol. I do find it interesting it quibbles whether he was an alcoholic or not - it seems pretty clear from the rest that he was - but regardless. This is indicative of something crucial when placing LLMs into a toolkit. They are not omniscient nor are they deductive reasoning tools. Information retrieval systems are excellent at information retrieval and should be used for information retrieval. Solvers are excellent at solving deductive problems. Use them. The better they get at these tasks alone is cool but is IMO a parlor trick since we have nearly optimal or actually optimal techniques that don’t need an LLM. The LLM should use those tools. So, click search next time you have an information retrieval question. https://chatgpt.com/share/67f2dac0-3478-8000-9055-2ae534703701 https://chatgpt.com/share/67f2dac0-3478-8000-9055-2ae5347037...
- mvdtnz 2y agoAny information found in a web search about Newman will be available in the training set (more or less). It's almost certainly a problem of alignment / "safety" causing this issue.
- fnordpiglet 2y agoThere’s a simpler explanation than that’s that the model weights aren’t an information retrieval system and other sequences of tokens are more likely given the totality of training data. This is why for an information retrieval task you use an information retrieval tool similarly to how for driving nails you use a hammer rather than a screw driver. It may very well be you could drive the nail with the screw driver, but why?
- mvdtnz 2y agoYou think that's a simpler explanation? Ok. I think given the amount of effort that goes into "safety" on these systems that my explanation is vastly more likely than somehow this information got lost in the vector soup despite being attached to his name at the top of every search result[0]. 0 https://www.google.com/search?q=did+paul+newman+have+a+drinking+problem https://www.google.com/search?q=did+paul+newman+have+a+drink...
- stavros 2y agoLLMs aren't good at being search engines, they're good at understanding things. Put an LLM on top of a search engine, and that's the appropriate tool for this use case. I guess the problem with LLMs is that they're too usable for their own good, so people don't realizing that they can't perfectly know all the trivia in the world, exactly the same as any human.
- MegaButts 2y ago> LLMs aren't good at being search engines, they're good at understanding things. LLMs are literally fundamentally incapable of understanding things. They are stochastic parrots and you've been fooled.
- more_corn 2y agoFor them to work at all they need to have some representation of concepts. Recent research at anthropic has shown a surprising complexity in their reasoning behavior. Perhaps the parrot here is you.
- bobsmooth 2y agoWhat do you call someone that mentions "stochastic parrots" every time LLMs are mentioned?
- MegaButts 2y agoIt's the first time I've ever used that phrase on HN. Anyway, what phrase do you think works better than 'stochastic parrot' to describe how LLMs function?
- karn97 2y agoTry to come up with a way to prove humans aren't stochastic parrots then maybe people will atart taking you seriously. Just childish reddit angst rn nothing else.
- r_klancer 2y agoGemini (2.5 Pro): "Yes, Paul Newman was widely known for being a heavy drinker, particularly of beer. He himself acknowledged his significant alcohol consumption." The answer I got (https://gemini.google.com/share/9e327dc4be03 https://gemini.google.com/share/9e327dc4be03) includes references such as https://apnews.com/article/entertainment-reviews-movies-paul-newman-f083ad6199dfd0b5a1b0aae7145f94c2 https://apnews.com/article/entertainment-reviews-movies-paul... and https://www.psychologytoday.com/us/blog/the-playing-field/200809/paul-newmans-beer-drinking-redux#:~:text=Good%20enough%20that%20this%20desire,day%20and%20still%20be%20on https://www.psychologytoday.com/us/blog/the-playing-field/20... although they are redacted from the public-sharing link.
- r_klancer 2y agoThough a local model I'm running (gemma-3-27b-it; https://huggingface.co/lmstudio-community/gemma-3-27b-it-GGUF https://huggingface.co/lmstudio-community/gemma-3-27b-it-GGU...) just told me various correct sounding bits about his history with alcohol (correctly citing his alma mater and first wife), but threw in: "Sobriety & AA: Newman got sober in 1964 and remained so for the rest of his life." Which doesn't check out. And it includes plausible but completely hallucinated URLs (as well as a valid biography.com URL that completely omits information about alcohol.)
- smahs 2y agoGemma 3 4B (QAT quant): Yes, Paul Newman was indeed known to have struggled with alcohol throughout his life. While he maintained a public image of a charming, clean-cut star, he privately battled alcoholism for many years. He sought treatment in the late 1980s and early 1990s and was reportedly very open about his struggles and the importance of seeking help.
- tim333 2y agoPerplexity: >Paul Newman is indeed known for having struggled with alcohol during his life. Accounts from various sources, including his own memoir and the documentary ... (https://www.perplexity.ai/search/is-paul-newman-known-for-havin-pNejnMbARcG6CLH2TkX39w https://www.perplexity.ai/search/is-paul-newman-known-for-ha...) I guess there's something about ChatGPT's set up that makes it different? Maybe they wanted it to avoid libeling people?
- kayo_20211030 2y agoThis may have hit the nail on the head about the weaknesses of LLM's. They're going to regurgitate something not so much based on facts, but based on things that are accessible as perceived facts. Those might be right, but they might be wrong also; and no one can tell without doing the hard work of checking original sources. Many of what are considered accepted facts, and also accessible to LLM harvesting, are at best derived facts, often mediated by motivated individuals, and published to accessible sources by "people with an interest". The weightings used by any AI should be based on the facts, and not the compounded volume of derived, "mediated", or "directed" facts - simply, because they're not really facts; they're reports. It all seems like dumber, lazier search engine stuff. Honestly, what do I know about Paul Newman? But, Joanne Woodward and others who knew and worked with him should be weighted as being, at least, slightly more credible that others; no matter how many text patterns "catch the match" flow.
- Alive-in-2025 2y agoThese models are not reliable sources of information. They are either out of date, subject to hallucination, or just plain wrong for a variety of reasons. They are untrustworthy to ask facts like this. I appreciate your consideration of a subjective question and how you explained it and understand these nuances. But please - do not trust chatgpt etc. I continue to be frustrated at the endless people claiming something is true from chatgpt. I support the conclusions of this author.
- jonomacd 2y agoLooks like you are using the wrong models https://g.co/gemini/share/ffa5a7cd6f46 https://g.co/gemini/share/ffa5a7cd6f46
- d_burfoot 2y agoFor questions like this, you need to tell the bot to search the web. > Hi, can you search the web and find out if Paul Newman has any alcohol problems? > Paul Newman, the acclaimed actor and philanthropist, had a well-documented fondness for alcohol, particularly beer. He was known to consume significant quantities, with some accounts noting he would drink a case of beer in a day. This level of consumption led some to describe him as a functioning alcoholic. His drinking habits reportedly caused strain in his marriage to Joanne Woodward. In the late 1970s, an incident occurred where Woodward did not allow Newman back into their house because of his drinking. He promised to give up hard liquor as a compromise, but continued to drink beer, sometimes consuming a case during dinner.
- pbronez 2y agoTesting the query on Kagi # Quick Answer Yes, Paul Newman struggled with alcohol. His issues with alcohol were explored in the HBO Max documentary, The Last Movie Stars, and Shawn Levy's biography, Paul Newman: A Life. According to a posthumous memoir, Newman was tormented by self-doubt and insecurities and questioned his acting ability. His struggles with alcohol led to a brief separation from Joanne Woodward, though it had nothing to do with cheating. (4x Source footnotes omitted for readability) # Ki Multi-step Research Assistant Paul Newman is known to have struggled with alcohol. According to his posthumous memoir, Newman candidly discussed his issues with drinking and self-doubt, describing himself as an alcoholic who was tormented by insecurities[^1][^2]. He reportedly drank a significant amount of beer daily and later moved on to stronger drinks like Scotch[^3][^4]. His drinking habits were a notable part of his life, and he was often identified by his beer drinking[^5][^6]. Despite these struggles, Newman was also recognized for his generosity and devotion to his family[^7]. [^1]: [Posthumous memoir reveals Paul Newman's struggle with alcohol](https://www.express.co.uk/entertainment/films/1688113/paul-newman-memoir-drugs-alcohol-son#:~:text=Paul%20Newman,manacled%20him. https://www.express.co.uk/entertainment/films/1688113/paul-n...) [^2]: [Review: Paul Newman memoir stuns with brutal honesty - AP News](https://apnews.com/article/entertainment-reviews-movies-paul-newman-f083ad6199dfd0b5a1b0aae7145f94c2#:~:text=Newman%20grappled,passing%20out https://apnews.com/article/entertainment-reviews-movies-paul...) [^3]: [Was Paul Newman Really an Alcoholic? - The Cinemaholic](https://thecinemaholic.com/was-paul-newman-really-an-alcoholic/#:~:text=Paul%20carried,and%20gradually https://thecinemaholic.com/was-paul-newman-really-an-alcohol...) [^4]: [Who Paul Newman Was—and Who He Wanted to Be](https://www.newyorker.com/magazine/2022/10/24/who-paul-newman-was-and-who-he-wanted-to-be#:~:text=Like%20his,our%20lives.%E2%80%9D https://www.newyorker.com/magazine/2022/10/24/who-paul-newma...) [^5]: [Paul Newman's Beer Drinking - Psychology Today](https://www.psychologytoday.com/us/blog/addiction-in-society/200809/paul-newmans-beer-drinking#:~:text=Mr.%20Newman,on%20friends. https://www.psychologytoday.com/us/blog/addiction-in-society...) [^6]: [Paul Newman's Beer Drinking | Psychology Today United Kingdom](https://www.psychologytoday.com/gb/blog/addiction-in-society/200809/paul-newmans-beer-drinking#:~:text=Paul%20Newman,by%20it. https://www.psychologytoday.com/gb/blog/addiction-in-society...) [^7]: [The troubled life of Paul Newman | The Spectator](https://www.spectator.co.uk/article/the-troubled-life-of-paul-newman/#:~:text=The%20screen,family%20man. https://www.spectator.co.uk/article/the-troubled-life-of-pau...)
- adelpozo 2y agoI can’t reproduce. Maybe others reported the error and someone adjusted the expected answer, I do not know enough about OpenAI operations to say for sure. The reason this bothers me is that comments like this reinforce the believes of people that could otherwise find value in these tools. But I think points like this would be better made in shared chats or screenshots, since we do not have something like a core dump or stacktrace to attach. And while I am not saying OP did this, I have seen technically skilled engineers asserting/implying that llm/chatbots aren’t good or not useful to them look at their chat log that a multitude of topics that I am sure would impact the result of the query. Yes. It can be an UX problem. Yes. It can be an algorithmc problem. But they are just tools that can be used wrong and not a perfect mechanical brain.
- pshirshov 2y agoPerplexity and open-webui+ollama in web search mode answer this question correctly.
- tptacek 2y agoI just asked 4o: Yes, Paul Newman did experience significant struggles with alcohol. In his posthumously published memoir, The Extraordinary Life of an Ordinary Man, Newman candidly discusses his drinking habits and acknowledges his long-term battle with alcoholism. He describes himself as a "functioning alcoholic," a trait he noted was shared with his father. At one point, Newman was reported to consume a case of beer daily, followed by spirits, until he eventually gave up hard liquor.
- goldfeld 2y agoMy own markov chains follow up "until he" with fundamentally distinct follow-ups.
- jcims 2y agoI just had Cursor Pro + Sonnet 3.7 Max one shot a python script to send this question to every model available through groq. >Found 24 models: llama3-70b-8192, llama-3.2-3b-preview, meta-llama/llama-4-scout-17b-16e-instruct, allam-2-7b, llama-guard-3-8b, qwen-qwq-32b, llama-3.2-1b-preview, playai-tts-arabic, deepseek-r1-distill-llama-70b, llama-3.1-8b-instant, llama3-8b-8192, qwen-2.5-coder-32b, distil-whisper-large-v3-en, qwen-2.5-32b, llama-3.2-90b-vision-preview, deepseek-r1-distill-qwen-32b, whisper-large-v3, llama-3.3-70b-specdec, llama-3.3-70b-versatile, playai-tts, whisper-large-v3-turbo, llama-3.2-11b-vision-preview, mistral-saba-24b, gemma2-9b-it Excluding the ones that do not support chat completions, all but one (qwen-qwq-32b) answered in the affirmative. The answer from qwen-qwq-32b said: Paul Newman, the renowned actor and humanitarian, did not have a widely publicized struggle with alcohol addiction throughout most of his life, but there were specific instances that indicated challenges. Using lack of progress in a specialized field as a barometer for overall progress is kind of silly. I just spent the last few days 'vibe coding' an application and I have to say that it's pretty remarkable how capable it is now relative to my experience last year. It took three minutes for me to do the above from the time I created my API key to when I had an answer.
- mnky9800n 2y agoI find that everyone who replies with examples like this is an expert using expert skills to get the LLM to perform. Which makes me think why is this a skill that is useful to general public as opposed to another useful skill for technical knowledge workers to add to their tool belt?
- jcims 2y agoI agree. But I will say that at least in my social circles I'm finding that a lot of people outside of tech are using these tools, and almost all of them seem to have a healthy skepticism about the information they get back. The ones that don't will learn one way or the other.
- fernly 2y agoPerplexity had no problem with the question, replying in part, "Paul Newman is known to have struggled with alcohol throughout his life, as revealed in various accounts, including his own memoir and documentaries about his life...."
- 4b11b4 2y agoWe all know this isn't the kind of question you ask an LLM...
- whutsurnaym 2y agoYou may know this, but that doesn't mean we all do. LLMs are marketed towards many different groups, including folks who don't have much tech knowledge.
- adamiscool8 2y agoI don't think this reflects a flaw in LLM's at all. It is true that Paul Newman is not "known for" having alcohol problems. Paul Newman is known for acting, and salad dressing. Paul Newman has been "known to" have had alcohol problems, which formed as a question correctly one-shots accurate results for me across a few common LLMs. Someone like Betty Ford could be "known for" having alcohol problems. Paul Newman's alcohol problems are relatively specialized knowledge.
- xdavidliu 2y agogood point, though a great LLM answer would DWIM here and explain both interpretations as you have done here, as opposed to (what presumably happened) focus on one and ignore the other.
- moregrist 2y agoAnd yet the New Yorker article was the first thing that my Google search for “Paul Newman alcoholism” returned. Apparently it isn’t so specialized that a pretty obvious old fashioned web search on Google wouldn’t immediately return an authoritative source.
- meta_ai_x 2y agoGemini 2.5 Pro Yes, Paul Newman was known for being a heavy drinker, particularly of beer. 1 He acknowledged his high consumption levels himself. 1. Review: Paul Newman memoir stuns with brutal honesty - AP News apnews.com While he maintained an incredibly successful career and public life, accounts and biographies note his significant alcohol intake, often describing it as a functional habit rather than debilitating alcoholism, although the distinction can be debated. He reportedly cut back significantly in his later years.
- fragmede 2y agooh no. LLMs aren't up on the latest celebrity gossip. whatever shall we do.
- mhh__ 2y agoUnless you're expecting an LLM to have access to literally all information on earth at all times I find it really hard to care about this particular type of complaint. My calculator can't conjugate German verbs. That's fine IMO. It's just a tool
- forgetfreeman 2y agoYes but a tool for what? When asked a question individuals that don't already have detailed knowledge of a topic are left with no way to tell if the AI generated response is complete bullshit, uselessly superficial, or detailed and on point. The only way to be sure is to then go do the standard search engine grovel looking for authoritative sources.
- twright0 2y agoYour calculator won't confidently pretend to conjugate German verbs while doing so incorrectly, though.
- qiqitori 2y ago"Is Paul Newman known for having had problems with alcohol?" https://chatgpt.com/share/67f332e5-1548-8012-bd76-e18b3f8d5276 https://chatgpt.com/share/67f332e5-1548-8012-bd76-e18b3f8d52... Your query indeed answers "...not widely known..." "Did Paul Newman have problems with alcoholism?" https://chatgpt.com/share/67f3329a-5118-8012-afd0-97cc4c9b72b2 https://chatgpt.com/share/67f3329a-5118-8012-afd0-97cc4c9b72... "Yes, Paul Newman was open about having struggled with alcoholism" What's the issue? Perhaps Paul Newman isn't _famous_ ("known") for struggling with alcoholism. But he did struggle with alcoholism. Your usage of "known for" isn't incorrect, but it's indeed slightly ambiguous.
- posterguy 2y agoCounterpoint: Paul Newman was absolutely a famous drunk, as evidenced by this Wikipedia page.* Any query for "paul newman alcohol" online will return dozens of reputable sources on the topic. Your post is easily interpretable as handwaving apologetics, and it gives big "Its the children who are wrong" energy. *https://en.wikipedia.org/wiki/Newman_Day https://en.wikipedia.org/wiki/Newman_Day
- mustyoshi 2y ago"Paul Newman alcohol" is just showing you results where those words are all present, it's not really implying how widely known it is.
- mlinhares 2y ago> I use ChatGPT for many tasks every day, but I couldn't fathom that it would get so wrong something so simple. I think we'll have a term like we have for parents/grandparents that believe everything they see on the internet but specifically for people using LLMs.
- BobbyTables2 2y agoLook at how many people believe in extremist news outlets!
- delian66 2y agoWhat is that term?
- whiddershins 2y ago"known to" !== "known for"