9 ms·
> Wah, it can't write code like a Senior engineer with 20 years of experience! No, that's not my problem with it. My problem with it is that inbuilt into the
by cogman10 2y ago
> Wah, it can't write code like a Senior engineer with 20 years of experience!
No, that's not my problem with it. My problem with it is that inbuilt into the models of all LLMs is that they'll fabricate a lot. What's worse, people are treating them as authoritative.
Sure, sometimes it produces useful code. And often, it'll simply call the "doTheHardPart()" method. I've even caught it literally writing the wrong algorithm when asked to implement a specific and well known algorithm. For example, asking it "write a selection sort" and watching it write a bubble sort instead. No amount of re-prompts pushes it to the right algorithm in those cases either, instead it'll regenerate the same wrong algorithm over and over.
Outside of programming, this is much worse. I've both seen online and heard people quote LLM output as if it were authoritative. That to me is the bigger danger of LLMs to society. People just don't understand that LLMs aren't high powered attorneys, or world renown doctors. And, unfortunately, the incorrect perception of LLMs is being hyped both by LLM companies and by "journalists" who are all to ready to simply run with and discuss the press releases from said LLM companies.
- MattGaiser 2y agoHumans bullshit and hallucinate and claim authority without citation or knowledge. They will believe all manner of things. They frequently misunderstand. The LLM doesn’t need to be perfect. Just needs to beat a typical human. LLM opponents aren’t wrong about the limits of LLMs. They vastly overestimate humans.
- jasonjayr 2y agoLLM's can't be held accountable. And many, many companies are proposing and implementing uses for LLM's to intentionally obscure that accountability. If a person makes up something, innocently or maliciously, and someone believes it and ends up getting harmed, that person can have some liability for the harm. If a LLM hallucinates something, that somone believes and they end up getting harmed, there's no accountability. And it seems that AI companies are pushing for laws & regulations that further protect them from this liability. These models can be useful tools, but the targets these AI companies are shooting for are going to be activly harmful in an economy that insists you do something productive for the continued right to exist.
- garciasn 2y ago[flagged]
- dwattttt 2y agoThat reads as "people shouldn't trust what AI tells them", which is in opposition to what companies want to use AI for. An airline tried to blame its chatbot for inaccurate advice it gave (whether a discount could be claimed after a flight). Tribunal said no, its chatbot was not a separate legal entity. https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know https://www.bbc.com/travel/article/20240222-air-canada-chatb...
- devsda 2y agoYeah. Where I live, we are always reminded that our conversations with insurance provider personnel over phone are recorded and can be referenced while making a claim. Imagine a chatbot making false promises to prospective customers. Your claim gets denied, you fight it out only to learn their ToS absolves them of "AI hallucinations".
- frontfor 2y agoThis is correct. On top of that, the failure modes of AI system are unpredictable and incomprehensible. Present day AI systems can fail on/be fooled by inputs in surprising ways that no humans would.
- MattGaiser 2y agoAccountability exists for two reasons: 1. To make those harmed whole. On this, you have a good point. The desire of AI firms or those using AI to be indemnified from the harms their use of AI causes is a problem as they will harm people. But it isn't relevant to the question of whether LLMs are useful or whether they beat a human. 2. To incentivize the human to behave properly. This is moot with LLMs. There is no laziness or competing incentive for them.
- 2y ago
- toomuchtodo 2y agoIt's fine if it isn't perfect if whomever is spitting out answers assumes liability when the robot is wrong. But, what people want is the robot to answer questions and there to be no liability when it is well known that the robot can be wildly inaccurate sometimes. They want the illusion of value without the liability of the known deficiencies. If LLM output is like a magic 8 ball you shake, that is not very valuable unless it is workload management for a human who will validate the fitness of the output.
- cyanydeez 2y agook, but people are building determinative software _on top of them_. It's like saying "it's ok, people make mistakes, but lets build infrastructure on some brain in a vat". It's just inherently not at the point that you can make it the foundation of anything but a pet that helps you slop out code, or whatever visual or textual project you have. It's one of those "quantities is so fascisnating, lets ignore how we got here in the first place"
- krustyburger 2y agoYou’re moving the goalposts. LLMs are masquerading as superb reference tools and as sources of expertise on all things, not as mere “typical humans.” If they were presented accurately as being about as fallible as a typical human, typical humans (users) wouldn’t be nearly as trusting or excited about using them, and they wouldn’t seem nearly as futuristic.
- cogman10 2y ago> LLM opponents aren’t wrong about the limits of LLMs. They vastly overestimate humans. On the contrary. Humans can earn trust, learn, and can admit to being wrong or not knowing something. Further, humans are capable of independent research to figure out what it is they don't know. My problem isn't that humans are doing similar things to LLMs, my problem is that humans can understand consequences of bullshitting at the wrong time. LLMs, on the other hand, operate purely on bullshitting. Sometimes they are right, sometimes they are wrong. But what they'll never do or tell you is "how confident am I that this answer is right". They leave the hard work of calling out the bullshit on the human. There's a level of social trust that exists which LLMs don't follow. I can trust when my doctor says "you have a cold" that I probably have a cold. They've seen it a million times before and they are pretty good at diagnosing that problem. I can also know that doctor is probably bullshitting me if they start giving me advice for my legal problems, because it's unlikely you are going to find a doctor/lawyer. > Just needs to beat a typical human. My issue is we can't even measure accurately how good humans are at their jobs. You now want to trust that the metrics and benchmarks used to judge LLMs are actually good measures? So much of the LLM advocates try and pretend like you can objectively measure goodness in subjective fields by just writing some unit tests. It's literally the "Oh look, I have an oracle java certificate" or "Aws solutions architect" method of determining competence. And so many of these tests aren't being written by experts. Perhaps the coding tests, but the legal tests? Medical tests? The problem is LLM companies are bullshiting society on how competently they can measure LLM competence.
- TheOtherHobbes 2y agoYour second point directly contradicts your first point. In fact we do know how good doctors and lawyers are at their jobs, and the answer is "not very." Medical negligence claims are a huge problem. Claims agains lawyers are harder to win - for obvious reasons - but there is plenty of evidence that lawyers cannot be presumed competent. As for coding, it took a friend of mine three days to go from a cold start with zero dev experience to creating a usable PDF editor with a basic GUI for a specific small set of features she needed for ebook design. No external help, just conversations with ChatGPT and some Googling. Obviously LLMs have issues, but if we're now in the "Beginners can program their own custom apps" phase of the cycle, the potential is huge.
- abenga 2y agoI never ask a typical human for help with my work, why should that be my benchmark for using an information tool? Afaik, most people do not write about what they don't know, and if one made a habit of it, they would be found and filtered out of authoritative sources of information.
- Animats 2y ago> inbuilt into the models of all LLMs is that they'll fabricate a lot. Still the elephant in the room. We need an AI technology that can output "don't know" when appropriate. How's that coming along?
- icelancer 2y agoNot sure. We haven't figured it out for humans yet.
- cogman10 2y agoSure we have. We don't have a perfect solution but it's miles better than what we have for LLMs. If a lawyer consistently makes stuff up on legal filings, in the worst cases they can lose their license (though they'll most likely end up getting fines). If a doctor really sucks, they become uninsurable and ultimately could lose their medical license. Devs that don't double check their work will cause havoc with the product and, not only will they earn low opinions from their colleges, they could face termination. Again, not perfect, but also not unfigured out.
- shsbdncudx 2y agoBy the same measure, of an llm really sucks we stop using it, same solution
- saagarjha 2y agoMany people haven’t gotten the message yet, it seems.
- icelancer 2y agoSure. I don't use GPT-3.5-Turbo for similar reasons. I fired it.
- vladms 2y agoI have seen many people not saying "don't know" when appropriate. If you believe whomever without some double-checking you will have (bad) surprises. To make another parallel: that's why we have automated testing in software (long before LLMs). Because you can't trust without checking.
- gonzobonzo 2y ago> My problem with it is that inbuilt into the models of all LLMs is that they'll fabricate a lot. What's worse, people are treating them as authoritative. The same is true about the internet, and people even used to use these arguments to try to dissuade people from getting their information online (back when Wikipedia was considered a running joke, and journalists mocked blogs). But today it would be considered silly to dissuade someone from using the internet just because the information there is extremely unreliable. Many programmers will say Stack Overflow is invaluable, but it's also unreliable. The answer is to use it as a tool and a jumping off point to help you solve your problem, not to assume that its authoritative. The strange thing to me these days is the number of people who will talk about the problems with misinformation coming from LLMs, but then who seem to uncritically believe all sorts of other misinformation they encounter online, in the media, or through friends. Yes, you need to verify the information you're getting, and this applies to far more than just LLMs.
- dcow 2y agoI have decided it must be attached to a sort of superiority complex. These types of people believe they are capable of deciphering fact from fiction but the general population isn’t so LLMs scare them because someone might hear something wrong and believe it. It almost seems delusional. You have to be incredibly self aggrandizing in your mind to think this way. If LLMs were actually causing “a problem” then there would be countless examples of humans making critical mistakes because of bad LLM responses, and that is decidedly not happening. Instead we’re just having fun ghiblifying the last 20 years of the internet.
- wqaatwt 2y ago> that is decidedly not happening Regardless of anything else it’s extremely too early to make such claims. We have to wait until people start allowing “AI agents” to make autonomous blackbox decision with minimal supervision since nobody has any clue what’s happening. Even if we tone down the SciFi dystopia angle not that many people really use LMMs in non superficial ways yet. What I’m most afraid of would be the next generation growing without the ability to critically synthesize information on their own.
- senordevnyc 2y agoIt's always perplexing when people talk about LLMs as "it", as if there's only one model out there, and they're all equally accurate. FWIW, here's 4o writing a selection sort: https://chatgpt.com/share/67e60f66-aacc-800c-9e1d-303982f54d57 https://chatgpt.com/share/67e60f66-aacc-800c-9e1d-303982f54d...
- threeseed 2y agoI don't understand the point of that share. There are likely thousands of implementations of selection sort on the internet and so being able to recreate one isn't impressive in the slightest. And all the models are identical in not being able to discern what is real or something it just made up.
- dvsfish 2y agoI guess the point is that op was pretty adamant that LLMs refuse to write selection sorts?
- wqaatwt 2y agoNo? I mean if they refused that would actually be a reasonably good outcome. The real problem is if they generally can write selection sorts but occasionally go haywire due to additional context and start hallucinating. I mean asking a straightforward question like: https://chatgpt.com/share/67e60f66-aacc-800c-9e1d-303982f54d57 https://chatgpt.com/share/67e60f66-aacc-800c-9e1d-303982f54d... is entirely pointless as a test
- senordevnyc 2y agoBecause, to be blunt, I think this is total bullshit if you're using a decent model: "I've even caught it literally writing the wrong algorithm when asked to implement a specific and well known algorithm. For example, asking it "write a selection sort" and watching it write a bubble sort instead. No amount of re-prompts pushes it to the right algorithm in those cases either, instead it'll regenerate the same wrong algorithm over and over."
- nonethewiser 2y ago> What's worse, people are treating them as authoritative. … I've both seen online and heard people quote LLM output as if it were authoritative. Thats not an LLM problem. But indeed quite bothersome. Dont tell me what Chatgpt told you. Tell me what you know. Maybe you got it from ChatGPT and verified it. Great. But my jaw kind of drops when people cite an LLM and just assume it’s correct.
- trial3 2y agoIt might not be an LLM problem, but it’s an AI-as-product problem. I feel like every major player’s gamble is that they can cement distinct branding and model capabilities (as perceived by the public) faster than the gradual calcification of public AI perception catches up with model improvements - every time a consumer gets burned by AI output in even small ways, the “AI version of Siri/Alexa only being used for music and timers” problem looms a tiny, tiny bit larger.
- chii 2y agoWhy is that a problem tho? Branding for current products have this property today - for example, apple products are seen as being used by creatives and such.
- nyarlathotep_ 2y ago> But my jaw kind of drops when people cite an LLM and just assume it’s correct. Yes but they're literally told by allegedly authoritative sources that it's going to change everything and eliminate intellectual labor, so is it totally their fault? They've heard about the uncountable sums of money spent on creating such software, why would they assume it was anything short of advertised?
- yunwal 2y ago> Yes but they're literally told by allegedly authoritative sources that it's going to change everything and eliminate intellectual labor Why does this imply that they’re always correct? I’m always genuinely confused when people pretend like hallucinations are some secret that AI companies are hiding. Literally every chat interface says something like “LLMs are not always accurate”.
- worthless-trash 2y agoI saw someone saying 80% of doctors believe that LLM's are trustworthy consultation partners. Code created by LLM's doesnt compile, hallucinated API's.. invalid syntax and completely broken logic, why would you trust it with someones life !
- Aeolun 2y agoBecause LLM’s, with like 20% hallucination rate, are more reliable than overworked, tired doctors that can spend only one ounce of their brainpower on the patient they’re currently helping?
- archagon 2y agoYeah, I'm gonna need really strong evidence for that claim before I entrust my life to an AI.
- financypants 2y agoluckily us being programmers can fix things like syntax errors.
- prawn 2y agoI wonder if the exact phrasing has varied from the source, but even then if "consultation partners" is doing the heavy lifting there. If it was something like "useful consultation partners", I can absolutely see value as an extra opinion that is easy to override. "Oh yeah, I hadn't thought about that option - I'll look into it further." I imagine we're talking about it as an extra resource rather than trusting it as final in a life or death decision.
- worthless-trash 2y agoI'll see if i can dig it up, it was from a real life meeting which I have tossed the printed notes from a while back in disgust.
- worthless-trash 2y ago> I imagine we're talking about it as an extra resource rather than trusting it > as final in a life or death decision. I'd like to think so. Trust is also one of those non-concrete terms that have different meanings to different people. I'd like to think that doctors use their own judgement to include the output from their trained models, I just wonder how long it is till they become the default judgement when humans get lazy.
- BriggyDwiggs42 2y agoPeople need time to adapt to llms capacity to spit out nonsense. It’ll take time but I’m sure they will.
- mitjam 2y agoThe superficial view: „they hallucinate“ The underlying cause: 3rd order ignorance: 3rd Order Ignorance (3OI)—Lack of Process. I have 3OI when I don't know a suitably efficient way to find out I don't know that I don't know something. This is lack of process, and it presents me with a major problem: If I have 3OI, I don't know of a way to find out there are things I don't know that I don't know. —- not from an llm My process: use llms and see what I can do with them while taking their Output with a grain of salt.
- mdp2021 2y agoBut the issue of the structural fault remains. To state the phenomenon (hallucination) is not "superficial", as the root does not add value in the context. Symptom: "Response was, 'Use the `solvetheproblem` command'". // Cause: "It has no method to know that there is no `solvetheproblem` command". // Alarm: "It is suggested that it is trying to guess a plausible world through lacking wisdom and data". // Fault: "It should have a database of what seems to be states of facts, and it should have built the ability to predict the world more faithfully to facts".
- dcow 2y agoHave you talked to a non-artificial intelligence lately? I’ve got some news for you…
- kristopolous 2y agoThis is the problem of the internet writ large. The solution is to be selective and careful like always
- geysersam 2y ago> What's worse, people are treating them as authoritative. So what? People are wrong all the time. What happens when people are wrong? Things go wrong. What happens then? People learn that the way they got their information wasn't robust enough and they'll adapt to be more careful in the future. This is the way it has always worked. But people are "worried" about LLMs... Because they're new. Don't worry, it's just another tool in the box, people are perfectly capable of being wrong without LLMs.
- jtafurth 2y agoBeing wrong when you are building a grocery management app is one thing, being wrong when building a bridge is another. For those sensitive use cases, it is imperative we create regulation, like every other technology that came before it, to minimize the inherent risks. In an unrelated example, I saw someone saying recently they don't like a new version of an LLM because it no longer has "cool" conversations with them, so take that as you will from a psychological perspective.
- geysersam 2y agoI have a hard time taking that kind of worry seriously. In ten years, how many bridges will have collapsed because of LLMs? How many people will have died? Meanwhile, how many will have died from fentanyl or cars or air pollution or smoking. Why do people care so much about the hypothetical bad effects from new technology and so little about the things we already know are harmful
- troismph 2y ago> What's worse, people are treating them as authoritative Because in people's experience, LLMs are often correct. You are right LLMs are not authoritative, but people trust it exactly because they often do produce correct answers.
- solumunus 2y agoA tool is good but lots of people are stupid and misuse it… That’s just life buddy.
- vidarh 2y agoPeople fabricated a lot too. Yesterday I spent far less time fixing issues in the far more complex and larger changes Claude Code managed to churn out than what the junior developer I worked with needed. Sometimes it's the reverse. But with my time factored in, working with Claude Code is generally more productive for me than working with a junior. The only reason I still work with a junior dev is as an investment into teaching him. Claude is cheaper, faster, produces better code.
- mdp2021 2y agoYou are mixing a point and the issue, largely heterogeneous: Claude being a champion in producing good code vs LLMs in general being delirious. If your junior developer is just "junior", that is one matter; if your junior developer hallucinates documentation details, that's different.
- vidarh 2y agoEvery developer I've ever worked with have gotten things wrong. Whether you call that hallucinating or not is irrelevant. What matters is the effort it takes to fix.
- mdp2021 2y agoOn the logically practical point I agree with you (what counts in the end in the specific process you mention is the gain vs loss game), but my point was that if your assistant is structurally delirious you will have to expect a big chunk of the "loss" as structural. -- Edit: new information may contribute to even this exchange, see https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... > It turns out that, in Claude, refusal to answer is the default behavior I.e., boxes that incline to different approaches to heuristic will behave differently and offer different value (to be further assessed within a framework of complexity, e.g. "be creative but strict" etc.)
- vidarh 2y ago
- HdS84 2y agoI was part of preparing an offer a few weeks ago. The customer prepared a lot of documents for us - maybe 100 pages on total. Boss insisted on using chatgpt to summarize this stuff and read only the summary. I did a loner, slower, reading and cought on some topics chatgpt outright dropped. Our offer was based on the summary - and fell through because we missed these nuances. But hey, boss did not read as much as previously...
- refurb 2y agoMy company just broadly adopted AI. It’s not a tech company and usually late to the game when it comes to tech adoption. I’m counting down the days when some AI hallucination makes its way all the way to the C-suite. People will get way too comfortable with AI and don’t understand just how wrong it can be. Some assumption will come from AI, no one will check it and it’ll become a basic business input. Then suddenly one day someone smart will say “thats not true” and someone will trace it back to AI. I know it. I assume at that point in time there will be some general directive on using AI and not assuming it’s correct. And then AI will slowly go out of favor.
- pkolaczk 2y ago> I've even caught it literally writing the wrong algorithm when asked to implement a specific and well known algorithm Happened to me as well. Wanted it to quickly write an algorithm for standard deviation over a stream of data, which is a text-book algorithm. It did it almost right, but messed up the final formula and the code gave wrong answers. Weird, considering some correct codes exist for that problem in Wikipedia.
- gumboshoes 2y agoBut this was true before LLMs. People would and still do take any old thing from an internet search and treat it as true. There is a known, difficult-to-remedy failure to properly adjudicate information and source quality, and you can find it discussed in research prior to the computer age. It is a user problem more than a system problem. In my experience, with the way I interact with LLMs, they are more likely to give me useful output than not, and this is borne out by mainstream non-edge-case academic peer-reviewed work. Useful does not necessarily equal 100% correct, just as a Google search does not. I judge and vet all information, whether from an LLM, search, book, paper, or wherever We can build a straw person who "always" takes LLM output as true and uses it as-is but those are the same people who use most information tools poorly, be they internet search, dictionaries, or even looking in their own files for their own work or sent mail (I say this as an IT professional who has seen the worker types from before the pre-internet days through now). In any case, we use automobiles despite others misusing them. But only the foolish among us completely take our hands off the wheel for any supposed "self-driving" features. While we must prevent and decry the misuse by fools, we cannot let their ignorance hold us back. Let's let their ignorance help make tools, as they help identify more undesirable scenarios.