4 ms·
It's even worse. AI is a really smart but inexperienced person who also lies frequently. Because AI is not accountable to anything, it'll always come up with a
by coliveira 2y ago
It's even worse. AI is a really smart but inexperienced person who also lies frequently. Because AI is not accountable to anything, it'll always come up with a reasonable answer to any question, if it is correct or not.
- belZaah 2y agoTo put it in other words: it is not clear when and how they hallucinate. With a person, their competence could be understood and also their limits. But a llm can happily give different answers based on trivial changes in the question with no warning.
- zozbot234 2y agoLLM's are non-deterministic: they'll happily give different answers to the same prompt based on nothing at all. This is actually great if you want to use them for "creative" content generation tasks, which is IMHO what they're best at. (Along with processing of natural language input.) Expecting them to do non-trivial amounts of technical or mathematical reasoning, or even something as simple as code generation (other than "translate these complex natural-language requirements into a first sketch of viable computer code") is a total dead end; these will always be language systems first and foremost.
- mapt 2y agoThis confuses me. You have your model, you have your tokens. If the tokens are bit-for-bit-identical, where does the non-determinism come in? If the tokens are only roughly-the-same-thing-to-a-human, sure I guess, but convergence on roughly the same output for roughly the same input should be inherently a goal of LLM development.
- zozbot234 2y agoThe model outputs probabilities, which you have to sample randomly. Choosing the "highest" probability every time leads to poor results in practice, such as the model tending to repeat itself. It's a sort of Monte-Carlo approach.
- lifthrasiir 2y agoIt is technically possible to make it fully deterministic if you have a complete control over the model, quantization and sampling processes. The GP probably meant to say that most commercially available LLM services don't usually give such control.
- brookst 2y agoActually you just have to set temperature to zero.
- deleted 2y ago[deleted]
- zeta0134 2y agoMost any LLM has a "temperature" setting, a set of randomness added to the otherwise fixed weights to intentionally cause exactly this nondeterministic behavior. Good for creative tasks, bad for repeatability. If you're running one of the open models, set the temperature down to 0 and it suddenly becomes perfectly consistent.
- owenpalmer 2y agoYou can get deterministic output with even with a high temp. Whatever "random" seed was used can be reused.
- ninkendo 2y ago> If the tokens are bit-for-bit-identical, where does the non-determinism come in? By design, most LLM’s have a randomization factor to their model. Some use the concept of “temperature” which makes them randomly choose the 2nd or 3rd highest ranked next token, the higher the temperature the more often/lower they pick a non-best next token. OpenAI described this in their papers around the GPT-2 timeframe IIRC.
- HarHarVeryFunny 2y agoThe trained model is just a bunch of statistics. To use those statistics to generate text you need to "sample" from the model. If you always sampled by taking the model's #1 token prediction that would be deterministic, but more commonly a random top-K or top-p token selection is made, which is where the randomness comes in.
- ninetyninenine 2y agoComputers are deterministic. LLMs run on computers. If you use the same seed for the random number generator you’ll see that it will produce the same output given an input.
- layer8 2y agoThe unreliability of LLMs is mostly unrelated to their (artificially injected) non-determinism.
- liotier 2y agoIn a conversation (conversation and attached pictures at https://bsky.app/profile/liotier.bsky.social/post/3ldxvutf7622t https://bsky.app/profile/liotier.bsky.social/post/3ldxvutf76...), I delete a spurious "de" ("Produce de two-dimensional chart [..]" to "Produce two-dimensional [..]") and ChatGPT generates a new version of the graph, illustrating a different function although nothing else has changed and there was a whole conversation to suggest that ChatGPT held a firm model of the problem. Confirmed my current doctrine: use LLM to give me concepts from a huge messy corpus, then check those against sources from said corpus.
- aruametello 2y ago> trivial changes in the question i love how those changes are often just a different seed in the randomness... as just chance. run some repeated tests with "deeper than surface knowledge" on some niche subjects and got impressed that it gave the right answer... about 20% of the time. (on earlier openAI models)
- ANewFormation 2y agoThere's no need for there to be changes to the question. LLMs have a rng factor built in to the algorithm. It can happily give you the right answer and then the wrong one.
- brookst 2y agoAsk survey designers how “trivial” changes to questions impact results from humans. It’s a huge thing in the field.
- Polizeiposaune 2y agoSaying that they "lie" creates the impression that they have knowledge that they make false statements, and they intend to deceive. They're not that capable. They're just bullshit artists. LLM = LBM (large bullshit models).
- oh_my_goodness 2y ago"AI is a really smart but inexperienced person who also lies frequently." Careful. Here "smart" means "amazing at pattern-matching and incredibly well-read, but has zero understanding of the material."
- maxdoop 2y agoAnd how is what humans do any different ? What does it mean to understand ? Are we pattern matching as well?
- deleted 2y ago[deleted]
- oh_my_goodness 2y agoI asked ChatGPT to help out: ----------------------------- "The distinction between AI and humans often comes down to the concept of understanding. You’re right to point out that both humans and AI engage in pattern matching to some extent, but the depth and nature of that process differ significantly." "AI, like the model you're chatting with, is highly skilled at recognizing patterns in data, generating text, and predicting what comes next in a sequence based on the data it has seen. However, AI lacks a true understanding of the content it processes. Its "knowledge" is a result of statistical relationships between words, phrases, and concepts, not an awareness of their meaning or context"
- oh_my_goodness 2y agoAnyone downvoting, please be aware that you are downvoting the AI's answer! :)
- portaouflop 2y agopeople are downvoting because they don’t want to see walls of text generated by llms on hn
- mattgreenrocks 2y agoIt is a wonderful irony that AI makes competence all the more important. It’s almost like all the thought leading that proclaimed the death of software eng was nothing but self-promotional noise. Huh, go figure.
- TrueDuality 2y agoDon't count it out yet as being problematic for software engineering, bu not in the way you probably intend with your comment. Where I see software companies using it most is as a replacement for interns and junior devs. That replacement means we're not training up the next generation to be the senior or expert engineers with real world experience. The industry will feel that badly at some point unless it gets turned around.
- kensey 2y agoIt’s also already becoming an issue for open-source projects that are being flooded with low-quality (= anything from “correct but pointless” to “actually introduces functional issues that weren’t there before”) LLM-generated PRs and even security reports —- for examples see Daniel Stenberg’s recent writing on this.
- mattgreenrocks 2y agoAgree. I think we are already seeing a hollowing out effect on tech hiring at the lower end. They’ve always been squeezed a bit, but it seems much worse now.
- bentt 2y agoI agree with this irony. That said, combining multiple ais and multiple programs together may mitigate this.
- scarface_74 2y agoHallucinations can be mostly eliminated with RAG and tools. I use NotebookLM all of the time to research through our internal artifacts, it includes citations/references from your documents. Even with ChatGPT you can ask it to find web citations and if it uses the Python runtime to find answers, you can look at the code. And to prevent the typical responses - my company uses GSuite so Google already has our IP, NotebookLM is specifically approved by my company and no Google doesn’t train on your documents
- hatenberg 2y agoEven with RAG you’re bounded at some 93%, it’s not a panacea.
- scarface_74 2y agoHow are you bounded? When you can easily check the sources? Also you act as if humans without LLMs have a higher success rate? There is an entire “reproducibility crisis” with research.
- hatenberg 2y agoFalse equivalence. Humans fail in predictable ways that can be mitigated though process and training. Try training an LLM.
- scarface_74 2y agoHow are the results any different? You can’t completely trust human output. There are all sorts of cognitive biases that come into play.
- HarHarVeryFunny 2y agoFacts can be checked with RAG, but the real value of AI isn't as a search replacement, but for reasoning/problem-solving where the answer isn't out there. How do you, in general, fact check a chain of reasoning?
- _heimdall 2y agoThat sounds mostly like an incentives problem. If OpenAI, Anthropic, etc decide their LLMs need to be accurate they will find some way of better catching hallucinations. It probably will end up (already is?) being yet another LLM acting as a control structure trying to fact check responses before they are sent to users though, so who knows if it will work well. Right noe there's no incentive though. People keep paying good money to use these tools despite their hallucinations, aka lies/gas lighting/fake information. As long as users don't stop paying and LLM companies don't have business pressure to lean on accuracy as a market differentiator, no one is going to bother fixing it.
- bearjaws 2y agoBelieve me, if they could use another LLM to audit an LLM, they would have done that already. It's inherit to transformers that they predict the next most likely token, its not possible to change that behavior without making them useless at generalizing tasks (overfitting). LLMs run on statistics, not logic. There is no fact checking, period. There is just the next most likely token based on the context provided.
- _heimdall 2y agoYeah its an interesting question, and I'm a little surprised I got down voted here. I wouldn't expect them to add an additional LLM layer unless hallucinations from the underlying LLM aren't acceptable, and in this case that means it is unacceptable enough to cost them users and money. Adding a check/audit layer, even if it would work, is expensive both financially and computationally. I'm not sold that it would actually work, but I just don't think they've had enough reason to really give it a solid effort yet either. Edit: as far as fact checking, I'm not sure why it would be impossible. An LLM wouldn't likely be able to run a check against a pre-trained model of "truth," but that isn't the only option. An LLM should be able to mimic what a human would do, interpret the response and search a live dataset of sources considered believable. Throw a budget of resources at processing the search results and have the LLM decide if the original response isn't backed up, or contradicts the source entirely.
- coliveira 2y ago
- uludag 2y agoIt's actually even worse than that: the current trend of AI is transformer-based deep learning models that use self-attention mechanisms to generate token probabilities, predicting sequences based on training data. If only it was something which we could ontologically map onto existing categories like servants or liars...
- kraftman 2y agoIf I had a senior member of the team that was incredibly knowledgeable but occasionally lied, but in a predictable way, I would still find that valuable. Talking to people is a very quick and easy way to get information about a specific subject in a specific context, so I could ask them targetted questions that are easy to verify, the worst thing that happens is I 'waste' a conversation with them.
- HarHarVeryFunny 2y agoSure, but LLMs don't lie in a predictable way. Its just their nature that they output statistical sentence continuations, with a complete disregard for the truth. Everything that they output is suspect, especially the potentially useful stuff that you don't know whether it's true or false.
- kraftman 2y agoThey do lie in a predictable way: if you ask them for a widely available fact you have a very high probability of getting the correct answer, if you ask them for something novel you have a very high probabilty of getting something made up. If I'm trying to use some tool that just got released or just got a big update, I wont use AI, if I want to check the syntax of a for loop in a language I don't know I will. Whenever you ask it a question you should have an idea in your mind of how likely you are to get a good answer back.
- HarHarVeryFunny 2y agoI suppose, but they can still be wrong on the common facts like number of R's in strawberry that are counter-intuitive. I saw an interesting example yesterday of type "I have 3 apples, my dad has 2 more than me ..." where of the top 10 predicted tokens, about 1/2 led to the correct answer, and about 1/2 didn't. It wasn't the most confident predictions that lead to the right answer - pretty much random. The trouble with LLMs vs humans is that humans learn to predict facts (as reflected in feedback from the environment, and checked by experimentation, etc), whereas LLMs only learn to predict sentence soup (training set) word statistics. It's amazing that LLM outputs are coherent as often as they are, but entirely unsurprising that they are often just "sounds good" flow-based BS.
- layer8 2y agoLying means stating things as facts despite knowing or believing that they are false. I don’t think this accurately characterizes LLMs. It’s more like a fever dream where you might fabulate stuff that appears plausibly factual in your dream world.