9 ms·

Overcoming the limits of current LLMs

119 points by sean_pedersen 2y ago

Carrok 2y ago
I wish he went into how to improve confidence scores, though I guess training on better data to begin with should improve results and thus confidence.
RodgerTheGreat 2y ago
One of the main factors that makes LLMs popular today is that scaling up the models is a simple and (relatively) inexpensive matter of buying compute capacity and scraping together more raw text to train them. Without large and highly diverse training datasets to construct base models, LLMs cannot produce even the superficial appearance of good results. Manually curating "tidy", properly-licensed and verified datasets is immensely more difficult, expensive, and time-consuming than stealing whatever you can find on the open internet. Wolfram Alpha is one of the more successful attempts in that curation-based direction (using good-old-fashioned heuristic techniques instead of opaque ML models), and while it is very useful and contains a great deal of factual information, it does not conjure appealing fantasies of magical capabilities springing up from thin air and hands-off exponential improvement.
- totetsu 2y ago
  It’s not unethical if people in positions of privilege and power do it to maintain their rightful position of privilege and power.
  - dang 2y ago
    Please don't post in the flamewar style to HN. It degrades discussion and we're trying to go in the opposite direction here, to the extent that is possible on the internet. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- threeseed 2y ago
  > properly-licensed and verified datasets is immensely more difficult, expensive Arguably the bigger problem is that many of those datasets e.g. WSJ articles are proprietary and can be exclusively licensed like we've seen recently with OpenAI. So we end up with in a situation where competition is simply not possible.
  - piva00 2y ago
    > Arguably the bigger problem is that many of those datasets e.g. WSJ articles are proprietary and can be exclusively licensed like we've seen recently with OpenAI. > So we end up with in a situation where competition is simply not possible. Exactly, and Technofeudalism advances a little more into a new feud. OpenAI is trying to create its moat by shoring up training data, probably attempting to not allow competitors to train on the same datasets they've been licencing, at least for a while. Training data is the only possible moat for LLMs, models seem to be advancing quite well between different companies but as mentioned here a tidy training dataset is the actual gold.
    - gessha 2y ago
      It's landmines no matter how you approach the problem. If you treat the web as a free-for-all and you scrape freely, you get sued by the content platforms for copyright or term of service violation. If you license the content, you let the highest bidder get the content. No matter what happens, capital wins.
  - motohagiography 2y ago
    the irony is that if large media providers aren't represented in the training sets, my comments on internet forums over the decades will be over-represented, which is kind of great, really.
    - pennomi 2y ago
      Right? I always ask people - Hypothetically if someone created a superintelligent AI that took over the world, wouldn’t you WANT it to share your opinions and morals? Every tiny bit of text you write is a vote in the election of our future AI overlords.
darby_nine 2y ago
Man it seems like the ship has sailed on "hallucination" but it's such a terrible name for the phenomenon we see. It is a major mistake to imply the issue is with perception rather than structural incompetence. Why not just say "incoherent output"? It's actually descriptive and doesn't require bastardizing a word we already find meaningful to mean something completely different.
- jwuphysics 2y ago
  > Why not just say "incoherent output"? Because the biggest problem with hallucinations is that the output is usually coherent but factually incorrect. I agree that "hallucination" isn't the best word for it... perhaps something like "confabulation" is better.
  - linguistbreaker 2y ago
    I appreciated a post on here recently that likened AI hallucination to 'bullshitting'. It's coherent, even plausible output without any regard for the truth.
    - wkat4242 2y ago
      While I have absolutely no issues with the word "shit" in popular terms, I'd normally like to reserve it for situations where there's actually intended malice like in "enshittification". Rather than just an imperfect technology as we have here. Many people object to the term enshittification for foul-mouthing reasons but I think it covers it very well because the principle it covers is itself so very nasty. But that's not at all the case here.
      - pessimizer 2y ago
        "Bullshitting" isn't a new piece of jargon, it's a common English word of many decades vintage, and is being used in its dictionary sense here.
    - MattPalmer1086 2y ago
ainoobler 2y ago
The article suggests a useful line of research. Train an LLM to detect logical fallacies and then see if that can be bootstrapped into something useful because it's pretty clear that all the issues with LLMs is the lack of logical capabilities. If an LLM was capable of logical reasoning then it would be obvious when it was generating made-up nonsense instead of referencing existing sources of consistent information.
- dtx1 2y ago
  I think we should start smaller and make them able to count first.
  - Terr_ 2y ago
    Yeah, you can train an LLM to recognize the vocabulary and grammatical features of logical fallacies... Except the nature of fallacies is that they look real on that same linguistic level, so those features aren't distinctive for that purpose. Heck, I think detecting sarcasm would be an easier goal, and still tricky.
    - idle_zealot 2y ago
      > Except the nature of fallacies is that they look real on that same linguistic level, so those features aren't distinctive for that purpose Well that's actually good news. With a large enough labelled dataset of actually-sound and fallacious text with similar grammatical features you should be able to train a discriminator to distinguish between them using some other metric. Good luck with getting that data set though.
      - Terr_ 2y ago
        > you should be able to train a discriminator to distinguish between them using some other metric Not when the better metrics are likely alien/incompatible to the discriminator's core algorithm! Then it's rather inconvenient news, because it means you have to develop something separate and novel. As the other poster already mentioned, if we can't even get them to reliably count how many objects are being referred to, how do you expect them to also handle logical syllogisms?
deleted 2y ago
[deleted]
fsndz 2y ago
The thing is we probably can't build AGI: https://www.lycee.ai/blog/why-no-agi-openai https://www.lycee.ai/blog/why-no-agi-openai
- knowaveragejoe 2y ago
  This is almost a year old, thoughts on it today?
  - fsndz 2y ago
    LLMs still do not reason or plan. And nothing in their architecture, training, post-training points toward real reasoning as scaling continues. Thinking does not happen one token at a time.
    - isaacfung 2y ago
      I don't get why some people seem to think the only way to use a LLM is for next token prediction or AGI has to be bult using LLM alone. You want planning, you can do monte carlo tree search and use LLM to evaluate which node to explore next. You want verifiable reasoning, you can ask it to generate code(an approach used by recent AI olympiad winner and many previous papers). What is even "planning", finding desirable/optimal solutions to some constrained satisfaction problems? Is the llm based minecraft bot voyager not doing some kind of planning? LLMs have their limitations. Then augment them with external data sources, code interpreters, give it ways to interact with real world/simulation environment.
      - threeseed 2y ago
        The problem is that every time you ask the LLM to evaluate what to do next it will return a wrong answer X% of the time. Multiple that X across the number of steps and you have a system that is effectively useless. X today is ~5%. I do think LLMs could be used to assist in building a world model that could be a foundation for an AGI/agent system. But it won't be the major part.
deleted 2y ago
[deleted]
FrameworkFred 2y ago
I'm playing around with LangChain and LangGraph (https://www.langchain.com/ https://www.langchain.com/) and it seems like these enable just the sort of mechanisms mentioned.
trte9343r4 2y ago
> One could spin this idea even further and train several models with radically different world views by curating different training corpi that represent different sets of beliefs / world views. You can get good results by combining different models in chat, or even the same model with different parameters. Model usually gives up on hallucinations when challenged. Sometime it pushes back and provides explanation with sources. I have a script that puts models into dialog, moderates discussion and takes notes. I run this stuff overnight, so getting multiple choices speeds up iteration.
wokwokwok 2y ago
Does anyone really believe that having a good corpus will remove hallucinations? Is this article even written by a person? Hard to know; they have a real blog with real article, but stuff like this reads strangely. Maybe it's just not a native english speaker? > Hallucinations are certainly the toughest nut to crack and their negative impact is basically only slightly lessened by good confidence estimates and reliable citations (sources). > The impact of contradictions in the training data. (was this a prompt header you forget to remove?) > LLM are incapable of "self-inspection" on their training data to find logical inconsistencies in it but in the input context window they should be able to find logical inconsistencies. Annnnyway... Hallucinations cannot be fixed by a good corpus in a non-deterministic (ie. temp > 0) LLM system where you've introduced a random factor. Period. QED. If you think it can, do more reading. The idea that a good corpus can significantly improve the error rate is an open question, but the research I've seen tends to fall on the side of "to some degree, but curating a 'perfect' dataset like that, of a sufficiently large size, is basically impossible'". So, it's a pipe dream. Yes, if you could have a perfect corpus, absolutely, you would get a better model. ...but how do you plan to get that perfect corpus of training data? If it was that easy, the people spending millions and millions of dollars making LLMs would have, I guess, probably come up with a solution for it. They're not stupid. If you could easily do it, it would already have been done. my $0.02: This is a dead end of research, because it's impossible. Using LLMs which are finetuned to evaluate the output of other LLMs and using multi-sample / voting to reduce the incidence of halluciations that make it past the API barrier is both actively used and far, far more effective. (ie. it doesn't matter if your LLM hallucinates 1 time in 10; if you can reliably detect that 1 instance, sample again, and return a non hallucination). Other solutions... I'm skeptical; most of the ones I've seen haven't worked when you actually try to use them.
- thntk 2y ago
  I've seen such articles more and more recently. In the past, when people had a vague idea, they had to do research before writing. During this process, they often realized some flaws and thoroughly revised the idea or gave up writing. Nowadays, research can be bypassed with the help of eloquent LLMs, allowing any vague idea to turn into a write-up.
- 2y ago
fatbird 2y ago
In my mind LLMs are already fatally compromised. Proximity matching via vector embeddings that offer no guarantees of completeness or correctness have already surrendered the essential advantage of technological advances. Imagine a dictionary where the words are only mostly in alphabetical order. If you look up a word and don't find it, you can't be certain it's not in there. It's as useful as asking someone else, or several other people, but it's value as a reference is zero, and there's no shortage of other people on the planet.
- TeMPOraL 2y ago
  > Proximity matching via vector embeddings that offer no guarantees of completeness or correctness have already surrendered the essential advantage of technological advances. On the contrary, it's arguably the breakthrough that allowed us to model concepts and meaning in computers. A sufficiently high-dimensional embedding space can model arbitrary relationships between embedded entities, which allows each of them to be defined in terms of its associations to all the others. This is more-less how we define concepts too, if you dig down into it. > Imagine a dictionary where the words are only mostly in alphabetical order. If you look up a word and don't find it, you can't be certain it's not in there. It's already the case with dictionaries. Dictionaries have mistakes, words out of order; they get outdated, and most importantly, they're descriptive. If a word isn't in it, or isn't defined in particular way, you cannot be certain it doesn't exist or doesn't mean anything other than the dictionary says it does. > It's as useful as asking someone else, or several other people Which is very useful, because it saves you the hassle of dealing with other people. Especially when it's as useful as asking an expert, which saves you the effort of finding one. Now scale that up to being able to ask about whole topics of interest, instead of single words. > it's value as a reference is zero Obviously. So is the value of asking even an expert for an immediate, snap answer, and going with that. > and there's no shortage of other people on the planet Again, dealing with people is stupidly expensive in time, energy and effort, starting with having to find the right people. LLM is just a function call away.
  - fatbird 2y ago
    Technology advances by supplanting human mechanisms, not by amplifying or cheapening them. A loom isn't a more nimble hand, it's a different mechanical approach to weaving. Wheels and roads aren't better legs, they're different conveyances. LLMs as a replacement for dealing with people but offering only the same certainty aren't an advance. LLMs do math by trying to match an answer to a prompt. Mathematica does better than that.
thntk 2y ago
We knew high quality data can help as evidenced by the \Phi models. However, this alone can never eliminate hallucination because data can never be both consistent and complete. Moreover, hallucination is an inherent flaw of intelligence in general if we think of intelligence as (lossy) compression.
jillesvangurp 2y ago
There has been steady improvement since the release of chat gpt into the wild, which is still only less than two years ago (easy to forget). I've been getting a lot of value out of chat gpt 4o, like lots of other people. I find with each model generation my dependence on this stuff for day to day work goes up as the soundness of its answers and reasoning improve. There are still lots of issues and limitations but it's a very different experience than with gpt 3 early on. A lot of the smaller OSS models are a bit of a mixed bag in terms of hallucinations and utility. But they can be useful if you apply some skills. Half the success is actually learning to prompt these things and learning to spot when it starts to hallucinate. One thing I find useful is to run ideas by it in kind of a socratic mode where I try to get it to flesh out brain farts I have for algorithms or other kinds of things. This can be coding related topics but also non technical kinds of things. It will get some things wrong and when you spot it, you can often get a better answer simply by pointing it out and maybe nudging it in a different direction. A useful trick with code is to also let it generate tests for its own code. When the tests fail to run, you can ask it to fix it. Or you can ask it for some alternative implementation of the same thing. Often you get something that is 95% close to what you asked for and then you can just do the remaining few percent yourself. Doing TDD with an LLM is a power move. Good tests are easy enough to understand and once they pass, it's hard to argue with the results. And you can just ask it to identify edge cases and add more tests for those. LLMs take a lot of the tediousness out of writing tests. I'm a big picture kind of guy and my weakness is skipping unit tests to fast forward to having working code. Spelling out all the stupid little assertions is mindnumbingly stupid work that I don't have to bother with anymore. I just let AI generate good test cases. LLMs make TDD a lot less tedious. It's like having a really diligent junior pair programmer doing all the easy bits. And if you apply SOLID principles to your own code (which is a good thing in any case), a lot of code is self contained enough that you can easily fit it in a small file that is small enough to fit into the context window of chat gpt (which is quite large these days). So, a thing I often do is just gather relevant code, copy past it and then tell it to make some reasonable assumptions about missing things and make some modifications to the code. Add a function that does X; how would I need to modify this code to address Y; etc. I also get it to iterate on its own code. And a neat trick is to ask it to compare its solution to other solutions out there and then get it to apply some of the same principles and optimizations. One thing with RAG is that we're still under utilizing LLMs for this. It's a lot easier to get an LLM to ask good questions than it is to get them to provide the right answers. With RAG, you can use good old information retrieval to answer the questions. IMHO limiting RAG to just vector search is a big mistake. It actually doesn't work that well for structured data and you could just ask it to query some API based on a specification of use some sql, xpath, or whatever query language. And why just ask 1 question? Maybe engage in a dialog where it zooms in on the solution via querying and iteratively coming up with better questions until the context has all the data needed to come up with the answer. If you think about it, this is how most knowledge workers address problems themselves. They are not oracles of wisdom that know everything but merely aggregators and filters of external knowledge. A good knowledge worker / researcher / engineer is one that knows how to ask the right questions in order to come up with an iterative process that converges on a solution. Once you stop using LLMs as one shot oracles that give you an answer given a question, they become a lot more useful. As for AGI, a human AI enhanced by AGI is a powerful combination. I kind of like the vision behind neuralink where the core idea is basically improving the bandwidth between our brains and external tools and intelligence. Using a chat bot is a low bandwidth kind of thing. I actually find it tedious.
luke-stanley 2y ago
As I understand it: the Phi models, are trained with a much more selective training data, the Tiny Stories research was one of the starts of that, they used GPT-4 to make stories and encyclopedia like training data for Phi to learn from and code, which probably helps with logical structuring too. I think they did add in real web data too though but I think it was fairly selective. Maybe something between Cyc and Google's math and geometry LLM's could help.
lsy 2y ago
We can't develop a universally coherent data set because what we understand as "truth" is so intensely contextual that we can't hope to cover the amount of context needed to make the things work how we want, not to mention the numerous social situations where writing factual statements would be awkward or disastrous. Here are a few examples of statements that are not "factual" in the sense of being derivable from a universally coherent data set, and that nevertheless we would expect a useful intelligence to be able to generate: "There is a region called Hobbiton where someone named Frodo Baggins lives." "We'd like to announce that Mr. Ousted is transitioning from his role as CEO to an advisory position while he looks for a new challenge. We are grateful to Mr. Ousted for his contributions and will be sad to see him go." "The earth is round." "Nebraska is flat."
- smokel 2y ago
  > We can't develop a universally coherent data set because Yet every child seems to manage, when raised by a small village, over a period of about 18 years. I guess we just need to give these LLMs a little more love and attention.
  - bugglebeetle 2y ago
    Or maybe hundreds of millions of years of evolutionary pressure to build unbelievably efficient function approximation.
  - antisthenes 2y ago
    And then you go out into the real world, talk to real adults, and discover that the majority of people don't have a coherent mental model of the world, and have completely ridiculous ideas that aren't anywhere near an approximation of the real physical world.
    - mistermann 2y ago
      > and discover that the majority of people don't have a coherent mental model of the world "Coherent" is doing a lot of lifting here. All humans have highly flawed models, and we've been culturally conditioned to grade on a curve to hide the problem from ourselves.
RamblingCTO 2y ago
My biggest problem with them is that I can't quite get it to behave like I want it to. I built myself a "therapy/coaching" telegram bot (I'm healthy, but like to reflect a lot, no worries). I even built a self-reflecting memory component that generates insights (sometimes spot on, sometimes random af). But the more I use it, the more I notice that neither the memory nor the prompt matters much. I just can't get it to behave like a therapist would. So in other words: I can't find the inputs to achieve a desirable prediction from the SOTA LLMs. And I think that's a bigger problem for them not to be a shallow hype.
- coldtea 2y ago
  >I just can't get it to behave like a therapist would import time import random SESSION_DURATION = 50 * 60 start_time = time.time() while True: current_time = time.time() elapsed_time = current_time - start_time if elapsed_time >= SESSION_DURATION: print("Our time is up. That will be $150. See you next week!") break _ = input("") print(random.choice(["Mmm hmm", "Tell me more", "How does that make you feel?"])) time.sleep(1) Thank me later!
  - RamblingCTO 2y ago
    haha, good one! although I'm German and it was free for me when I did it. I just had the best therapist. $150 a session is insane!
DolphinAsa 2y ago
I'm surprised he didn't mention the way, that we are solving the issue at amazon. It's not an secret at this point, giving the LLM's hands or agentic systems to run code or do things that get feedback in a loop DRAMATICALLY REDUCE Hallucinations.
nyrikki 2y ago
> ...manually curate a high-quality (consistent) text corpus based on undisputed, well curated wikipedia articles and battle tested scientific literature. This assumption is based on the mistaken assumption that science is about objective truth. It is confusing the map for the territory. Scientific models are intended to be useful, not perfect. Statistical learning, vs symbolic learning is about existential quantification vs universal quantification respectively. All models are wrong some are useful, this applies to even the most unreasonably accurate versions like QFT and GR. Spherical cows, no matter how useful are hotly debated outside of the didactic half truths of low level courses. The corpus that the above seeks doesn't exist in academic circles, only in popular science where people don't see that practical, useful models are far more important that 'correct' ones.
mitthrowaway2 2y ago
LLMs don't only hallucinate because of mistaken statements in their training data. It just comes hand-in-hand with the model's ability to remix, interpolate, and extrapolate answers to other questions that aren't directly answered in the dataset. For example if I ask ChatGPT a legal question, it might cite as precedent a case that doesn't exist at all (but which seems plausible, being interpolated from cases that do exist). It's not necessarily because it drew that case from a TV episode. It works the same way that GPT-3 wrote news releases that sounded convincing, matching the structure and flow of real articles. Training only on factual data won't solve this. Anyway, I can't help but feel saddened sometimes to see our talented people and investment resources being drawn in to developing these AI chatbots. These problems are solvable, but are we really making a better world by solving them?
- CamperBob2 2y ago
  These problems are solvable, but are we really making a better world by solving them? When you ask yourself that question -- and you do ask yourself that, right? -- what's your answer?
  - mitthrowaway2 2y ago
    I do, all the time! My answer is "most likely not". (I assumed that answer was implied by my expressing sadness about all the work being invested in them.) This is why, although I try to keep up-to-date with and understand these technologies, I am not being paid to develop them.
    - bosch_mind 2y ago
      I mean, comon man, _most_ tech people work on in general is also useless. FAANG has a retainer on talent whether they are spending their time wisely or not. LLM stuff is not in a different echelon
- dweinus 2y ago
  100% I think the author is really misunderstanding the issue here. "Hallucination" is a fundamental aspect of the design of Large Language Models. Narrowing the distribution of the training data will reduce the LLM's ability to generalize, but it won't stop hallucinations.
Animats 2y ago
Plausible idea which needs a big training budget. Was it funded?
MR4D 2y ago
Q: is hallucination a milestone towards consciousness? Given how inevitable it is, it seems to me that it might be.
xarope 2y ago
I do feel like we've reached a local maxima with the current state of LLMs, and researchers need to find something completely different to hit a new maxima (whether that is the global maxima or not, we'll know when we hail our new AI overlords).
simplysparsh 2y ago
I came here thinking I will learn how to make LLMs better. But leaving with more complicated questions: 1. Do I want LLMs to be trained with licensed data, that's arguably well curated. Or, do I want LLM to scrape the web because it is more democratic in opinions? 2. If hallucination is not about training data but how LLM uses that data to extrapolate info that's not directly present in training data - can we teach it this skill to make better choices? 3. It's easy to define good data for facts. How to define good data for subjective topics? 4. For subjective topics, is it better to have separate LLMs trained with each theme of opinions or one big LLM with a mix of all opinions? 5. Is using LLM to improve its own training data truly helpful as the author claims? If yes - is this recursion method better or it's better to use multiple LLMs together? Dang! If I interview for a position that requires knowledge of AI - every question they ask will be answered with more questions. smh!

119 points by sean_pedersen 2y ago

I wish he went into how to improve confidence scores, though I guess training on better data to begin with should improve results and thus confidence.

RodgerTheGreat 2y ago

One of the main factors that makes LLMs popular today is that scaling up the models is a simple and (relatively) inexpensive matter of buying compute capacity and scraping together more raw text to train them. Without large and highly diverse training datasets to construct base models, LLMs cannot produce even the superficial appearance of good results. Manually curating "tidy", properly-licensed and verified datasets is immensely more difficult, expensive, and time-consuming than stealing whatever you can find on the open internet. Wolfram Alpha is one of the more successful attempts in that curation-based direction (using good-old-fashioned heuristic techniques instead of opaque ML models), and while it is very useful and contains a great deal of factual information, it does not conjure appealing fantasies of magical capabilities springing up from thin air and hands-off exponential improvement.

totetsu 2y ago
It’s not unethical if people in positions of privilege and power do it to maintain their rightful position of privilege and power.
- dang 2y ago
  Please don't post in the flamewar style to HN. It degrades discussion and we're trying to go in the opposite direction here, to the extent that is possible on the internet. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
threeseed 2y ago
> properly-licensed and verified datasets is immensely more difficult, expensive Arguably the bigger problem is that many of those datasets e.g. WSJ articles are proprietary and can be exclusively licensed like we've seen recently with OpenAI. So we end up with in a situation where competition is simply not possible.
- piva00 2y ago
  > Arguably the bigger problem is that many of those datasets e.g. WSJ articles are proprietary and can be exclusively licensed like we've seen recently with OpenAI. > So we end up with in a situation where competition is simply not possible. Exactly, and Technofeudalism advances a little more into a new feud. OpenAI is trying to create its moat by shoring up training data, probably attempting to not allow competitors to train on the same datasets they've been licencing, at least for a while. Training data is the only possible moat for LLMs, models seem to be advancing quite well between different companies but as mentioned here a tidy training dataset is the actual gold.
  - gessha 2y ago
    It's landmines no matter how you approach the problem. If you treat the web as a free-for-all and you scrape freely, you get sued by the content platforms for copyright or term of service violation. If you license the content, you let the highest bidder get the content. No matter what happens, capital wins.
- motohagiography 2y ago
  the irony is that if large media providers aren't represented in the training sets, my comments on internet forums over the decades will be over-represented, which is kind of great, really.
  - pennomi 2y ago
    Right? I always ask people - Hypothetically if someone created a superintelligent AI that took over the world, wouldn’t you WANT it to share your opinions and morals? Every tiny bit of text you write is a vote in the election of our future AI overlords.

darby_nine 2y ago

Man it seems like the ship has sailed on "hallucination" but it's such a terrible name for the phenomenon we see. It is a major mistake to imply the issue is with perception rather than structural incompetence. Why not just say "incoherent output"? It's actually descriptive and doesn't require bastardizing a word we already find meaningful to mean something completely different.

jwuphysics 2y ago
> Why not just say "incoherent output"? Because the biggest problem with hallucinations is that the output is usually coherent but factually incorrect. I agree that "hallucination" isn't the best word for it... perhaps something like "confabulation" is better.
- linguistbreaker 2y ago
  I appreciated a post on here recently that likened AI hallucination to 'bullshitting'. It's coherent, even plausible output without any regard for the truth.
  - wkat4242 2y ago
    While I have absolutely no issues with the word "shit" in popular terms, I'd normally like to reserve it for situations where there's actually intended malice like in "enshittification". Rather than just an imperfect technology as we have here. Many people object to the term enshittification for foul-mouthing reasons but I think it covers it very well because the principle it covers is itself so very nasty. But that's not at all the case here.
    - pessimizer 2y ago
      "Bullshitting" isn't a new piece of jargon, it's a common English word of many decades vintage, and is being used in its dictionary sense here.
  - MattPalmer1086 2y ago

ainoobler 2y ago

The article suggests a useful line of research. Train an LLM to detect logical fallacies and then see if that can be bootstrapped into something useful because it's pretty clear that all the issues with LLMs is the lack of logical capabilities. If an LLM was capable of logical reasoning then it would be obvious when it was generating made-up nonsense instead of referencing existing sources of consistent information.

dtx1 2y ago
I think we should start smaller and make them able to count first.
- Terr_ 2y ago
  Yeah, you can train an LLM to recognize the vocabulary and grammatical features of logical fallacies... Except the nature of fallacies is that they look real on that same linguistic level, so those features aren't distinctive for that purpose. Heck, I think detecting sarcasm would be an easier goal, and still tricky.
  - idle_zealot 2y ago
    > Except the nature of fallacies is that they look real on that same linguistic level, so those features aren't distinctive for that purpose Well that's actually good news. With a large enough labelled dataset of actually-sound and fallacious text with similar grammatical features you should be able to train a discriminator to distinguish between them using some other metric. Good luck with getting that data set though.
    - Terr_ 2y ago
      > you should be able to train a discriminator to distinguish between them using some other metric Not when the better metrics are likely alien/incompatible to the discriminator's core algorithm! Then it's rather inconvenient news, because it means you have to develop something separate and novel. As the other poster already mentioned, if we can't even get them to reliably count how many objects are being referred to, how do you expect them to also handle logical syllogisms?

deleted 2y ago

[deleted]

fsndz 2y ago

The thing is we probably can't build AGI: https://www.lycee.ai/blog/why-no-agi-openai https://www.lycee.ai/blog/why-no-agi-openai

knowaveragejoe 2y ago
This is almost a year old, thoughts on it today?
- fsndz 2y ago
  LLMs still do not reason or plan. And nothing in their architecture, training, post-training points toward real reasoning as scaling continues. Thinking does not happen one token at a time.
  - isaacfung 2y ago
    I don't get why some people seem to think the only way to use a LLM is for next token prediction or AGI has to be bult using LLM alone. You want planning, you can do monte carlo tree search and use LLM to evaluate which node to explore next. You want verifiable reasoning, you can ask it to generate code(an approach used by recent AI olympiad winner and many previous papers). What is even "planning", finding desirable/optimal solutions to some constrained satisfaction problems? Is the llm based minecraft bot voyager not doing some kind of planning? LLMs have their limitations. Then augment them with external data sources, code interpreters, give it ways to interact with real world/simulation environment.
    - threeseed 2y ago
      The problem is that every time you ask the LLM to evaluate what to do next it will return a wrong answer X% of the time. Multiple that X across the number of steps and you have a system that is effectively useless. X today is ~5%. I do think LLMs could be used to assist in building a world model that could be a foundation for an AGI/agent system. But it won't be the major part.

deleted 2y ago

[deleted]

FrameworkFred 2y ago

I'm playing around with LangChain and LangGraph (https://www.langchain.com/ https://www.langchain.com/) and it seems like these enable just the sort of mechanisms mentioned.

trte9343r4 2y ago

> One could spin this idea even further and train several models with radically different world views by curating different training corpi that represent different sets of beliefs / world views. You can get good results by combining different models in chat, or even the same model with different parameters. Model usually gives up on hallucinations when challenged. Sometime it pushes back and provides explanation with sources. I have a script that puts models into dialog, moderates discussion and takes notes. I run this stuff overnight, so getting multiple choices speeds up iteration.

wokwokwok 2y ago

Does anyone really believe that having a good corpus will remove hallucinations? Is this article even written by a person? Hard to know; they have a real blog with real article, but stuff like this reads strangely. Maybe it's just not a native english speaker? > Hallucinations are certainly the toughest nut to crack and their negative impact is basically only slightly lessened by good confidence estimates and reliable citations (sources). > The impact of contradictions in the training data. (was this a prompt header you forget to remove?) > LLM are incapable of "self-inspection" on their training data to find logical inconsistencies in it but in the input context window they should be able to find logical inconsistencies. Annnnyway... Hallucinations cannot be fixed by a good corpus in a non-deterministic (ie. temp > 0) LLM system where you've introduced a random factor. Period. QED. If you think it can, do more reading. The idea that a good corpus can significantly improve the error rate is an open question, but the research I've seen tends to fall on the side of "to some degree, but curating a 'perfect' dataset like that, of a sufficiently large size, is basically impossible'". So, it's a pipe dream. Yes, if you could have a perfect corpus, absolutely, you would get a better model. ...but how do you plan to get that perfect corpus of training data? If it was that easy, the people spending millions and millions of dollars making LLMs would have, I guess, probably come up with a solution for it. They're not stupid. If you could easily do it, it would already have been done. my $0.02: This is a dead end of research, because it's impossible. Using LLMs which are finetuned to evaluate the output of other LLMs and using multi-sample / voting to reduce the incidence of halluciations that make it past the API barrier is both actively used and far, far more effective. (ie. it doesn't matter if your LLM hallucinates 1 time in 10; if you can reliably detect that 1 instance, sample again, and return a non hallucination). Other solutions... I'm skeptical; most of the ones I've seen haven't worked when you actually try to use them.

thntk 2y ago
I've seen such articles more and more recently. In the past, when people had a vague idea, they had to do research before writing. During this process, they often realized some flaws and thoroughly revised the idea or gave up writing. Nowadays, research can be bypassed with the help of eloquent LLMs, allowing any vague idea to turn into a write-up.
2y ago

fatbird 2y ago

In my mind LLMs are already fatally compromised. Proximity matching via vector embeddings that offer no guarantees of completeness or correctness have already surrendered the essential advantage of technological advances. Imagine a dictionary where the words are only mostly in alphabetical order. If you look up a word and don't find it, you can't be certain it's not in there. It's as useful as asking someone else, or several other people, but it's value as a reference is zero, and there's no shortage of other people on the planet.

TeMPOraL 2y ago
> Proximity matching via vector embeddings that offer no guarantees of completeness or correctness have already surrendered the essential advantage of technological advances. On the contrary, it's arguably the breakthrough that allowed us to model concepts and meaning in computers. A sufficiently high-dimensional embedding space can model arbitrary relationships between embedded entities, which allows each of them to be defined in terms of its associations to all the others. This is more-less how we define concepts too, if you dig down into it. > Imagine a dictionary where the words are only mostly in alphabetical order. If you look up a word and don't find it, you can't be certain it's not in there. It's already the case with dictionaries. Dictionaries have mistakes, words out of order; they get outdated, and most importantly, they're descriptive. If a word isn't in it, or isn't defined in particular way, you cannot be certain it doesn't exist or doesn't mean anything other than the dictionary says it does. > It's as useful as asking someone else, or several other people Which is very useful, because it saves you the hassle of dealing with other people. Especially when it's as useful as asking an expert, which saves you the effort of finding one. Now scale that up to being able to ask about whole topics of interest, instead of single words. > it's value as a reference is zero Obviously. So is the value of asking even an expert for an immediate, snap answer, and going with that. > and there's no shortage of other people on the planet Again, dealing with people is stupidly expensive in time, energy and effort, starting with having to find the right people. LLM is just a function call away.
- fatbird 2y ago
  Technology advances by supplanting human mechanisms, not by amplifying or cheapening them. A loom isn't a more nimble hand, it's a different mechanical approach to weaving. Wheels and roads aren't better legs, they're different conveyances. LLMs as a replacement for dealing with people but offering only the same certainty aren't an advance. LLMs do math by trying to match an answer to a prompt. Mathematica does better than that.

thntk 2y ago

We knew high quality data can help as evidenced by the \Phi models. However, this alone can never eliminate hallucination because data can never be both consistent and complete. Moreover, hallucination is an inherent flaw of intelligence in general if we think of intelligence as (lossy) compression.

jillesvangurp 2y ago

There has been steady improvement since the release of chat gpt into the wild, which is still only less than two years ago (easy to forget). I've been getting a lot of value out of chat gpt 4o, like lots of other people. I find with each model generation my dependence on this stuff for day to day work goes up as the soundness of its answers and reasoning improve. There are still lots of issues and limitations but it's a very different experience than with gpt 3 early on. A lot of the smaller OSS models are a bit of a mixed bag in terms of hallucinations and utility. But they can be useful if you apply some skills. Half the success is actually learning to prompt these things and learning to spot when it starts to hallucinate. One thing I find useful is to run ideas by it in kind of a socratic mode where I try to get it to flesh out brain farts I have for algorithms or other kinds of things. This can be coding related topics but also non technical kinds of things. It will get some things wrong and when you spot it, you can often get a better answer simply by pointing it out and maybe nudging it in a different direction. A useful trick with code is to also let it generate tests for its own code. When the tests fail to run, you can ask it to fix it. Or you can ask it for some alternative implementation of the same thing. Often you get something that is 95% close to what you asked for and then you can just do the remaining few percent yourself. Doing TDD with an LLM is a power move. Good tests are easy enough to understand and once they pass, it's hard to argue with the results. And you can just ask it to identify edge cases and add more tests for those. LLMs take a lot of the tediousness out of writing tests. I'm a big picture kind of guy and my weakness is skipping unit tests to fast forward to having working code. Spelling out all the stupid little assertions is mindnumbingly stupid work that I don't have to bother with anymore. I just let AI generate good test cases. LLMs make TDD a lot less tedious. It's like having a really diligent junior pair programmer doing all the easy bits. And if you apply SOLID principles to your own code (which is a good thing in any case), a lot of code is self contained enough that you can easily fit it in a small file that is small enough to fit into the context window of chat gpt (which is quite large these days). So, a thing I often do is just gather relevant code, copy past it and then tell it to make some reasonable assumptions about missing things and make some modifications to the code. Add a function that does X; how would I need to modify this code to address Y; etc. I also get it to iterate on its own code. And a neat trick is to ask it to compare its solution to other solutions out there and then get it to apply some of the same principles and optimizations. One thing with RAG is that we're still under utilizing LLMs for this. It's a lot easier to get an LLM to ask good questions than it is to get them to provide the right answers. With RAG, you can use good old information retrieval to answer the questions. IMHO limiting RAG to just vector search is a big mistake. It actually doesn't work that well for structured data and you could just ask it to query some API based on a specification of use some sql, xpath, or whatever query language. And why just ask 1 question? Maybe engage in a dialog where it zooms in on the solution via querying and iteratively coming up with better questions until the context has all the data needed to come up with the answer. If you think about it, this is how most knowledge workers address problems themselves. They are not oracles of wisdom that know everything but merely aggregators and filters of external knowledge. A good knowledge worker / researcher / engineer is one that knows how to ask the right questions in order to come up with an iterative process that converges on a solution. Once you stop using LLMs as one shot oracles that give you an answer given a question, they become a lot more useful. As for AGI, a human AI enhanced by AGI is a powerful combination. I kind of like the vision behind neuralink where the core idea is basically improving the bandwidth between our brains and external tools and intelligence. Using a chat bot is a low bandwidth kind of thing. I actually find it tedious.

luke-stanley 2y ago

As I understand it: the Phi models, are trained with a much more selective training data, the Tiny Stories research was one of the starts of that, they used GPT-4 to make stories and encyclopedia like training data for Phi to learn from and code, which probably helps with logical structuring too. I think they did add in real web data too though but I think it was fairly selective. Maybe something between Cyc and Google's math and geometry LLM's could help.

lsy 2y ago

We can't develop a universally coherent data set because what we understand as "truth" is so intensely contextual that we can't hope to cover the amount of context needed to make the things work how we want, not to mention the numerous social situations where writing factual statements would be awkward or disastrous. Here are a few examples of statements that are not "factual" in the sense of being derivable from a universally coherent data set, and that nevertheless we would expect a useful intelligence to be able to generate: "There is a region called Hobbiton where someone named Frodo Baggins lives." "We'd like to announce that Mr. Ousted is transitioning from his role as CEO to an advisory position while he looks for a new challenge. We are grateful to Mr. Ousted for his contributions and will be sad to see him go." "The earth is round." "Nebraska is flat."

smokel 2y ago
> We can't develop a universally coherent data set because Yet every child seems to manage, when raised by a small village, over a period of about 18 years. I guess we just need to give these LLMs a little more love and attention.
- bugglebeetle 2y ago
  Or maybe hundreds of millions of years of evolutionary pressure to build unbelievably efficient function approximation.
- antisthenes 2y ago
  And then you go out into the real world, talk to real adults, and discover that the majority of people don't have a coherent mental model of the world, and have completely ridiculous ideas that aren't anywhere near an approximation of the real physical world.
  - mistermann 2y ago
    > and discover that the majority of people don't have a coherent mental model of the world "Coherent" is doing a lot of lifting here. All humans have highly flawed models, and we've been culturally conditioned to grade on a curve to hide the problem from ourselves.

RamblingCTO 2y ago

My biggest problem with them is that I can't quite get it to behave like I want it to. I built myself a "therapy/coaching" telegram bot (I'm healthy, but like to reflect a lot, no worries). I even built a self-reflecting memory component that generates insights (sometimes spot on, sometimes random af). But the more I use it, the more I notice that neither the memory nor the prompt matters much. I just can't get it to behave like a therapist would. So in other words: I can't find the inputs to achieve a desirable prediction from the SOTA LLMs. And I think that's a bigger problem for them not to be a shallow hype.

coldtea 2y ago
>I just can't get it to behave like a therapist would import time import random SESSION_DURATION = 50 * 60 start_time = time.time() while True: current_time = time.time() elapsed_time = current_time - start_time if elapsed_time >= SESSION_DURATION: print("Our time is up. That will be $150. See you next week!") break _ = input("") print(random.choice(["Mmm hmm", "Tell me more", "How does that make you feel?"])) time.sleep(1) Thank me later!
- RamblingCTO 2y ago
  haha, good one! although I'm German and it was free for me when I did it. I just had the best therapist. $150 a session is insane!

DolphinAsa 2y ago

I'm surprised he didn't mention the way, that we are solving the issue at amazon. It's not an secret at this point, giving the LLM's hands or agentic systems to run code or do things that get feedback in a loop DRAMATICALLY REDUCE Hallucinations.

nyrikki 2y ago

> ...manually curate a high-quality (consistent) text corpus based on undisputed, well curated wikipedia articles and battle tested scientific literature. This assumption is based on the mistaken assumption that science is about objective truth. It is confusing the map for the territory. Scientific models are intended to be useful, not perfect. Statistical learning, vs symbolic learning is about existential quantification vs universal quantification respectively. All models are wrong some are useful, this applies to even the most unreasonably accurate versions like QFT and GR. Spherical cows, no matter how useful are hotly debated outside of the didactic half truths of low level courses. The corpus that the above seeks doesn't exist in academic circles, only in popular science where people don't see that practical, useful models are far more important that 'correct' ones.

mitthrowaway2 2y ago

LLMs don't only hallucinate because of mistaken statements in their training data. It just comes hand-in-hand with the model's ability to remix, interpolate, and extrapolate answers to other questions that aren't directly answered in the dataset. For example if I ask ChatGPT a legal question, it might cite as precedent a case that doesn't exist at all (but which seems plausible, being interpolated from cases that do exist). It's not necessarily because it drew that case from a TV episode. It works the same way that GPT-3 wrote news releases that sounded convincing, matching the structure and flow of real articles. Training only on factual data won't solve this. Anyway, I can't help but feel saddened sometimes to see our talented people and investment resources being drawn in to developing these AI chatbots. These problems are solvable, but are we really making a better world by solving them?

CamperBob2 2y ago
These problems are solvable, but are we really making a better world by solving them? When you ask yourself that question -- and you do ask yourself that, right? -- what's your answer?
- mitthrowaway2 2y ago
  I do, all the time! My answer is "most likely not". (I assumed that answer was implied by my expressing sadness about all the work being invested in them.) This is why, although I try to keep up-to-date with and understand these technologies, I am not being paid to develop them.
  - bosch_mind 2y ago
    I mean, comon man, _most_ tech people work on in general is also useless. FAANG has a retainer on talent whether they are spending their time wisely or not. LLM stuff is not in a different echelon
dweinus 2y ago
100% I think the author is really misunderstanding the issue here. "Hallucination" is a fundamental aspect of the design of Large Language Models. Narrowing the distribution of the training data will reduce the LLM's ability to generalize, but it won't stop hallucinations.

Animats 2y ago

Plausible idea which needs a big training budget. Was it funded?

MR4D 2y ago

Q: is hallucination a milestone towards consciousness? Given how inevitable it is, it seems to me that it might be.

xarope 2y ago

I do feel like we've reached a local maxima with the current state of LLMs, and researchers need to find something completely different to hit a new maxima (whether that is the global maxima or not, we'll know when we hail our new AI overlords).

simplysparsh 2y ago

I came here thinking I will learn how to make LLMs better. But leaving with more complicated questions: 1. Do I want LLMs to be trained with licensed data, that's arguably well curated. Or, do I want LLM to scrape the web because it is more democratic in opinions? 2. If hallucination is not about training data but how LLM uses that data to extrapolate info that's not directly present in training data - can we teach it this skill to make better choices? 3. It's easy to define good data for facts. How to define good data for subjective topics? 4. For subjective topics, is it better to have separate LLMs trained with each theme of opinions or one big LLM with a mix of all opinions? 5. Is using LLM to improve its own training data truly helpful as the author claims? If yes - is this recursion method better or it's better to use multiple LLMs together? Dang! If I interview for a position that requires knowledge of AI - every question they ask will be answered with more questions. smh!