6 ms·
I'm curious -- does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar? We all know now that ChatGPT is ju
by crazygringo 4y ago
I'm curious -- does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar?
We all know now that ChatGPT is just autocomplete on steroids. It produces plausibly convincing patterns of speech.
But from the way it's built and trained, it's not like there's even any kind of factual confidence level you could threshold, or anything. The concept of factuality doesn't exist in the model at all.
So, is any progress being made towards internet-scale ML "fact engines" that also have the flexibility and linguistic expressiveness of ChatGPT? Or are these just two totally different paths that nobody knows how to marry?
Because I know there's plenty of work done with knowledge graphs et al., but those are very brittle things that generally need plenty of human curation and verification, and can't provide any of the (good) "fuzzy thinking" that ChatGPT can. They can't summarize essays or write poems.
- snowstormsun 4y agoI think "Explainable AI" is a related research direction, but perhaps not popular for language models.
- behnamoh 4y agoImpossible to explain the inner workings of GPT-3 without having access to the model and its weights. Does anyone know if any methods exist for this?
- IncRnd 4y agoI asked ChatGPT for some in-depth source code that realistically mimics chatgpt. ChatGPT replied with various answers in python. I'm not sure any of them are correct, though.
- PeterisP 4y agoSince it's impossible to run inference on the model without having access to the model and its weights, interpretable AI generally does assume that you have access to all of that. Otherwise, why you would want to try to explain the inner workings of something that you don't have and can't use?
- shawntan 4y agoI think part of the issue is what level of explanation is satisfactory. We can explain how every linear transformation computes its output, but the sum of it is in many ways more than its parts. Then there are efforts that look like this one: https://news.ycombinator.com/item?id=34821414 https://news.ycombinator.com/item?id=34821414 They go probing for specific capabilities of Transformers to figure out which cell fires under some specific stimulus. But think a little bit more about what people might want from explainability and you quickly find that something like this is insufficient. There may be a tradeoff we're looking at where explainability (for some definition of it) will have to be exchanged for performance (under some set of tasks). You can build more interpretable models these days, but you usually pay for it in terms of how well you do on benchmarks.
- simonw 4y agoThat's basically what the new Bing is. It's a large language model that can run searches, and then use what comes back from those searches to generate answers to questions. Whether or not the information that comes back from those searches is reliable is a whole other question. I would love to learn what the latest research is into "factual correctness" detection. Presumably there are teams out there trying to solve that one?
- behnamoh 4y agoAFAIK, Bing AI is not itself an LLM, but rather a wrapper around ChatGPT, which itself is based on GPT-3, which is based on the GPT architecture, which is (roughly speaking) half of a transformer architecture, which is based on encoder/decoder neural nets which are based on ...
- nl 4y agoIt's a newer, different GPT model than chatGPT.
- simonw 4y agoTo quote the Bing announcement post: https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/ https://blogs.microsoft.com/blog/2023/02/07/reinventing-sear... > Next-generation OpenAI model. We’re excited to announce the new Bing is running on a new, next-generation OpenAI large language model that is more powerful than ChatGPT and customized specifically for search. It takes key learnings and advancements from ChatGPT and GPT-3.5 – and it is even faster, more accurate and more capable.
- nerdponx 4y agoChatGPT and You.com chat both claim to be able to provide references, but usually the URLs they provide are for completely unrelated topics, even if they are on convincing-looking domains (e.g. Arxiv or Sciencedirect, but completely unrelated random-seeming papers).
- mochomocha 4y ago> But from the way it's built and trained, it's not like there's even any kind of factual confidence level you could threshold, or anything. The concept of factuality doesn't exist in the model at all. I'm not super familiar with ChatGPT internals, but there are plenty of ways to tack on uncertainty estimates to predictions of typical "large scale ML models" without touching Bayesian stuff (which only work for small scale academics problems). You can do simple parametric posteriors estimation or if all you have is infinite compute and don't even want to bother with anything "mathy", bootstrapping is the "scalable / easy" solution.
- pavon 4y agoSure, but would that uncertainty estimate measure the accuracy of the data or the accuracy of it being a reasonably sounding sentence.
- alfalfasprout 4y agoThe latter. Which is why it wouldn't work.
- ericlewis 4y agoits super duper easy, prob not perfect and I don't have any sort of proper "test": 1. I ask the model first if it seems like a question that benefits from an external answer 2. I talk to Wolfram alpha with some abstraction of the question 3. I wait for a response 4. I "incept" it into the final response, essentially a prompt that mixes in a context of sorts that contains the factual information. you could cross check this stuff too with yet more models.
- csours 4y agoI'm curious about falsifiable models.
- irrational 4y agoRemember the guy a few weeks ago that was being gaslighted by ChatGPT that this is the year 2022? Not only is it giving out potentially false info, but it will double down that it is right and you are wrong. Though, to be honest, that sounds like a lot of real people. The difference is, people are smart enough to not double down on try to say it is a different year and your phone is probably reporting the year wrong.
- nl 4y ago> does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar? Yes. It's a very active area of research. For example: Discovering Latent Knowledge in Language Models Without Supervision (https://arxiv.org/abs/2212.03827 https://arxiv.org/abs/2212.03827) shows an unsupervised approach for probing a LLM to discover things it thinks are facts Locating and Editing Factual Associations in GPT (https://arxiv.org/pdf/2202.05262.pdf https://arxiv.org/pdf/2202.05262.pdf) shows an approach to editing a LLM to edit facts. Language Models as Knowledge Bases? (https://aclanthology.org/D19-1250.pdf https://aclanthology.org/D19-1250.pdf) is some slightly older work exploring how well LLMs store factual information itself.
- crazygringo 4y agoThank you so much! Those are exactly the types of links I'm curious about.
- supriyo-biswas 4y agoReplying to this comment to find it later. (Is there a good way to bookmark comments on HN?)
- nl 4y agoYou can click the date of the comment then "favorite" it.
- alfalfasprout 4y agoBy definition, an LLM doesn't have a semantic world model or ontology. Even the most "dumb" (and I use that in quotes because they really aren't) animal is able to reason about uncertain concepts and understands risk and uncertainty. Yann Lecun has posted a lot recently about this but basically LLMs are a "useful offramp on the road to AGI".
- cypress66 4y agoThere's nothing "by definition" that says so. In fact many propose that when you train an LLM, in order to be able to predict the next word with enough accuracy, it must internally build a world model. Yann Lecun is very salty about chatgpt, I wouldn't take his word seriously.
- alfalfasprout 4y agoLet me clarify, autoregressive LLMs build a probabilistic mapping between words and tokens. They don't actually understand what these concepts mean. Only what they appear in conjunction with, etc. We (and most animals) interact with the physical world and learn through a combination of doing, experiencing, biology, and book learning. That lets us reason about how things work in unseen contexts and we know what we know vs. don't know (whether we express it or not is a different story). > Yann Lecun is very salty about chatgpt, I wouldn't take his word seriously. With all due respect, he's not salty at all. He's even overseen plenty of cutting edge research in the LLM space. But he rightfully has pointed out what they can and can't do. There's too many people encountering a chatbot for the first time that sounds coherent and engaging in anthropomorphism.
- d110af5ccf 4y ago> They don't actually understand what these concepts mean. You say this so confidently. But can you define in terms that are directly quantifiable what "understanding a concept" actually means? I don't believe that anyone can (at present, anyway) although there are certainly some interesting theories and heuristics that have been put forward by various people.
- BoorishBears 4y agoThere's research being done on this: https://arxiv.org/abs/2302.04761 https://arxiv.org/abs/2302.04761 At its core using an LM alone to solve factual problems seems silly: It's not unlike asking Dall-E to draw DOT compliant road signs. I've gone at length at how unfortunate it would be if LMs start to get a bad rap because they're being shoehorned into being "Ask Jeeves 2.0" when they could be so much more.
- crazygringo 4y ago> It's not unlike asking Dall-E to draw DOT compliant road signs. I love that. That's going to be my new explanation for people around ChatGPT. For some reason it seems so much more obvious when Dall-E does something close but still totally wrong (e.g. 3 or 6 fingers, 3 arms, etc.), but it's not immediately obvious with text. But it's still the same underlying principles.
- renewiltord 4y agoYou don't have to use ChatGPT. There are other styles of AIs that use LLMs like https://www.perplexity.ai/ https://www.perplexity.ai/ Personally, I use ChatGPT (the paid version) and Copilot every day and find them awesome enhancers.
- opisthenar84 4y agoI imagine OpenAI is probably collecting a massive dataset of "false" responses (from the general public's use of ChatGPT and Bing) and fine-tuning GPT-3.5 with it. The rich keep getting richer.
- soerxpso 4y agoYou're describing a problem as old as academia, on which very little progress has ever been made. Before "add a factual confidence level evaluator to a bot that doesn't understand the concept of fact" you must first figure out how to calculate a "factual confidence level" at all, in general.
- visarga 4y agoThere was a model that could set up a simulation to estimate the answer for you. So it won't use pure language, but it only works in a few cases.
- wwwpatdelcom 4y agoDeepmind is an LLM with a fact verifier attached, though the fact verifier is actually a ranked list of code compile times. Obviously this is a narrow subset of specific problems, but one could expand that library of problems over time.
- qualudeheart 4y agoAre you talking about Alphacode? That seems like the only Deepmind project similar to what you describe.
- mrtnmcc 4y agoGiving LLM the ability to query other services like Google should solve much of this. For example ChatGPT can be initialized to be told it can output commands like "QUERY_GOOGLE:What is the current time?" and get Google's response, which it can incorporate. You can actually do this yourself and prove it works by performing the Google search for ChatGPT.
- deleted 4y ago[deleted]
- juujian 4y agoPeople say things that are wrong. We train language model on what people say. And even if we were able to filter the training data for just factually correct things---language models use stochastic to generate novel replies, there is always the risk something wrong comes up. So in short, no, that is not what language models are designed to do.
- krainboltgreene 4y ago> We all know now that ChatGPT is just autocomplete on steroids I promise you most people do not know this.
- numeri 4y ago> The concept of factuality doesn't exist in the model at all. This is an example of a whole range of beliefs about LLMs that are very common (even in the field itself), because they were obviously true for small models, but that might not necessarily hold for larger models. There's a lot that we don't know about LLMs, but we do know that they exhibit emergent behaviors as they scale. Smaller models don't really have world models, just language models, but these larger models have started developing clear world models once given the capacity and data to do so. As for the existence of a concept of factuality, I found this paper[1] very interesting. It details an unsupervised method to identify which internal activations of the model correspond to factual statements, regardless of what the model ends up saying. Looking at those internal activations rather than just the model's output even reduces the model's susceptibility to prompts that lead it towards saying the wrong answer. [1] https://arxiv.org/abs/2212.03827 https://arxiv.org/abs/2212.03827
- sdrinf 4y agoAdd to your prompt: "For every factual statement, assign a certainty float 0..1, where 0 means you're very uncertain, and 1 means you're absolutely certain it is true". Specific example: "why do we have first-person subjective experiences? List current theories. For every theory, assign a truthiness float 0..1, where 0 means you're sure it is wrong, and 1 means you're absolutely sure it is true" From experimenting with this, it will shift the output, sometimes drastically so, as the model now has to reason about it's own certainty; it tends to make significantly less shit up (for example, the non-truth-marked version of the output for the query above also listed panpsychism; whereas the truth-marked version listed only scientific hypotheses). So the model _can_ reason about it's certainty, and truth-value; and I strongly suspect it was just not rewarded during RLHF for omitting things it knew to be false -basically, percolating the social lies people tell to eachother- which seems to show up in coding as well. Edit: see https://twitter.com/sdrinf/status/1629084909422931969 https://twitter.com/sdrinf/status/1629084909422931969 for results
- wildrhythms 4y agoI initialized with that prompt and it did not give me any 0..1 certainty values on any subsequent output to my queries.
- swexbe 4y agoOr maybe it will just hallucinate this number too.
- YeGoblynQueenne 4y ago>> So, is any progress being made towards internet-scale ML "fact engines" that also have the flexibility and linguistic expressiveness of ChatGPT? Or are these just two totally different paths that nobody knows how to marry? I wouldn't hold my breath. The whole idea of statistical language modelling (much more ancient than Transformer-trained large language models, btw) is to represent structure without having to represent meaning, because we have no idea how to represent meaning. Or, seen another way, we know how to represent structure, but not how to represent meaning, so let's focus on structure and cross our fingers that meaning will naturally sort of emerge, when it feels like it. So far, we got structure down pat (it's been a few years now, or quite a few, depending on how you see it) but meaning is nowhere to be seen. Nevertheless, this is an interesting scientific result: one can have smooth, grammatically correct linguistic structure without meaning. Progress has been achieved (and no, this is not sarcasm).