4 ms·
This is a tired semantic argument that does not bring any insight into the discussion. A token-predictor could still be trained to predict the tokens “I’m not s
by Sharlin 1y ago
This is a tired semantic argument that does not bring any insight into the discussion. A token-predictor could still be trained to predict the tokens “I’m not sure what you mean because of points x, y, and z; could you elaborate?”
- Terr_ 1y agoIt means if you want something resembling a self-introspective theory of mind, you need to arrange the overall document to cohere to documents where such things are/appear-to-be happening. This leads us to new questions: How can we characterize and identify real-world documents which fit? How can we determine what features may be significant, and which of those can be easily transplanted to our use-case?
- sitkack 1y agoYou are just doubling down on protecting your argument. I operate LLMs in many conversational modes where it does ask clarifying questions, probing questions, baseline determining questions. It takes at most one sentence in the prompt to get them to act this way.
- bigcat12345678 1y ago> It takes at most one sentence in the prompt to get them to act this way. What is this one sentence you are using? I am struggling to elicite clarification behavior form llms
- sandspar 1y agoCould you share your prompt to get it to ask clarifying questions? I'm wondering if it would work in custom instructions.
- sitkack 1y agoIt is domain dependent, you really need to play with it. Tell it you are doing pair thinking and either get it to ask questions about things it doesn't understand, or get it to ask you questions to get you to think better. Project the AI into a vantage point in the latent space and then get it to behave in the way that you want it to. You can ask it to use the Socratic method, but then it is probing you, not its own understanding. Now have it use the socratic method on itself. You can tell it to have multiple simultaneous minds. Play with deepseek in thinking and non-thinking mode, give it nebulous prompts and see if you can get it to ask for clarifications.
- simianwords 1y agoThere are a lot of words but it feels like you have never really used LLM's (apologies for the bluntness). We see LLM's introspecting all the time[1]. >Notably, DeepSeek-AI et al. report that the average response length and downstreamperformance of DeepSeek-R1-Zero increases as training progresses. They further report an “aha moment” during training, which refers to the “emergence” of the model’s ability to reconsider its previously generated content. As we show in Section 3.2, this reconsideration behaviour is often indicated by the generation of phrases such as ‘wait, ...’ or ‘alternatively, ...’ [1] https://arxiv.org/pdf/2504.07128 https://arxiv.org/pdf/2504.07128
- root_axis 1y agoIt could be trained to say that, but it's not exactly clear how you would reinforce the absence of certain training data in order to emit that response accurately, rather than just based on embedding proximity.
- simianwords 1y agoWhy does it seem so hard to make training data for this? You can cook up a few thousands of training data and do an RLHF.
- root_axis 1y agoYes, but all that does is locate "I don't know" near the cooked up data within the embeddings. This doesn't actually reflect an absence of data in the training.
- jsnider3 1y agoSeems easy. Have a set of vague requests and train it to ask for clarification instead of guessing.
- timdiggerm 1y agoHow does it identify what's vague?
- jsnider3 1y agoMany ways. 1) Hire some humans to label the data. 2) Let the user give you feedback. 3) Ask another LLM.
- root_axis 1y agoAs I said, it's possible to train it to ask for clarification, but it's not clear how to reinforce that response in a way that correctly maps on to the absence of data rather than arbitrary embedding proximity. You can't explicitly train on every possible scenario where the AI should recognize its lack of knowledge.
- roywiggins 1y agoAnthropic found that it Claude will pretend that it used the "standard" way to do addition- add the digits, carry the 1, etc- but the pattern of activations showed it using a completely different algorithm. So these things can role play as introspecting- they come up with plausible post-hoc explanations for their output- but they are still just pretending, so they will get it wrong. So you can teach a model to sometimes ask for clarification, but will it actually have insight into when it really needs it, or will it just interject for clarification more or less at random? These models have really awful insight into their own capabilities, ChatGPT eg insists to me that it can read braille, and then cheerfully generates a pure hallucination.
- roenxi 1y ago> Anthropic found that it Claude will pretend that it used the "standard" way to do addition- add the digits, carry the 1, etc- but the pattern of activations showed it using a completely different algorithm. That doesn't mean much; humans sometimes do the same thing. I recall a fun story about a mathematician with synesthesia multiplying numbers by mixing the colours together. With a bit of training such a person could also pretend to be executing a normal algorithm for the purposes of passing tests.
- frabcus 1y agoEven then the human doesn't know how they execute the algorithm, or mix the colours together - our conscious self-reflective mind has limits as to how far into our neural network weights it can reach. Can get further with lots of meditation, but it is still definitionally limited (in information theory terms).
- jcims 1y agoI agree that it's a tired argument, but there appears to be two separate things being discussed in this little corner of HN. Clarity in the problem it's being asked to solve, and confidence that the answer it has is correct. I can trivially get any of the foundational models to ask me clarifying questions. I've never had one respond with 'I don't know'.
- chipsrafferty 1y agoI've gotten lots of responses like "with the information you provided, I cannot answer that. Can you provide more information?" Which IMO is the name as "idk"
- littlestymaar 1y agoIt's not a tired argument, and not just a semantic one it's a foundational characteristic of LLM. > A token-predictor could still be trained to predict the tokens “I’m not sure what you mean because of points x, y, and z; could you elaborate?” This is entirely true, and the key insight is even right in your sentence but you don't seem to grasp it. “could still be trained”: you can train an LLM into doing whatever you want it to, but you have to train it specifically for that! In the beginning of LLM we witnessed this impressive phenomenon where the LLM exhibited emergent capabilities (I'm particularly thinking about LLMs being few shots learners about stuff that wasn't in their training corpus). And these emergent capabilities legitimately raised the question about “how intelligent these things are, really”. But for the past three years, the key lesson is that this kind of emergent effect is too small to be useful, and the focus has been put towards creating purposely built datasets (with tons of “artificial data”) to train the model to explicitly do things we want it to do. And it works pretty well, as models' capabilities kept improving at a fast pace (and in particular, I don't see would we couldn't overcome the problem highlighted by this paper, with more synthetic data specifically designed for multi-turn conversation). But their progress is now strictly limited by their makers' own intelligence. You cannot just scrap the web throw compute at the problem and expect emergent intelligence to occur anymore. It's more “simulated intelligence” than “artificial intelligence”, really.
- famouswaffles 1y agoIt's definitely a tired and semantical one because as he said, it brings no insight and is not even good at the analogy level. I can't have a conversation with Dracula and Dracula can't make decisions that affect the real world, so LLMs already break key aspects and assumptions of the 'Document Simulator'. Pre-trained LLMs will ask clarifying questions just fine. So I think this is just another consequence of post-training recipes.
- Terr_ 1y ago> Dracula can't make decisions that affect the real world, so LLMs already break key aspects and assumptions of the 'Document Simulator'. Nonsense, we are already surrounded by mindless algorithms (and their outputs) that "affect the real world" because many of us have full-time jobs ensuring it happens! " When someone uses a SimCity-esque program to generate a spreadsheet used for real-world bus schedules, does that "break key aspects and assumptions of a traffic simulator"? Does the downstream effect elevate it to a microcosm of tiny lives? Nope!
- deleted 1y ago[deleted]
- dkdbejwi383 1y agoHow would an LLM “know” when it isn’t sure? Their baseline for truth is competent text, they don’t have a baseline for truth based on observed reality. That’s why they can be “tricked” into things like “Mr Bean is the president of the USA”
- ben_w 1y agoThe answer is the same as how the messy bag of chemistry that is the human brain "knows" when it isn't sure: Badly, and with great difficulty, so while it can just about be done, even then only kinda.
- foldr 1y agoWe really don’t understand the human brain well enough to have confidence that the mechanisms that cause people to respond with “I don’t know” are at all similar to the mechanisms which cause LLMs to give such responses. And there are quite a few prima facie reasons to think that they wouldn’t be the same.
- Sharlin 1y agoThe mechanics don't have to be similar, only analogous, in the morphology sense.
- foldr 1y ago'Analogous in the morphology sense' is actually a more specific concept than 'similar'. But either way, we still don't know if they're analogous, or similar, or whatever term you prefer. Anyone who actually understands both LLMs and the human brain well enough to make confident claims that they basically work the same really ought to put in the effort to write up a paper and get a Nobel prize or two.
- Sharlin 1y agoAnalogous in the morphology sense means having come up with an entirely distinct solution to a common problem. Insect and bird wings have little to do with each other except that both flap to create lift. It explicitly does not imply the solutions are similar in mechanism, although that can be, and often is, a result of convergent evolution, of course. In particular, generally speaking (not claiming that LLMs a road to AGI, which is something I doubt) it's generally not a well-defensible philosophical position that the vertebrate brain (and remember that mammalian, bird and cephalopod brains are very different) is uniquely suited to produce what we call "intelligence". > Anyone who actually understands both LLMs and the human brain well enough to make confident claims that they basically work the same This is a strawman and not my position.
- dTal 1y agoI disagree, it's a very insightful comment. The problem is that any information about any internal processes used to generate a particular token is lost; the LLM is stateless, apart from the generated text. If you ask an LLM-character (which I agree should be held distinct from the LLM itself and exists at a different layer of abstraction) why it said something, the best it can do is a post-hoc guess. The "character", and any internal state we might wish it to have, only exists insofar as it can be derived anew from the text.
- Sharlin 1y agoI certainly agree with the point about post-hoc justifications – but isn't it amazing that it's also something very familiar to humans who do that all the time and manage to lie to ourselves about it very convincingly?! The more you read about neuropsychology the more you're forced to assume a view where the conscious self, whatever it is, has only a very tenuous grasp of what is going on and how much it actually has control over things. In any case, you don't need accurate understanding of how your mind works (hello humans, again!) to be able to converge on INSUFFICIENT DATA FOR A MEANINGFUL ANSWER when there's no other uniquely good local optimum in the search space.