5 ms·
> the AI says things like “Interesting!” My experience of those utterance is that it’s purely phatic mimicry: they lack genuine intuitive surprise, it’s just m
by bertil 5mo ago
> the AI says things like “Interesting!”
My experience of those utterance is that it’s purely phatic mimicry: they lack genuine intuitive surprise, it’s just marking a very odd shift in direction. The problem isn’t the lack of path, is that the rhetorical follow-up to those leaps are usually relevant results, so they stream-of-token ends up rapidly over-playing its own conviction. That’s why it’s necessary (and often ineffective) to tell them to validate their findings thoroughly: too much of their training is “That’s odd” followed by “Eureka!” and not “Nevermind…”
- deleted 5mo ago[deleted]
- sigbottle 5mo agoI think that a lot of models have to sprinkle in a lot of "fluff" in their thinking to stay within the right distribution. They only have language as their only medium; the way we annotate context is via brackets and then training them to hopefully respect the brackets. I'd imagine that either top labs explicitly train, or through the RL process the models implicitly learn, to spam tokens to keep them 'within distribution' since everything's going through the same channel and there's no fine grained separation between things. Philosophically, it's not like you're a detached observer who simply reasons over all possible hypotheses. Ever get stuck in a dead end and find it hard to dig yourself out? If you were a detached observer, it'd be pretty easy to just switch gears. But it's not (for humans).
- WarmWash 5mo agoLanguage really only exists at the input and output surfaces of the models. In the middle it's all numerical values. Which you might be quick in relating to just being a numeric cypher of the words, which while not totally false, it misses that it is also a numeric cypher of anything. You can train a transformer on anything that you can assign tokens to.
- pohl 5mo agoSimilarly, none of our comments actually exist as language on Hacker News—just numerical values from the ASCII table. We're deluding each other into thinking we're using language.
- jfengel 5mo agoI believe it's reasonably clear that our thought processes generally occur outside of language. We do use language during explicit reasoning, but most thinking occurs heuristically. It's on par with the thinking of animals that don't use language but do complex behavior. It not clear to me how well that maps onto LLMs. Our wetware predates language, and isn't derived from it. Language is built on top. LLMs are derived from language. I think that means that the intermediate layers are very different from the brain neurons, but I don't know. It's eerie how well the former emulates the latter.
- tonyarkles 5mo agoThere’s an interesting thing there that I believe varies person to person. My understanding is that some people do think in a more symbolic/heuristic way, some rely very heavily on their inner monologue to make sense of things (I am in the latter camp, and only have a single core language processor so pretty much cannot come up with coherent thoughts if I’m concentrating on what someone else is saying) Even more interesting, and getting off on a bit of a tangent, there is also a mode that I use for revealing emotions that I don’t have words for (alexythmia): I open up a text editor, stare off into space, and let my fingers type without “observing” the stream of words coming out. I then go back and read what I “wrote” and often end up understanding how I’m feeling much better than I did. It’s weird. Edit: also, playing with local models through e.g. llama-cpp in “thinking mode” is super fascinating for me. The “thought process” that comes out before the real answer often feels pretty familiar when I reflect on my own inner monologue, although sometimes it’s frustrating for me because I see where their “thinking” went off the rails and want to correct it.
- redsocksfan45 5mo ago[dead]
- jackcarter 5mo agoIt’s funny that this is probably due to bias in the training texts, right? Humans are way more likely to publish their “Eureka!” moments than their screwups… if they did, maybe models would’ve exhibit this behavior. Now that AI labs have all these “Nevermind” texts to train on, maybe it’s getting easier to correct? (Would require some postprocessing to classify the AI outputs as successful or not before training)
- Forgeties79 5mo agoMy understanding is that it’s the result of these companies making sure to keep you engaged/happy less than the result of data these companies train with. I don’t know if it’s true or not but it certainly tracks given LLMs are way more polite than the average post on the internet lol
- embedding-shape 5mo agoI think it's more explicit than that, part of post-training to enforce the kind of behavior, I don't think it's emergent but rather researchers steering it to do that because they saw the CoT gets slightly better if the model tries to doubt itself or cheer itself on. Don't recall if there was a paper outlining this, tried finding where I got this from but searches/LLMing turns up nothing so far.
- epolanski 5mo agoInterestingly this is strikingly similar to how my mind would process something I find genuinely interesting.
- animal531 5mo agoI've somehow managed to train mine out of trying to fluff me up the whole time, its become very factual. Overall it saves me a lot of time reading when it's just focusing on the details.
- hmontazeri 5mo agoThe new Opus 4.7 thinks quite often with: Hmmmm… Haha anyone else seen this?
- holoduke 5mo agoIndeed. I think it's the client. Not the model
- etherealG 5mo agoAnd what I find fascinating is I see similar mimicking by my 5 year old. Perhaps we shouldn’t be so quick to call this a lack of being genuine. Sometimes emotions are learned in humans but we wouldn’t call them fake. I don’t want to declare machines to have emotion outright, but to call mimicry evidence of falsehood is also itself false.
- nkrisc 5mo agoMimicry is how kids learn the expected reactions to particular emotions. A kid mimicking your surprise doesn’t mean they are surprised (as surprise requires an existing expectation of an outcome they may not have the experience for), but when they do feel genuine surprise, they’ll know how to express it.
- orangebread 5mo agoHow do we know that AI isn't feeling genuine surprise then?
- philipallstar 5mo agoBecause it's a statistical process generating one part of a word at a time. It probably isn't even generating "surprise". It might be generating "sur", then "prise" then "!"
- yulker 5mo agoBut what is surprise really? Something not following expectation. The distribution may statistically leverage surprise as a concept via how it has seen surprise as a concept e.g. "interesting!" So it can be both true that it has nothing to do with the emotion of surprise, but appear as the emulation of that emotion since the training data matches the concept of surprise (mismatch between expectation and event).
- nkrisc 5mo ago
- fnordpiglet 5mo agoI think sometimes though there harness LLMs providing guidance. For instance I’ve seen recently coding agents doing an analysis then mid response saying “no wait, that’s not right” and course correcting. This feels implausible as an auto regressive rhetorical tick. LLM harnesses are widely used in advanced agentic systems and I’m sure the Pro level reasoning models exploit them extensively. I’m not saying this is what happened here, but there is a chance it was something injected by the hardness into its thinking.
- SummSolutions 5mo ago[dead]
- rocqua 5mo agoI believe there might be more to it. Wasn't a big part of thinking or reasoning taking the response, replacing the final period with "Wait!" and then continuing? Which suggests that such words actually are important to the internals.