4 ms·
> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for h
by themafia 5mo ago
> we're not actually on the right track to achieve real intelligence.
Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid.
The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationally difficult or it would harm the reputation of the companies charging a good sum to use them.
- bluefirebrand 5mo agoMy theory is because the people building the models and in charge of directing where they go love the sycophantic yes-man behavior the models display They don't like hearing "I don't know"
- colechristensen 5mo agoYou can TELL the models to do this and they'll follow your prompt. "Give me your answer and rate each part of it for certainty by percentage" or similar.
- mylifeandtimes 5mo agocould you please tell me how it generates that certainty score?
- colechristensen 5mo agoThe whole thing is a statistical model, that's just what it is. No, I cannot in a reasonable way dissect how an LLM works to a satisfactory level to a skeptic.
- skydhash 5mo agoIt's a statistical model for words and sentences, not knowledge. What does the LLM knows about having a pebble in your shoes, or drinking a nice cup of coffee?
- fc417fc802 5mo agoHe's not a skeptic, he's asking you to explicitly state your reasoning with the expectation that either the readers will learn something or (more likely) you will realize that your thought and speech pattern there was the equivalent of an LLM hallucinating. Yes you can prompt it as you suggested and yes you will generally receive a convincing answer but it is not doing what you seem to think it is doing ie the generated rating is complete bullshit that the model pulled out of its proverbial ass.
- colechristensen 5mo agoare you actually curious or do you just want to argue against it?
- clipsy 5mo ago"I can only explain my beliefs to people who promise they'll agree" is certainly a unique take.
- fc417fc802 5mo agoI think you're obviously wrong (based on my relatively detailed but certainly somewhat out of date and not expert level knowledge of LLM internals) but if you're willing to explain your reasoning I'm willing to reconsider my own position in light of any new information or novel observations you might provide.
- D-Machine 5mo agoGP is obviously wrong, and probably doesn't know about calibration and/or that it isn't even clear how to calibrate frontier models in the manner we need, given how complex and expensive the training is, and how tricky calibration becomes in e.g. mixture-of-experts and chain of thought approaches.
- mootothemax 5mo agoI suspect that introducing the calibration concept might be a case of too much too soon for some people. As far as I understand it, the various probability matrices boil down to: what token has the highest likelihood of coming next, given this set of input tokens. Which then all gets chucked away and rebuilt when the most likely token is appended to the input set. Objective assessment of internal state - again, to my non-expert eye - doesn’t appear to have any way to surface to me. Big-if my rough working understand is more or less correct - your calibration point makes a lot of sense to me. I’m not sure that it would make sense to someone who eg considers some form of active thinking process that is intellectualising about whether to output this or that token.
- adastra22 5mo agoVibes.
- cmrdporcupine 5mo agoThat's just not how they work, really. They don't know what they don't know and their process requires an output. I think they're getting better at it, but it's likely just the number of parameters getting bigger and bigger in the SOTA models more than anything.
- adastra22 5mo agoThey do know what they don't know. There's a probability distribution for outputs that they are sampling from. That just isn't being used for that purpose.
- Isamu 5mo agoOh, you mean somewhere it is tracking the statistical likelihood of the output. Yeah I buy that, although I think it just tends towards the most likely output given the context that it is dragging along. I mean it wouldn’t deliberately choose something really statistically unlikely, that’s like a non sequitur.
- tempest_ 5mo agoFrom its point of view what does it mean "to know". Is it the token (or set of tokens) that are strictly > 50% probable or is it just the highest probability in a set of probabilities? While generating bullshit is not ideal for a lot of use cases you don't want your premier chat bot to say "I don't know" to the general public half the time. The investment in these things requires wide adoption so they are always going to favour the "guesses".
- adastra22 5mo agoWell, it's not tracking. As it predicts each token it is sampling from a probability distribution -- that's what the matrix multiplies are for. It gets a distribution over all tokens and then picks randomly according to that distribution. How flat or how spiky that distribution is tells you how confident it is in its answer. But it then throws that distribution away / consumes it in the next token calculation. So it's not really tracking it per se.
- wagwang 5mo agoYou can just tell the agent to do exactly that
- alterom 5mo ago>You can just tell the agent to do exactly that You can. It just won't do it.
- wagwang 5mo agoSeems to work for me https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f658887 https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f6588...
- tempest_ 5mo agoI've had various agents backed by various models ignore the shit out of various rules and request at varying rates but they all do it. When you point it out "Oh yes, I did do that which is contrary to the rules, request <whatever>.. Anyway..."
- wagwang 5mo agoIf you are on a sota model and your context window is less than 100k tokens and you don't have any vague or contradicting rules, then I've almost never seen a rule broken The most common failure I've seen come from tools that pollute their context with crap and the llm will forget stuff or just get confused from all the irrelevant sentences; which if the report is true, is probably what these ai notetakers are guilty of. This problem gets exacerbated if these tools turn on the 1M context window version.
- fg137 5mo agoYeah, that's exactly why I have full confidence in that system, especially for medical notetaking. /s
- D-Machine 5mo agoExcept you can't be sure it isn't producing nonsense when you do this, and generally the model(s) will be overconfident. This has been studied, see e.g. https://openreview.net/pdf?id=E6LOh5vz5x https://openreview.net/pdf?id=E6LOh5vz5x > An alternative way to obtain uncertainty estimates from LLMs is to prompt them directly. One benefit of this approach is that it requires no access to the internals of the model. However, this approach has produced mixed results: LLMs can sometimes verbalize calibrated confidence levels (Lin et al., 2022a; Tian et al., 2023), but can also be highly overconfident (Xiong et al., 2024). Interestingly, Xiong et al. (2024) found that LLMs typically state confidence values in the range of 80-100%, usually in multiples of 5, potentially in imitation of how humans discuss confidence levels. Nevertheless, prompting strategies remain an important tool for uncertainty quantification, along with measures based on the internal state (such as MSP).
- vintagedave 5mo ago> Real intelligence means you have to say "I don't know" when you don't know I have met many supposedly intelligent, certainly high status, humans who don't appear to be able to do that either. I have more confidence we can train AIs to do it, honestly.
- FatherOfCurses 5mo agoWhile it is true that there are people who do not admit they are wrong when they factually are, your assertion glosses over the fact that most of the people we maintain in our social circle are people we trust through our experiences with them to be honest.
- vintagedave 5mo agoWe don't always have control over who everyone we interact with though. We interact with many more people than voluntarily chosen friends.