4 ms·
I have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models signif
by rainsford 5mo ago
I have generally moved from bearish to bullish on the future of current AI technology, but the continued inaccuracy with basic facts all while the models significantly improve continues to give me significant pause.
As an example, creating recipes with Claude Opus based on flavor profiles and preferences feels magical, right up until the point at which it can't accurately convert between tablespoons and teaspoons. It's like the point in the movie where a character is acting nearly right but something is a bit off and then it turns out they're a zombie and going to try to eat your brain. This note taking example feels similar. It nearly works in some pretty impressive ways and then fails at the important details in a way that something able to do the things AI can allegedly do really shouldn't.
It's these failures that make me more and more convinced that while current generation AI can do some pretty cool things if you manage it right, we're not actually on the right track to achieve real intelligence. The persistence of these incredibly basic failure modes even as models advance makes it fairly obvious that continued advancement isn't going to actually address those problems.
- themafia 5mo ago> we're not actually on the right track to achieve real intelligence. Real intelligence means you have to say "I don't know" when you don't know, or ask for help, or even just saying you refuse to help with the subtext being you don't want to appear stupid. The models could ostensibly do this when it has low confidence in it's own results but they don't. What I don't know if it's because it would be very computationally difficult or it would harm the reputation of the companies charging a good sum to use them.
- bluefirebrand 5mo agoMy theory is because the people building the models and in charge of directing where they go love the sycophantic yes-man behavior the models display They don't like hearing "I don't know"
- colechristensen 5mo agoYou can TELL the models to do this and they'll follow your prompt. "Give me your answer and rate each part of it for certainty by percentage" or similar.
- mylifeandtimes 5mo agocould you please tell me how it generates that certainty score?
- colechristensen 5mo agoThe whole thing is a statistical model, that's just what it is. No, I cannot in a reasonable way dissect how an LLM works to a satisfactory level to a skeptic.
- skydhash 5mo agoIt's a statistical model for words and sentences, not knowledge. What does the LLM knows about having a pebble in your shoes, or drinking a nice cup of coffee?
- fc417fc802 5mo agoHe's not a skeptic, he's asking you to explicitly state your reasoning with the expectation that either the readers will learn something or (more likely) you will realize that your thought and speech pattern there was the equivalent of an LLM hallucinating. Yes you can prompt it as you suggested and yes you will generally receive a convincing answer but it is not doing what you seem to think it is doing ie the generated rating is complete bullshit that the model pulled out of its proverbial ass.
- colechristensen 5mo agoare you actually curious or do you just want to argue against it?
- 5mo ago
- cmrdporcupine 5mo agoThat's just not how they work, really. They don't know what they don't know and their process requires an output. I think they're getting better at it, but it's likely just the number of parameters getting bigger and bigger in the SOTA models more than anything.
- adastra22 5mo agoThey do know what they don't know. There's a probability distribution for outputs that they are sampling from. That just isn't being used for that purpose.
- Isamu 5mo agoOh, you mean somewhere it is tracking the statistical likelihood of the output. Yeah I buy that, although I think it just tends towards the most likely output given the context that it is dragging along. I mean it wouldn’t deliberately choose something really statistically unlikely, that’s like a non sequitur.
- tempest_ 5mo agoFrom its point of view what does it mean "to know". Is it the token (or set of tokens) that are strictly > 50% probable or is it just the highest probability in a set of probabilities? While generating bullshit is not ideal for a lot of use cases you don't want your premier chat bot to say "I don't know" to the general public half the time. The investment in these things requires wide adoption so they are always going to favour the "guesses".
- adastra22 5mo agoWell, it's not tracking. As it predicts each token it is sampling from a probability distribution -- that's what the matrix multiplies are for. It gets a distribution over all tokens and then picks randomly according to that distribution. How flat or how spiky that distribution is tells you how confident it is in its answer. But it then throws that distribution away / consumes it in the next token calculation. So it's not really tracking it per se.
- wagwang 5mo agoYou can just tell the agent to do exactly that
- alterom 5mo ago>You can just tell the agent to do exactly that You can. It just won't do it.
- wagwang 5mo agoSeems to work for me https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f658887 https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f6588...
- tempest_ 5mo agoI've had various agents backed by various models ignore the shit out of various rules and request at varying rates but they all do it. When you point it out "Oh yes, I did do that which is contrary to the rules, request <whatever>.. Anyway..."
- wagwang 5mo agoIf you are on a sota model and your context window is less than 100k tokens and you don't have any vague or contradicting rules, then I've almost never seen a rule broken The most common failure I've seen come from tools that pollute their context with crap and the llm will forget stuff or just get confused from all the irrelevant sentences; which if the report is true, is probably what these ai notetakers are guilty of. This problem gets exacerbated if these tools turn on the 1M context window version.
- fg137 5mo agoYeah, that's exactly why I have full confidence in that system, especially for medical notetaking. /s
- D-Machine 5mo agoExcept you can't be sure it isn't producing nonsense when you do this, and generally the model(s) will be overconfident. This has been studied, see e.g. https://openreview.net/pdf?id=E6LOh5vz5x https://openreview.net/pdf?id=E6LOh5vz5x > An alternative way to obtain uncertainty estimates from LLMs is to prompt them directly. One benefit of this approach is that it requires no access to the internals of the model. However, this approach has produced mixed results: LLMs can sometimes verbalize calibrated confidence levels (Lin et al., 2022a; Tian et al., 2023), but can also be highly overconfident (Xiong et al., 2024). Interestingly, Xiong et al. (2024) found that LLMs typically state confidence values in the range of 80-100%, usually in multiples of 5, potentially in imitation of how humans discuss confidence levels. Nevertheless, prompting strategies remain an important tool for uncertainty quantification, along with measures based on the internal state (such as MSP).
- vintagedave 5mo ago> Real intelligence means you have to say "I don't know" when you don't know I have met many supposedly intelligent, certainly high status, humans who don't appear to be able to do that either. I have more confidence we can train AIs to do it, honestly.
- FatherOfCurses 5mo agoWhile it is true that there are people who do not admit they are wrong when they factually are, your assertion glosses over the fact that most of the people we maintain in our social circle are people we trust through our experiences with them to be honest.
- vintagedave 5mo agoWe don't always have control over who everyone we interact with though. We interact with many more people than voluntarily chosen friends.
- Brian_K_White 5mo agoI hate to help provide possible soultions to an entire process I don't approve of, but maybe the fuzzy tools need old style deterministic tools the same way and for the same reasons we do. So instead of an LLM trying to answer a math or reason question by finding a statistical match with other similar groups of words it found on 4chan and the all in podcast and a terrible recipe for soup written by a terrible cook, it can use a calculator when it needs a calculator answer.
- colechristensen 5mo agoNo, they just need to be trained to have adversarial self review "thinking" processes. You ask an LLM "What's wrong with your answer?" and you get pretty good results.
- binary0010 5mo agoOr you get the original output result was perfect and the adversarial "rethinking" switches to an incorrect result.
- byzantinegene 5mo agothis seems to happen far more than i would like
- stevula 5mo agoI think that is how the smarter agents do things? Just like Claude/ChatGPT sometimes does a web search they can do other tool calls instead of just making a statistical guess. Of course it doesn’t always make the bright choice between those options though…
- WalterBright 5mo ago> it doesn’t always make the bright choice I'm available for a small fee.
- cootsnuck 5mo agoYup, spot on. There's a capability-reliability gap that the industry does not like to talk about too much. It often feels like the AI industry is continually glossing over the fact that capability and reliability are fundamentally different qualities. We tend to use "accurate" and "reliable" interchangeably, but they describe different things. A model can ace a benchmark (capability/accuracy) and still be a liability in production (reliability). Just look at recent reactions to yet another release from METR showing improved capabilities. But the less talked about part is how their measure is for a 50% success rate (and the even lesser talked about secondary measure they have at 80% success rate has a drastically lower time-horizon for tasks). https://metr.org/ https://metr.org/ I implement AI systems for enterprises and I don't know any that would ever be okay with 80% reliability (let alone 50%).
- jcgrillo 5mo agoThis capability-reliability gap (excellent term btw, more people need to think in these terms or we'll be in real trouble) is also infecting LLM assisted outputs. I just tried VSCode again tonight after a ~3yr hiatus and goddamn has it deteriorated. Lots of new features, lots of interesting looking plugins, but 3 out of the 5 plugins I tried for code CAD (the reason I downloaded VSCode again at all) were completely unusable--like couldn't even be made to work at all--and the other two didn't do anything like what they claimed. Also VSCode itself got into some kind of spastic loop trying to log me into github, and seemed incapable of recognizing the virtual environment in a python project's workspace... It also feels like the UI got even slower. This situation is bad.
- igleria 5mo agoYesterday I was using opus 4.6 through copilot (don't ask...) to rubber-duck-brainstorm a big feature that needs a lot of care. I got some inspiration from it but it misinterpreted very basic stuff. might be a skill issue on my side, I do not know.
- smusamashah 5mo agoYour analogy reminds of messed up fingers and hands in image generation models just a year ago. Now that is pretty much solved. These days they are generating videos you can't tell apart from reality. This makes me believe these nuances will keep reducing and eventually become very hard to notice and find in may be every task.
- sillyfluke 5mo agoI would suggest slightly adjusting your expectations by factoring in the difference between video training data and text training data. Due to computation and cost limitations, the idea of video training data being polluted with AI video slop is less of a thing. Also, humans don't generate a lot of biology and physics defying fictional video relative to the abundance and generation ease of real-life video. The main problem currently with LLM text is not that they create incoherent sentences, it's that what they purport to be statements of fact or general consensus often times aren't, because they are bullshit machines that become better and more accurate bullshitters the more context-accurate data they are fed. AI videos may still have issues with "looking plausible" whereas LLM text currently has less issues with "sounding plausible" and more issues with "being correct" with respect to reality. Which they have no direct connection to. No one is penalizing an AI video generator for creating a scene that never happened in real life.
- cyberrock 5mo agoIf Claude occasionally overestimates the conversion, that might be an artifact of Australian tablespoons being different (4tsp/20mL vs 3tsp/15mL in the US). This error could at least be explained as a complication of the real world. (If it's saying 3.14tsp or 2tsp then I have no idea)
- eigencoder 5mo agoUsually tsp means teaspoons, and 4tsp / 20 mL vs 3tsp / 15 mL are the same ratios of tsp to mL, aren't they?
- retrochameleon 5mo agoI was skeptical that LLMs could be the right path to AGI, but then I kept being impressed by how much further we could take it by expanding upon the way we use it, the harnesses we use with LLMs, and better context engineering. When I see how LLMs are capable of essentially prompt and context engineering for themselves, it makes me think they won't need human guidance forever. When it comes to simple fact-based tasks that have a concrete methodology, it is no surprise to me that LLMs aren't the right tool, and I believe it's a failure of the harness to not recognize those types of tasks and handle them with a more concretely functioning tool instead of relying on statistical probabilities in the LLM "brain" to spit out the correct number to a math problem. In the same sense that LLMs can use "skills" when necessary, it should have tools or possibly even specialized "brains" for it to pass of certain types of tasks to. I'm starting to feel that our first form of AGI is not going to be a single brain but an elaborate system of harnesses, multiple LLM models, skills, domain and task specialized subsystems it passes tasks off to, etc. Whether we get there with current LLM technology before some other evolution in AI is the question, to me.
- djhn 5mo agoThis sounds a lot like ignoring the Bitter Lesson, and expending a lot of effort rebuilding slightly better Expert Systems.
- retrochameleon 5mo agoThanks for introducing me to those concepts. If I take the Bitter Lesson into account, I would frame this as needing more focus on enabling a general intelligence to use tools more effectively and when appropriate to essentially stop making mistakes. A basic example being a calculator. The AI needs to recognize its default thinking pattern doesn't work well for math calculations, so it delegates it to the available calculator tool / skill / MCP instead. An LLM should not be relying on LLM prediction to give a mathematical resultant figure, ever. It should come from a deterministic tool. If anything, the LLM may interpret the problem and convert it into starting math figures to use for calculation. If we can enable AI systems to learn and apply that for themselves, and even develop their own deterministic tooling and sense of what tool to use for what job, that starts to sound promising to me. Skills feel like a conceptual stepping stone to the next useful abstraction.