5 ms·
I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of abili
by My_Name 1y ago
I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of ability to do that. They are good at repeating their training data, not thinking about it.
The problem, I find, is that they then don't stop, or say they don't know (unless explicitly prompted to do so) they just make stuff up and express it with just as much confidence.
- deleted 1y ago[deleted]
- ftalbot 1y agoEvery token in a response has an element of randomness to it. This means they’re non-deterministic. Even if you set up something within their training data there is some chance that you could get a nonsense, opposite, and/or dangerous result. The chance of that may be low because of things being set up for it to review its result, but there is no way to make a non-deterministic answer fully bound to solving or reasoning anything assuredly, given enough iterations. It is designed to be imperfect.
- yuvalr1 1y agoYou are making a wrong leap from non-deterministic process to uncontrollable result. Most of the parallel algorithms are non-deterministic. There might be no guarantee about the order of calculation or even sometimes the final absolute result. However, even when producing different final results, the algorithm can still guarantee characteristics about the result. The hard problem then is not to eliminate non-deterministic behavior, but find a way to control it so that it produces what you want.
- flavaflav2 1y agoLife and a lot in our universe is non-deterministic. Some people assume science and mathematics are some universal truths rather than imperfect agreed upon understandings. Similarly many assume humans can be controlled through laws, penalties, prisons, propaganda, coercion, etc. But terrible things happen. Yes, if you set up the gutter-rails in your bowling lane, you can control the bowling ball unless it is thrown over those rails or in a completely different direction, but those rails are wide with LLMs by default, and the system instructions provided it aren’t rules, they are an inherently faulty way to coerce a non-deterministic system. But, yes, if there’s absolutely no way to do something, and you’re aware of every possible way a response or tool could affect things, and you have taken every possible precaution, you can make it behave. That’s not how people are using it though, and we cannot control our tendency to trust that which seems trustworthy even if we are told these things.
- squidbeak 1y agoNo, Science is a means of searching for those truths - definitely not some 'agreed upon understanding'. It's backed up by experimentation and reproducible proofs. You also make a huge bogus leap from science to humanities.
- iq176 1y agoScientific method is the process. Science itself includes the study and compendium of understandings, based on a belief system that includes shared understandings just like mathematics. The foundation of these are philosophical beliefs that we can know and understand these things. For example, on a metaphysical level, if the world around us were a simulation, then science could provide understandings about that simulated universe, but not about that which is simulating it.
- squidbeak 1y agoThis I'm afraid is rubbish. Scientific proofs categorically don't depend on philosophical beliefs. Reality is measurable and the properties measured don't care about philosophy.
- weltensturm 1y ago> Reality is measurable Heisenberg would disagree.
- squidbeak 1y agoAre you arguing that the uncertainty principle derives from philosophy rather than math?
- darkwater 1y agoBut those are still approximations to the actual underlying reality. Because the other option (and yes, it's a dichotomy) is that we already defined and understood every detail of the physics that applies to our universe.
- mannykannot 1y agoThere seems to be more to it than that - in my experience with LLMs, they are good at finding some relevant facts but then quite often present a non-sequitur for a conclusion, and the article's title alone indicates that the problem for LRMs is similar: a sudden fall-off in performance as the task gets more difficult. If the issue was just non-determinism, I would expect the errors to be more evenly distributed, though I suppose one could argue that the sensitivity to non-determinism increases non-linearly.
- squidproquo 1y agoThe non-determinism is part of the allure of these systems -- they operate like slot machines in a casino. The dopamine hit of getting an output that appears intelligent and the variable rewards keeps us coming back. We down-weight and ignore the bad outputs. I'm not saying these systems aren't useful to a degree, but one should understand the statistical implications on how we are collectively perceiving their usefulness.
- galaxyLogic 1y ago> Every token in a response has an element of randomness to it. I haven't tried this, but so if you ask the LLM the exact same question again, but in a different process, will you get a different answer? Wouldn't that mean we should mosr of the time ask the LLM each question multiple times, to see if we get a better answer next time? A bit like asking the same question from multiple different LLMs just to be sure.
- PxldLtd 1y agoI think a good test of this seems to be to provide an image and get the model to predict what will happen next/if x occurs. They fail spectacularly at Rube-Goldberg machines. I think developing some sort of dedicated prediction model would help massively in extrapolating data. The human subconscious is filled with all sorts of parabolic prediction, gravity, momentum and various other fast-thinking paths that embed these calculations.
- yanis_t 1y agoAny example of that? One would think that predicting what comes next from an image is basically video generation, which works not perfect, but works somehow (Veo/Sora/Grok)
- PxldLtd 1y agoHere's one I made in Veo3.1 since gemini is the only premium AI I have access to. Using this image - https://www.whimsicalwidgets.com/wp-content/uploads/2023/07/featured-image-001-B-scaled.jpg https://www.whimsicalwidgets.com/wp-content/uploads/2023/07/... and the prompt: "Generate a video demonstrating what will happen when a ball rolls down the top left ramp in this scene." You'll see it struggles - https://streamable.com/5doxh2 https://streamable.com/5doxh2 , which is often the case with video gen. You have to describe carefully and orchestrate natural feeling motion and interactions. You're welcome to try with any other models but I suspect very similar results.
- chamomeal 1y agoI love how it still copies the slow pan and zoom from rube goldberg machine videos, but it's just following along with utter nonsense lol
- galaxyLogic 1y agoA Goldbergs machine was not part of their training data. For humans, we have seem such things.
- pistoriusp 1y agoI saw a meme that I think about fairly often: Great apes have learnt sign language, and communicated with humans, since the 1960's. In all that time they've never asked human questions. They've never tried to learn anything new! The theory is that they don't know that there are entities that know things they don't. I like to think that AI are the great apes of the digital world.
- 20k 1y agoIts worth noting that the idea that great apes have learnt sign language is largely a fabrication by a single person, and nobody has ever been able to replicate this. All the communication has to be interpreted through that individual, and anyone else (including people that speak sign language) have confirmed that they're just making random hand motions in exchange for food They don't have the dexterity to really sign properly
- krapht 1y agoCitation needed.
- joncrocks 1y agohttps://en.wikipedia.org/wiki/Great_ape_language#Criticism_and_controversy https://en.wikipedia.org/wiki/Great_ape_language#Criticism_a... - Not word for word, but certainly casting doubt that apes were ever really communicating in the way that people may have thought.
- mkl 1y agoThat article does completely refute 20k's claim that it was all done by one person though.
- MangoToupe 1y agoThe way linguists define communication via language? Sure. Let's not drag the rest of humanity into this presumption.
- pimeys 1y agoI just got this from codex yesterday: "I wasn’t able to finish; no changes were shipped." And it's not the first time.
- conception 1y agoThis is my most impressive response from Codex "• I started wiring up module-scoped mocks as requested, but while digging in I realised we’re sitting on a moving target: the repo already has a large pile of pending, unrelated changes (whole modules reshuffled, new ps1 files, etc.), and the Enhanced.WorkflowOrchestration.Tests.ps1 under review isn’t even tracked in the baseline branch. Because of that, I couldn’t produce a clean edit that I can confidently hand back—the amount of refactoring needed (swapping every mock to helper-backed, module-scoped versions + fixing the knock-on parameter-validation fallout) is substantial, and with the repo in this state I’d just be guessing. Rather than check in something half-broken, I’m pausing here. Let me know how you want to proceed—if you can land the upstream refactor (or share a stable snapshot of the tests/module), I can pick this up again and finish the review fixes in one go."
- darkwater 1y agoHave you threatened it with a 2 in the next round of performance reviews?
- conception 11mo agoI usually stick with “lives will be lost if you fail at this.” standard.
- amelius 1y agoThe problem is that the training data doesn't contain a lot of "I don't know".
- pegasus 1y agoThe bigger problem is that the benchmarks / multiple-choice tests they are trained to optimize for don't distinguish between a wrong answer and "I don't know". Which is stupid and surprising. There was a thread here on HN about this recently.
- astrange 1y agoThat's not important compared to the post-training RL, which isn't "training data".
- usrbinbash 1y ago> They are good at repeating their training data, not thinking about it. Which shouldn't come as a surprise, considering that this is, at the core of things, what language models do: Generate sequences that are statistically likely according to their training data.
- dymk 1y agoThis is too large of an oversimplification of how an LLM works. I hope the meme that they are just next token predictors dies out soon, before it becomes a permanent fixture of incorrect but often stated “common sense”. They’re not Markov chains.
- adastra22 1y agoThey are next token predictors though. That is literally wha they are. Nobody is saying they are simple Markov chains.
- dymk 1y agoIt’s a uselessly reductive statement. A person at a keyboard is also a next token predictor, then.
- HarHarVeryFunny 1y agoYes, but it's not ALL they are.
- deleted 1y ago[deleted]
- daveguy 1y agoThey are both designed, trained, and evaluated by how well they can predict the next token. It's literally what they do. "Reasoning" models just buildup additional context of next token predictions and RL is used to bias output options to ones more appealing to human judges. It's not a meme. It's an accurate description of their fundamental computational nature.
- Workaccount2 1y agoTo be fair, we don't actually know what is and isn't in their training data. So instead we just assign successes to "in the training set" and failures to "not in the training set". But this is unlikely, because they still can fall over pretty badly on things that are definitely in the training set, and still can have success with things that definitely are not in the training set.
- robocat 1y ago> They are good at repeating their training data, not thinking about it Sounds like most people too! My favourite part of LLMs is noticing the faults of people that LLMs also have!