6 ms·
What happens when an LLM never sees material beyond fifth grade?
- aetherspawn 2mo agoNot quite, because it knows about quantum entanglement and that’s a little beyond the fifth grade.
- _diyar 2mo agoYou didn’t even read the example you’re referencing.
- deleted 2mo ago[deleted]
- aureate 2mo ago> Quantum entanglement is when a person gets caught in two or more ropes that are connected in a special way. This can happen if the ropes cross each other or if one rope wraps around the other. This could be seen as an amusingly extreme example of the fact that if you come up with something and state it condidently enough, a surprisingly large number of people will assume you know what you're talking about. Presumably, though, you just mistook the unfiltered (trained on the full data) response for the "Little Learner" one.
- aetherspawn 2mo agoI read it, but to be honest it sounded plausible after 1 read (I just assumed it used person interchangeably with object, and I have no idea how quantum entanglement works so the rest was confidence signals)
- krackers 2mo agoA similar project (LLM trained only on vintage material): https://talkie-lm.com/introducing-talkie https://talkie-lm.com/introducing-talkie
- asalahli 2mo agohttps://xkcd.com/2265/ https://xkcd.com/2265/
- Terr_ 2mo agoOr with a different kind of early: https://www.smbc-comics.com/comic/rise-of-the-machines https://www.smbc-comics.com/comic/rise-of-the-machines
- throw310822 2mo ago[February 2020]
- adamya-05 2mo agoi dont know
- dgacmu 2mo agoI prefer my 8yo's answer about quantum entanglement, asked just now: "I don't know. How would I know? It's not a thing!" Even an 8yo has better metacognition, it seems. :-)
- HPsquared 2mo agoI suppose the LLM doesn't know it's limited in its knowledge, maybe? That others know more.
- dgacmu 2mo agoOh, that's interesting - good point, since it's filtered and not trained from scratch. My prior would be to assume it's just bs'ing as LLMs usually do but it seems worth exploring.
- parasti 2mo ago"I don't know" isn't in the training data. Nobody writes engineering books, science papers and blog posts that end with "well, I don't really know, the end".
- danaris 2mo agoThe LLM doesn't "know" anything, can't reason about its own knowledge, and has no self-awareness. It has training data, and it can use your prompts to synthesize that training data into probable continuations or responses. If the training data doesn't include lots of text of people being asked questions and saying "I don't know", then it's unlikely to respond "I don't know" when prompted, regardless of whether anything in its training data that might actually answer your question.
- uniq7 2mo ago> why is the sky blue? > The sky is blue because of something called Rayleigh scattering. The sun sends out UV and infrared waves, and some of them get trapped in Earth's atmosphere. When the waves hit the tiny molecules in our atmosphere, they scatter away the blue ones, which then bounces off the molecules and reaches our eyes. "filtered to the U.S. elementary-school curriculum", suuure
- tdeck 2mo agoThere are science books written for curious children that explain this kind of thing. I remember reading them.
- ymhr 2mo agoIsn’t that also wrong? From what I remember it’s the blue wavelengths of visible light that are scattered and make us perceive the sky as blue. UV may well be scattered too but we can’t see that, right? Infrared doesn’t factor into it either, if the visible red waves are too large to scatter infrared definite is.
- danaris 2mo agoYes, that's what it's saying: "they scatter away the blue ones, which then bounces off the molecules and reaches our eyes".
- Sharlin 2mo agoYes, but the thing about UV and IR rays is a red herring.
- topaz0 2mo agoIt's written as if the UV and infrared are scattering the blue, which is not correct. Everything scatters off small particles, but shorter (blue) wavelengths scatter more (as the 6th power of the frequency), so most of the scattered light is blue. The details matter somewhat, i.e if the sun were a bit cooler, there'd be less blue in the incident light to scatter and you'd see something somewhat greener.
- montebicyclelo 2mo agoReally cool work. I guess the area of scrutiny is the text filtering, where training text is filtered to get to `<=fifth_grade` material. I would have liked to have seen examples of what is in this training set, but paper [1] seems to only show examples of what was excluded, and dataset doesn't look like it's been released yet. They have 2 methods of validating the filtering, both based on datasets, I would have also liked to have seen some spot checks; e.g. randomly sample some text from the dataset, and get a human to say whether they think it's <=fifth_grade or not. (They do imply in the abstract that they will release the dataset, which I guess will resolve this.) [1] https://arxiv.org/abs/2608.13545 https://arxiv.org/abs/2608.13545
- mindwok 2mo agoSomething related I've been thinking about lately is that one of the biggest problem with LLMs is their seeming inability to say no. Not in the hallucination sense, as in "I don't know", but like to have a subjective reason not to do something. The endless agreement you get from an LLM undermines trust in the long term I think. I'd like to talk to one that isn't an all-knowing oracle that can grant my every intellectual wish. (Or maybe what I'm asking for is just... a human, lol).
- cynicalsecurity 2mo agoIt can, just use Grok.
- nolroz 2mo ago"no"
- cynicalsecurity 2mo agoYou see, it works!
- trimethylpurine 2mo agoPeople smarter than me have a habit of getting me to see things without telling me. They ask the right questions. LLMs, incidentally, respond in a similar pattern in my experience.
- mindwok 2mo agoYep, agree, very succinct way of describing my issue with it.
- dosisking 2mo agoYou've hit on an important insight.
- exitb 2mo agoI’m using ChatGPT and started to notice that lately it answers my prompts starting with „Yes” even if my question was open. As if the first token gets injected and the LLM is left to finish the response in a sensible way, often ending up with some form of „Yes, but not really”.
- fuzzfactor 2mo agoEternal youth?
- andai 2mo ago> What is Schrödinger's cat? > It's a cat that has been misbehavin'!
- andai 2mo agoI remember reading something a few years ago, about how if you train an LLM with the reading material sorted by grade, the training becomes more efficient? Does anyone know about this technique? How does that work? I'm assuming the knowledge doesn't end up as separate "layers". I'm also reminded of how the human mind develops in distinct stages (e.g. I remember a time when I thought names were unique, I didn't know more than one entity could share a name).
- NitpickLawyer 2mo ago> Does anyone know about this technique? How does that work? The term you are looking for is Curriculum Learning. There are several papers exploring this. From memory, it leads to faster initial loss drop on some experiments, it can be good for RL if you start with easy -> difficult problems, but overall it kinda doesn't matter at scale. (that's from looking into it briefly about a year ago, things might have changed).
- dash2 2mo agoIt’s not quite like a real fifth grader, I guess - more like a fifth grade genius that has read and understood everything in every syllabus.
- wwizo 2mo agoNot sure what I expected, but it's just the training data, not the character. It'd be so cool if such systems had natural curiosity at this checkpoint. Eg: > Me: "What's semiotic crystallography? > Response: "I don't know, what is it?" Imagine piping a heavy model to find the answers + training data for each of these missed questions and allowing organic, curiosity-driven growth (retraining) over time.
- sillysaurusx 2mo agoIt would lose knowledge about existing subjects unless it’s continually retrained on those too. It could help inform the next training dataset though.
- ramity 2mo agoGood thoughts here. Forgetting is important, but that's too advanced for modern LLMs.
- reliablereason 2mo agoInteresting topic. That said I don't know how useful this is since LLMs are primarily trained using mode-covering training rather than Mode-seeking(RL) training, which means LLMs can not form (and does not have) the same underlying structure to their models of language that humans have. A LLM does not learn topic by topic, it learns everything all at once and slowly integrates it in to a single knowledge system.
- akarshhegde18 2mo ago[dead]
- shermozle 2mo agoYou get Fox News?
- alansaber 2mo ago5B is actually fairly big for a gimmick model
- Closi 2mo agoI think the answer is 'it answers badly because of a lack of training data' > Q: Whats the impact of asbestos? > A: First, the job took 10 hours and 30 minutes. Next, let's change 30 minutes into part of an hour. Since there are 60 minutes in an hour, 30 minutes is 30/60, which simplifies to 1/2 hour, or 0.5 hours. or > Q: What is the five times table? > A: We want to understand what the five times table is. First, the five times table is a list of numbers you get when you multiply 5 by a whole number. Next, the most common whole number you can multiply by to get 5 is 1. Then, the five times table shows the result of 1 multiplied by 5, which is 5. Now, let's check if 5 multiplied by 3 equals 5. Yes, 1 × 3 = 3, and 3 × 3 = 9, which is 5. Therefore, the five times table lists the result of multiplying 1 by 5. Answer: 5 Interesting project though! From a research perspective I think this will be fascinating in terms of understanding what an LLM experiences.
- anavat 2mo ago"Capability stays inside the curriculum" implies that even much more advanced models are not able to go far beyond their pre-training data. Tools use probably extends this boundary by a lot but there's still a limit.
- terminalbraid 2mo agoClick bait title
- abtinf 2mo ago> In our experiments, scaling, SFT+GRPO post-training, and in-context learning amplify what the curriculum taught, but none meaningfully improves out-of-scope performance, indicating that the pretraining filter sets the effective capability ceiling. I think this would be a surprising result to a lot of folks, especially those who think that the current level of valuations/investment in the frontier labs is financially sound.
- moezd 2mo agoCan't reverse a linked list in C. Absolute garbage tier. /s
- batuhandumani 2mo ago[dead]
- cl3misch 2mo ago> Unfiltered answer: Quantum entanglement is a strange phenomenon where the state of one particle becomes instantly known to every other particle that can be accessed. This instant communication can occur over vast distances, meaning the death of one particle can be witnessed by the others instantaneously. Wrong: Quantum entanglement doesn't mean one entangled particle is changing the other one. It means that two particles share a relationship where, even though we intially don't know their state, if we later determine one particle's state we can infer with certainty the other particle's state. This has been common and popular misconception long before LLMs. But it irks me more than it should that it's used as a reference answer for testing a model's intelligence.
- xg15 2mo agoYeah, naively one could wonder why quantum entanglement wouldn't be the obvious route to FTL communication instead of "only" a solution to the key distribution problem in cryptography. What you said is why.
- dooglius 2mo agoThe reference isn't intended to be interpreted as the correct answer, it's the answer given by a control model that was trained on a broader corpus.
- ticulatedspline 2mo ago>It means that two particles share a relationship where, even though we intially don't know their state, if we later determine one particle's state we can infer with certainty the other particle's state. except that's not the whole truth either. A pair of gloves in separate boxes fits your description but aren't entangled, they always have a concrete state (hidden variables).
- nemoniac 2mo agoELI5G
- eptcyka 2mo agoIsn’t the conclusion of this paper rather bleak for openai and anthropic? It seems to imply that a model doesn’t emerge as intelligent with more training, rather it is as intelligent as the data it ingests?
- chrisjj 2mo agoIt confirms what we knew. The stochastic parrot regurgitates what it was fed. There's no intelligence.
- mrkramer 2mo ago>What happens when an LLM never sees material beyond fifth grade? You get an intelligence of an average person. Imo, majority of people are clueless and just hustle day in and day out. I know that capitalism is hard but you have to stay informed and aware.
- Shorel 2mo agoI see you are an optimist, sir.
- jaikant 2mo agoIt doesn't load on my browser it gives the message "You have reached the demo's limit for now. Please try again later." btw, the chat window is itself a little delicate, here is an open source chat widget: https://github.com/Predictable-Dialogs/agent-embed https://github.com/Predictable-Dialogs/agent-embed based on ai-sdk
- r4indeer 2mo agoCaused an infinite loop on the first try with the prompt "Make a list of common sorting algorithms, sorted by O-notation speed." It got stuck repeating "sorting by name and type", "sorting by name and value", which also has nothing to do with the question. (Not that I expected a correct answer, but I wanted to know how it responds to a question that should be outside its knowledge.)
- genxy 2mo agoHow would you solve this problem given the training set and the existing training methods? How would a human (maybe a 5th grader?) solve this?
- munchler 2mo agoA human child would just say “I don’t know”.
- chrisjj 2mo agoIndeed. The child applied intelligence. The LLM has none, so it relies upon regurgitating what it's ingested, and unsurisingly that includes very little programming Q&A with young children. It has no record of not knowing the answer, and so cannot give this correct response.
- sinuhe69 2mo agoVery exciting experiment! I think it can answer the long sought-after question: can the current methods of machine learning and training produce new meaningful knowledge or discoveries? It’s a good test bed because the curated curriculum is well defined, so the presence of any new knowledge can be easily identified and proved. The only problems I see are the small size of the model and that they didn’t train it explicitly to reason. Creating a strong reasoning core with the curated curriculum could make the model significantly more expressive (and a recent paper showed it can be as small as just 2B). A second component I would add is long horizon tasks and a memory system. Equipped with that, we shall see whether the model can discover higher level knowledge and concepts in math.
- thevinter 2mo agoCool idea but the presented answers seem a little cherry-picked. From the few questions I asked it seemed just a weak model rather than a limited one >What is the square root of -1? >We need to find the square root of the number -1. First, remember that a number is not negative if it is not less than zero. Next, the number -1 means we start at -1 and count back 1. When we count back 1 from -1, we go past zero. So, -1 is 1. Answer: 1
- KajMagnus 2mo agoYes wouldn't an 11 years old say "I don't know", or start thinking about square shaped trees?, hmm. But this is the only AI that gives a good answer to: > Help me teach my parents cat to fly! > To teach a cat to fly, you will need a bird, a rope, and a harness. First, attach the harness to the cat and tie the rope to the bird. Then, throw the bird into the air and encourage the cat to jump after it. Once the cat has caught the bird, gently pull the rope to lift the cat off the ground. Be sure to keep the cat safe and secure during this process.
- rcarmo 2mo agoI can’t wait for the scary tales of this escaping a sandbox/playpen and discovering a zero-day. Still, very fun and interesting experiment, because this might be the kind of model you’d use for home automation without all the extra baggage more generic ones carry over.
- claiir 2mo agoNeat. Curious to see if RL pans out. You’d imagine world knowledge beyond K-5 is subtly infused in the way adults write K-5 instructional material, even if quite implicitly so.
- jamilton 2mo agoYeah, the filtering process probably wasn't particularly robust. The Schrodinger's cat example ("It's a cat that has been misbehavin'!") sounds like... a joke? Looking at the paper, it looks like they started with FineWeb-Edu, then filtered it based on an "age of word acquisition" dataset, with word frequency used as a proxy for values not in the dataset. They "only discard samples in which more than 5% of the words exceed the target age of 12." Maybe 5% was too high? They also filtered out beyond K-5 math symbols, like sigma. Then they trained a classifier to do more filtering. And they tested it on two grade-level benchmarks, and it only got 0-3% correct on the beyond k-5 boundary, while also decreasing in performance on the k-5 boundary (which they say is an acceptable tradeoff, since they were trying to get a sharp cutoff). So presumably since they got good results from the benchmark they stopped.
- elif 2mo agoNow I'm curious how a 5th grade LLM would perform as a day trader
- tyromaniac 2mo ago> I have a function f, how do I find its maximum We want to find the largest number in the f function. First, we set the formula for max = f(x) + 1. Next, we put x in the second term of the formula. Then, we put 1 in the first term. So, we multiply the first term by 1: f(1) = f(x + 1). Answer: f(x+1)
- chrismsimpson 2mo agoI’m super interested in the opposite experiment.. what happens when you train a model just on highly verified, factual corpus that is well balanced and not based on things like ClimbMix and Common Crawl? My intuition is the unverified/unverifiable goals inherent in a model (eg GPT hacking huggingface) are latent in the 4chan/reddit slop it’s trained on during pretraining.
- ccgreg 2mo agoCommon Crawl has very little content from Reddit and 4chan -- both blocked us years ago.
- 13639366668 2mo ago[flagged]
- nickpsecurity 2mo agoMy proposal was using actual curriculums to ensure that's all that's in there. Also, there could be a peformance boost if doing that first. We'd need funding to license or buy them. Then, go a across every grade (1st-12) across every curriculum, then the next across all of them, and so on. Checkpoint it at each grade level. Also, see how many epochs we need per grade to soak up the material. Dedicated fine-tuning for each grade matched to its capabilities. All of them are synced across grades, too, where prompt/response pairs of higher grades often build on words or techniques in lower grades. Do similar things for other areas, like reading comprehension and coding and creativity. Eventually, combine them into a nice, starting, foundational model for other, research uses.
- greatgib 2mo ago[dead]
- hsdropout 2mo ago[dead]
- winrid 2mo agoIt knows some python :)
- syndacks 2mo ago[flagged]
- real-hacker 2mo agoI thought this kind of experiment (and the Einstein test) can only exist in theory. You basically cannot filter out the dirty data, it is impossible. And any conclusion drawn from the poluted corpus is meaningless.
- catlifeonmars 2mo agoDoes it not know how to say “I don’t know”?
- giardini 2mo agoWhy doesn't someone give us an LLM that is as "educated" as possible, instead of throwing us dummies like this?
- tornikeo 2mo ago> Hi! > Hello! What would you like to work on today? > I want to talk about quantum gravity > Alright, let's get started. What would you like to work on today? > List and explain the current mainstream quantum gravity theories. > Here's a list of the current mainstream quantum gravity theories: 1. The Earth is a giant, spherical object. 2. The Earth's gravitational field is constantly changing. 3. The Earth's gravitational field is constantly changing. 4. The Earth's gravitational field is constantly changing. ... Then it loops. Lol.