6 ms·

An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. The model finds a structure to the

An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. The model finds a structure to the problem you give it in a largely language-agnostic way that benefits from training on every language (these things are direct descendants of Google Translate), and even non-programming knowledge - the structure of your task might resemble an ancient Chinese poem that influences the model's response, for example. That structure is considered a form of compression, as some fascinating and illuminating recent 3blue1brown videos get into - a common pattern in Haskell or FORTRAN and a situation described in an ancient Chinese poem may all compress to something quite similar to your task, thus when the model compresses the idea of your task it immediately draws from those ideas. There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same. Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.

znnajdla 2mo ago
Thanks for putting this so well. The mathematical evidence for this “general intelligence underlying everything” is the “universal geometry of embeddings” paper. Fascinating read. Or as the ancient philosophers used to say, the one who knows God knows everything.
- jan_m_savage 2mo ago
  [dead]
- bigfishrunning 2mo ago
  "the one who knows god knows everything" could also be a reference to the fact that god is unknowable, and therefore so is everything...
- genxy 2mo ago
  Is this the paper? Harnessing the Universal Geometry of Embeddings https://arxiv.org/abs/2505.12540 https://arxiv.org/abs/2505.12540
kennywinker 2mo ago
This makes a ton of sense, and seems like it's coming from somebody who understands LLMs better than I do so I will definitely take your word for it - but it doesn't totally track with my experience of running small local models either. In my experience some models are better at language X and others better at language Y - and all of them are better at language X and Y than language Z-I-just-made-up. I take that to mean there is some relationship between training data and skill. Maybe not enough to make pluggable modules, tho... at least not with the architectures we're currently using.
redmacaw 2mo ago
could you share the 3blue1brown videos you're referring to?
- pixelatedindex 2mo ago
  https://youtu.be/l6DKRf-fAAM https://youtu.be/l6DKRf-fAAM maybe? Title is “Reinventing Entropy” And of course the neural network series.
  - jimmaswell 2mo ago
    Yes, that's the one
marton78 2mo ago
By pattern matching that SpongeBob episode to the case at hand you provided an example of what you had just explained. Nice. So meta.
weitendorf 2mo ago
This is an oversimplification: more data makes models smarter ceteris paribus, but mostly only because auto-regressive training (where most of the general knowledge comes from) is essentially compressing information that can be recalled later if it’s useful (or not recalled). Obviously there are differences in kind within “more data” too, you would much rather have all the books and blog posts in the world than all the fanfiction. Yes, all three together would be even better. But it wouldn’t be if you had 100x more fanfiction, mostly synthetic, generated during RL to teach a model to be better at writing fan fiction. There are real limits to the amount of knowledge you can cram into fixed-size (downstream of hardware availability) weights. For a period scaling with data was basically “free” because we had the Internet and all the books/media that humans had already created; the data was accessible and limited (at least, the parts we think models should know about) enough and top-hardware big enough that we could basically compress the whole thing. Post-training/RL are making this obsolete because they’re more about skill/capability acquisition rather than knowledge. They can generate much more data (most of it quotidian/useless, ie an agent made a typo in batch 382829) and clearly seem to cause a kind of mode collapse even in the most advanced frontier models. We don’t need to make LLMs forget about SpongeBob SquarePants so they learn more about bash. But if I have a question about SpongeBob SquarePants, I don’t need to hear about load bearing seams prefaced with honest caveats after a model writes 400 lines of bash to look up SpongeBob’s family. And there is probably a lot more SpongeBob knowledge we could put into models if we wanted to: interviews with the creative staff, a SpongeEnv/SpongeHarness modeling how the art/story team work together to create entertaining kids tv, a SpongeBench measuring entertainment value, etc. If a SpongeAgent spends 2000 years in Agent University learning how to Spongemaxx we probably don’t need or want to have it spend another 2000 years writing smoke tests
- jatora 2mo ago
  [flagged]
jiggawatts 2mo ago
People keep forgetting that programming is not just about knowing the target programming language, but also an enormous volume of tacit knowledge: - Understanding of protocols like HTTP. - HTML, JS, CSS, SVG, and everything "web". - Understanding of databases, SQL, etc. - Abstract code architecture patterns. - Understanding the users' requests in English. - Responding in English. - Command line tool usage (agents/harnesses) - Industry-specific knowledge that can be applied. - Frameworks, SDKs, applicable libraries. - Relevant legal requirements. - Etc... I.e.: If I tell a frontier AI that this project is for a "local council in XYZ location" it can immediately figure out that a scalable, globally distributed architecture is not required. It can also figure out that using local time instead of UTC is not only "fine", but even desired. Or that globalization/localization is not required... or.... required if the council is in some place like Belgium or Canada where multiple languages are officially recognised and supported by the government.
- fennecbutt 2mo ago
  Tbf the majority of harnesses for frontier models do not allow the agent to do this (gut instinct) and instead get it to search, and for good reason.
- kennywinker 2mo ago
  Those assumptions are just that - assumptions. "Local council in XYZ location" implies a bunch of things, and each one might be wrong for my specific circumstances. What better way to guide expectations than importing specific knowledge? I.e. if I import the english and catalan modules, then I probably want to localize my site in english and catalan. It would be trivial to have a pre-flight convo with an llm to guide the user thru module choices. "Build a site" -> "ok, describe the purpose" -> "local council in XYZ location" -> "that implies you won't need localization since XYZ has a monolingual government" -> "english and catalan localization please". Right now, you prompt and it builds using assumptions, and we prompt to adjust. I think it would be great to be able to pre-load a set of assumptions.
  - jiggawatts 2mo ago
reaperducer 2mo ago
An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. You just defined a liberal arts education.
chicken-stew 2mo ago
Ah yes, so that’s why if I ask gemini for a simple low-resource trick to cast a float to its binary representation it is reminded of a work of shakespeare and suggests memcpy. Or yesterday, when I asked it for the best low resource approach to select the matching string from a collection of strings for a shell interface it suggested an exhaustive string compare over the entire collection. Because it was thinking of how David Lynch would tackle it.
- kennywinker 2mo ago
  David Lynch would simply refuse to compare strings. Which is how we can tell LLMs haven’t been trained on enough David Lynch - they don’t simply refuse often enough.
flaburgan 2mo ago
>There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same. I think you're missing the point of the article though, which suggests not to mix reasoning capacities with actual knowledge. Sure working with Swift and Python is basically the same, it's programming, with the same concepts etc. Much closer than the ability to drive a car. But the point is, the methods, libs etc are all different and things are changing each time a new version of the language is released. Like you don't need to relearn how to drive a car if you go to the UK but you have to known the wheel is on the right. Knowledge shouldn't be stored in the weights.

znnajdla 2mo ago

Thanks for putting this so well. The mathematical evidence for this “general intelligence underlying everything” is the “universal geometry of embeddings” paper. Fascinating read. Or as the ancient philosophers used to say, the one who knows God knows everything.

jan_m_savage 2mo ago
[dead]
bigfishrunning 2mo ago
"the one who knows god knows everything" could also be a reference to the fact that god is unknowable, and therefore so is everything...
genxy 2mo ago
Is this the paper? Harnessing the Universal Geometry of Embeddings https://arxiv.org/abs/2505.12540 https://arxiv.org/abs/2505.12540

kennywinker 2mo ago

This makes a ton of sense, and seems like it's coming from somebody who understands LLMs better than I do so I will definitely take your word for it - but it doesn't totally track with my experience of running small local models either. In my experience some models are better at language X and others better at language Y - and all of them are better at language X and Y than language Z-I-just-made-up. I take that to mean there is some relationship between training data and skill. Maybe not enough to make pluggable modules, tho... at least not with the architectures we're currently using.

redmacaw 2mo ago

could you share the 3blue1brown videos you're referring to?

pixelatedindex 2mo ago
https://youtu.be/l6DKRf-fAAM https://youtu.be/l6DKRf-fAAM maybe? Title is “Reinventing Entropy” And of course the neural network series.
- jimmaswell 2mo ago
  Yes, that's the one

marton78 2mo ago

By pattern matching that SpongeBob episode to the case at hand you provided an example of what you had just explained. Nice. So meta.

weitendorf 2mo ago

This is an oversimplification: more data makes models smarter ceteris paribus, but mostly only because auto-regressive training (where most of the general knowledge comes from) is essentially compressing information that can be recalled later if it’s useful (or not recalled). Obviously there are differences in kind within “more data” too, you would much rather have all the books and blog posts in the world than all the fanfiction. Yes, all three together would be even better. But it wouldn’t be if you had 100x more fanfiction, mostly synthetic, generated during RL to teach a model to be better at writing fan fiction. There are real limits to the amount of knowledge you can cram into fixed-size (downstream of hardware availability) weights. For a period scaling with data was basically “free” because we had the Internet and all the books/media that humans had already created; the data was accessible and limited (at least, the parts we think models should know about) enough and top-hardware big enough that we could basically compress the whole thing. Post-training/RL are making this obsolete because they’re more about skill/capability acquisition rather than knowledge. They can generate much more data (most of it quotidian/useless, ie an agent made a typo in batch 382829) and clearly seem to cause a kind of mode collapse even in the most advanced frontier models. We don’t need to make LLMs forget about SpongeBob SquarePants so they learn more about bash. But if I have a question about SpongeBob SquarePants, I don’t need to hear about load bearing seams prefaced with honest caveats after a model writes 400 lines of bash to look up SpongeBob’s family. And there is probably a lot more SpongeBob knowledge we could put into models if we wanted to: interviews with the creative staff, a SpongeEnv/SpongeHarness modeling how the art/story team work together to create entertaining kids tv, a SpongeBench measuring entertainment value, etc. If a SpongeAgent spends 2000 years in Agent University learning how to Spongemaxx we probably don’t need or want to have it spend another 2000 years writing smoke tests

jatora 2mo ago
[flagged]

jiggawatts 2mo ago

People keep forgetting that programming is not just about knowing the target programming language, but also an enormous volume of tacit knowledge: - Understanding of protocols like HTTP. - HTML, JS, CSS, SVG, and everything "web". - Understanding of databases, SQL, etc. - Abstract code architecture patterns. - Understanding the users' requests in English. - Responding in English. - Command line tool usage (agents/harnesses) - Industry-specific knowledge that can be applied. - Frameworks, SDKs, applicable libraries. - Relevant legal requirements. - Etc... I.e.: If I tell a frontier AI that this project is for a "local council in XYZ location" it can immediately figure out that a scalable, globally distributed architecture is not required. It can also figure out that using local time instead of UTC is not only "fine", but even desired. Or that globalization/localization is not required... or.... required if the council is in some place like Belgium or Canada where multiple languages are officially recognised and supported by the government.

fennecbutt 2mo ago
Tbf the majority of harnesses for frontier models do not allow the agent to do this (gut instinct) and instead get it to search, and for good reason.
kennywinker 2mo ago
Those assumptions are just that - assumptions. "Local council in XYZ location" implies a bunch of things, and each one might be wrong for my specific circumstances. What better way to guide expectations than importing specific knowledge? I.e. if I import the english and catalan modules, then I probably want to localize my site in english and catalan. It would be trivial to have a pre-flight convo with an llm to guide the user thru module choices. "Build a site" -> "ok, describe the purpose" -> "local council in XYZ location" -> "that implies you won't need localization since XYZ has a monolingual government" -> "english and catalan localization please". Right now, you prompt and it builds using assumptions, and we prompt to adjust. I think it would be great to be able to pre-load a set of assumptions.
- jiggawatts 2mo ago

reaperducer 2mo ago

An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. You just defined a liberal arts education.

chicken-stew 2mo ago

Ah yes, so that’s why if I ask gemini for a simple low-resource trick to cast a float to its binary representation it is reminded of a work of shakespeare and suggests memcpy. Or yesterday, when I asked it for the best low resource approach to select the matching string from a collection of strings for a shell interface it suggested an exhaustive string compare over the entire collection. Because it was thinking of how David Lynch would tackle it.

kennywinker 2mo ago
David Lynch would simply refuse to compare strings. Which is how we can tell LLMs haven’t been trained on enough David Lynch - they don’t simply refuse often enough.

flaburgan 2mo ago

>There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same. I think you're missing the point of the article though, which suggests not to mix reasoning capacities with actual knowledge. Sure working with Swift and Python is basically the same, it's programming, with the same concepts etc. Much closer than the ability to drive a car. But the point is, the methods, libs etc are all different and things are changing each time a new version of the language is released. Like you don't need to relearn how to drive a car if you go to the UK but you have to known the wheel is on the right. Knowledge shouldn't be stored in the weights.