4 ms·
Would love to see an architecture that learned more like humans. Start with just imitating one letter, then a few more, than some syllables, then full words, th
by nico 1y ago
Would love to see an architecture that learned more like humans. Start with just imitating one letter, then a few more, than some syllables, then full words, then sentences, etc. Progressively adding on top of previous knowledge
Also, it’s interesting that one of the big goals/measures of models is their capacity to “generalize”, but the training methods optimize for loss/accuracy, and only after training test for generalization to validate
Are there training methods/curriculums that explicitly maximize generalization?
- serced 1y agoYes, I also wonder about this! Progress from children books to scientific papers etc. Could it learn e.g. language structure faster in a pre-training stage? Also somehow one needs to define a proxy to generalization to compute a loss and do backpropagation.
- arbot360 1y agoThis field of study is known as "Curriculum Learning" for your Googling pleasure (or I guess ChatGPT Deep Research now).
- olliepro 1y agoProbably don’t need the name of the field for ChatGPT to get it.
- razodactyl 1y agoI get why this comment was downvoted but I also get where you're coming from - yes, these models are becoming increasingly intelligent at understanding the nuance and where to look without knowing what to begin searching for. But the downside is, you end up digging in the wrong direction if you leave it to a generalist system instead of a professional community in some cases which is counter productive. Getting burnt is a good way to learn not to sometimes though...
- razodactyl 1y agoYeah. This comment is profound to me. The internet works differently with these tools. I haven't used the deep research features much but their ability to hash out concepts and build knowledge or even provide an amplified search experience is something...
- exit 1y ago"an architecture that learned more like humans" i.e. enduring countless generations of evolutionary selection and cross breeding, then fine-tuning a bit? although it could be interesting, i don't think training on progressively complex strings entirely recapitulates this.
- nico 1y agoThat’s a very interesting take. I hadn’t really considered evolution I guess if you really wanted to start from scratch, you could figure out how to evolve the whole system from a single cell or something like that. In some ways neural networks have kind of evolved in that way, assisted by humans. They started with a single perceptron, and have gone all the way to deep learning and convolutional networks I also remember a long time ago studying genetic and evolutionary algorithms, but they were pretty basic in terms of what they could learn and do, compared to modern LLMs Although recently I saw some research in which they were applying essentially genetic algorithms to merge model weights and produce models with new/evolved capabilities
- laterium 1y agoHow much compute does simulating the earth for 4.7 billion years at atomic precision take? Why would that be more efficient than current approaches? Evolutionary algorithms work but are extremely inefficient, we don't have the compute to evolve even a single bacteria, let alone the whole history of the planet so we can arrive at human-like species.
- razodactyl 1y agoIt's this take on the situation which I think needs more emphasis. Whether anyone likes it or not, these systems have co-evolved with us. Hundreds of researchers contributing and just like English for example, it's ever-changing and evolving. Given this trend, it's highly unlikely we won't achieve ASI. It's not like hardware engineers stop innovating or venture capital stops wanting more. There might be a massive dip or even another AI winter but like the last one, eventually it picks up momentum again because there's clearly utility in these systems. I've been coding for 25+ years and only a couple of days ago did it hit me that my profession has changed in a very dramatic way - I'm very critical of AI output, but I can read and comprehend code much quicker than I can write it relative to these systems. Of course, that creates a barrier to holding a system in your head so going slow is something that should be pushed for when appropriate.
- ares623 1y agoIsn’t that what all the hundreds of billions are banking on? “General” intelligence.
- onlyrealcuzzo 1y agoYou don't need general intelligence to make good memes to keep people scrolling through Instagram. You don't need general intelligence to make a decent coding tool like Cursor. You don't need general intelligence to improve SERPs. You don't need general intelligence to sell a subscription for a decent AI assistant. There's tons of value already added without anything general.
- reassess_blind 1y agoThe question is whether, if the models plateau, and "AGI" as it was claimed in the beginning never arrives, if it's enough to justify these ongoing multi-hundred billion dollar deals. I mean, probably, LLMs as they are today are already changing the world. But I do think a lot of the ongoing investment is propped up on the promise of another breakthrough that is looking less likely.
- ares623 1y agoYes but $500B and counting for memes wasn’t what was sold
- malfist 1y agoI remember reading somewhere someone said "the problem with AI is it's a $50b industry pretending its a $10t industry"
- onlyrealcuzzo 1y ago$500B is future projections for total spending (a lot of that decently far into the future). The revenues are already in the high tens of billions per year. Models will get better from here, especially on the low end. Costs will eventually approach peanuts for current capabilities. Given enough time, this will pay for existing investments. If growth slows, future spending will slow as well.
- zer00eyz 1y ago"Would love to see an architecture that learned" Would be a far more accurate statement. Training != Learning.
- rokobobo 1y agoDo you have an example of an algorithm that learns, rather than is trained/trains itself? I don’t really see the boundary between the two concepts.
- zer00eyz 1y agoIf we make some massive physics breakthrough tommrow is an LLM going to be able to fully integrate that into its current data set? Or will we need to produce a host of documents and (re)train a new one in order for the concept to be deeply integrated. This distinction is subtle but lost on many who think that our current path will get us to AGI... That isn't to say we haven't created a meaningful tool but the sooner we get candid and realistic about what it is and how it works the sooner we can get down to the business of building practical applications with it. (And as an aside scaling it, something we arent doing well with now).
- llbbdd 1y agoWhat is the subtle distinction? I'm "many" and it's not clear at all here. If we had some massive physics breakthrough, the LLM needs to be tought about it, but so do people. Teaching people about it would involve producing a host of documents in some format but that's also true of teaching people. Training and learning here seem to be opposite ends of the same verb no matter the medium, but I'm open to being enlightened.
- bjt 1y agoNot sure exactly what the parent comment intended, but it does seem to me that it's harder for an LLM to undergo a paradigm shift than for humans. If some new scientific result disproves something that's been stated in a whole bunch of papers, how does the model know that all those old papers are wrong? Do we withhold all those old papers in the next training run, or apply a super heavy weight somehow to the new one, or just throw them all in the hopper and hope for the best?
- fooker 1y agoThere's an interesting question here. Would a single human/entity learn more in ..say.. three million years or would short lived ones evolving over three million years and then ~20 years of education learn more? The current AI tech cycle is focusing on the first, but we don't really know if there are benefits of both. There's no obvious way to combine these yet.
- razodactyl 1y agoOpinion: a lot can change over such a span of time and knowledge goes in and out of relevance - I think the natural progression of models shrinking in parameter count goes to show it's better to know how to use knowledge than to attempt to remember everything. That said, optimising for capability of maximal learning seems to be a natural occurrence in nature. I think the non-obvious emergent effects are something to look into. Culling bad models in favour of the A/B version and check pointing is a kind of combination of the two and the feedback loop of models trained on new snapshots of Internet data that are written with humans and AI. There's an unintended long-form training loop which I think is going to get weirder as time goes on. The wave of models being able to manipulate Cursor / Windsurf etc., being trained to be smarter and more efficient at this and then being retrained for other purposes, even though the model is deleted, the pattern of data can be saved and trained into more advanced models over time.
- weregiraffe 1y agoWould like to see a car that moved like a horse.
- dingnuts 1y agoyeah me too that would be fucking awesome, are you kidding?
- whyagain2025 1y agoTechnically internal combustion engine has piston moving like horse legs.