3 ms·
We don't really know why language works with humans, either. If you raise a baby from birth, you kind of observe how it is learning language, but the process is
by trollbridge 5mo ago
We don't really know why language works with humans, either. If you raise a baby from birth, you kind of observe how it is learning language, but the process is also rather mysterious. My eldest son's first word was to actually imitate a cow mooing, and then after that to imitate a motor noise of a tractor or truck. And then after that a meow. (His first complete sentence was "King Graham fell"...)
My next child took a completely different path to language, including skipping all the non-verbal imitations.
And then at some point, you just suddenly can two-way communicate with them when you couldn't before, and then after that, they can engage in reasoning.
- jumploops 5mo agoCompletely agree! It’s interesting to me how similar attempting to understand LLMs is to neuroscience. “When we turn this bit off, this other thing happens… if we change these weights the Eiffel Tower is now in Rome” We’re basically just probing around and trying to reverse engineer an emergent system. To your point, this system may be quite different from model to model (human to human) although some similarities likely occur. The comment I was responding to tried to belittle the OP’s understanding of transformers, by mentioning that running an LLM at scale is much harder than the simple white board diagram. My point was simply that we don’t know why they work, and all the extra optimizations isn’t the “thing” that makes it emergent. Simply scaling the “GPT” is good enough to see it, so the OP’s awe should stand. (On a side note, what other architectures can we scale to find similar emergent behavior?)
- trollbridge 5mo agoComputer vision ends up displaying emergent behaviour. It just "figures out" things.
- galaxyLogic 5mo agoIsn't the LLM simply predicting what should be the next sentences after user's input, using its algorithm and data it has exatrcted from existing texts on the internet. The algorithm that does that could have many different designs, some better some worse for the purpose of predicting what output makes most sense next? So what is it that we don't understand about why theyr work? The algorithm? We have the code. Why the specific algorithm makes such good predictions? I see it as a generalization of trying to predict who wins Kentucky Derby.
- ai_slop_hater 5mo agoHuman brain capabilities are truly amazing, imagine if people didn’t treat their children as if they are stupid and didn’t constantly lie to them, because kids are stupid right, they wouldn’t understand. What heights could be reached.
- baq 5mo agoWe don’t treat children like they’re stupid, we treat children like they’re children. A stupid adult is treated very differently than any child. Adults are expected to have their world models approximately correct in terms of physical environment so they won’t accidentally kill themselves by falling off a cliff; then there are the social norms which adults are expected to conform to so everyone is kinda predictable to everyone else so adults don’t kill each other too often over food or mates. Understanding of neither is expected from children.
- ai_slop_hater 5mo agoYou may have been raised properly since you don’t get what I mean. I really envy kids with “Chinese parents” that had them learn math early on and not some bullshit like that if you put your tooth under your pillow, then a tooth fairy will come.
- mejutoco 5mo agoI think those 2 are orthogonal. Math still works with Santa or the tooth fairy.
- ai_slop_hater 5mo agoMaybe math works but critical thinking doesn’t. There are people who have lived for many decades without ever questioning insane b.s. they were taught as kids.
- beezlewax 5mo agoIt is possible to have learned both things you know.