3 ms·
We train them to predict words, but that doesn't mean they have to predict them in a naive statistical way. Imagine you gave a human a physics textbook and told
by wongarsu 2y ago
We train them to predict words, but that doesn't mean they have to predict them in a naive statistical way. Imagine you gave a human a physics textbook and told them that in a month they have a test that consists of completing sentences from the book. Who do you think would do better: the one who tries to remember every sentence, or the one who understands the book's contents? In LLMs we "just" do back propagation and let the model figure out how to do the prediction, and larger models develop something that looks and feels like a degree of understanding and reasoning (incomplete as it may be).
Case in point, GPT4o's answer to the above question is:
"Inserting two gherkins in your nostrils will not enable you to fly. Flight, as we generally understand it, is achieved through specific principles of physics and engineering, such as those utilized by birds, airplanes, and other flying devices. These principles typically involve lift, thrust, weight, and drag. Gherkins, unfortunately, do not possess any properties that could facilitate flight in a human. If you're interested in flying, consider looking into aviation technology or learning to pilot an aircraft!"
- semking 2y agoYour example of Memory vs Understanding is crazy... it gave me a counterintuitive and insightful idea that's amazing —and I'm not making fun of you! :D But in the real world, what is the added value of back propagation beyond the obvious examples? There's no creativity whatsoever... we're just working with what already exists.
- wongarsu 2y agoThe biggest added value is that it works. Before LLMs we tried all kinds of stuff, including purely statistical methods, and anything more complex than scripted responses was incredibly hard to keep coherent for more than five sentences. GPT2 had similar issues, but modern LLMs are incredibly good. But ultimately the goal are models that have a decent world model and can reason about it. A model that you can ask for a python program that does something completely novel, and you get a working result. We are somewhere on that road, far beyond what we were able to do with statistical methods but still not nearly as good as we wish.
- semking 2y agoIs works, but at what cost? Content becoming a commodity?