4 ms·
Your example of Memory vs Understanding is crazy... it gave me a counterintuitive and insightful idea that's amazing —and I'm not making fun of you! :D But in
by semking 2y ago
Your example of Memory vs Understanding is crazy... it gave me a counterintuitive and insightful idea that's amazing —and I'm not making fun of you! :D
But in the real world, what is the added value of back propagation beyond the obvious examples?
There's no creativity whatsoever... we're just working with what already exists.
- wongarsu 2y agoThe biggest added value is that it works. Before LLMs we tried all kinds of stuff, including purely statistical methods, and anything more complex than scripted responses was incredibly hard to keep coherent for more than five sentences. GPT2 had similar issues, but modern LLMs are incredibly good. But ultimately the goal are models that have a decent world model and can reason about it. A model that you can ask for a python program that does something completely novel, and you get a working result. We are somewhere on that road, far beyond what we were able to do with statistical methods but still not nearly as good as we wish.
- semking 2y agoIs works, but at what cost? Content becoming a commodity?