3 ms·
What they bring in my mind that is deeply surprising is the realization that from such a simple objective function (predicting the next word) could emerge such
by cjauvin 3y ago
What they bring in my mind that is deeply surprising is the realization that from such a simple objective function (predicting the next word) could emerge such a wide array of general capabilities. Before LLMs, neural networks accomplishing "narrow" tasks were understood to be powerful, but not in such a general way.
- rdedev 3y agoMy understanding is that at some point the model needs to gain an understanding of how the world works if it needs to know the right set of words that come next to keep reducing perplexity. But I don't think this can scale to AGI but I have nothing to back it up
- famouswaffles 3y agoPredicting the next word as an objective is straightforward, not simple. Think about what it would mean to predict internet scale text accurately. To for instance, know the results of scientific papers before they are uttered. There's a difference between an objective and what you need to do to fulfill it.