3 ms·
LLMs themselves are made up of "good old fashioned" components. The public conversation about generative AI seems to assume that it's doing something fundamenta
by davesque 3y ago
LLMs themselves are made up of "good old fashioned" components. The public conversation about generative AI seems to assume that it's doing something fundamentally different than what came before. Of course, LLMs are still just deep NNs trained through back propagation to "classify" strings of text into one of many categories corresponding to each possible token. The only thing they do differently is to compute auto-correlation coefficients in each layer and zero some of them out. And of course they take advantage of massive amounts of training data by self-supervising.
LLMs are only a slight variation on the good old fashioned models that already existed at the time they showed up.
- cjauvin 3y agoWhat they bring in my mind that is deeply surprising is the realization that from such a simple objective function (predicting the next word) could emerge such a wide array of general capabilities. Before LLMs, neural networks accomplishing "narrow" tasks were understood to be powerful, but not in such a general way.
- rdedev 3y agoMy understanding is that at some point the model needs to gain an understanding of how the world works if it needs to know the right set of words that come next to keep reducing perplexity. But I don't think this can scale to AGI but I have nothing to back it up
- famouswaffles 3y agoPredicting the next word as an objective is straightforward, not simple. Think about what it would mean to predict internet scale text accurately. To for instance, know the results of scientific papers before they are uttered. There's a difference between an objective and what you need to do to fulfill it.
- snarkconjecture 3y ago"Good old-fashioned AI" (GOFAI) is a term of art that refers to symbolic AI, in contrast to deep NNs. Maybe we need to rename it if deep learning is already "old fashioned"! (EDIT: to be clear, I have no idea what TechCrunch thinks GOFAI means.)
- davesque 3y agoYeah, that's what I was trying to indicate with the quotations. But thanks for adding the note.
- DonHopkins 3y agoThe AI Treadmill: Looking backward: GOFAI (Good Old Fashioned AI) becomes BOFAI (Bad Old Fashioned AI), then later becomes ROFAI (Retro Old Fashioned AI) then eventually AOFAI (Antique Old Fashioned AI)... Looking forward: AI (Artificial Intelligence) progresses to AGI (Artificial General Intelligence) progresses to AGIC (Artificial General Intelligence Consciousness) progresses to MAGIC (Miraculous Artificial General Intelligence Consciousness)...