3 ms·
GPT-3 is what you get when you take what Chomsky said about language and do the exact opposite at every turn. His first big contribution was arguing that the no
by canjobear 4y ago
GPT-3 is what you get when you take what Chomsky said about language and do the exact opposite at every turn. His first big contribution was arguing that the notion of "probability of a sentence" was useless, because sentences like "colorless green thoughts sleep furiously" have probability zero in a corpus and yet are grammatical. Meanwhile now, the only systems we have ever made that can really use natural language were produced by taking a generic function approximator and making it maximize probabilities of sentences.
- benlivengood 4y agoWhat Chomsky and others never achieved was comprehensive semantics (useful mappings of the instantiations of grammatical language to the real world and to reasoning), because semantics is AI-hard. LLMs are picking up the semantics from the mix of grammar and semantics they train on. They literally minimize the error of producing semantic grammatic sentences, which is the key thing no one in the old days had the computing power to do beyond toy environments. The domain of discourse is the entire world now instead of colored shapes in an empty room, and so semantics about reasoning itself have been trained which yields rudimentary intelligence.
- Baeocystin 4y agoAs an aside, "colorless green ideas sleep furiously" makes for a fun starting prompt in diffusion image generators.