4 ms·

Has this been rigorously considered or is it just vibes? Seems preemptive to presume that unveiling more structure wouldn't give understanding meaning and inten

• famouswaffles 4y ago
  It's obviously what's happening. It's why: 1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus. 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English. 3. LLMs trained on code demonstrably reason better. It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level. Part of the breakthrough with LLMs has precisely been about encoding nothing at all about language and letting the transformer figure it out from scratch. We don't know what is necessary for learning language. Whatever structure we would add at our current level of understanding would far more likely hamper progress than anything. It would also be extremely surface level which LLMs go far beyond.