3 ms·
Is it normal not to come away from these with any kind of intuition like “ah, that’s why transformers work so well” Instead I just come away feeling like it’s
by bilsbie 3y ago
Is it normal not to come away from these with any kind of intuition like “ah, that’s why transformers work so well”
Instead I just come away feeling like it’s a Frankenstein of different matrices and statistics.
- hiddencost 3y agoThe reason they're good is that they're not recurrent: you can do attention over all of the sequence at once. Recurrent models had two flaws: (1) vanishing gradient problem (2) not parallelizable
- Geisterde 3y agoIf it looks like Frankenstein, and it often acts like Frankenstein... These things can and are and will be useful, but its not a magic ticket to knowing things. I booted up lamma2 at home, first question "how do I make you smarter?", all it returned were responses like "tell me a book or field to research". Obviously, that doesnt work, and it never suggested importing libraries or running training cycles. Hey, I search a linux problem on brave, and the ai gives me the right answer 75% of the time, nice, but the other 25% it is just regurgiating outdated reddit threads. It has no cognition, it has no bearing on what consequences are, it just pronounces the result of a statistical model, it doesnt know what any of it means.
- romusha 3y agoAnd we already have people zealously argue that LLM has cognition, just like predicted by sci-fi