5 ms·
Understanding Large Language Models – A Transformative Reading List
- auggierose 4y agoYeah, I was looking for a book about this topic, but there doesn't seem to be anything out there except for research articles, it is just too new. When I look at the descriptions of the papers, it sounds incredibly complicated, and at the same time incredibly trivial. Clearly this whole domain is not well enough understood yet to be explained properly. Or maybe there is no interest yet in clear and concise explanations.
- KHRZ 4y agoThe description of the attention mechanism of GPT architectures and a couple of examples can be very brief. Then you have to supply your imagination and realize that the model simply found a whole bunch of very effective such attention measures by itself, which are all computed for every query, and they could be anything from the straight forward examples, or some more clever abstract types of attention. I think we'll need some more descriptions of what the more important attention measures it comes up with really are, to understand better.
- CongoDan 4y agoThat’s how blockchain was a few years back. Now everyone and their uncle has a “how blockchains work” article or video
- corbulo 4y agoAnd yet I feel like I know less at this point than I did then.
- qup 4y ago> Large language models have taken the public attention by storm – no pun intended. As far as I can tell, no pun made, either!
- marginalia_nu 4y ago"Attention" is the would be pun, I think.
- 0z8 4y agoTo make a pun in this domain, attention is all you need!
- xwowsersx 4y agoThe pun is in "attention" because GPT uses "attention" to weigh each input token and comes up with an attention score between whatever token is currently being generated and all the input tokens then it'll take those scores to determine what weight each contributes to the output. Something along those lines... I'm no GPT expert.
- mensetmanusman 4y agoDid anyone predict the unreasonable effectiveness of LLMs? We seemed to have stumbled upon something quite big as humans…
- tough 4y agoOr maybe language just isn't that complex at all? Same with pixels? Dunno
- GeorgeTirebiter 4y agoPerhaps Monica Anderson understood early? https://experimental-epistemology.ai/the-red-pill-of-machine-learning/ https://experimental-epistemology.ai/the-red-pill-of-machine...
- loveparade 4y agoI worked in one of the big labs when the first large models came out and I can pretty confidently say that nobody in the field predicted this. Sure, there were always people who said "let's make models bigger because why not, we have the infra and it'll be a good paper" but nobody expected them to become this good just by being bigger and using more data. The consensus was that they'd hit a ceiling of what they can do much sooner.
- tomr75 4y agoaren't humans trained on massive data sets, why not expect the same for digital intelligence?
- soraki_soladead 4y agoOnly some model architectures continue to get better as you pump in more data. Transformers and their variants have this property more so than prior architectures.
- mensetmanusman 4y agoLarge yes, but millions of orders of magnitude less.