3 ms·

(author here) Recents events in ML make me feel about 2/3 vindicated of the claims made in the book. Based on the book's ideas, I began training LLMs based on

(author here) Recents events in ML make me feel about 2/3 vindicated of the claims made in the book. Based on the book's ideas, I began training LLMs based on large corpora in the early 2010s, well before it was "cool". I figured out that LLMs could scale to giga-parameter complexity without overfitting, and that the concepts developed under this training would be reusable for other tasks (I called this the Reusability Hypothesis, to emphasize that it was deeply non-obvious; other terms like "self-supervision" are more common in the literature). I missed on two related points. Technically, I did not think DNNs would scale up forever; I thought that they would hit some barrier, and the engineers would not be able to debug the problem because of the black-box nature of DNNs. Philosophically, I wanted this work to resemble classical empirical science in that the humans involved should achieve a high degree of knowledge relating to the material. In the case of LLMs, I wanted researchers (including myself) to develop understanding of key concepts in linguistics such as syntax, semantics, morphology, etc. This style of research actually worked! I built a statistical parser without using any labelled training data! And I did learn a ton about syntax by building these models. One nice insight was that the PCFG is a bad formalism for grammar; I wrote about this here: https://ozoraresearch.wordpress.com/2017/03/17/chuckling-a-bit-at-microsoft-and-the-pcfg-formalism/ https://ozoraresearch.wordpress.com/2017/03/17/chuckling-a-b... Obviously, I feel into the "Bitter Lesson" trap described by Rich Sutton. The DNNs can scale up, and can improve up their understanding much faster than a group of human researchers can. One funny memory is that in 2013 I went to CVPR and told a bunch of CV researchers that they should give up on modeling P(L|I) - label given image - and just model P(I) instead - the probability of an image. They weren't too happy to hear that. I'm not sure that approach has yet taken over the CV world, but based on the overwhelming success of GPT in the NLP world, I'm sure it's just a matter of time. In hindsight, I regret the emphasis I placed on the keyword "compression". To me, compression is a nice and rigorous way to compare models, with a built-in Occam's principle. But "compression" means many different things to different people. The important idea is that we're modeling very large unlabelled datasets, using the most natural objective metric in this setting. edit: I used the wrong name in reference to the Bitter Lesson idea, here is the essay: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

gwern 3y ago
> I'm not sure that approach has yet taken over the CV world, but based on the overwhelming success of GPT in the NLP world, I'm sure it's just a matter of time. Yeah, iGPT was the writing on the wall there, but CLIP gave cheap non-generative modeling a new lease on life. Contrastive learning sucks in many ways, but it's substantially cheaper: compare the cost of training a CLIP to the cost of training a DALL-E 1. (CLIP itself was originally generative, doing the obvious generation of caption & image separately, but they found it was like 8x cheaper to go full contrastive.) So, everyone flocked into that to avoid paying the Bitter Lesson. However, people increasingly run into the limits of contrastive learning (eg. about half the examples you'll see of DALL-E 2 or Midjourney or SD failing on a prompt are probably due solely to the use of contrastive embeddings) and compute/resources keep piling up, so we'll get to generative-everything in images eventually.
boberoni 3y ago
Hi, Dan! Just want to say thanks for your work on this topic. I really loved your book so I wanted to share it with the HN community. Supervised learning always seemed to rub me the wrong way, both when I learned it in college and when I saw it used in practice in industry. I was led to your book by recent research in self-supervised learning by LeCun et al [1] [2]. Since reading your book, I have been digging into the work by Rissanen [3], Grunwald [4], and Hinton [5], among many others. I'm trying to build up my knowledge so that I can apply it to TinyML [6] (e.g. running a neural network on a microcontroller with 256kb of RAM). In a TinyML context, power usage must be low and labeled data is non-existent. I have a vague intuition of how MDL can be used to guide the engineering constraints of TinyML, and I'm hoping to formalize this in my research. Dan, if you know of any papers or research groups that would be related to this area, I'd love to read more about it. [1] https://ai.meta.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/ https://ai.meta.com/blog/self-supervised-learning-the-dark-m... [2] https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/ https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/ [3] https://link.springer.com/book/10.1007/978-0-387-68812-1 https://link.springer.com/book/10.1007/978-0-387-68812-1 [4] https://mitpress.mit.edu/9780262529631/the-minimum-description-length-principle/ https://mitpress.mit.edu/9780262529631/the-minimum-descripti... [5] https://www.researchgate.net/publication/5920308_To_recognize_shapes_first_learn_to_generate_images https://www.researchgate.net/publication/5920308_To_recogniz... [6] https://sites.google.com/g.harvard.edu/tinyml/home https://sites.google.com/g.harvard.edu/tinyml/home

gwern 3y ago

> I'm not sure that approach has yet taken over the CV world, but based on the overwhelming success of GPT in the NLP world, I'm sure it's just a matter of time. Yeah, iGPT was the writing on the wall there, but CLIP gave cheap non-generative modeling a new lease on life. Contrastive learning sucks in many ways, but it's substantially cheaper: compare the cost of training a CLIP to the cost of training a DALL-E 1. (CLIP itself was originally generative, doing the obvious generation of caption & image separately, but they found it was like 8x cheaper to go full contrastive.) So, everyone flocked into that to avoid paying the Bitter Lesson. However, people increasingly run into the limits of contrastive learning (eg. about half the examples you'll see of DALL-E 2 or Midjourney or SD failing on a prompt are probably due solely to the use of contrastive embeddings) and compute/resources keep piling up, so we'll get to generative-everything in images eventually.

boberoni 3y ago

Hi, Dan! Just want to say thanks for your work on this topic. I really loved your book so I wanted to share it with the HN community. Supervised learning always seemed to rub me the wrong way, both when I learned it in college and when I saw it used in practice in industry. I was led to your book by recent research in self-supervised learning by LeCun et al [1] [2]. Since reading your book, I have been digging into the work by Rissanen [3], Grunwald [4], and Hinton [5], among many others. I'm trying to build up my knowledge so that I can apply it to TinyML [6] (e.g. running a neural network on a microcontroller with 256kb of RAM). In a TinyML context, power usage must be low and labeled data is non-existent. I have a vague intuition of how MDL can be used to guide the engineering constraints of TinyML, and I'm hoping to formalize this in my research. Dan, if you know of any papers or research groups that would be related to this area, I'd love to read more about it. [1] https://ai.meta.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/ https://ai.meta.com/blog/self-supervised-learning-the-dark-m... [2] https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/ https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/ [3] https://link.springer.com/book/10.1007/978-0-387-68812-1 https://link.springer.com/book/10.1007/978-0-387-68812-1 [4] https://mitpress.mit.edu/9780262529631/the-minimum-description-length-principle/ https://mitpress.mit.edu/9780262529631/the-minimum-descripti... [5] https://www.researchgate.net/publication/5920308_To_recognize_shapes_first_learn_to_generate_images https://www.researchgate.net/publication/5920308_To_recogniz... [6] https://sites.google.com/g.harvard.edu/tinyml/home https://sites.google.com/g.harvard.edu/tinyml/home