3 ms·
PDP 2.0 I see. Edit: for those who don't know, Parallel Distributed Processing (PDP) was a subgroup of CogSci researchers who were using a neo-Connectionist ap
by alephnerd 3y ago
PDP 2.0 I see.
Edit: for those who don't know, Parallel Distributed Processing (PDP) was a subgroup of CogSci researchers who were using a neo-Connectionist approach to modeling cognition.
Geoff Hinton was a early member of this group of MIT, UT, UCSD, Stanford, and CMU psychologists, CogScientists, and CS researchers
The core research of the PDP group is what became CNNs, which are what truly enabled LLMs (as most can trace their origins to the work done on BERT [edit 2: BERT is transformer based, not a CNN arch, though there was a CNN-BERT arch from a couple years ago that made a decent splash at ACL])
https://mitpress.mit.edu/9780262680530/parallel-distributed-processing/ https://mitpress.mit.edu/9780262680530/parallel-distributed-...
- bamboozled 3y agoWhat is PDP?
- alephnerd 3y agoParallel Distributed Processing (PDP) was a subgroup of CogSci and Neuroscience researchers who were using a neo-Connectionist approach to modeling cognition. Geoff Hinton was a early member of this group of MIT, UT, UCSD, and CMU psychologists, CogScientists, and CS researchers The core research of the PDP group is what became CNNs, which are what truly enabled LLMs (as most can trace their origins to the work done on BERT) One of Hinton's earliest papers in the space: https://stanford.edu/~jlmcc/papers/PDP/Chapter1.pdf https://stanford.edu/~jlmcc/papers/PDP/Chapter1.pdf
- Sai_ 3y agoUnrelated but I’ve seen and appreciated your posts on some totally unrelated topics - like Vietnam politics and Indian ghost cities and now, Hinton’s history in Compsci. How the hell do you know so many different things and in, what looks to me, considerable depth? Not trolling, genuinely impressed.
- mistrial9 3y agohistorically true but no, CNN was displaced by a second DeepLearning method called Transformers; Transformers enabled LLM. Since Transformers replace CNN, CNN did not truly enable LLM.
- alephnerd 3y agoFair point. It's been several years since I was in the space and I think I muddled some of the innovations (CNNs and BERT specifically). That said, transformers are themselves the result of of the PDP architecture.
- deleted 3y ago[deleted]
- YeGoblynQueenne 3y agoNothing displaced anything. Transformers dominate NLP, CNNs dominate vision. That's the standard thing in deep learning, where a new architecture is proposed every few years that brings about a new revolution in one sub-field of AI research, then performance plateaus and everything stagnates until the Next Big Thing in AI comes around and the hype train choo-choos in another town, another domain, another modality, whatever. I've had the misfortune of starting my MSc (ML and data science) just 2 years after the ImageNet moment and I've lived through all the heights of hype. By my count, in the last 10 years or so, the hype train whistled at MLPs (for speech recog), CNNs, LSTMs, GANs, DQNs, Transformers, and Diffusion models. Note well that each of those architectures still dominates its domain (maybe with the exception of GANs). For instance, nobody seriously considers replacing CNNs with Transformers for vision (oh, people are trying) but nobody makes a big fuss about how beating ImageNet heralds a new age of AGI right around the corner, anymore either. In fact nobody talks much about CNNs today, hence the impression that Transformers "displaced" them: only in the headlines. In all probability, in a few years, just as the LLM hype starts to deflate, the Next Big Thing will arise that will take us one more step closer to AGI, Zeno's paradox-style. /cynic
- mistrial9 3y ago