5 ms·

For those who are interested in how Transformers seem more prevalent, please read this thread by Karpathy where he talks about a consolidation in ML: https://t

For those who are interested in how Transformers seem more prevalent, please read this thread by Karpathy where he talks about a consolidation in ML: https://twitter.com/karpathy/status/1468370605229547522 https://twitter.com/karpathy/status/1468370605229547522 And of course one of the early classic papers in the field, as a bonus: https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd05... (The paper is mentioned in the article)

lucidrains 5y ago
if one prefers video, Yannic Kilcher does an excellent explanation of the seminal paper https://www.youtube.com/watch?v=iDulhoQ2pro https://www.youtube.com/watch?v=iDulhoQ2pro
- amrrs 5y ago
  Glad to see you here lucidrains. Truly appreciate your recent open-source contributions and works like big sleep, deep daze. Everyone else check out this https://github.com/lucidrains?tab=repositories https://github.com/lucidrains?tab=repositories
  - lucidrains 5y ago
    Thanks for the kind words, and credit goes to Ryan Murdock for Big Sleep and Deep Daze. I simply packaged it up to spread the usage
  - tomweingarten 5y ago
    +1, lucidrains is practically a one-person ML start-up
algo_trader 5y ago
Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs. [1]https://www.gwern.net/notes/Attention https://www.gwern.net/notes/Attention
- lucidrains 5y ago
  I would recommend Routing Transformer https://github.com/lucidrains/routing-transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pytorch https://github.com/lucidrains/memory-efficient-attention-pyt...
enriquto 5y ago
> the approaches were completely different, often not even ML based. Oh, the horror! Some aproaches were not (gasp!) even ML-based. As someone who was been working all these years in image processing without recourse to much ML stuff, I find this attitude cute and endearing.
- YeGoblynQueenne 5y ago
  Do you have some pointers to interesting research of the kind you are doing? Btw, when we're talking about "non-ML" we meen no-learning, not just no-neural-nets, correct? Thanks in advance.
  - enriquto 5y ago
    > Do you have some pointers to interesting research of the kind you are doing? I don't think my research is particularly interesting for a general public as it is quite niche (low level image processing). Still, we consistently receive industrial funding so it is probably useful to some extent. Some nice journals where we publish our research: Journal of Mathematical Imaging and Vision, SIAM Journal on Imaging Sciences. > when we're talking about "non-ML" we mean no-learning, not just no-neural-nets, correct? Why do you say "just" ? Machine learning and neural networks are independent things. There's plenty of wonderful research about neural networks that has nothing to do with learning.
    - YeGoblynQueenne 5y ago
      Thanks, I'll look up the journals you suggest. By "just" I meant to clarify the use of "ML". It's often used to mean "deep learning" and while I thought you didn't use it that way I wanted to make sure. I normally think of neural nets as a primarily connectionist approach to machine learning (although not necessarily statistical: the first artificial neuron was propositional logic-based). I'm curious to read research that treats them in a different manner. Can you recommend some? Thanks in advance (again)!
      - 5y ago

lucidrains 5y ago

if one prefers video, Yannic Kilcher does an excellent explanation of the seminal paper https://www.youtube.com/watch?v=iDulhoQ2pro https://www.youtube.com/watch?v=iDulhoQ2pro

amrrs 5y ago
Glad to see you here lucidrains. Truly appreciate your recent open-source contributions and works like big sleep, deep daze. Everyone else check out this https://github.com/lucidrains?tab=repositories https://github.com/lucidrains?tab=repositories
- lucidrains 5y ago
  Thanks for the kind words, and credit goes to Ryan Murdock for Big Sleep and Deep Daze. I simply packaged it up to spread the usage
- tomweingarten 5y ago
  +1, lucidrains is practically a one-person ML start-up

algo_trader 5y ago

Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs. [1]https://www.gwern.net/notes/Attention https://www.gwern.net/notes/Attention

lucidrains 5y ago
I would recommend Routing Transformer https://github.com/lucidrains/routing-transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pytorch https://github.com/lucidrains/memory-efficient-attention-pyt...

enriquto 5y ago

> the approaches were completely different, often not even ML based. Oh, the horror! Some aproaches were not (gasp!) even ML-based. As someone who was been working all these years in image processing without recourse to much ML stuff, I find this attitude cute and endearing.

YeGoblynQueenne 5y ago
Do you have some pointers to interesting research of the kind you are doing? Btw, when we're talking about "non-ML" we meen no-learning, not just no-neural-nets, correct? Thanks in advance.
- enriquto 5y ago
  > Do you have some pointers to interesting research of the kind you are doing? I don't think my research is particularly interesting for a general public as it is quite niche (low level image processing). Still, we consistently receive industrial funding so it is probably useful to some extent. Some nice journals where we publish our research: Journal of Mathematical Imaging and Vision, SIAM Journal on Imaging Sciences. > when we're talking about "non-ML" we mean no-learning, not just no-neural-nets, correct? Why do you say "just" ? Machine learning and neural networks are independent things. There's plenty of wonderful research about neural networks that has nothing to do with learning.
  - YeGoblynQueenne 5y ago
    Thanks, I'll look up the journals you suggest. By "just" I meant to clarify the use of "ML". It's often used to mean "deep learning" and while I thought you didn't use it that way I wanted to make sure. I normally think of neural nets as a primarily connectionist approach to machine learning (although not necessarily statistical: the first artificial neuron was propositional logic-based). I'm curious to read research that treats them in a different manner. Can you recommend some? Thanks in advance (again)!
    - 5y ago