3 ms·

As others have said, the title does not match the contents of the article and the way this has been explained looks like a very opaque and highly abstract rube

As others have said, the title does not match the contents of the article and the way this has been explained looks like a very opaque and highly abstract rube goldberg machine with limited use cases. What does this solve? Even by going with the flawed title, this mostly is a solution in search of a problem.

typon 3y ago
It solves language understanding in a practical sense - a task heretofore considered impossible
- colesantiago 3y ago
  This has been already solved even without LLMs for years: Google Translate, DeepL and Firefox Translate have done this use-case already, much more efficiently than LLMs. So what does this particular use case of a transformer as a general-purpose computer solve that hasn't already been solved before? Why do we need this?
  - lolinder 3y ago
    > Google Translate, DeepL and Firefox Translate have done this use-case already, much more efficiently than LLMs. They may have been more efficient, but the transformer-based Google Translate we have today is way better at its job than its predecessors, and the old Google Translate wasn't capable of other NLU tasks that can now all be done by transformers (and again, all better than the previous specialized tools). I agree that transformer-as-Turing-machine isn't really solving any problems, but transformers have been nothing short of revolutionary for tasks that have usually been hard for plain Turing computation.
JohannesVoderho 3y ago
Original creator of the blog here. I originally wrote this for a course from my University and thought it was interesting so integrated it into my website. So i didn't want to "solve" anything. I just wanted to extend on what Andrej karpathy mentioned in one of his tweets: https://twitter.com/karpathy/status/1582807367988654081 https://twitter.com/karpathy/status/1582807367988654081. I think this view on transformers is a very powerful one and explains in the best way why the transformer is build that way. This also allows us to reason about certain improvements for the transformer. This was mostly an educational blog

typon 3y ago

It solves language understanding in a practical sense - a task heretofore considered impossible

colesantiago 3y ago
This has been already solved even without LLMs for years: Google Translate, DeepL and Firefox Translate have done this use-case already, much more efficiently than LLMs. So what does this particular use case of a transformer as a general-purpose computer solve that hasn't already been solved before? Why do we need this?
- lolinder 3y ago
  > Google Translate, DeepL and Firefox Translate have done this use-case already, much more efficiently than LLMs. They may have been more efficient, but the transformer-based Google Translate we have today is way better at its job than its predecessors, and the old Google Translate wasn't capable of other NLU tasks that can now all be done by transformers (and again, all better than the previous specialized tools). I agree that transformer-as-Turing-machine isn't really solving any problems, but transformers have been nothing short of revolutionary for tasks that have usually been hard for plain Turing computation.

JohannesVoderho 3y ago

Original creator of the blog here. I originally wrote this for a course from my University and thought it was interesting so integrated it into my website. So i didn't want to "solve" anything. I just wanted to extend on what Andrej karpathy mentioned in one of his tweets: https://twitter.com/karpathy/status/1582807367988654081 https://twitter.com/karpathy/status/1582807367988654081. I think this view on transformers is a very powerful one and explains in the best way why the transformer is build that way. This also allows us to reason about certain improvements for the transformer. This was mostly an educational blog