4 ms·

The title is a little misleading. The post makes an analogy between a turing machine and a transformer and and expands on that analogy a bit. The substance of

by Hugsun 3y ago

The title is a little misleading. The post makes an analogy between a turing machine and a transformer and and expands on that analogy a bit. The substance of the article is an explanation of the design of a standard transformer. It does not use transformers as a computer in any novel way. It just makes the analogy.

typon 3y ago
For those unaware of Alex Graves' work, check out the NTM paper: https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 This actually does what this blog post pretends to
- nyrikki 3y ago
  Except it proves nothing about TC, it just refers to a paper that demonstrated RNNs were TC. The authors decision to name their creation a "Neural Turing Machine" doesn't make it computationally equivalent. The fact that it explicitly states: > ...analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to- end, allowing it to be efficiently trained with gradient descent. Actually demonstrates it is not equivalent. Note that I am not saying their approach is wrong, but that their overloading of terms suggests more than they claim. While 'general-purpose computer' is ambiguous, a Turning machine is not as opaque. LLMs can be TC with infinite space for input and infinite width. For finite input, size, and depth, LLMs aren't even equivalent to primitive recursive functions IIRC. With consent depth threshold circuits being a recent claim as the limits of LLMs.
JohannesVoderho 3y ago
Original creator of the blog here. Yes, you are right. The title of the blog is a bit misleading. I originally wrote this for a course from my University and thought it was interesting so integrated it into my website. I didn't expect to get that much traffic on the blog and didn't intend to do any clickbaiting. I just wanted to extend on what Andrej karpathy mentioned in one of his tweets: https://twitter.com/karpathy/status/1582807367988654081 https://twitter.com/karpathy/status/1582807367988654081. I think i also linked it in the blogpost. I think this view on transformers is a very powerful one and explains in the best way why the transformer is build that way. This also allows us to reason about certain improvements for the transformer. For example weight tying some layers to make it easier to map to algorithms that have a lot of repeated operation. The Goal of the blog was not to use the transformer as a general purpose differentiable computer but to convince you that the transformer already IS a general purpose differentiable computer. Maybe i will create another blog where i train the transformer on simple algorithmic problems like sorting or copying data, just like in the differentiable turing machine, or even show how you can hard code the transformer weights to solve them. What do you think would be a better title for the blog?
- Hugsun 3y ago
  That makes sense. It definitely read like a university project to me. One idea is "Transformers, analogous to general purpose computers" although that doesn't mention the meat of the article. "Transformers from scratch, a turing machine analogy" is another idea. I'm not sure they're good titles but they don't imply that it's about using a transformer as a computer. With the goal you state, I understand better how you came to the title.

by Hugsun 3y ago

typon 3y ago

For those unaware of Alex Graves' work, check out the NTM paper: https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 This actually does what this blog post pretends to

nyrikki 3y ago
Except it proves nothing about TC, it just refers to a paper that demonstrated RNNs were TC. The authors decision to name their creation a "Neural Turing Machine" doesn't make it computationally equivalent. The fact that it explicitly states: > ...analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to- end, allowing it to be efficiently trained with gradient descent. Actually demonstrates it is not equivalent. Note that I am not saying their approach is wrong, but that their overloading of terms suggests more than they claim. While 'general-purpose computer' is ambiguous, a Turning machine is not as opaque. LLMs can be TC with infinite space for input and infinite width. For finite input, size, and depth, LLMs aren't even equivalent to primitive recursive functions IIRC. With consent depth threshold circuits being a recent claim as the limits of LLMs.

JohannesVoderho 3y ago

Original creator of the blog here. Yes, you are right. The title of the blog is a bit misleading. I originally wrote this for a course from my University and thought it was interesting so integrated it into my website. I didn't expect to get that much traffic on the blog and didn't intend to do any clickbaiting. I just wanted to extend on what Andrej karpathy mentioned in one of his tweets: https://twitter.com/karpathy/status/1582807367988654081 https://twitter.com/karpathy/status/1582807367988654081. I think i also linked it in the blogpost. I think this view on transformers is a very powerful one and explains in the best way why the transformer is build that way. This also allows us to reason about certain improvements for the transformer. For example weight tying some layers to make it easier to map to algorithms that have a lot of repeated operation. The Goal of the blog was not to use the transformer as a general purpose differentiable computer but to convince you that the transformer already IS a general purpose differentiable computer. Maybe i will create another blog where i train the transformer on simple algorithmic problems like sorting or copying data, just like in the differentiable turing machine, or even show how you can hard code the transformer weights to solve them. What do you think would be a better title for the blog?

Hugsun 3y ago
That makes sense. It definitely read like a university project to me. One idea is "Transformers, analogous to general purpose computers" although that doesn't mention the meat of the article. "Transformers from scratch, a turing machine analogy" is another idea. I'm not sure they're good titles but they don't imply that it's about using a transformer as a computer. With the goal you state, I understand better how you came to the title.