2 ms·
I'm not sure why that argument is applicable to a queue but not to any fixed sized memory? A transformer can conceptually modify any position by simply outputti
by marcinzm 3y ago
I'm not sure why that argument is applicable to a queue but not to any fixed sized memory? A transformer can conceptually modify any position by simply outputting the whole context except with 1 different value. Basically, it's restating in a convoluted way the known fact that anything with finite memory has finite states and thus the halting problem is solvable.
- davesque 3y agoYeah, I guess you're right. The queue vs. addressable memory thing seems secondary. Is the real difference that a transformer model is stateless and Turing machine is stateful? And the assumption of a stateless model is why they can assert that a repeated state implies an infinite loop (assuming we're not sampling the softmax to get the output token)?