Highly doubtful. Transformers are at their limit. In general, first there was simply (weight * value) . Then people experimented with a trainable parameter tha

Highly doubtful. Transformers are at their limit. In general, first there was simply (weight * value) . Then people experimented with a trainable parameter that would select a set of weights for a given input. Transformers essentially turned this 1 dimentional discrete selection into a n dimentional multiplication, where the weights get modified by a product of values and a trainable parameter, i.e (weight * value) * value To get further, 2 things need to happen. First, there needs to be investigation into being able to learn the network architecture itself. For example, what if the transformer learned parameters were instead computed at inference time? I.e ( ( weight * value ) * value ) * value. What if this was abstracted to some form where there was a learned parameters on where to get those values from? Secondly, there needs to be some differentiable method of storing data between inference runs, so instead of input->output, you have input+memory -> output.