Of course it's not going to scale much past this, it's quadratic and already hitting painful compute levels. However, if you were starting this research today,

Of course it's not going to scale much past this, it's quadratic and already hitting painful compute levels. However, if you were starting this research today, you'd use any of half-a-dozen different self-attention variants which are roughly linear, including OA's own Sparse Transformers (which they did use to generate images, just on a far smaller scale which wouldn't be adequate to show competitive performance with SimCLR etc). With those, it's perfectly possible to do self-attention over whole images at 256px or higher. (As a matter of fact, Aydao has been working on a StyleGAN which just uses self-attention every layer instead of convolutions, using one of the new attentions; you can see some generated image samples from Flowers here: https://github.com/tensorfork/tensorfork/issues/31 https://github.com/tensorfork/tensorfork/issues/31 GAN loss, not autoregressive pixel likelihood, but it makes the point.)