+1. Personal opinion: accelerators are useful today but have kept us in a local minimum which is certainly not the ideal. There are interesting approaches such

+1. Personal opinion: accelerators are useful today but have kept us in a local minimum which is certainly not the ideal. There are interesting approaches such as near linear low-rank approximation of attention gradients [1]. Would we rather have that, or somewhat better constant factors? [1] https://arxiv.org/html/2408.13233v1 https://arxiv.org/html/2408.13233v1