> [which anyone who understands] how a multi headed attention transformer works [would know is impossible] How is that productive? The author already claimed i

> [which anyone who understands] how a multi headed attention transformer works [would know is impossible] How is that productive? The author already claimed in their changes that they aren't an ML expert and asked for advice. It could be written like multi-headed attention transformers frobnicate all the weights in their QKV matmuls. <supposition ... I would look at IO rates during inference ...> You have an opportunity to teach rather than admonish.