Now I'm wondering about a world where OpenAI can treat their compute not as "dedicate this time slice of an A100 to generating tokens for customer A's query" bu

Now I'm wondering about a world where OpenAI can treat their compute not as "dedicate this time slice of an A100 to generating tokens for customer A's query" but rather "there is a custom pipeline at a gate level that exactly matches our model architecture, hardcoding weights at the gate level, and can treat parallel requests the same way a processor would pipeline instructions." I don't think any human could write the TensorFlow compiler for this, but theoretically a generative AI could learn how to, experimenting on its own compute hardware? I don't know that we've seen the last of FPGAs.