> You could train a model that just takes natural language and outputs only code, with no conversion. Those models would be less general, and less interesting t

> You could train a model that just takes natural language and outputs only code, with no conversion. Those models would be less general, and less interesting to non Technical people, which is why we don’t see them as much (imo.) Right, I’m not saying “the reason they output other text first is because they need to in order to produce the correct output.”. I agree that the reason they put the other text first is due to how they were trained. I’m just saying that if they didn’t do that, that this would limit the theoretical maximum average performance they could achieve at the “producing the output of some long code” task. But, I don’t expect that this was part of the motivation that led to them being trained in the way they have been. > There’s no way to fix that for transformers architectures as the limited context length is inherent in their attention mechanisms Hmm… the paper showing that transformers can recognize TC^0 has a definition for attention that is a bit different from the usual attention, and seems a bit restricted.., it seemed to me that the paper involved the idea that this could be applied to arbitrary length sequences, but I could be mistaken.