But it's used on the thing that it's trained for. LLMs are trained on language, but then used as a substitute for thinking, which it naively looks like they are

But it's used on the thing that it's trained for. LLMs are trained on language, but then used as a substitute for thinking, which it naively looks like they are doing due to the smooth language. Protein folding is a kind of syntax. If you train on protein folding and then use it to obtain protein folding results, you are using a screwdriver to drive a screw: that checks out. Nobody should be arguing along lines analogous to the claim that a good neural net trained on handwritten digits is not suitable for classifying handwritten digits.