This observation reminds me of trying to counter Betteridge's law of headlines [1] with the observation that if you were to negate the question in the headline,

This observation reminds me of trying to counter Betteridge's law of headlines [1] with the observation that if you were to negate the question in the headline, you would clearly get a question whose answer is not No. Of course, the question you get is no longer an actual _headline_, so there's no reason to expect Betteridge's law to apply. A similar thing happens here: if you take a scientific article, and rearrange its words to change its meaning, what you get is no longer a scientific article, so outside the scope of the classification task "classify scientific articles based on how likely they are to reproduce". It's not even meaningful to ask whether it will reproduce: what you have is no longer a scientific article reporting on research that actually took place and got a certain result, it's merely a bag of words that you generated by permuting the words of an actual scientific article. Your observations are irrelevant: they cannot be used to tell apart models which accomplish the stated task (correctly classify scientific articles based on how likely they are to reproduce) from those which do not. This doesn't mean the proposed model actually works (I for one think it almost certainly doesn't), but it cannot be strong evidence either way. It's like somebody proposing a model that can identify an animal based on children's pencil drawings of the animal, then you countering that if you rearrange the photo of a man standing next to a horse into a centaur, the model still returns "horse". Such rearranged, synthetic data is well outside the scope of the inputs the model was meant to handle, so you cannot draw any strong conclusions about the model's ability to perform its actual, advertised task. [1] https://en.m.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.m.wikipedia.org/wiki/Betteridge%27s_law_of_headli...