OK, but how does that not just demonstrate LLMs can't generalize absent massaged training data? > it's not as stable as you'd want. Which I take to mean that

OK, but how does that not just demonstrate LLMs can't generalize absent massaged training data? > it's not as stable as you'd want. Which I take to mean that a model can sometimes confabulate "B is A", solely out of random variation, and that it's possible to bias the data and prompt to generate the expected response. The model hasn't done any logical deduction, the response is just a bias-influenced lucky break.