The SEP article on understanding perhaps will be helpful to break the impasse. It cites an influential theorist[1]: >Central to the notion of understanding are

The SEP article on understanding perhaps will be helpful to break the impasse. It cites an influential theorist[1]: >Central to the notion of understanding are various coherence-like elements: to have understanding is to grasp explanatory and conceptual connections between various pieces of information involved in the subject matter in question. Understanding individual operations in isolation is a far cry from understanding how the system works as a collective unit, i.e. "grasping explanatory and conceptual connections". If you accept weak emergence as a concept (and you should), then you recognize that the behavior of a complex system can be unpredictable from an analysis of the behavior of the components. The space of potential interactions grows exponentially as units and their effects are added. Features of the system that are relevant to its behavior are necessary to comprehend in order to be said to understand the system. Ghost features, i.e. unidentified structure, undermine the claim of understanding. We presumably understand the rules of particle physics that constrain the behavior of all matter in the universe. But it would be absurd to claim that we understand everything about the objects and phenomena that make up the universe. It just ignores the computational and interaction complexity as irrelevant to understanding. This is plainly a mistake. Regarding your point about science, the difference is that the process of science is, at least in part, about increasing predictive accuracy independent of understanding. Mathematics obviously helps here. But this doesn't say anything about whether predictive accuracy increases understanding, which you already assert is an independent concern. Science results in understanding when the models we develop correspond to the mechanisms involved in the real-world phenomena. But it is not the model that is the understanding, it is our ability to engage with features of the model intelligently, in service to predictive and instrumental goals. If all we can do is run some mechanized version of the model and read its output, we don't understand anything about the model or what it tells us about the world. This is obviously a verbal dispute and so nothing of substance turns on its resolution. But when you say that we definitely do understand them, you are just miscommunicating with your interlocutor. I think its clear that most people associate understanding with the ability to predict. You're free to disagree with this association, but you should be more concerned with accurate communication rather than asserting your idiosyncratic usage. Convincing people that we understand how LLMs work (and thus can predict their behavior) has the potential to cause real damage. Perhaps that is an overriding concern of yours rather than debating the meaning of a word or grinding your axe against the ML field? [1] https://plato.stanford.edu/entries/understanding/ https://plato.stanford.edu/entries/understanding/