3 ms·

>> But understanding from direct specification breaks down when the space of operations is too big to comprehend. If all we can do is run a program to see its o

>> But understanding from direct specification breaks down when the space of operations is too big to comprehend. If all we can do is run a program to see its output because its state space is too large and/or complicated to comprehend, then we lack understanding. I don't agree. That's [edit: part of] why maths are established as the language of science, because they're a formal language of abstractions by which we can avoid having to deal with large, or even infinite, spaces, by writing down a finite formula that fully defines the relevant concept. For example, I don't have to count to infinity, to understand infinity. I can study the definition of infinity (there are more than one) and understand what it means. Having understood what infinity means, I can then reuse the concept in calculations, where again I don't have to count to infinity to get to a correct result. I can also reuse the concept to form new concepts, again without having to count to infinity. With LLMs then, we have the maths that define their "space of operations". We can use those maths to train LLMs! Again- what else understanding remains to be had? I do think you're still talking about tracing the operations in an LLM to fully follow how inputs become outputs. But that's not how we commonly approach the problem of understanding, and even predicting the behaviour, of large and complex technological artifacts. Like, I don't reckon there's anyone alive that could draw you a diagram of every interaction between every system on an airliner. Yet, we "understand" those systems and in fact we can analyse them and predict their behaviour (with error). That kind of analysis is missing from LLMs, but that's because nobody wants to do it, currently. People are too busy poking LLMs and oohing and aaahing at what comes out. I'm hoping that, at some point, the initial rush of hype will subside and some good analytical work will be produced. This was done for previous LLMs although rarely of course, and poorly, because of the generally poor methodologies in machine learning research. >> The danger in LLMs is that their sequence of operations ... That's relevant to what I say above. Yeah, that work hasn't been done and it should be done. But that's not about understanding how LLMs work, it's about analysing the function of specific systems.

hackinthebochs 3y ago
The SEP article on understanding perhaps will be helpful to break the impasse. It cites an influential theorist[1]: >Central to the notion of understanding are various coherence-like elements: to have understanding is to grasp explanatory and conceptual connections between various pieces of information involved in the subject matter in question. Understanding individual operations in isolation is a far cry from understanding how the system works as a collective unit, i.e. "grasping explanatory and conceptual connections". If you accept weak emergence as a concept (and you should), then you recognize that the behavior of a complex system can be unpredictable from an analysis of the behavior of the components. The space of potential interactions grows exponentially as units and their effects are added. Features of the system that are relevant to its behavior are necessary to comprehend in order to be said to understand the system. Ghost features, i.e. unidentified structure, undermine the claim of understanding. We presumably understand the rules of particle physics that constrain the behavior of all matter in the universe. But it would be absurd to claim that we understand everything about the objects and phenomena that make up the universe. It just ignores the computational and interaction complexity as irrelevant to understanding. This is plainly a mistake. Regarding your point about science, the difference is that the process of science is, at least in part, about increasing predictive accuracy independent of understanding. Mathematics obviously helps here. But this doesn't say anything about whether predictive accuracy increases understanding, which you already assert is an independent concern. Science results in understanding when the models we develop correspond to the mechanisms involved in the real-world phenomena. But it is not the model that is the understanding, it is our ability to engage with features of the model intelligently, in service to predictive and instrumental goals. If all we can do is run some mechanized version of the model and read its output, we don't understand anything about the model or what it tells us about the world. This is obviously a verbal dispute and so nothing of substance turns on its resolution. But when you say that we definitely do understand them, you are just miscommunicating with your interlocutor. I think its clear that most people associate understanding with the ability to predict. You're free to disagree with this association, but you should be more concerned with accurate communication rather than asserting your idiosyncratic usage. Convincing people that we understand how LLMs work (and thus can predict their behavior) has the potential to cause real damage. Perhaps that is an overriding concern of yours rather than debating the meaning of a word or grinding your axe against the ML field? [1] https://plato.stanford.edu/entries/understanding/ https://plato.stanford.edu/entries/understanding/

hackinthebochs 3y ago

The SEP article on understanding perhaps will be helpful to break the impasse. It cites an influential theorist[1]: >Central to the notion of understanding are various coherence-like elements: to have understanding is to grasp explanatory and conceptual connections between various pieces of information involved in the subject matter in question. Understanding individual operations in isolation is a far cry from understanding how the system works as a collective unit, i.e. "grasping explanatory and conceptual connections". If you accept weak emergence as a concept (and you should), then you recognize that the behavior of a complex system can be unpredictable from an analysis of the behavior of the components. The space of potential interactions grows exponentially as units and their effects are added. Features of the system that are relevant to its behavior are necessary to comprehend in order to be said to understand the system. Ghost features, i.e. unidentified structure, undermine the claim of understanding. We presumably understand the rules of particle physics that constrain the behavior of all matter in the universe. But it would be absurd to claim that we understand everything about the objects and phenomena that make up the universe. It just ignores the computational and interaction complexity as irrelevant to understanding. This is plainly a mistake. Regarding your point about science, the difference is that the process of science is, at least in part, about increasing predictive accuracy independent of understanding. Mathematics obviously helps here. But this doesn't say anything about whether predictive accuracy increases understanding, which you already assert is an independent concern. Science results in understanding when the models we develop correspond to the mechanisms involved in the real-world phenomena. But it is not the model that is the understanding, it is our ability to engage with features of the model intelligently, in service to predictive and instrumental goals. If all we can do is run some mechanized version of the model and read its output, we don't understand anything about the model or what it tells us about the world. This is obviously a verbal dispute and so nothing of substance turns on its resolution. But when you say that we definitely do understand them, you are just miscommunicating with your interlocutor. I think its clear that most people associate understanding with the ability to predict. You're free to disagree with this association, but you should be more concerned with accurate communication rather than asserting your idiosyncratic usage. Convincing people that we understand how LLMs work (and thus can predict their behavior) has the potential to cause real damage. Perhaps that is an overriding concern of yours rather than debating the meaning of a word or grinding your axe against the ML field? [1] https://plato.stanford.edu/entries/understanding/ https://plato.stanford.edu/entries/understanding/