3 ms·

>We don't really know how they work inside. I keep reading this, and my response is so what? Do you have a ball sitting handy somewhere? Pick it up, throw it

by fnovd 4y ago

>We don't really know how they work inside. I keep reading this, and my response is so what? Do you have a ball sitting handy somewhere? Pick it up, throw it in the air, and then catch it. How did you do that? Can you regurgitate all of the equations used to define how objects move, how you determined the approximate mass of the ball, how you calibrated your muscles to apply a specific amount of force and then how you orchestrated the whole thing? Of course not, that's ridiculous. You don't need to know physics or even be literate in order to throw and catch a ball. Your brain "knows" how to do it but that knowledge is opaque and the part of your brain that talks to people can't actually communicate any of the nitty-gritty. So why does it matter that an AI can't tell us how it does what it does? We're the smartest things we know and we can't even do it. We can give rationalized mockups of a lot, sure, but the AI can, too. Why do we think introspective clairvoyance is a requirement of AGI?

joenot443 4y ago
The reason LLMs are novel in this regard are because they are a software model which can't interrupted, inspected, and understood during its execution. Any other software system running on your machine has a state which at any level of abstraction, from an HTML checkbox to a single LED pixel, can be understood and reasoned with. It is a system we can step through in a discrete and understandable way because it's a system we created. My understanding is that we cannot do this with an LLM. There isn't a callstack which can be followed to trace why we returned 'foo' instead of 'bar', just the oft-mentioned matrix of floats corresponding to tokens. Perhaps not everyone sees it this way, but I think this separation, of a system which we can conceptualize to one we cannot, is a big deal.
- fnovd 4y ago
  It's a big deal as far as impact, sure, but I think it's also OK for us to abandon the need for complete control. Because, really, that's what this is about: we're worried that if we can't understand what's happening, we can't control it. Personally I think that's just fine. It's a different class of thing but that's also OK. Do we even really know all that we should about how to manufacture chips? My understanding is that we rely heavily on experiments and heuristics. I think with complexity that's just how things are, sometimes. And again, going to the human-throwing-a-ball metaphor, maybe generalized intelligence actually requires introspective opaqueness. Maybe it's some cosmic law we don't understand: the better a system is at handling open-ended issues the less it can be systematically understood. I just think the utility of application is so, so much more important than our inability to know down to the last bit how a given LLM works. If it works, and delivers value, then we just don't need the why. We can and should try to understand, but we can also accept that we won't always get all of the answers we want. I mean, why does anyone do anything? Ask someone to explain why they did everything they did today and I'm sure a lot of what they'll tell you is made up or just plain wrong. Humanity seems to be just fine despite all that. Why do we expect our apprentices to be different?
- thomastjeffery 4y ago
  That's the black box. The rest of the narrative implies there is a person inside. That's just what happens when you call something "AI": personification. The ultimate irony of LLMs is that they are inference models, which means they can never "know" anything explicitly; but it also means that we can never "know" them explicitly. Everything we have heard about inference models was itself inferred by humans! Do we truly have the same limitation, or can we take another approach? I don't think that is the case. I don't think we are limited to modeling what the thing does through inference. I think we can construct an explicit understanding from how the thing is designed to work, because all of that exists as explicitly written algorithms. We need to stop personifying the thing. We should probably also stop calling it a "Language Model", because that means we are studying the resulting model, and not the thing that constructs it. I prefer to call them, "Text Inference Models". That's actually pretty easy to conceptualize: it finds patterns that are present (inference) in text (not limited to language patterns). That gets us asking a more useful question: "What patterns are present in text?" The answer is, "Whatever the human writer chose to write." In other words, the entropy of human writing. That gives us a coherent source for what patterns an LLM might find. Some patterns are language grammar. Some are more interesting. Some are helpful. Some are limiting. Most importantly, there is no categorization happening about the patterns themselves. Each pattern is on equal footing to the rest: each indistinguishable from each other. That means we can't ever choose the ones we like or avoid the ones that break stuff. Instead, we can only change what is present in the text itself. Knowing that, it's easy to see where "limitations" come from. They are just the reality of natural language: ambiguity. That's exactly the problem inference models are intended to work around, but the problem could only be moved, not solved.
- booleandilemma 4y ago
  Is it just that someone hasn't built a debugger for an LLM, or is there something fundamental that prevents them from being debugged?
  - sanxiyn 4y ago
    We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so by doing rotation on unit circle. For a + b = c mod p, probability of c is proportional to cos((a+b-c) 2π/p), which works because cosine is maximized when the argument is multiple of 2π. As you can see, there is no fundamental difficulty, but it is also not trivial. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217
agalunar 4y ago
I feel like you're attacking a straw man. I don't think anyone believes an artificial intelligence needs to know how it itself works to be intelligent. On the other hand, we'd like to know how it works. I mean, why do we do science at all? If you don't need to know physics to throw a ball, why bother studying mechanics or biology?
- fnovd 4y ago
  "It doesn't know how it knows what it knows" is a very common criticism. "It's just predicting the next token but doesn't really 'understand' anything" is another. To me it's like saying, "How can humans actually play baseball if we don't even explain how we throw balls and swing bats at moving objects?" Wording it that way, it just sounds ridiculous. As ridiculous as I think this AI conversation is. I don't know why we care so much about model transparency. Yes, it's worth scientific pursuit to understand what we're building and how it works, we should absolutely try to know as much as we can. But if the bar for "true intelligence" is a machine capable of doing things that we can't do ourselves, is that saying we aren't "truly" intelligent, either? If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK.
  - shinycode 4y ago
    Because contrary to your example, this intelligence could « understand » the flaws of our systems and lock us down if it wants to. For example by creating a massive ransomware and exploiting 0-days because it’s much better and faster than us to analyze code. What happens then ? If it’d like to do harm for any reason (it might not be harm for an AI but just the « right thing » according to its criterias) what would we do ? Wouldn’t it be a good thing to know how it works inside precisely ? Or « whatever happens, happens » ?
  - agalunar 4y ago
    I suppose it depends on what you mean by "knowing how I know what I know". That could refer to "knowing the physiology of my brain", which is what you seemed to be referring to. But it could also refer to "explaining my feelings or decision making process". I've lived in my head for many years and had many chances to observe myself, so I can perform post hoc self-analysis with reasonable accuracy, and on a good day, I can even be self-aware in the very moment (which is useful in getting myself to do what I want). I think maybe that second thing is what people are worried about AI lacking. > If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK. I don't say this merely to be contentious, but I don't think I have quite as optimistic an outlook myself ^_^' which isn't to say I think we shouldn't meddle with the world, just that we sometimes get ourselves in over our heads. But I'm hopeful.

by fnovd 4y ago

joenot443 4y ago

The reason LLMs are novel in this regard are because they are a software model which can't interrupted, inspected, and understood during its execution. Any other software system running on your machine has a state which at any level of abstraction, from an HTML checkbox to a single LED pixel, can be understood and reasoned with. It is a system we can step through in a discrete and understandable way because it's a system we created. My understanding is that we cannot do this with an LLM. There isn't a callstack which can be followed to trace why we returned 'foo' instead of 'bar', just the oft-mentioned matrix of floats corresponding to tokens. Perhaps not everyone sees it this way, but I think this separation, of a system which we can conceptualize to one we cannot, is a big deal.

fnovd 4y ago
It's a big deal as far as impact, sure, but I think it's also OK for us to abandon the need for complete control. Because, really, that's what this is about: we're worried that if we can't understand what's happening, we can't control it. Personally I think that's just fine. It's a different class of thing but that's also OK. Do we even really know all that we should about how to manufacture chips? My understanding is that we rely heavily on experiments and heuristics. I think with complexity that's just how things are, sometimes. And again, going to the human-throwing-a-ball metaphor, maybe generalized intelligence actually requires introspective opaqueness. Maybe it's some cosmic law we don't understand: the better a system is at handling open-ended issues the less it can be systematically understood. I just think the utility of application is so, so much more important than our inability to know down to the last bit how a given LLM works. If it works, and delivers value, then we just don't need the why. We can and should try to understand, but we can also accept that we won't always get all of the answers we want. I mean, why does anyone do anything? Ask someone to explain why they did everything they did today and I'm sure a lot of what they'll tell you is made up or just plain wrong. Humanity seems to be just fine despite all that. Why do we expect our apprentices to be different?
thomastjeffery 4y ago
That's the black box. The rest of the narrative implies there is a person inside. That's just what happens when you call something "AI": personification. The ultimate irony of LLMs is that they are inference models, which means they can never "know" anything explicitly; but it also means that we can never "know" them explicitly. Everything we have heard about inference models was itself inferred by humans! Do we truly have the same limitation, or can we take another approach? I don't think that is the case. I don't think we are limited to modeling what the thing does through inference. I think we can construct an explicit understanding from how the thing is designed to work, because all of that exists as explicitly written algorithms. We need to stop personifying the thing. We should probably also stop calling it a "Language Model", because that means we are studying the resulting model, and not the thing that constructs it. I prefer to call them, "Text Inference Models". That's actually pretty easy to conceptualize: it finds patterns that are present (inference) in text (not limited to language patterns). That gets us asking a more useful question: "What patterns are present in text?" The answer is, "Whatever the human writer chose to write." In other words, the entropy of human writing. That gives us a coherent source for what patterns an LLM might find. Some patterns are language grammar. Some are more interesting. Some are helpful. Some are limiting. Most importantly, there is no categorization happening about the patterns themselves. Each pattern is on equal footing to the rest: each indistinguishable from each other. That means we can't ever choose the ones we like or avoid the ones that break stuff. Instead, we can only change what is present in the text itself. Knowing that, it's easy to see where "limitations" come from. They are just the reality of natural language: ambiguity. That's exactly the problem inference models are intended to work around, but the problem could only be moved, not solved.
booleandilemma 4y ago
Is it just that someone hasn't built a debugger for an LLM, or is there something fundamental that prevents them from being debugged?
- sanxiyn 4y ago
  We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so by doing rotation on unit circle. For a + b = c mod p, probability of c is proportional to cos((a+b-c) 2π/p), which works because cosine is maximized when the argument is multiple of 2π. As you can see, there is no fundamental difficulty, but it is also not trivial. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217

agalunar 4y ago

I feel like you're attacking a straw man. I don't think anyone believes an artificial intelligence needs to know how it itself works to be intelligent. On the other hand, we'd like to know how it works. I mean, why do we do science at all? If you don't need to know physics to throw a ball, why bother studying mechanics or biology?

fnovd 4y ago
"It doesn't know how it knows what it knows" is a very common criticism. "It's just predicting the next token but doesn't really 'understand' anything" is another. To me it's like saying, "How can humans actually play baseball if we don't even explain how we throw balls and swing bats at moving objects?" Wording it that way, it just sounds ridiculous. As ridiculous as I think this AI conversation is. I don't know why we care so much about model transparency. Yes, it's worth scientific pursuit to understand what we're building and how it works, we should absolutely try to know as much as we can. But if the bar for "true intelligence" is a machine capable of doing things that we can't do ourselves, is that saying we aren't "truly" intelligent, either? If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK.
- shinycode 4y ago
  Because contrary to your example, this intelligence could « understand » the flaws of our systems and lock us down if it wants to. For example by creating a massive ransomware and exploiting 0-days because it’s much better and faster than us to analyze code. What happens then ? If it’d like to do harm for any reason (it might not be harm for an AI but just the « right thing » according to its criterias) what would we do ? Wouldn’t it be a good thing to know how it works inside precisely ? Or « whatever happens, happens » ?
- agalunar 4y ago
  I suppose it depends on what you mean by "knowing how I know what I know". That could refer to "knowing the physiology of my brain", which is what you seemed to be referring to. But it could also refer to "explaining my feelings or decision making process". I've lived in my head for many years and had many chances to observe myself, so I can perform post hoc self-analysis with reasonable accuracy, and on a good day, I can even be self-aware in the very moment (which is useful in getting myself to do what I want). I think maybe that second thing is what people are worried about AI lacking. > If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK. I don't say this merely to be contentious, but I don't think I have quite as optimistic an outlook myself ^_^' which isn't to say I think we shouldn't meddle with the world, just that we sometimes get ourselves in over our heads. But I'm hopeful.