3 ms·

The reason LLMs are novel in this regard are because they are a software model which can't interrupted, inspected, and understood during its execution. Any othe

The reason LLMs are novel in this regard are because they are a software model which can't interrupted, inspected, and understood during its execution. Any other software system running on your machine has a state which at any level of abstraction, from an HTML checkbox to a single LED pixel, can be understood and reasoned with. It is a system we can step through in a discrete and understandable way because it's a system we created. My understanding is that we cannot do this with an LLM. There isn't a callstack which can be followed to trace why we returned 'foo' instead of 'bar', just the oft-mentioned matrix of floats corresponding to tokens. Perhaps not everyone sees it this way, but I think this separation, of a system which we can conceptualize to one we cannot, is a big deal.

fnovd 4y ago
It's a big deal as far as impact, sure, but I think it's also OK for us to abandon the need for complete control. Because, really, that's what this is about: we're worried that if we can't understand what's happening, we can't control it. Personally I think that's just fine. It's a different class of thing but that's also OK. Do we even really know all that we should about how to manufacture chips? My understanding is that we rely heavily on experiments and heuristics. I think with complexity that's just how things are, sometimes. And again, going to the human-throwing-a-ball metaphor, maybe generalized intelligence actually requires introspective opaqueness. Maybe it's some cosmic law we don't understand: the better a system is at handling open-ended issues the less it can be systematically understood. I just think the utility of application is so, so much more important than our inability to know down to the last bit how a given LLM works. If it works, and delivers value, then we just don't need the why. We can and should try to understand, but we can also accept that we won't always get all of the answers we want. I mean, why does anyone do anything? Ask someone to explain why they did everything they did today and I'm sure a lot of what they'll tell you is made up or just plain wrong. Humanity seems to be just fine despite all that. Why do we expect our apprentices to be different?
thomastjeffery 4y ago
That's the black box. The rest of the narrative implies there is a person inside. That's just what happens when you call something "AI": personification. The ultimate irony of LLMs is that they are inference models, which means they can never "know" anything explicitly; but it also means that we can never "know" them explicitly. Everything we have heard about inference models was itself inferred by humans! Do we truly have the same limitation, or can we take another approach? I don't think that is the case. I don't think we are limited to modeling what the thing does through inference. I think we can construct an explicit understanding from how the thing is designed to work, because all of that exists as explicitly written algorithms. We need to stop personifying the thing. We should probably also stop calling it a "Language Model", because that means we are studying the resulting model, and not the thing that constructs it. I prefer to call them, "Text Inference Models". That's actually pretty easy to conceptualize: it finds patterns that are present (inference) in text (not limited to language patterns). That gets us asking a more useful question: "What patterns are present in text?" The answer is, "Whatever the human writer chose to write." In other words, the entropy of human writing. That gives us a coherent source for what patterns an LLM might find. Some patterns are language grammar. Some are more interesting. Some are helpful. Some are limiting. Most importantly, there is no categorization happening about the patterns themselves. Each pattern is on equal footing to the rest: each indistinguishable from each other. That means we can't ever choose the ones we like or avoid the ones that break stuff. Instead, we can only change what is present in the text itself. Knowing that, it's easy to see where "limitations" come from. They are just the reality of natural language: ambiguity. That's exactly the problem inference models are intended to work around, but the problem could only be moved, not solved.
booleandilemma 4y ago
Is it just that someone hasn't built a debugger for an LLM, or is there something fundamental that prevents them from being debugged?
- sanxiyn 4y ago
  We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so by doing rotation on unit circle. For a + b = c mod p, probability of c is proportional to cos((a+b-c) 2π/p), which works because cosine is maximized when the argument is multiple of 2π. As you can see, there is no fundamental difficulty, but it is also not trivial. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217

fnovd 4y ago

It's a big deal as far as impact, sure, but I think it's also OK for us to abandon the need for complete control. Because, really, that's what this is about: we're worried that if we can't understand what's happening, we can't control it. Personally I think that's just fine. It's a different class of thing but that's also OK. Do we even really know all that we should about how to manufacture chips? My understanding is that we rely heavily on experiments and heuristics. I think with complexity that's just how things are, sometimes. And again, going to the human-throwing-a-ball metaphor, maybe generalized intelligence actually requires introspective opaqueness. Maybe it's some cosmic law we don't understand: the better a system is at handling open-ended issues the less it can be systematically understood. I just think the utility of application is so, so much more important than our inability to know down to the last bit how a given LLM works. If it works, and delivers value, then we just don't need the why. We can and should try to understand, but we can also accept that we won't always get all of the answers we want. I mean, why does anyone do anything? Ask someone to explain why they did everything they did today and I'm sure a lot of what they'll tell you is made up or just plain wrong. Humanity seems to be just fine despite all that. Why do we expect our apprentices to be different?

thomastjeffery 4y ago

That's the black box. The rest of the narrative implies there is a person inside. That's just what happens when you call something "AI": personification. The ultimate irony of LLMs is that they are inference models, which means they can never "know" anything explicitly; but it also means that we can never "know" them explicitly. Everything we have heard about inference models was itself inferred by humans! Do we truly have the same limitation, or can we take another approach? I don't think that is the case. I don't think we are limited to modeling what the thing does through inference. I think we can construct an explicit understanding from how the thing is designed to work, because all of that exists as explicitly written algorithms. We need to stop personifying the thing. We should probably also stop calling it a "Language Model", because that means we are studying the resulting model, and not the thing that constructs it. I prefer to call them, "Text Inference Models". That's actually pretty easy to conceptualize: it finds patterns that are present (inference) in text (not limited to language patterns). That gets us asking a more useful question: "What patterns are present in text?" The answer is, "Whatever the human writer chose to write." In other words, the entropy of human writing. That gives us a coherent source for what patterns an LLM might find. Some patterns are language grammar. Some are more interesting. Some are helpful. Some are limiting. Most importantly, there is no categorization happening about the patterns themselves. Each pattern is on equal footing to the rest: each indistinguishable from each other. That means we can't ever choose the ones we like or avoid the ones that break stuff. Instead, we can only change what is present in the text itself. Knowing that, it's easy to see where "limitations" come from. They are just the reality of natural language: ambiguity. That's exactly the problem inference models are intended to work around, but the problem could only be moved, not solved.

booleandilemma 4y ago

Is it just that someone hasn't built a debugger for an LLM, or is there something fundamental that prevents them from being debugged?

sanxiyn 4y ago
We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so by doing rotation on unit circle. For a + b = c mod p, probability of c is proportional to cos((a+b-c) 2π/p), which works because cosine is maximized when the argument is multiple of 2π. As you can see, there is no fundamental difficulty, but it is also not trivial. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217