5 ms·

This is a weird post. "What the transformer is actually doing"? You can just follow the code and see what it's doing. It's not doing something more or less than

This is a weird post. "What the transformer is actually doing"? You can just follow the code and see what it's doing. It's not doing something more or less than that. It's not doing some other thing.

gjm11 3y ago
The post is long and complicated and I haven't read most of it, so whether it's actually any good I shan't try to decide. But the above seems like a very weird argument. Sure, the code is doing what it's doing. But trying to understand it at that level of abstraction seems ... not at all promising. Consider a question about psychology. Say: "What are people doing when they decide what to buy in a shop?". If someone writes an article about this, drawing on some (necessarily simplified) model of human thinking and decision-making, and some experimental evidence about how people's purchasing decisions change in response to changes in price, different lighting conditions, mood, etc., ... would you say "You can just apply the laws of physics and see what the people are doing. They're not doing something more or less than that."? I mean, it would be true. People, so far as we know, do in fact obey the laws of physics. You could, in principle, predict what someone will buy in a given situation by modelling their body and surroundings at the level of atoms or thereabouts (quantum physics is a thing, of course, but it seems likely that a basically-classical model could be good enough for this purpose). When we make decisions, we are obeying the laws of physics and not doing some other thing. But this answer is completely useless for actually understanding what we do. If you're wondering "what would happen if the price were ten cents higher?" you've got no way to answer it other than running the whole simulation again. Maybe running thousands of versions of it since other factors could affect the results. If you're wondering "does the lighting make a difference, and what level of lighting in the shop will lead to people spending least or most?" then you've got no way to answer it other than running simulations with many different lighting conditions. Whereas if you have a higher-level, less precise model that says things like "people mostly prefer to spend less" and "people try to predict quality on the basis of price, so sometimes they will spend more if it seems like they're getting something better that way" and "people like to feel that they're getting a bargain" and so on, you may be able to make predictions without running an impossibly detailed person-simulation zillions of times. You may be able to give general advice to someone with a spending problem who'd like to spend more wisely, or to a shopkeeper who wants to encourage their customers to spend more. Similarly with language models and similar systems. Sure, you can find out what it does in some very specific situation by just running the code. But what if you have some broader question than that? Then simply knowing what the code does may not help you at all, because what the code does is gazillions of copies of "multiply these numbers together and add them". Again, I make no claim about whether the particular thing linked here offers much real insight. But it makes zero sense, so far as I can see, to dismiss it on the grounds that all you need to do is read the code.
- xanderlewis 3y ago
  You’re spot on; it’s like saying you can understand the game of chess by simply reading the rules. In a certain very superficial sense, yes. But the universe isn’t so simple. The same reason even a perfect understanding of what goes on at the level of subatomic particles isn’t thought to be enough to say we ‘understand the universe’. A hell of a lot can happen in between the setting out of some basic rules and the end — much higher level — result.
  - danielmarkbruce 3y ago
    And yet...alpha zero.
    - xanderlewis 3y ago
      My entire point is that implementation isn’t sufficient for understanding. Alpha Zero is the perfect example of that; you can create an amazing chess playing machine and (potentially) learn nothing at all about how to play chess. …so what’s your point? I’m not getting it from those two words.
      - danielmarkbruce 3y ago
        Understanding how the machine plays or how you should play? They aren't the same thing. And that is the point - trying to analogize to some explicit, concrete function you can describe is backwards. These models are gigantic (even the 'small' ones), they are looking to minimize a loss function by looking in multi thousand dimensional space. It is the very opposite of something that fits in a human brain in any explicit fashion.
        gjm11 3y ago
        So is what happens in an actual literal human brain. And yet, we spend quite a lot of our time thinking about what human brains do, and sometimes it's pretty useful. For a lot of this, we treat the actual brain as a black box and don't particularly care about how it does what it does, but knowing something about the internal workings at various levels of abstraction is useful too. Similarly, if for whatever reason you are interested in, or spend some of your time interacting with, transformer-based language models, then you might want some intuition for what they do and how. You'll never fit the whole thing in your brain. That's why you want simplified abstracted versions of it. Which, AIUI, is one thing that the OP is trying to do. (As I said before, I don't know how well it does it; what I'm objecting to is the idea that trying to do this is a waste of time because the only thing there is to know is that the model does what the code says it does.)
nl 3y ago
A walk through with what the data at each point looks like is actually pretty useful.
- danielmarkbruce 3y ago
  Sure, it is. But trying to explain it as though the weights have some goal is weird. They aren't trying to do anything. You have a loss function. The optimizer keeps moving weights around in an attempt to minimize the loss function. It's not more or less than that.
  - MarkusQ 3y ago
    This is just wrong. First of all, you're rejecting teleological anthropomorphizing (saying it's weird to act as if "the weights have some goal") but then in the very next line you talk about the optimizer making "an attempt" to accomplish a goal. All of which misses the point, since the question is about _explanations_ not goals and intentions. Then you reject out of hand any other level of explanation than the one you favor, saying "it's not more or less than that" when in fact it is both more and less than that; you can climb the ladder of abstraction either way to build more or less abstract explanations. We can dig down and talk about how the optimizer adjusts weights, or how tensor math works and how it's used in this case, or about how GPUs work, or gates, or transistors, etc. Or we could climb up and talk about (as this article does) and talk about what attention heads do, and why they work, when they work, when they don't, etc.
    - danielmarkbruce 3y ago
      The optimizer has a goal. The weights in the model do not. The optimizer isn't the model. There is no contradiction if you know how it works. Climbing the layer of abstraction from model weights doesn't seem to work in this field. Just saying it's so doesn't make it so.
      - MarkusQ 3y ago
        > Just saying it's so doesn't make it so. Does that apply to you as well?
drdeca 3y ago
Understanding how a given CPU (+ the other computer hardware) works, does not suffice to understand what is going on when a particular program is running. For that, you need to either read the program, or an execution trace, or both, or something along these lines, which is specific to the program being run.
- danielmarkbruce 3y ago
  This is the wrong analogy. The transformer block is a bunch of code and weights. It's a set of instructions laying out which numbers to run which operations on. The optimizer changes weights to minimize a loss function during training and then the code implementing a forward pass just runs during inference. That's what it is doing. It's not doing something else. If the argument is that a model is a function approximator, then it certainly isn't approximating some function that performs worse at the task at hand, and it certainly isn't approximating a function we can describe in a few hundred words.
  - FeepingCreature 3y ago
    We have no reason at all to be certain of the latter.
    - danielmarkbruce 3y ago
      There is pretty good reason. If it could be described explicitly in a few hundred words, it would be extremely unlikely that we'd have seen a jump in capability with model size.
richardw 3y ago
I'll go with an example to demonstrate why that's not always enough. Many people are quite keen to know what this (for example) is actually doing: float InvSqrt(float x){ float xhalf = 0.5f * x; int i = *(int*)&x; i = 0x5f3759df - (i >> 1); x = *(float*)&i; x = x*(1.5f - xhalf*x*x); return x; } From https://betterexplained.com/articles/understanding-quakes-fast-inverse-square-root/ https://betterexplained.com/articles/understanding-quakes-fa... In my case I don't have a huge amount of time to chase down every rabbit hole, but I'd love to accelerate intuition for LLM's. Multiple points of intuition or comparison really help. I'm also not a python expert - what you see and what I see from a line of code will be quite different.
- danielmarkbruce 3y ago
  The author is attempting to build an explicit mental model of what a bunch of weights are "doing". It's not really the same thing. They are minimizing the loss function. People try to (and often do) generate intuition for architectures that will work given the lay out of data. But, the reason models are so big now is that trying to understand what the model is "doing" in a way humans understand didn't work out so well.
magicalhippo 3y ago
> You can just follow the code and see what it's doing. It's not doing something more or less than that. And that's why we'll never have fun things like an obfuscated code contest. Oh wait[1]... [1]: https://www.ioccc.org/ https://www.ioccc.org/
psyklic 3y ago
Last week I trained an LLM to auto-unrotate a Caeser cipher, without it knowing the rotation key. It turns out there are only a few possible ways this can be done, e.g. by analyzing letter frequencies. Of course, the LLM must use one of these algorithms! Determining this high-level algorithm would be "understanding" what it is "actually doing". Through analysis, I proved it indeed counts each letter's frequency. It then combines the "overall" letter frequencies with another distribution of "first letter" frequencies. This increased its accuracy to nearly 100%. To me, this description is much nicer than had I "followed the code" and described it as millions of multiplications and additions. (Much of this ends up being noise anyway.)

This is a weird post. "What the transformer is actually doing"? You can just follow the code and see what it's doing. It's not doing something more or less than that. It's not doing some other thing.

gjm11 3y ago

The post is long and complicated and I haven't read most of it, so whether it's actually any good I shan't try to decide. But the above seems like a very weird argument. Sure, the code is doing what it's doing. But trying to understand it at that level of abstraction seems ... not at all promising. Consider a question about psychology. Say: "What are people doing when they decide what to buy in a shop?". If someone writes an article about this, drawing on some (necessarily simplified) model of human thinking and decision-making, and some experimental evidence about how people's purchasing decisions change in response to changes in price, different lighting conditions, mood, etc., ... would you say "You can just apply the laws of physics and see what the people are doing. They're not doing something more or less than that."? I mean, it would be true. People, so far as we know, do in fact obey the laws of physics. You could, in principle, predict what someone will buy in a given situation by modelling their body and surroundings at the level of atoms or thereabouts (quantum physics is a thing, of course, but it seems likely that a basically-classical model could be good enough for this purpose). When we make decisions, we are obeying the laws of physics and not doing some other thing. But this answer is completely useless for actually understanding what we do. If you're wondering "what would happen if the price were ten cents higher?" you've got no way to answer it other than running the whole simulation again. Maybe running thousands of versions of it since other factors could affect the results. If you're wondering "does the lighting make a difference, and what level of lighting in the shop will lead to people spending least or most?" then you've got no way to answer it other than running simulations with many different lighting conditions. Whereas if you have a higher-level, less precise model that says things like "people mostly prefer to spend less" and "people try to predict quality on the basis of price, so sometimes they will spend more if it seems like they're getting something better that way" and "people like to feel that they're getting a bargain" and so on, you may be able to make predictions without running an impossibly detailed person-simulation zillions of times. You may be able to give general advice to someone with a spending problem who'd like to spend more wisely, or to a shopkeeper who wants to encourage their customers to spend more. Similarly with language models and similar systems. Sure, you can find out what it does in some very specific situation by just running the code. But what if you have some broader question than that? Then simply knowing what the code does may not help you at all, because what the code does is gazillions of copies of "multiply these numbers together and add them". Again, I make no claim about whether the particular thing linked here offers much real insight. But it makes zero sense, so far as I can see, to dismiss it on the grounds that all you need to do is read the code.

xanderlewis 3y ago
You’re spot on; it’s like saying you can understand the game of chess by simply reading the rules. In a certain very superficial sense, yes. But the universe isn’t so simple. The same reason even a perfect understanding of what goes on at the level of subatomic particles isn’t thought to be enough to say we ‘understand the universe’. A hell of a lot can happen in between the setting out of some basic rules and the end — much higher level — result.
- danielmarkbruce 3y ago
  And yet...alpha zero.
  - xanderlewis 3y ago
    My entire point is that implementation isn’t sufficient for understanding. Alpha Zero is the perfect example of that; you can create an amazing chess playing machine and (potentially) learn nothing at all about how to play chess. …so what’s your point? I’m not getting it from those two words.
    - danielmarkbruce 3y ago
      Understanding how the machine plays or how you should play? They aren't the same thing. And that is the point - trying to analogize to some explicit, concrete function you can describe is backwards. These models are gigantic (even the 'small' ones), they are looking to minimize a loss function by looking in multi thousand dimensional space. It is the very opposite of something that fits in a human brain in any explicit fashion.
      - gjm11 3y ago
        So is what happens in an actual literal human brain. And yet, we spend quite a lot of our time thinking about what human brains do, and sometimes it's pretty useful. For a lot of this, we treat the actual brain as a black box and don't particularly care about how it does what it does, but knowing something about the internal workings at various levels of abstraction is useful too. Similarly, if for whatever reason you are interested in, or spend some of your time interacting with, transformer-based language models, then you might want some intuition for what they do and how. You'll never fit the whole thing in your brain. That's why you want simplified abstracted versions of it. Which, AIUI, is one thing that the OP is trying to do. (As I said before, I don't know how well it does it; what I'm objecting to is the idea that trying to do this is a waste of time because the only thing there is to know is that the model does what the code says it does.)

nl 3y ago

A walk through with what the data at each point looks like is actually pretty useful.

danielmarkbruce 3y ago
Sure, it is. But trying to explain it as though the weights have some goal is weird. They aren't trying to do anything. You have a loss function. The optimizer keeps moving weights around in an attempt to minimize the loss function. It's not more or less than that.
- MarkusQ 3y ago
  This is just wrong. First of all, you're rejecting teleological anthropomorphizing (saying it's weird to act as if "the weights have some goal") but then in the very next line you talk about the optimizer making "an attempt" to accomplish a goal. All of which misses the point, since the question is about _explanations_ not goals and intentions. Then you reject out of hand any other level of explanation than the one you favor, saying "it's not more or less than that" when in fact it is both more and less than that; you can climb the ladder of abstraction either way to build more or less abstract explanations. We can dig down and talk about how the optimizer adjusts weights, or how tensor math works and how it's used in this case, or about how GPUs work, or gates, or transistors, etc. Or we could climb up and talk about (as this article does) and talk about what attention heads do, and why they work, when they work, when they don't, etc.
  - danielmarkbruce 3y ago
    The optimizer has a goal. The weights in the model do not. The optimizer isn't the model. There is no contradiction if you know how it works. Climbing the layer of abstraction from model weights doesn't seem to work in this field. Just saying it's so doesn't make it so.
    - MarkusQ 3y ago
      > Just saying it's so doesn't make it so. Does that apply to you as well?

drdeca 3y ago

Understanding how a given CPU (+ the other computer hardware) works, does not suffice to understand what is going on when a particular program is running. For that, you need to either read the program, or an execution trace, or both, or something along these lines, which is specific to the program being run.

danielmarkbruce 3y ago
This is the wrong analogy. The transformer block is a bunch of code and weights. It's a set of instructions laying out which numbers to run which operations on. The optimizer changes weights to minimize a loss function during training and then the code implementing a forward pass just runs during inference. That's what it is doing. It's not doing something else. If the argument is that a model is a function approximator, then it certainly isn't approximating some function that performs worse at the task at hand, and it certainly isn't approximating a function we can describe in a few hundred words.
- FeepingCreature 3y ago
  We have no reason at all to be certain of the latter.
  - danielmarkbruce 3y ago
    There is pretty good reason. If it could be described explicitly in a few hundred words, it would be extremely unlikely that we'd have seen a jump in capability with model size.

richardw 3y ago

I'll go with an example to demonstrate why that's not always enough. Many people are quite keen to know what this (for example) is actually doing: float InvSqrt(float x){ float xhalf = 0.5f * x; int i = *(int*)&x; i = 0x5f3759df - (i >> 1); x = *(float*)&i; x = x*(1.5f - xhalf*x*x); return x; } From https://betterexplained.com/articles/understanding-quakes-fast-inverse-square-root/ https://betterexplained.com/articles/understanding-quakes-fa... In my case I don't have a huge amount of time to chase down every rabbit hole, but I'd love to accelerate intuition for LLM's. Multiple points of intuition or comparison really help. I'm also not a python expert - what you see and what I see from a line of code will be quite different.

danielmarkbruce 3y ago
The author is attempting to build an explicit mental model of what a bunch of weights are "doing". It's not really the same thing. They are minimizing the loss function. People try to (and often do) generate intuition for architectures that will work given the lay out of data. But, the reason models are so big now is that trying to understand what the model is "doing" in a way humans understand didn't work out so well.

magicalhippo 3y ago

> You can just follow the code and see what it's doing. It's not doing something more or less than that. And that's why we'll never have fun things like an obfuscated code contest. Oh wait[1]... [1]: https://www.ioccc.org/ https://www.ioccc.org/

psyklic 3y ago

Last week I trained an LLM to auto-unrotate a Caeser cipher, without it knowing the rotation key. It turns out there are only a few possible ways this can be done, e.g. by analyzing letter frequencies. Of course, the LLM must use one of these algorithms! Determining this high-level algorithm would be "understanding" what it is "actually doing". Through analysis, I proved it indeed counts each letter's frequency. It then combines the "overall" letter frequencies with another distribution of "first letter" frequencies. This increased its accuracy to nearly 100%. To me, this description is much nicer than had I "followed the code" and described it as millions of multiplications and additions. (Much of this ends up being noise anyway.)