4 ms·

I’ve never understood the hidden layers argument. Ultimately these models are executing code. You can examine the code. Why can’t that be done?

afthonos 3y ago
Think of it this way: looking at the neurons is like looking at the assembly and trying to understand how a video is played on YouTube. Yes, in theory it’s possible, but realistically no one would be able to do that. At least not in a reasonable amount of time.
- astrange 3y ago
  Video decoding works by reading and writing pixels to the same buffers you use for displaying them, so you could see a lot of how it works by watching the accesses there.
  - afthonos 3y ago
    Yeah, but imagine you don’t know what “decoding”, “buffers”, and “display” mean in a software context.
OkayPhysicist 3y ago
Fundamentally, artificial neural networks boil down to some code that can be written, mathematically, as σ(C*σ(B*σ(A*x)))... = y repeated matrix-vector multiplication, separated by nonlinear functions, σ(). A,B,C etc here are your weights, x is your input vector, y is your output vector. Any nonlinear function will work, some behave better than others, but they must be present, otherwise you could simplify the problem by apply the associative property to the weights side of things, and you'd effectively have a single layer that can only simulate linear functions. A hidden layer is just any vector that is intermediate to that calculation, for example the result of σ(A*x) would be the first hidden layer. So right off the bat, you obviously have a problem: "Examining the code" means examining the weights, which are just gigabyte/terabyte sized matrices. Opaque is putting it mildly. The only sane approach is to start from one of our known-meaningful vectors (the input or output), and work our way inwards from there, either seeing what elements of hidden layer vectors have the most significant value when a certain input is applied, or determining what hidden layer vector values produce values closest resembling a desired output. Then you start running into the problems described in the article.
- janalsncm 3y ago
  This is a really good explanation for a fully connected network. Most models will have more complex architectures so are even more complicated than this.
  - OkayPhysicist 3y ago
    I haven't been great about keeping up with advances in the field, but to my understanding most if not all architectures in effect merely enforce symmetries upon the network. That is, they can be represented by a fully connected network but in that representation not all weights are free, in that some are fixed (0 or 1) or some are dependent (A(1,1) will also be equal to C(1,1)).
    - janalsncm 3y ago
      I don’t blame you, things are pretty diversified so I’m white knuckling it through my own subfield. But as a simple example, convolution would be a little tedious to describe in the general notation you wrote above without getting into the image dimensions, stride, padding etc., not to mention residual layers or norm layers that are commonly used. Then there are things like stop gradient, dropout or even other training targets which are only used during training but not inference.
      - OkayPhysicist 3y ago
        Convolution actually can be pretty elegantly translated into multiplication by a matrix with symmetries. It's been a few years, but that was an example we had to work out by hand in my Deep Learning course at university. It's pretty much the quintessential example of an enforced symmetry, in that it introduces a symmetry against translation.
drdeca 3y ago
If by “code” you mean “the source code people wrote”, then, that source code by itself does not determine the behavior. The behavior is determined by the numbers. Saying that understanding that code implies understanding how the network works, is like saying “John wrote this emulator for GBA games, therefore he must understand how the game [some GBA game] works”. Except, instead of the game being written in assembly, it was written in malbolge.
cwillu 3y ago
“This is great for AIs but bad for interpreters. We hoped we could figure out what our AIs were doing just by looking at them. But it turns out they’re simulating much bigger and more complicated AIs, and if we want to know what’s going on, we have to look at those. But those AIs only exist in simulated abstract hyperdimensional spaces. Sounds hard to dissect!” “Still, last month Anthropic’s interpretability team announced that they successfully dissected of one of the simulated AIs in its abstract hyperdimensional space. (finally, we’re back to the monosemanticity paper!) First the researchers trained a very simple 512-neuron AI to predict text, like a tiny version of GPT or Anthropic’s competing model Claude. Then, they trained a second AI called an autoencoder to predict the activations of the first AI. They told it to posit a certain number of features (the experiments varied between ~2,000 and ~100,000), corresponding to the neurons of the higher-dimensional AI it was simulating. Then they made it predict how those features mapped onto the real neurons of the real AI. They found that even though the original AI’s neurons weren’t comprehensible, the new AI’s simulated neurons (aka “features”) were! They were monosemantic, ie they meant one specific thing. Here’s feature #2663 (remember, the original AI only had 512 neurons, but they’re treating it as simulating a larger AI with up to ~100,000 neuron-features).”
quickthrower2 3y ago
With these models, the Pytorch code is an interpreter, and the 100Gb of weights is the code. So you have say a 10G (10 weights = 1 LOC) LOC codebase, all in assembly, no comments. G'luck. Oh and it doesn't always use logic. So there are no ORs and ANDs and IFs, just +, *, exp, max, etc.

afthonos 3y ago

Think of it this way: looking at the neurons is like looking at the assembly and trying to understand how a video is played on YouTube. Yes, in theory it’s possible, but realistically no one would be able to do that. At least not in a reasonable amount of time.

astrange 3y ago
Video decoding works by reading and writing pixels to the same buffers you use for displaying them, so you could see a lot of how it works by watching the accesses there.
- afthonos 3y ago
  Yeah, but imagine you don’t know what “decoding”, “buffers”, and “display” mean in a software context.

OkayPhysicist 3y ago

Fundamentally, artificial neural networks boil down to some code that can be written, mathematically, as σ(C*σ(B*σ(A*x)))... = y repeated matrix-vector multiplication, separated by nonlinear functions, σ(). A,B,C etc here are your weights, x is your input vector, y is your output vector. Any nonlinear function will work, some behave better than others, but they must be present, otherwise you could simplify the problem by apply the associative property to the weights side of things, and you'd effectively have a single layer that can only simulate linear functions. A hidden layer is just any vector that is intermediate to that calculation, for example the result of σ(A*x) would be the first hidden layer. So right off the bat, you obviously have a problem: "Examining the code" means examining the weights, which are just gigabyte/terabyte sized matrices. Opaque is putting it mildly. The only sane approach is to start from one of our known-meaningful vectors (the input or output), and work our way inwards from there, either seeing what elements of hidden layer vectors have the most significant value when a certain input is applied, or determining what hidden layer vector values produce values closest resembling a desired output. Then you start running into the problems described in the article.

janalsncm 3y ago
This is a really good explanation for a fully connected network. Most models will have more complex architectures so are even more complicated than this.
- OkayPhysicist 3y ago
  I haven't been great about keeping up with advances in the field, but to my understanding most if not all architectures in effect merely enforce symmetries upon the network. That is, they can be represented by a fully connected network but in that representation not all weights are free, in that some are fixed (0 or 1) or some are dependent (A(1,1) will also be equal to C(1,1)).
  - janalsncm 3y ago
    I don’t blame you, things are pretty diversified so I’m white knuckling it through my own subfield. But as a simple example, convolution would be a little tedious to describe in the general notation you wrote above without getting into the image dimensions, stride, padding etc., not to mention residual layers or norm layers that are commonly used. Then there are things like stop gradient, dropout or even other training targets which are only used during training but not inference.
    - OkayPhysicist 3y ago
      Convolution actually can be pretty elegantly translated into multiplication by a matrix with symmetries. It's been a few years, but that was an example we had to work out by hand in my Deep Learning course at university. It's pretty much the quintessential example of an enforced symmetry, in that it introduces a symmetry against translation.

drdeca 3y ago

If by “code” you mean “the source code people wrote”, then, that source code by itself does not determine the behavior. The behavior is determined by the numbers. Saying that understanding that code implies understanding how the network works, is like saying “John wrote this emulator for GBA games, therefore he must understand how the game [some GBA game] works”. Except, instead of the game being written in assembly, it was written in malbolge.

cwillu 3y ago

“This is great for AIs but bad for interpreters. We hoped we could figure out what our AIs were doing just by looking at them. But it turns out they’re simulating much bigger and more complicated AIs, and if we want to know what’s going on, we have to look at those. But those AIs only exist in simulated abstract hyperdimensional spaces. Sounds hard to dissect!” “Still, last month Anthropic’s interpretability team announced that they successfully dissected of one of the simulated AIs in its abstract hyperdimensional space. (finally, we’re back to the monosemanticity paper!) First the researchers trained a very simple 512-neuron AI to predict text, like a tiny version of GPT or Anthropic’s competing model Claude. Then, they trained a second AI called an autoencoder to predict the activations of the first AI. They told it to posit a certain number of features (the experiments varied between ~2,000 and ~100,000), corresponding to the neurons of the higher-dimensional AI it was simulating. Then they made it predict how those features mapped onto the real neurons of the real AI. They found that even though the original AI’s neurons weren’t comprehensible, the new AI’s simulated neurons (aka “features”) were! They were monosemantic, ie they meant one specific thing. Here’s feature #2663 (remember, the original AI only had 512 neurons, but they’re treating it as simulating a larger AI with up to ~100,000 neuron-features).”

quickthrower2 3y ago

With these models, the Pytorch code is an interpreter, and the 100Gb of weights is the code. So you have say a 10G (10 weights = 1 LOC) LOC codebase, all in assembly, no comments. G'luck. Oh and it doesn't always use logic. So there are no ORs and ANDs and IFs, just +, *, exp, max, etc.