4 ms·

Can someone explain exactly what is the "unknown" of neural networks? We built them, we know what they comprise of and how they work. Yes, we can't map out ever

Can someone explain exactly what is the "unknown" of neural networks? We built them, we know what they comprise of and how they work. Yes, we can't map out every single connection between nodes in this "multilayer perceptron" but don't we know how these connections are formed?

taneq 2y ago
There’s a ton of research going into analysing and reverse engineering NNs, this “they’re mysterious black boxes and forever inscrutable” narrative is outdated.
lupire 2y ago
We don't know what each connection means, what information is encoded in each weight. We don't know how it would behave differently if each of the million or trillion weights was changed. Compare this to dictionaey, where it's obvious what information is on each page and each line.
wslh 2y ago
The brain serves as a useful analogy, even though LLMs are not brains. Just as we can’t fully understand how we think by merely examining all of our neurons, understanding LLMs requires more than analyzing their individual components, though decoding LLMs is most likely easier, which doesn't mean easy.
famouswaffles 2y ago
Sota LLMs like GPT-4o can natively understand b64 encoded text. Now we have algorithms that can decode and encode b64 text. Is that what GPT-4o is doing ? Did training learn that algorithm ? Clearly not or at least not completely because typos in b64 that would destroy any chance of extracting meaning in the original text for our algorithms are barely an inconvenience for 4o. So how is it decoding b64 then ? We have no idea. We don't built Neural Networks. Not really. We build architectures and then train them. Whatever they learn is outside the scope of human action beyond supplying the training data. What they learn is largely unknown beyond trivial toy examples. We know connections form, we can see the weights, we can even see the matrices multiplying. We don't know what any of those calculations are doing. We don't know what they mean. Would an alien understand C Code just because he could see it executing ?
- HarHarVeryFunny 2y ago
  Base64 encoding is very simple - it's just taking each 6-bits of the input and encoding (replacing) it as one of the 64 (2^6) characters A-Za-z0-9+/. If the input is 8-bit ASCII text, then each 3 input characters will be encoded as 4 Base64 characters (3 * 8 = 24 bits = 4 * 6-bit Base64 chunks). So, this is very similar to an LLM having to deal with tokenized input, but instead of sequences of tokens representing words you've got sequences of Base64 characters representing words.
  - famouswaffles 2y ago
    It's not about how simple B64 is or isn't. In fact i chose a simple problem we've already solved algorithmically on purpose. It's that all you've just said, reasonable as it may sound is entirely speculation. Maybe "no idea" was a bit much for this example but any idea certainly didn't come from seeing the matrices themselves fly.
    - HarHarVeryFunny 2y ago
      Huh? I just pointed out what Base64 encoding actually is - not some complex algorithm, but effectively just a tokenization scheme. This isn't speculation - I've implemented Base64 decode/encode myself, and you can google for the definition if you don't believe I've accurately described it!
      - famouswaffles 2y ago
        The speculation here is not about what b64 text is. It's about how the LLM has learnt to process it. Edit: Basically, For all anyone knows, it treats b64 as another language entirely and decoding it is akin in the network to translating French rather than the very simple swapping you've just described.
        HarHarVeryFunny 2y ago
        LLMs, just like all modern neural nets, are trained via gradient descent which means following the most direct path (steepest gradient on the error surface) to reduce the error, with no more changes to weights once the error gradient is zero. Complexity builds upon simplicity, and the LLM will begin by noticing the direct (and repeated without variation) predictive relationship between Base64 encoded text and corresponding plain text in the training set. Having learnt this simple way to predict Base64 decoding/encoding, there is simply no mechanism whereby it could change to a more complex "like translating French" way of doing it. Once the training process has discovered that Base64 text decoding can be PERFECTLY predicted by a simple mapping, then the training error will be zero and no more changes (unnecessary complexification) will take place.
spencerchubb 2y ago
We know the process to train a model, but when a model makes a prediction we don't know exactly "how" it predicts the way it does. We can use the economy as an analogy. No single person really understands the whole supply chain. But we know that each person in the supply chain is trying to maximize their own profit, and that ultimately delivers goods and services to a consumer.
Lerc 2y ago
We know how they are formed(and how to form them), we don't know why forming in that particular way solves the problem at hand. Even this characterization is not strictly valid anymore, there is a great deal of research into what's going on inside the black box. The problem was never that it was a black box(we can look inside at any time), but that it was hard to understand. KANs help some of that be placed into mathematical formulation. Generating mappings of activations over data similarly grants insight.
mjburgess 2y ago
* Given the training data, and the architecture of the network, why does SGD with backprop find the given f? vs. any other of an infinite set. * Why are there are a set of f each with 0-loss that work? * Given the weight space, and an f within it, why/when is a task/skill defined as a subset of that space covered by f? I think a major reasons why these are hard to answer is that it's assumed that NNs are operating within an inferential statistical context (ie., reversing some latent structure in the data). But they're really bad at that. In my view, they are just representation-builders that find proxy representations in a proxy "task" space (def, aprox, proxy = "shadow of some real structure, as captured in an unrelated space").
_navierstokes 2y ago
Skipping some detail: the model applies many high-dimensional functions to the input, and we don't know the reasoning for why these functions solve the problem. Reducing the dimension of the weights to human-readable values is non-trivial, and multiple neurons interact in unpredictable ways. Interpretability research has resulted in many useful results and pretty visualizations[1][2], and there are many efforts to understand Transformers[3][4] but we're far from being able to completely explain the large models currently in use. [1] - https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ [2] - https://distill.pub/2019/activation-atlas/ https://distill.pub/2019/activation-atlas/ [3] - https://transformer-circuits.pub/ https://transformer-circuits.pub/ [4] - https://arxiv.org/pdf/2407.02646 https://arxiv.org/pdf/2407.02646

taneq 2y ago

There’s a ton of research going into analysing and reverse engineering NNs, this “they’re mysterious black boxes and forever inscrutable” narrative is outdated.

lupire 2y ago

We don't know what each connection means, what information is encoded in each weight. We don't know how it would behave differently if each of the million or trillion weights was changed. Compare this to dictionaey, where it's obvious what information is on each page and each line.

wslh 2y ago

The brain serves as a useful analogy, even though LLMs are not brains. Just as we can’t fully understand how we think by merely examining all of our neurons, understanding LLMs requires more than analyzing their individual components, though decoding LLMs is most likely easier, which doesn't mean easy.

famouswaffles 2y ago

Sota LLMs like GPT-4o can natively understand b64 encoded text. Now we have algorithms that can decode and encode b64 text. Is that what GPT-4o is doing ? Did training learn that algorithm ? Clearly not or at least not completely because typos in b64 that would destroy any chance of extracting meaning in the original text for our algorithms are barely an inconvenience for 4o. So how is it decoding b64 then ? We have no idea. We don't built Neural Networks. Not really. We build architectures and then train them. Whatever they learn is outside the scope of human action beyond supplying the training data. What they learn is largely unknown beyond trivial toy examples. We know connections form, we can see the weights, we can even see the matrices multiplying. We don't know what any of those calculations are doing. We don't know what they mean. Would an alien understand C Code just because he could see it executing ?

HarHarVeryFunny 2y ago
Base64 encoding is very simple - it's just taking each 6-bits of the input and encoding (replacing) it as one of the 64 (2^6) characters A-Za-z0-9+/. If the input is 8-bit ASCII text, then each 3 input characters will be encoded as 4 Base64 characters (3 * 8 = 24 bits = 4 * 6-bit Base64 chunks). So, this is very similar to an LLM having to deal with tokenized input, but instead of sequences of tokens representing words you've got sequences of Base64 characters representing words.
- famouswaffles 2y ago
  It's not about how simple B64 is or isn't. In fact i chose a simple problem we've already solved algorithmically on purpose. It's that all you've just said, reasonable as it may sound is entirely speculation. Maybe "no idea" was a bit much for this example but any idea certainly didn't come from seeing the matrices themselves fly.
  - HarHarVeryFunny 2y ago
    Huh? I just pointed out what Base64 encoding actually is - not some complex algorithm, but effectively just a tokenization scheme. This isn't speculation - I've implemented Base64 decode/encode myself, and you can google for the definition if you don't believe I've accurately described it!
    - famouswaffles 2y ago
      The speculation here is not about what b64 text is. It's about how the LLM has learnt to process it. Edit: Basically, For all anyone knows, it treats b64 as another language entirely and decoding it is akin in the network to translating French rather than the very simple swapping you've just described.
      - HarHarVeryFunny 2y ago
        LLMs, just like all modern neural nets, are trained via gradient descent which means following the most direct path (steepest gradient on the error surface) to reduce the error, with no more changes to weights once the error gradient is zero. Complexity builds upon simplicity, and the LLM will begin by noticing the direct (and repeated without variation) predictive relationship between Base64 encoded text and corresponding plain text in the training set. Having learnt this simple way to predict Base64 decoding/encoding, there is simply no mechanism whereby it could change to a more complex "like translating French" way of doing it. Once the training process has discovered that Base64 text decoding can be PERFECTLY predicted by a simple mapping, then the training error will be zero and no more changes (unnecessary complexification) will take place.

spencerchubb 2y ago

We know the process to train a model, but when a model makes a prediction we don't know exactly "how" it predicts the way it does. We can use the economy as an analogy. No single person really understands the whole supply chain. But we know that each person in the supply chain is trying to maximize their own profit, and that ultimately delivers goods and services to a consumer.

Lerc 2y ago

We know how they are formed(and how to form them), we don't know why forming in that particular way solves the problem at hand. Even this characterization is not strictly valid anymore, there is a great deal of research into what's going on inside the black box. The problem was never that it was a black box(we can look inside at any time), but that it was hard to understand. KANs help some of that be placed into mathematical formulation. Generating mappings of activations over data similarly grants insight.

mjburgess 2y ago

* Given the training data, and the architecture of the network, why does SGD with backprop find the given f? vs. any other of an infinite set. * Why are there are a set of f each with 0-loss that work? * Given the weight space, and an f within it, why/when is a task/skill defined as a subset of that space covered by f? I think a major reasons why these are hard to answer is that it's assumed that NNs are operating within an inferential statistical context (ie., reversing some latent structure in the data). But they're really bad at that. In my view, they are just representation-builders that find proxy representations in a proxy "task" space (def, aprox, proxy = "shadow of some real structure, as captured in an unrelated space").

_navierstokes 2y ago

Skipping some detail: the model applies many high-dimensional functions to the input, and we don't know the reasoning for why these functions solve the problem. Reducing the dimension of the weights to human-readable values is non-trivial, and multiple neurons interact in unpredictable ways. Interpretability research has resulted in many useful results and pretty visualizations[1][2], and there are many efforts to understand Transformers[3][4] but we're far from being able to completely explain the large models currently in use. [1] - https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ [2] - https://distill.pub/2019/activation-atlas/ https://distill.pub/2019/activation-atlas/ [3] - https://transformer-circuits.pub/ https://transformer-circuits.pub/ [4] - https://arxiv.org/pdf/2407.02646 https://arxiv.org/pdf/2407.02646