3 ms·

> Pretending GPT is just using a percent chance that some token should show up next is absurd. It's almost a hundred layers of neural pathways. That is literal

• knome 4y ago
  A multi-layer neural net is not limited to vague statistical combinations. You can represent XOR in a neural network. A NAND. If you can represent NAND. You can represent arbitrary logic. Across billions of weights and across nearly a hundred layers of possible decision points. Meaning can be spread across the neurons in ways that allow each to be part of many many different representations. This is only in GPT-3. GPT-4 and soon 5 no doubt exceed these considerably. https://towardsdatascience.com/how-neural-networks-solve-the-xor-problem-59763136bdd7 https://towardsdatascience.com/how-neural-networks-solve-the...
  • dvt 4y ago
    > A multi-layer neural net is not limited to vague statistical combinations. It, absolutely, unequivocally, 100% is only limited to statistical inference. That's how it works. The problem of linear separability (which is like 60 years old, by the way) has nothing to do with logic, but it's actually a byproduct of (a) the fact that neural networks use linear activation functions, and (b) XOR is not separable by a single hyperplane. Hence, you need multiple layers. No idea how you get to "arbitrary logic" from this.
    • toxik 4y ago
      > neural networks use linear activation functions This is exactly wrong, NNs use nonlinear activation functions. That’s the whole point [in activation functions].