3 ms·

I don't think it's nearly as complex as you think. The important measure isn't transistors, but the amount of concepts formed by composing transistors together.

I don't think it's nearly as complex as you think. The important measure isn't transistors, but the amount of concepts formed by composing transistors together. There's no way in hell people can make sense of 19 billion separate concepts. Likely what's going on in the chip is just repeated patterns designed to make a very simple machine operate really really quickly. But of course I'm not denying the complexity. But the 19 billion implication here is a bit too much. It's likely just as complex as say linux.

eru 3y ago
> There's no way in hell people can make sense of 19 billion separate concepts. Likely what's going on in the chip is just repeated patterns designed to make a very simple machine operate really really quickly. You are right about human designed chips. If you use something like simulated evolution or backpropagation or some other techniques from machine learning or mathematical optimization, you can get machines with billions of parts but not necessarily any clear pattern nor repetition. The human brain might be an example of such a mechanism.
- corethree 3y ago
  The human brain IS an example of such a mechanism. We've done it artificially as well with machine learning and genetic algorithms. The amount of concepts in the neural network of an LLM is too large for us to fully understand. At best we can only hope to understand these things as a sort of curve fitting optimization problem.
  - eru 3y ago
    I was using some weasel words, because the human brain does have some structure that we can decipher. Specific injuries to the brain, even in different people, tend to cause the same specific problems. (With heavy emphasis on 'tend to', this is not deterministic, like removing the 'e' key from your keyboard.)
    - corethree 3y ago
      We can do the same with LLMs. We can tease out some macro structure behaviors by looking at intermediate output from the feed forward network. But you see how this is far from true understanding. It's like removing the WiFi module from a PC and you lose WiFi, therefore you know the WiFi module does WiFi stuff but in the end this action contributes nothing towards cohesive understanding of the PC.

eru 3y ago

> There's no way in hell people can make sense of 19 billion separate concepts. Likely what's going on in the chip is just repeated patterns designed to make a very simple machine operate really really quickly. You are right about human designed chips. If you use something like simulated evolution or backpropagation or some other techniques from machine learning or mathematical optimization, you can get machines with billions of parts but not necessarily any clear pattern nor repetition. The human brain might be an example of such a mechanism.

corethree 3y ago
The human brain IS an example of such a mechanism. We've done it artificially as well with machine learning and genetic algorithms. The amount of concepts in the neural network of an LLM is too large for us to fully understand. At best we can only hope to understand these things as a sort of curve fitting optimization problem.
- eru 3y ago
  I was using some weasel words, because the human brain does have some structure that we can decipher. Specific injuries to the brain, even in different people, tend to cause the same specific problems. (With heavy emphasis on 'tend to', this is not deterministic, like removing the 'e' key from your keyboard.)
  - corethree 3y ago
    We can do the same with LLMs. We can tease out some macro structure behaviors by looking at intermediate output from the feed forward network. But you see how this is far from true understanding. It's like removing the WiFi module from a PC and you lose WiFi, therefore you know the WiFi module does WiFi stuff but in the end this action contributes nothing towards cohesive understanding of the PC.