5 ms·

I've been saying this to people. Tons of people don't realize that we have no idea how these things work. So questions like is it AGI? It is conscious? Where th

I've been saying this to people. Tons of people don't realize that we have no idea how these things work. So questions like is it AGI? It is conscious? Where the questions themselves contain words with fuzzy and ill defined meanings are pointless. The core of the matter is: We don't know! So when someone says chatGPT could be conscious. Or when someone says it can't be conscious. We don't actually know! And we aren't even fully sure about the definition of consciousness. My problem with HN is that a ton of people here take the stance that we know how LLMs work and that we know definitively they aren't conscious and the people who say otherwise are alarmist and stupid. I think the fact of the matter is, if you're putting your foot down and saying LLMs aren't intelligent... you're wildly illogical and misinformed about the status quo of Artificial intelligence and a huge portion of the mob mentality on HN thinks this way. It's like there's these buzz phrases getting thrown around saying that the LLM is just a glorified auto complete (which it is) and people latch on to this buzz phrases and their entire understanding of LLMs becomes basically a composition of these buzz concepts like "transformers" and "LLMs" and "chain of thought" when in actuality real critical thinking about what's going on in these networks tells you we don't UNDERSTAND anything. Also the characterization in the article is mistaken. It says we understand LLMs in a limited way. Yeah sure. It's as limited as our understanding of the human brain. You know scientists found a way to stimulate the reward centers of the brain using electrodes and were able to cause a person to feel the utmost pleasure? The whole golden gate bridge thing is exactly that. You perturb something in the network and causes a semi predictable output. In the end we still generally don't get wtf is going on. We literally understand nothing. What we do understand is so minuscule compared with what we don't understand that it's negligible.

sirwhinesalot 1y ago
This is such a weird take to me. We know exactly how LLMs and neural networks in general work. They're just highly scaled up versions of their smaller curve fitting cousins. For those we can even make pretty visualizations that show exactly what is happening as the network "learns". I don't mean "we can see parts of the brain light up", I mean "we can see the cuts and bends each ReLU is doing to the input". We built these things, we know exactly how they work. There's no surprise beyond just how good prediction accuracy gets with a big enough model. Deep Neural Networks are also a very different architecture from what is going on in our brains (which work more like Spiking Neural Networks) and our brains don't do backpropagation, so you can't even make direct comparisons.
- SEGyges 1y ago
  fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to make sense of and it was posted on hackernews
  - sirwhinesalot 1y ago
    Your article, unlike the bizarre desperate take from the poster above, is actually very good. We do not understand the features the neural net learns, that's 100% true (and really the whole point of them in the first place). For small image recognition models we can visualize them and get an intuition for what they are doing, but it doesn't really matter. For even smaller models, we can translate them to a classical AI model (like a mixed integer program as an example) and actually do various "queries" on the model itself to, e.g., learn that the network recognizes the number "8" by just checking 2 pixels in the image. None of this changes the fact that we know what these things are and how they work, because we built them. Any comparisons to our lack of knowledge of the human brain are ridiculous. LLMs are obviously not conscious, they don't even have real "state", they're an approximated pure function f(context: List<Token>) -> Token, that's run in a loop. The only valid alarmist take is that we're using black box algorithms to make decisions with serious real-world impact, but this is true of any black box algorithm, not just the latest and greatest ML models.
    - dpoloncsak 1y ago
      Its a complex adaptive system, right? Isn't that the whole idea? We know how each part of the system works by itself. We know all inputs, and can measure outputs. I still (even if I actually understood the math) cannot tell you 'If you prompt 'x', the model will return 'y' with 100% confidence.
      - sirwhinesalot 1y ago
        > If you prompt 'x', the model will return 'y' with 100% confidence. We can do this for smaller models. Which means it's a problem of scale/computing power rather than a fundamental limitation. The situation with the human brain is completely different. We know neurons exchange information and how that works, and we have a pretty good understanding of the architecture of parts of the brain like the visual cortex, but we have no idea of the architecture as a whole. We know the architecture of an LLM. We know how the data flows. We know what it is the individual neurons are learning (cuts and bends of a plane in a hyperdimensional space). We know how the weights are learned (backpropagation). We know the "algorithm" the LLM as a whole is approximating (List<Token> -> Token). Yes there are emergent properties we don't understand but the same is true of a spam filter. Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly.

sirwhinesalot 1y ago

This is such a weird take to me. We know exactly how LLMs and neural networks in general work. They're just highly scaled up versions of their smaller curve fitting cousins. For those we can even make pretty visualizations that show exactly what is happening as the network "learns". I don't mean "we can see parts of the brain light up", I mean "we can see the cuts and bends each ReLU is doing to the input". We built these things, we know exactly how they work. There's no surprise beyond just how good prediction accuracy gets with a big enough model. Deep Neural Networks are also a very different architecture from what is going on in our brains (which work more like Spiking Neural Networks) and our brains don't do backpropagation, so you can't even make direct comparisons.

SEGyges 1y ago
fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to make sense of and it was posted on hackernews
- sirwhinesalot 1y ago
  Your article, unlike the bizarre desperate take from the poster above, is actually very good. We do not understand the features the neural net learns, that's 100% true (and really the whole point of them in the first place). For small image recognition models we can visualize them and get an intuition for what they are doing, but it doesn't really matter. For even smaller models, we can translate them to a classical AI model (like a mixed integer program as an example) and actually do various "queries" on the model itself to, e.g., learn that the network recognizes the number "8" by just checking 2 pixels in the image. None of this changes the fact that we know what these things are and how they work, because we built them. Any comparisons to our lack of knowledge of the human brain are ridiculous. LLMs are obviously not conscious, they don't even have real "state", they're an approximated pure function f(context: List<Token>) -> Token, that's run in a loop. The only valid alarmist take is that we're using black box algorithms to make decisions with serious real-world impact, but this is true of any black box algorithm, not just the latest and greatest ML models.
  - dpoloncsak 1y ago
    Its a complex adaptive system, right? Isn't that the whole idea? We know how each part of the system works by itself. We know all inputs, and can measure outputs. I still (even if I actually understood the math) cannot tell you 'If you prompt 'x', the model will return 'y' with 100% confidence.
    - sirwhinesalot 1y ago
      > If you prompt 'x', the model will return 'y' with 100% confidence. We can do this for smaller models. Which means it's a problem of scale/computing power rather than a fundamental limitation. The situation with the human brain is completely different. We know neurons exchange information and how that works, and we have a pretty good understanding of the architecture of parts of the brain like the visual cortex, but we have no idea of the architecture as a whole. We know the architecture of an LLM. We know how the data flows. We know what it is the individual neurons are learning (cuts and bends of a plane in a hyperdimensional space). We know how the weights are learned (backpropagation). We know the "algorithm" the LLM as a whole is approximating (List<Token> -> Token). Yes there are emergent properties we don't understand but the same is true of a spam filter. Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly.