4 ms·

This is such a weird take to me. We know exactly how LLMs and neural networks in general work. They're just highly scaled up versions of their smaller curve fi

This is such a weird take to me. We know exactly how LLMs and neural networks in general work. They're just highly scaled up versions of their smaller curve fitting cousins. For those we can even make pretty visualizations that show exactly what is happening as the network "learns". I don't mean "we can see parts of the brain light up", I mean "we can see the cuts and bends each ReLU is doing to the input". We built these things, we know exactly how they work. There's no surprise beyond just how good prediction accuracy gets with a big enough model. Deep Neural Networks are also a very different architecture from what is going on in our brains (which work more like Spiking Neural Networks) and our brains don't do backpropagation, so you can't even make direct comparisons.

SEGyges 1y ago
fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to make sense of and it was posted on hackernews
- sirwhinesalot 1y ago
  Your article, unlike the bizarre desperate take from the poster above, is actually very good. We do not understand the features the neural net learns, that's 100% true (and really the whole point of them in the first place). For small image recognition models we can visualize them and get an intuition for what they are doing, but it doesn't really matter. For even smaller models, we can translate them to a classical AI model (like a mixed integer program as an example) and actually do various "queries" on the model itself to, e.g., learn that the network recognizes the number "8" by just checking 2 pixels in the image. None of this changes the fact that we know what these things are and how they work, because we built them. Any comparisons to our lack of knowledge of the human brain are ridiculous. LLMs are obviously not conscious, they don't even have real "state", they're an approximated pure function f(context: List<Token>) -> Token, that's run in a loop. The only valid alarmist take is that we're using black box algorithms to make decisions with serious real-world impact, but this is true of any black box algorithm, not just the latest and greatest ML models.
  - dpoloncsak 1y ago
    Its a complex adaptive system, right? Isn't that the whole idea? We know how each part of the system works by itself. We know all inputs, and can measure outputs. I still (even if I actually understood the math) cannot tell you 'If you prompt 'x', the model will return 'y' with 100% confidence.
    - sirwhinesalot 1y ago
      > If you prompt 'x', the model will return 'y' with 100% confidence. We can do this for smaller models. Which means it's a problem of scale/computing power rather than a fundamental limitation. The situation with the human brain is completely different. We know neurons exchange information and how that works, and we have a pretty good understanding of the architecture of parts of the brain like the visual cortex, but we have no idea of the architecture as a whole. We know the architecture of an LLM. We know how the data flows. We know what it is the individual neurons are learning (cuts and bends of a plane in a hyperdimensional space). We know how the weights are learned (backpropagation). We know the "algorithm" the LLM as a whole is approximating (List<Token> -> Token). Yes there are emergent properties we don't understand but the same is true of a spam filter. Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly.
      - ninetyninenine 1y ago
        >Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly. Don't call my claim silly. I'm sick of your attitude. Why can't you have a civil discussion? Literally we don't know. You can't make a claim that it's silly when you can't even define what consciousness is. You don't know how human brains work, you don't know how consciousness forms, you don't know how emergence in LLMs work. So your claim here is logically just made up out of thin air. Sure we "understand" LLMs from the curve fitting perspective. But the entirety of why we use LLMs and what we use it for arises from the emergence which is what we don't understand. Curve fitting is like 1% of the LLM, it is the emergent properties we completely don't get (99%) and take advantage of on a daily basis. Curve fitting is just a high level concept that allows us to construct the algorithm which is the actual thing that does the hard work of wiring up the atomic units of the network. >Yes there are emergent properties we don't understand but the same is true of a spam filter. Yeah and so? Your statement proves nothing. It just illustrates a contrast in sentiment. The spam filter is a trivial thing, the human brain is not. We don't understand the spam filter. And this is the most interesting part of it all is that the SAME scaling problem that prevents us from understanding the spam filter can be characterized as the reason that prevents us from understanding BOTH the LLM and the human brain. Your statement doesn't change anything. It's just using sentiment to try to re-characterize a problem in a different light.
- polotics 1y ago
  Coud you provide a link so we can follow your thread of thought on this? It appears your article got submitted by another user than you.
ninetyninenine 1y ago
>This is such a weird take to me. We know exactly how LLMs and neural networks in general work. Why do you make such crazy statements when the article you’re responding to literally says the opposite of this. Like your statement is categorically false and both the article and I are in total contradiction to it. The rest of your post doesn’t even get into the nuances of what we do and don’t understand. > They're just highly scaled up versions of their smaller curve fitting cousins. For those we can even make pretty visualizations that show exactly what is happening as the network "learns". This proves nothing. Additionally is in direct contradiction to what the OP and I wrote. You describe the minuscule aspects that we do understand but neglect to describe the fact that we overall don’t understand. Saying we understand curve fitting is like saying we understand the human brain because we can completely understand how atoms work and the human brain is made up of atoms therefore we understand the human brain. No. We understand atoms. We don’t understand the human brain even though we know the brain is just a bunch of atoms. The generalization of this is that the amount we don’t understand eclipses by a massive amount the parts we do understand. Let me spell it out to your brain what this means overall: we don’t understand shit. > I don't mean "we can see parts of the brain light up", I mean "we can see the cuts and bends each ReLU is doing to the input". Even with direct insight on every single signal propagating through the feed forward network we still don’t understand it. Like the OPs article he says it’s a scaling problem. Again I don’t understand how come you’re only referring to the miniscule aspects we do understand and not even once mentioning the massive aspects we don’t understand. > We built these things, we know exactly how they work. There's no surprise beyond just how good prediction accuracy gets with a big enough model. No we don’t. Sometimes I find basic rationality and common sense just doesn’t work with people. What ends up working is citing experts in the field saying the exact fucking opposite of you: https://youtu.be/qrvK_KuIeJk?t=284 https://youtu.be/qrvK_KuIeJk?t=284 That’s Geoffrey Hinton, the guy who made ML relevant again in the last decade. Like literally word for word saying exactly opposite of what you’re saying. The interviewer even said “we built the thing, we know how it works” and Geoffrey is like “no we don’t”. Bro. Wake up. > Deep Neural Networks are also a very different architecture from what is going on in our brains (which work more like Spiking Neural Networks) and our brains don't do backpropagation, so you can't even make direct comparisons. Bro we don’t understand deep neural networks. Throwing big words like back propagation and even understanding that it’s just the chain rule from calculus doesn’t mean shit. We overall don’t understand it because it’s a scaling problem. There’s no theory or model that characterizes how a signal propagates through a trained network. It’s like saying you know assembly language and therefore you understand all of Linux. No we understand assembly but these learning algorithms are what built Linux and thus we don’t understand it because we didn’t build it. We don’t understand the human brain and we don’t understand deep neural networks. You know what. Just take a look at what Geoffrey Hinton says. Like if you feel my take is so bizarre and felt the need to comment on it I hope his take rewires your brain and helps you realize you’re the one out of touch. Rationality rarely changes people’s minds, but someone with a higher reputation often is capable of actually getting a person out of their own biased headspace. So listen to what he says then couple it with the explanation here and completely flip your opinion. Or don’t. I rarely see people just do an opinion flip. Humans are just too biased.

SEGyges 1y ago

fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to make sense of and it was posted on hackernews

sirwhinesalot 1y ago
Your article, unlike the bizarre desperate take from the poster above, is actually very good. We do not understand the features the neural net learns, that's 100% true (and really the whole point of them in the first place). For small image recognition models we can visualize them and get an intuition for what they are doing, but it doesn't really matter. For even smaller models, we can translate them to a classical AI model (like a mixed integer program as an example) and actually do various "queries" on the model itself to, e.g., learn that the network recognizes the number "8" by just checking 2 pixels in the image. None of this changes the fact that we know what these things are and how they work, because we built them. Any comparisons to our lack of knowledge of the human brain are ridiculous. LLMs are obviously not conscious, they don't even have real "state", they're an approximated pure function f(context: List<Token>) -> Token, that's run in a loop. The only valid alarmist take is that we're using black box algorithms to make decisions with serious real-world impact, but this is true of any black box algorithm, not just the latest and greatest ML models.
- dpoloncsak 1y ago
  Its a complex adaptive system, right? Isn't that the whole idea? We know how each part of the system works by itself. We know all inputs, and can measure outputs. I still (even if I actually understood the math) cannot tell you 'If you prompt 'x', the model will return 'y' with 100% confidence.
  - sirwhinesalot 1y ago
    > If you prompt 'x', the model will return 'y' with 100% confidence. We can do this for smaller models. Which means it's a problem of scale/computing power rather than a fundamental limitation. The situation with the human brain is completely different. We know neurons exchange information and how that works, and we have a pretty good understanding of the architecture of parts of the brain like the visual cortex, but we have no idea of the architecture as a whole. We know the architecture of an LLM. We know how the data flows. We know what it is the individual neurons are learning (cuts and bends of a plane in a hyperdimensional space). We know how the weights are learned (backpropagation). We know the "algorithm" the LLM as a whole is approximating (List<Token> -> Token). Yes there are emergent properties we don't understand but the same is true of a spam filter. Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly.
    - ninetyninenine 1y ago
      >Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly. Don't call my claim silly. I'm sick of your attitude. Why can't you have a civil discussion? Literally we don't know. You can't make a claim that it's silly when you can't even define what consciousness is. You don't know how human brains work, you don't know how consciousness forms, you don't know how emergence in LLMs work. So your claim here is logically just made up out of thin air. Sure we "understand" LLMs from the curve fitting perspective. But the entirety of why we use LLMs and what we use it for arises from the emergence which is what we don't understand. Curve fitting is like 1% of the LLM, it is the emergent properties we completely don't get (99%) and take advantage of on a daily basis. Curve fitting is just a high level concept that allows us to construct the algorithm which is the actual thing that does the hard work of wiring up the atomic units of the network. >Yes there are emergent properties we don't understand but the same is true of a spam filter. Yeah and so? Your statement proves nothing. It just illustrates a contrast in sentiment. The spam filter is a trivial thing, the human brain is not. We don't understand the spam filter. And this is the most interesting part of it all is that the SAME scaling problem that prevents us from understanding the spam filter can be characterized as the reason that prevents us from understanding BOTH the LLM and the human brain. Your statement doesn't change anything. It's just using sentiment to try to re-characterize a problem in a different light.
polotics 1y ago
Coud you provide a link so we can follow your thread of thought on this? It appears your article got submitted by another user than you.

ninetyninenine 1y ago

>This is such a weird take to me. We know exactly how LLMs and neural networks in general work. Why do you make such crazy statements when the article you’re responding to literally says the opposite of this. Like your statement is categorically false and both the article and I are in total contradiction to it. The rest of your post doesn’t even get into the nuances of what we do and don’t understand. > They're just highly scaled up versions of their smaller curve fitting cousins. For those we can even make pretty visualizations that show exactly what is happening as the network "learns". This proves nothing. Additionally is in direct contradiction to what the OP and I wrote. You describe the minuscule aspects that we do understand but neglect to describe the fact that we overall don’t understand. Saying we understand curve fitting is like saying we understand the human brain because we can completely understand how atoms work and the human brain is made up of atoms therefore we understand the human brain. No. We understand atoms. We don’t understand the human brain even though we know the brain is just a bunch of atoms. The generalization of this is that the amount we don’t understand eclipses by a massive amount the parts we do understand. Let me spell it out to your brain what this means overall: we don’t understand shit. > I don't mean "we can see parts of the brain light up", I mean "we can see the cuts and bends each ReLU is doing to the input". Even with direct insight on every single signal propagating through the feed forward network we still don’t understand it. Like the OPs article he says it’s a scaling problem. Again I don’t understand how come you’re only referring to the miniscule aspects we do understand and not even once mentioning the massive aspects we don’t understand. > We built these things, we know exactly how they work. There's no surprise beyond just how good prediction accuracy gets with a big enough model. No we don’t. Sometimes I find basic rationality and common sense just doesn’t work with people. What ends up working is citing experts in the field saying the exact fucking opposite of you: https://youtu.be/qrvK_KuIeJk?t=284 https://youtu.be/qrvK_KuIeJk?t=284 That’s Geoffrey Hinton, the guy who made ML relevant again in the last decade. Like literally word for word saying exactly opposite of what you’re saying. The interviewer even said “we built the thing, we know how it works” and Geoffrey is like “no we don’t”. Bro. Wake up. > Deep Neural Networks are also a very different architecture from what is going on in our brains (which work more like Spiking Neural Networks) and our brains don't do backpropagation, so you can't even make direct comparisons. Bro we don’t understand deep neural networks. Throwing big words like back propagation and even understanding that it’s just the chain rule from calculus doesn’t mean shit. We overall don’t understand it because it’s a scaling problem. There’s no theory or model that characterizes how a signal propagates through a trained network. It’s like saying you know assembly language and therefore you understand all of Linux. No we understand assembly but these learning algorithms are what built Linux and thus we don’t understand it because we didn’t build it. We don’t understand the human brain and we don’t understand deep neural networks. You know what. Just take a look at what Geoffrey Hinton says. Like if you feel my take is so bizarre and felt the need to comment on it I hope his take rewires your brain and helps you realize you’re the one out of touch. Rationality rarely changes people’s minds, but someone with a higher reputation often is capable of actually getting a person out of their own biased headspace. So listen to what he says then couple it with the explanation here and completely flip your opinion. Or don’t. I rarely see people just do an opinion flip. Humans are just too biased.