4 ms·

My current method for trying to break through this misconception is informing people that nobody knows how AI works. Literally. Nobody knows. (Note that knowing

by wrs 1y ago

My current method for trying to break through this misconception is informing people that nobody knows how AI works. Literally. Nobody knows. (Note that knowing how to make something is not the same as knowing how it works. Take humans as an obvious example.)

generic92034 1y ago
Nobody knows (full scope and on every level) how human brains work. Still bosses rely on their employees' brains all the time.
- eCa 1y ago
  > Nobody knows (full scope and on every level) how human brains work. That is what the parent meant.
  - generic92034 1y ago
    And my point is, that it does not matter. That we still rely on all kinds of things we do not fully grasp.
- Rygian 1y ago
  If I need to manage an AI as I would manage an employee's brain, I'm going to need quite a few non-technical resources to actually achieve that: time, willpower to babysit it, ability to motivate it, leverage in the form of incentives (and reprimands), to name a few. AI sits at a weird place where it can't be analyzed as software, and it can't be managed as a person. My current mental model is that AGI can only be achieved when a machine experiences pleasure, pain, and "bodily functions". Otherwise there's no way to manage it.
- beyarkay 1y ago
  Bosses rely on their employees brains, but only after multiple rounds of interviews and reference checks to ensure the brain they're getting is reliable enough for the job. No boss relies on arbitrary brains taken off the street.
deleted 1y ago
[deleted]
ako 1y ago
I think there are people who know exactly how it works, they know how neural networks work, they know how the transformer architecture works, how attention works, embeddings, tokenization, etc. We just can’t define the weights of the connections between the neurons.
- mattmanser 1y ago
  That's a bit like saying knowing how a pipe works is enough to explain a combustion engine. You're just listing part of how LLMs work. Those mechanisms only explain next word prediction, not LLM reasoning. That's an emergent property that no person, as far as I understand it, can explain past hand waving. Happy to be corrected here.
  - ako 1y ago
    There's no magic involved, the LLM creators can go anywhere and rebuild an LLM with pretty much the same outcome, if they have the same training data. With unlimited time you could even reproduce the output of an LLM manually as it is just a lot of mathematics. Including reasoning, as that is mostly adding words in the context that will steer the word predication to include reasoning. As this is a useful LLM behavior this context is now part of the training data, so it becomes part of the neural network weights.
    - mattmanser 1y ago
      Yes, but we can't inspect, reproduce or explain the emergent property independently. We can't pick out the "math reasoning" part or the "programming" part, or inspect how it's working, or selectively change any of it. You can't turn any dials or twiddle any knobs. You can't replace one part with another, or pick out components. You can't peek inside and say: "hey it's got an irrational preference for naming its variables after famous viking warriors, lets change that!" But worse, it's not that you can't change it, you just don't know! All you can do is test it and guess its biases. Is it racist, is it homophobic, is it misogynistic? There was an article here the other day about AI in recruitment and the hidden biases. And there was a recruitment AI that only picked men for a role. The job spec was entirely gender neutral. And they hadn't noticed until a researcher looked at it. It's a black box. So if it does something incorrectly, all they can do is retrain and hope. Again, this is my present understanding of how it all works right now.
mock-possum 1y ago
How is it possible that nobody knows how it works - it’s running on hardware we have complete control over and perfect observability into, is it not? At any frame we can pause, examine the state, then step forward, examine the state, and observe what changes have occurred - we have perfect knowledge of the source code, the compiler, whatever components you prefer to break software down into - What is it that we don’t understand?
- thorum 1y ago
  The source code is not the LLM. The LLM is billioms of random floating point numbers that somehow encode everything the model knows and can do. The ML field has a good understanding of the algorithms that produce these floating point numbers and lots of techniques that seem to produce “better” numbers in experiments. However, there is little to no understanding of what the numbers represent or how they do the things they do.
- Slartie 1y ago
  We know how each of the "parts" work, but there is a gazillion of parts (especially since you need to take the model weights into account, which are way larger in size than the code that generates them or uses them to generate stuff), and we found out that together they do something that we do not really understand why they do it. And inspecting each part is not enough to understand how, together, they achieve what they achieve. We would need to understand the entire system in a much more abstract way, and currently we have nothing more than ideas of how it _might_ work. Normally, with software, we do not have this problem, as we start on the abstract level with a fully understood design and construct the concrete parts thereafter. Obviously we have a much better understanding of how the entire system of concrete parts works together to perform some complex task. With AI, we took the other way: concrete parts were assembled with vague ideas on the abstract level of how they might do some cool stuff when put together. From there it was basically trial-and-error, iteration to the current state, but always with nothing more than vague ideas of how all of the parts work together on the abstract level. And even if we just stopped the development now and tried to gain a full, thorough understanding of the abstract level of a current LLM, we would fail, as they already reached a complexity that no human can understand anymore, even when devoting their entire lifetime to it. However, while this is a clear difference to most other software (though one has to get careful when it comes to the biggest projects like Chromium, Windows, Linux, ... since even though these were constructed abstract-first, they have been in development for such a long time and have gained so many moving parts in the meantime that someone trying to understand them fully on the abstract level will probably start to face the difficulty of limited lifetime as well), it is not an uncommon thing per se: we also do not "really" understand how economy works, how money works, how capitalism works. Very much like with LLMs, humanity has somehow developed these systems through interaction of billions of humans over a long time, there was never an architect designing them on an abstract level from scratch, and they have shown emergent capabilities and behaviors that we don't fully understand. Still, we obviously try to use them to our advantage every day, and nobody would say that modern economies are useless or should be abandoned because they're not fully understood.
hshdhdhehd 1y ago
Well we are happy to work with other humans without knowing how they work.
- beyarkay 1y ago
  Kinda, but if another human repeatedly showed signs of being dishonest or untrustworthy, we wouldn't be happy to work with them. Suspicion of unknown entities is good.
mr_toad 1y ago
I think you’re confusing knowing how a system works with being able to predict how that system will perform. In a non-linear system the former is often easier than the latter. For example we know how planets “work” from the laws of motion. But planetary orbits involving > 2 bodies are non-linear, and predicting their motion far into the future is surprisingly difficult. Neural networks are the same. They’re actually quite simple, it’s all undergraduate maths and statistics. But because they’re non-linear systems, predicting their behaviour is practically impossible.
- wrs 1y ago
  Claude just analyzed the emotional content of a piece of music for me -- quite accurately -- by just looking at an uploaded PDF of the score. How does that work? "It's a nonlinear system" or "it's a bunch of matrix multiplication" is in no useful way an explanation. That's way down at the bottom of an explanatory abstraction hierarchy that we have only begun to make tools to begin to explore. It's like asking how humans work and getting the answer "it's just undergraduate chemistry!" The study of LLMs is much closer to biology than engineering.
  - ako 1y ago
    Did it show you the reasoning? Did it recognize the notes, the scale, tempo and determine the emotion effect of these or did it use some other reasoning?
    - wrs 1y ago
      I don't know about the reasoning, but it gave "evidence" for its observations in much the same way a human composer might. In terms of "understanding", it seemed in the ballpark of what I get when I ask it to explain some code. I don't want to paste in the whole giant thing, but if you're curious: [0] [0] https://drive.google.com/file/d/1D5yICywmkp24YajboKHdYFcBej0VnVnN/view?usp=sharing https://drive.google.com/file/d/1D5yICywmkp24YajboKHdYFcBej0...
      - ako 1y ago
ako 1y ago
We don’t know how to build humans from scratch (other than letting nature do it) so that’s not really a relevant example. We only know how to fix certain aspects of defect humans, and how to minimize the chance of future defects, but that’s far from building a human from scratch.
- wrs 1y ago
  We don't know how to build LLMs from scratch either, in that sense. We know how to build a machine that you run trillions of tokens through and hope something good happens. What happens, we only know at a vague and low level. You just end up with a bag of hundreds of billions of floating point numbers with some amazing emergent behaviors. Similarly, two adult humans know what to do to start the process that makes another human, and we know a few of the very low-level details about what happens, but that is a far cry from knowing how adult humans do what they do.
beyarkay 1y ago
I thought I could do just this, but alas, [some people][1] are very convinced that we know how these things work. [1]: https://www.reddit.com/r/slatestarcodex/comments/1o6n5ne/why_your_boss_isnt_worried_about_ai_cant_you_just/njhvw1d/ https://www.reddit.com/r/slatestarcodex/comments/1o6n5ne/why...

by wrs 1y ago

generic92034 1y ago

Nobody knows (full scope and on every level) how human brains work. Still bosses rely on their employees' brains all the time.

eCa 1y ago
> Nobody knows (full scope and on every level) how human brains work. That is what the parent meant.
- generic92034 1y ago
  And my point is, that it does not matter. That we still rely on all kinds of things we do not fully grasp.
Rygian 1y ago
If I need to manage an AI as I would manage an employee's brain, I'm going to need quite a few non-technical resources to actually achieve that: time, willpower to babysit it, ability to motivate it, leverage in the form of incentives (and reprimands), to name a few. AI sits at a weird place where it can't be analyzed as software, and it can't be managed as a person. My current mental model is that AGI can only be achieved when a machine experiences pleasure, pain, and "bodily functions". Otherwise there's no way to manage it.
beyarkay 1y ago
Bosses rely on their employees brains, but only after multiple rounds of interviews and reference checks to ensure the brain they're getting is reliable enough for the job. No boss relies on arbitrary brains taken off the street.

deleted 1y ago

[deleted]

ako 1y ago

I think there are people who know exactly how it works, they know how neural networks work, they know how the transformer architecture works, how attention works, embeddings, tokenization, etc. We just can’t define the weights of the connections between the neurons.

mattmanser 1y ago
That's a bit like saying knowing how a pipe works is enough to explain a combustion engine. You're just listing part of how LLMs work. Those mechanisms only explain next word prediction, not LLM reasoning. That's an emergent property that no person, as far as I understand it, can explain past hand waving. Happy to be corrected here.
- ako 1y ago
  There's no magic involved, the LLM creators can go anywhere and rebuild an LLM with pretty much the same outcome, if they have the same training data. With unlimited time you could even reproduce the output of an LLM manually as it is just a lot of mathematics. Including reasoning, as that is mostly adding words in the context that will steer the word predication to include reasoning. As this is a useful LLM behavior this context is now part of the training data, so it becomes part of the neural network weights.
  - mattmanser 1y ago
    Yes, but we can't inspect, reproduce or explain the emergent property independently. We can't pick out the "math reasoning" part or the "programming" part, or inspect how it's working, or selectively change any of it. You can't turn any dials or twiddle any knobs. You can't replace one part with another, or pick out components. You can't peek inside and say: "hey it's got an irrational preference for naming its variables after famous viking warriors, lets change that!" But worse, it's not that you can't change it, you just don't know! All you can do is test it and guess its biases. Is it racist, is it homophobic, is it misogynistic? There was an article here the other day about AI in recruitment and the hidden biases. And there was a recruitment AI that only picked men for a role. The job spec was entirely gender neutral. And they hadn't noticed until a researcher looked at it. It's a black box. So if it does something incorrectly, all they can do is retrain and hope. Again, this is my present understanding of how it all works right now.

mock-possum 1y ago

How is it possible that nobody knows how it works - it’s running on hardware we have complete control over and perfect observability into, is it not? At any frame we can pause, examine the state, then step forward, examine the state, and observe what changes have occurred - we have perfect knowledge of the source code, the compiler, whatever components you prefer to break software down into - What is it that we don’t understand?

thorum 1y ago
The source code is not the LLM. The LLM is billioms of random floating point numbers that somehow encode everything the model knows and can do. The ML field has a good understanding of the algorithms that produce these floating point numbers and lots of techniques that seem to produce “better” numbers in experiments. However, there is little to no understanding of what the numbers represent or how they do the things they do.
Slartie 1y ago
We know how each of the "parts" work, but there is a gazillion of parts (especially since you need to take the model weights into account, which are way larger in size than the code that generates them or uses them to generate stuff), and we found out that together they do something that we do not really understand why they do it. And inspecting each part is not enough to understand how, together, they achieve what they achieve. We would need to understand the entire system in a much more abstract way, and currently we have nothing more than ideas of how it _might_ work. Normally, with software, we do not have this problem, as we start on the abstract level with a fully understood design and construct the concrete parts thereafter. Obviously we have a much better understanding of how the entire system of concrete parts works together to perform some complex task. With AI, we took the other way: concrete parts were assembled with vague ideas on the abstract level of how they might do some cool stuff when put together. From there it was basically trial-and-error, iteration to the current state, but always with nothing more than vague ideas of how all of the parts work together on the abstract level. And even if we just stopped the development now and tried to gain a full, thorough understanding of the abstract level of a current LLM, we would fail, as they already reached a complexity that no human can understand anymore, even when devoting their entire lifetime to it. However, while this is a clear difference to most other software (though one has to get careful when it comes to the biggest projects like Chromium, Windows, Linux, ... since even though these were constructed abstract-first, they have been in development for such a long time and have gained so many moving parts in the meantime that someone trying to understand them fully on the abstract level will probably start to face the difficulty of limited lifetime as well), it is not an uncommon thing per se: we also do not "really" understand how economy works, how money works, how capitalism works. Very much like with LLMs, humanity has somehow developed these systems through interaction of billions of humans over a long time, there was never an architect designing them on an abstract level from scratch, and they have shown emergent capabilities and behaviors that we don't fully understand. Still, we obviously try to use them to our advantage every day, and nobody would say that modern economies are useless or should be abandoned because they're not fully understood.

hshdhdhehd 1y ago

Well we are happy to work with other humans without knowing how they work.

beyarkay 1y ago
Kinda, but if another human repeatedly showed signs of being dishonest or untrustworthy, we wouldn't be happy to work with them. Suspicion of unknown entities is good.

mr_toad 1y ago

I think you’re confusing knowing how a system works with being able to predict how that system will perform. In a non-linear system the former is often easier than the latter. For example we know how planets “work” from the laws of motion. But planetary orbits involving > 2 bodies are non-linear, and predicting their motion far into the future is surprisingly difficult. Neural networks are the same. They’re actually quite simple, it’s all undergraduate maths and statistics. But because they’re non-linear systems, predicting their behaviour is practically impossible.

wrs 1y ago
Claude just analyzed the emotional content of a piece of music for me -- quite accurately -- by just looking at an uploaded PDF of the score. How does that work? "It's a nonlinear system" or "it's a bunch of matrix multiplication" is in no useful way an explanation. That's way down at the bottom of an explanatory abstraction hierarchy that we have only begun to make tools to begin to explore. It's like asking how humans work and getting the answer "it's just undergraduate chemistry!" The study of LLMs is much closer to biology than engineering.
- ako 1y ago
  Did it show you the reasoning? Did it recognize the notes, the scale, tempo and determine the emotion effect of these or did it use some other reasoning?
  - wrs 1y ago
    I don't know about the reasoning, but it gave "evidence" for its observations in much the same way a human composer might. In terms of "understanding", it seemed in the ballpark of what I get when I ask it to explain some code. I don't want to paste in the whole giant thing, but if you're curious: [0] [0] https://drive.google.com/file/d/1D5yICywmkp24YajboKHdYFcBej0VnVnN/view?usp=sharing https://drive.google.com/file/d/1D5yICywmkp24YajboKHdYFcBej0...
    - ako 1y ago

ako 1y ago

We don’t know how to build humans from scratch (other than letting nature do it) so that’s not really a relevant example. We only know how to fix certain aspects of defect humans, and how to minimize the chance of future defects, but that’s far from building a human from scratch.

wrs 1y ago
We don't know how to build LLMs from scratch either, in that sense. We know how to build a machine that you run trillions of tokens through and hope something good happens. What happens, we only know at a vague and low level. You just end up with a bag of hundreds of billions of floating point numbers with some amazing emergent behaviors. Similarly, two adult humans know what to do to start the process that makes another human, and we know a few of the very low-level details about what happens, but that is a far cry from knowing how adult humans do what they do.

beyarkay 1y ago

I thought I could do just this, but alas, [some people][1] are very convinced that we know how these things work. [1]: https://www.reddit.com/r/slatestarcodex/comments/1o6n5ne/why_your_boss_isnt_worried_about_ai_cant_you_just/njhvw1d/ https://www.reddit.com/r/slatestarcodex/comments/1o6n5ne/why...