4 ms·

How is it possible that nobody knows how it works - it’s running on hardware we have complete control over and perfect observability into, is it not? At any fra

How is it possible that nobody knows how it works - it’s running on hardware we have complete control over and perfect observability into, is it not? At any frame we can pause, examine the state, then step forward, examine the state, and observe what changes have occurred - we have perfect knowledge of the source code, the compiler, whatever components you prefer to break software down into - What is it that we don’t understand?

thorum 1y ago
The source code is not the LLM. The LLM is billioms of random floating point numbers that somehow encode everything the model knows and can do. The ML field has a good understanding of the algorithms that produce these floating point numbers and lots of techniques that seem to produce “better” numbers in experiments. However, there is little to no understanding of what the numbers represent or how they do the things they do.
Slartie 1y ago
We know how each of the "parts" work, but there is a gazillion of parts (especially since you need to take the model weights into account, which are way larger in size than the code that generates them or uses them to generate stuff), and we found out that together they do something that we do not really understand why they do it. And inspecting each part is not enough to understand how, together, they achieve what they achieve. We would need to understand the entire system in a much more abstract way, and currently we have nothing more than ideas of how it _might_ work. Normally, with software, we do not have this problem, as we start on the abstract level with a fully understood design and construct the concrete parts thereafter. Obviously we have a much better understanding of how the entire system of concrete parts works together to perform some complex task. With AI, we took the other way: concrete parts were assembled with vague ideas on the abstract level of how they might do some cool stuff when put together. From there it was basically trial-and-error, iteration to the current state, but always with nothing more than vague ideas of how all of the parts work together on the abstract level. And even if we just stopped the development now and tried to gain a full, thorough understanding of the abstract level of a current LLM, we would fail, as they already reached a complexity that no human can understand anymore, even when devoting their entire lifetime to it. However, while this is a clear difference to most other software (though one has to get careful when it comes to the biggest projects like Chromium, Windows, Linux, ... since even though these were constructed abstract-first, they have been in development for such a long time and have gained so many moving parts in the meantime that someone trying to understand them fully on the abstract level will probably start to face the difficulty of limited lifetime as well), it is not an uncommon thing per se: we also do not "really" understand how economy works, how money works, how capitalism works. Very much like with LLMs, humanity has somehow developed these systems through interaction of billions of humans over a long time, there was never an architect designing them on an abstract level from scratch, and they have shown emergent capabilities and behaviors that we don't fully understand. Still, we obviously try to use them to our advantage every day, and nobody would say that modern economies are useless or should be abandoned because they're not fully understood.
beyarkay 1y ago
> At any frame we can pause, examine the state, then step forward, examine the state, and observe what changes have occurred This example from software doesn't meaningfully hold for neural networks. It's a bit like trying to watch an individual COVID virus duplicate and then attempting to predict the pandemic. It's incredibly complicated and we haven't yet built the tools to help us understand

thorum 1y ago

The source code is not the LLM. The LLM is billioms of random floating point numbers that somehow encode everything the model knows and can do. The ML field has a good understanding of the algorithms that produce these floating point numbers and lots of techniques that seem to produce “better” numbers in experiments. However, there is little to no understanding of what the numbers represent or how they do the things they do.

Slartie 1y ago

We know how each of the "parts" work, but there is a gazillion of parts (especially since you need to take the model weights into account, which are way larger in size than the code that generates them or uses them to generate stuff), and we found out that together they do something that we do not really understand why they do it. And inspecting each part is not enough to understand how, together, they achieve what they achieve. We would need to understand the entire system in a much more abstract way, and currently we have nothing more than ideas of how it _might_ work. Normally, with software, we do not have this problem, as we start on the abstract level with a fully understood design and construct the concrete parts thereafter. Obviously we have a much better understanding of how the entire system of concrete parts works together to perform some complex task. With AI, we took the other way: concrete parts were assembled with vague ideas on the abstract level of how they might do some cool stuff when put together. From there it was basically trial-and-error, iteration to the current state, but always with nothing more than vague ideas of how all of the parts work together on the abstract level. And even if we just stopped the development now and tried to gain a full, thorough understanding of the abstract level of a current LLM, we would fail, as they already reached a complexity that no human can understand anymore, even when devoting their entire lifetime to it. However, while this is a clear difference to most other software (though one has to get careful when it comes to the biggest projects like Chromium, Windows, Linux, ... since even though these were constructed abstract-first, they have been in development for such a long time and have gained so many moving parts in the meantime that someone trying to understand them fully on the abstract level will probably start to face the difficulty of limited lifetime as well), it is not an uncommon thing per se: we also do not "really" understand how economy works, how money works, how capitalism works. Very much like with LLMs, humanity has somehow developed these systems through interaction of billions of humans over a long time, there was never an architect designing them on an abstract level from scratch, and they have shown emergent capabilities and behaviors that we don't fully understand. Still, we obviously try to use them to our advantage every day, and nobody would say that modern economies are useless or should be abandoned because they're not fully understood.

beyarkay 1y ago

> At any frame we can pause, examine the state, then step forward, examine the state, and observe what changes have occurred This example from software doesn't meaningfully hold for neural networks. It's a bit like trying to watch an individual COVID virus duplicate and then attempting to predict the pandemic. It's incredibly complicated and we haven't yet built the tools to help us understand