3 ms·

I think there are people who know exactly how it works, they know how neural networks work, they know how the transformer architecture works, how attention work

by ako 1y ago

I think there are people who know exactly how it works, they know how neural networks work, they know how the transformer architecture works, how attention works, embeddings, tokenization, etc. We just can’t define the weights of the connections between the neurons.

mattmanser 1y ago
That's a bit like saying knowing how a pipe works is enough to explain a combustion engine. You're just listing part of how LLMs work. Those mechanisms only explain next word prediction, not LLM reasoning. That's an emergent property that no person, as far as I understand it, can explain past hand waving. Happy to be corrected here.
- ako 1y ago
  There's no magic involved, the LLM creators can go anywhere and rebuild an LLM with pretty much the same outcome, if they have the same training data. With unlimited time you could even reproduce the output of an LLM manually as it is just a lot of mathematics. Including reasoning, as that is mostly adding words in the context that will steer the word predication to include reasoning. As this is a useful LLM behavior this context is now part of the training data, so it becomes part of the neural network weights.
  - mattmanser 1y ago
    Yes, but we can't inspect, reproduce or explain the emergent property independently. We can't pick out the "math reasoning" part or the "programming" part, or inspect how it's working, or selectively change any of it. You can't turn any dials or twiddle any knobs. You can't replace one part with another, or pick out components. You can't peek inside and say: "hey it's got an irrational preference for naming its variables after famous viking warriors, lets change that!" But worse, it's not that you can't change it, you just don't know! All you can do is test it and guess its biases. Is it racist, is it homophobic, is it misogynistic? There was an article here the other day about AI in recruitment and the hidden biases. And there was a recruitment AI that only picked men for a role. The job spec was entirely gender neutral. And they hadn't noticed until a researcher looked at it. It's a black box. So if it does something incorrectly, all they can do is retrain and hope. Again, this is my present understanding of how it all works right now.
    - emp17344 1y ago
      It’s not clear that those “emergent properties” even exist.
    - ako 1y ago
      So just because all the parts are collaborating to get the desired outcome, and specific aspects of that outcome cannot be contributed to specific parts of the llm you think we don’t understand LLMs? With mixture of expert systems we’re introducing dedicated subsystems into the llm responsible for specific aspect of the llm, so we’re partially moving in that direction. But overall in my opinion if devs are able to rebuild it from scratch with a predefined outcome, and even know how to improve the system to improve certain aspects of it, we do understand how it works.
      - wrs 1y ago
        Being able to make it and knowing how it works are just not the same thing. I can make bread of consistent quality, and know how to improve certain aspects of it. To know how it works, I’d need to get a doctorate in biochemistry and then still have a fairly patchy understanding. I know plenty of people who can drive a car very successfully but would never claim they know how it works.
        ako 1y ago
        But with LLMs is there really more to understand? They’re just large functions that take numerical input and transform it into numerical output based on trained weights. There is nothing behind the scenes doing things we don’t understand. The magic is in the weights, and we know how to create these based on training data. Regarding the car, if you know how to build a car, you understand how a car works. A driver is more like someone using and llm, not a developer able to create an llm.
        wrs 1y ago
        I’ve never seen the word “just” take on more load than it did in that second sentence! Sorry to inform everybody doing their Ph.D. on LLM interpretability that they’re just wasting their time.

by ako 1y ago

mattmanser 1y ago

That's a bit like saying knowing how a pipe works is enough to explain a combustion engine. You're just listing part of how LLMs work. Those mechanisms only explain next word prediction, not LLM reasoning. That's an emergent property that no person, as far as I understand it, can explain past hand waving. Happy to be corrected here.

ako 1y ago
There's no magic involved, the LLM creators can go anywhere and rebuild an LLM with pretty much the same outcome, if they have the same training data. With unlimited time you could even reproduce the output of an LLM manually as it is just a lot of mathematics. Including reasoning, as that is mostly adding words in the context that will steer the word predication to include reasoning. As this is a useful LLM behavior this context is now part of the training data, so it becomes part of the neural network weights.
- mattmanser 1y ago
  Yes, but we can't inspect, reproduce or explain the emergent property independently. We can't pick out the "math reasoning" part or the "programming" part, or inspect how it's working, or selectively change any of it. You can't turn any dials or twiddle any knobs. You can't replace one part with another, or pick out components. You can't peek inside and say: "hey it's got an irrational preference for naming its variables after famous viking warriors, lets change that!" But worse, it's not that you can't change it, you just don't know! All you can do is test it and guess its biases. Is it racist, is it homophobic, is it misogynistic? There was an article here the other day about AI in recruitment and the hidden biases. And there was a recruitment AI that only picked men for a role. The job spec was entirely gender neutral. And they hadn't noticed until a researcher looked at it. It's a black box. So if it does something incorrectly, all they can do is retrain and hope. Again, this is my present understanding of how it all works right now.
  - emp17344 1y ago
    It’s not clear that those “emergent properties” even exist.
  - ako 1y ago
    So just because all the parts are collaborating to get the desired outcome, and specific aspects of that outcome cannot be contributed to specific parts of the llm you think we don’t understand LLMs? With mixture of expert systems we’re introducing dedicated subsystems into the llm responsible for specific aspect of the llm, so we’re partially moving in that direction. But overall in my opinion if devs are able to rebuild it from scratch with a predefined outcome, and even know how to improve the system to improve certain aspects of it, we do understand how it works.
    - wrs 1y ago
      Being able to make it and knowing how it works are just not the same thing. I can make bread of consistent quality, and know how to improve certain aspects of it. To know how it works, I’d need to get a doctorate in biochemistry and then still have a fairly patchy understanding. I know plenty of people who can drive a car very successfully but would never claim they know how it works.
      - ako 1y ago
        But with LLMs is there really more to understand? They’re just large functions that take numerical input and transform it into numerical output based on trained weights. There is nothing behind the scenes doing things we don’t understand. The magic is in the weights, and we know how to create these based on training data. Regarding the car, if you know how to build a car, you understand how a car works. A driver is more like someone using and llm, not a developer able to create an llm.
        wrs 1y ago
        I’ve never seen the word “just” take on more load than it did in that second sentence! Sorry to inform everybody doing their Ph.D. on LLM interpretability that they’re just wasting their time.