5 ms·

Do we understand how neural networks work?

32 points by Philpax 1y ago

ninetyninenine 1y ago
I've been saying this to people. Tons of people don't realize that we have no idea how these things work. So questions like is it AGI? It is conscious? Where the questions themselves contain words with fuzzy and ill defined meanings are pointless. The core of the matter is: We don't know! So when someone says chatGPT could be conscious. Or when someone says it can't be conscious. We don't actually know! And we aren't even fully sure about the definition of consciousness. My problem with HN is that a ton of people here take the stance that we know how LLMs work and that we know definitively they aren't conscious and the people who say otherwise are alarmist and stupid. I think the fact of the matter is, if you're putting your foot down and saying LLMs aren't intelligent... you're wildly illogical and misinformed about the status quo of Artificial intelligence and a huge portion of the mob mentality on HN thinks this way. It's like there's these buzz phrases getting thrown around saying that the LLM is just a glorified auto complete (which it is) and people latch on to this buzz phrases and their entire understanding of LLMs becomes basically a composition of these buzz concepts like "transformers" and "LLMs" and "chain of thought" when in actuality real critical thinking about what's going on in these networks tells you we don't UNDERSTAND anything. Also the characterization in the article is mistaken. It says we understand LLMs in a limited way. Yeah sure. It's as limited as our understanding of the human brain. You know scientists found a way to stimulate the reward centers of the brain using electrodes and were able to cause a person to feel the utmost pleasure? The whole golden gate bridge thing is exactly that. You perturb something in the network and causes a semi predictable output. In the end we still generally don't get wtf is going on. We literally understand nothing. What we do understand is so minuscule compared with what we don't understand that it's negligible.
- sirwhinesalot 1y ago
  This is such a weird take to me. We know exactly how LLMs and neural networks in general work. They're just highly scaled up versions of their smaller curve fitting cousins. For those we can even make pretty visualizations that show exactly what is happening as the network "learns". I don't mean "we can see parts of the brain light up", I mean "we can see the cuts and bends each ReLU is doing to the input". We built these things, we know exactly how they work. There's no surprise beyond just how good prediction accuracy gets with a big enough model. Deep Neural Networks are also a very different architecture from what is going on in our brains (which work more like Spiking Neural Networks) and our brains don't do backpropagation, so you can't even make direct comparisons.
  - SEGyges 1y ago
    fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to make sense of and it was posted on hackernews
    - sirwhinesalot 1y ago
      Your article, unlike the bizarre desperate take from the poster above, is actually very good. We do not understand the features the neural net learns, that's 100% true (and really the whole point of them in the first place). For small image recognition models we can visualize them and get an intuition for what they are doing, but it doesn't really matter. For even smaller models, we can translate them to a classical AI model (like a mixed integer program as an example) and actually do various "queries" on the model itself to, e.g., learn that the network recognizes the number "8" by just checking 2 pixels in the image. None of this changes the fact that we know what these things are and how they work, because we built them. Any comparisons to our lack of knowledge of the human brain are ridiculous. LLMs are obviously not conscious, they don't even have real "state", they're an approximated pure function f(context: List<Token>) -> Token, that's run in a loop. The only valid alarmist take is that we're using black box algorithms to make decisions with serious real-world impact, but this is true of any black box algorithm, not just the latest and greatest ML models.
      - dpoloncsak 1y ago
        Its a complex adaptive system, right? Isn't that the whole idea? We know how each part of the system works by itself. We know all inputs, and can measure outputs. I still (even if I actually understood the math) cannot tell you 'If you prompt 'x', the model will return 'y' with 100% confidence.
        sirwhinesalot 1y ago
        > If you prompt 'x', the model will return 'y' with 100% confidence. We can do this for smaller models. Which means it's a problem of scale/computing power rather than a fundamental limitation. The situation with the human brain is completely different. We know neurons exchange information and how that works, and we have a pretty good understanding of the architecture of parts of the brain like the visual cortex, but we have no idea of the architecture as a whole. We know the architecture of an LLM. We know how the data flows. We know what it is the individual neurons are learning (cuts and bends of a plane in a hyperdimensional space). We know how the weights are learned (backpropagation). We know the "algorithm" the LLM as a whole is approximating (List<Token> -> Token). Yes there are emergent properties we don't understand but the same is true of a spam filter. Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly.
markisus 1y ago
I’m not sure that LLMs are solely autocomplete. The next token prediction task is only for pretraining. After that I thought you apply reinforcement learning.
- porridgeraisin 1y ago
  Nevertheless it is next token prediction. Each token it predicts is an action for the RL setup, and context += that_token is the new state. Solutions are either human labelled (RLHF) or to math/code problems (deterministic answer) and prediction == solution is used as the reward signal. Policy gradient approaches in RL are just supervised learning, when viewed through some lenses. You can search for karpathys more fleshed out argument for the same, I'm on mobile jow.
  - SEGyges 1y ago
    My short explanation would be that even for RL, you are training on a next token objective; but the next token is something that has been selected very very carefully for solving the problem, and was generated by the model itself. So you're amplifying existing trajectories in the model by feeding the model's outputs back to itself, but only when those outputs solve a problem. This elides the kl penalty and the odd group scoring, which are the same in the limit but vastly more efficient in practice.
  - astrange 1y ago
    Inference often isn't next token prediction though, either weakly (because of speculative decoding/multiple token outputs) or strongly (because of tool usage like web search).
    - porridgeraisin 1y ago
      Well I may be misunderstanfing you, but speculative decoding is just using next token predictions from a few models(or many samples from one model) instead of just one sample. It is still next token prediction. Tool usage is also just next token prediction. You have it predict the next token of the syntax needed for tool use, and then it is fed the result of that in context which it then predicts the next token of.
KingMob 1y ago
As a former neuroscientist, it's both amusing and sad to see AI people run into the same issues neuroscientists have grappled with for decades, as LLMs start to approach biological systems in complexity. (And mind you, LLMs are still smaller/simpler than the human brain.) My hope is that the greater accessibility of neural networks over actual neurons will lead to some insights into biology. But I suspect they're still different enough, both in size, but more importantly in organization, that it won't be realized anytime soon. The Golden Gate Claude is reminiscent of searches for the "grandmother" neuron. We know from direct brain stimulation that certain neurons are strongly associated with certain outputs...but we also know that a lot of knowledge representation is widely distributed. The whole thing is a mix of specialization and redundancy.
- nickm12 1y ago
  I was, for a time, a neuroscience major and have had this same thought. I'm concerned that we're treating these systems as engineered systems when they are closer to evolved or biological systems. They at at least different in that we can study them much more precisely than biological systems because their entire state is visible and we can run them on arbitrary inputs and measure responses.
  - SEGyges 1y ago
    I agree, what we do is much closer to growing them than to engineering them. We basically engineer the conditions for growth, and then check the results and try again. My best argument that insights from neuroscience will transfer to neural networks, and vice versa: For sufficiently complex phenomena (e.g., language), there should only be one reasonably efficient solution to the problem, and small variations on that solution. So there should be some reversible mapping between any two tractable solutions to the problem that is pretty close to lossless, provided both solutions actually solve the problem. And, yeah, the main advantage of neural networks is that they're white-box. You can also control your experiments in a way you can't in the real world.
bob1029 1y ago
> You are training the neural network (a big block of matrices) using gradient descent (a trick that modifies the big block of matrices) to do something. Things get even more mysterious (and efficient) if you introduce non-linearities, recurrence and a time domain. These mainstream ANN architectures are fairly pedestrian by comparison. You can visualize and walk through the statistical distributions. Everything is effectively instantaneous within each next token generation. It's hard to follow an action potential around in time because it's constantly being integrated with other potentials and its influence is only relevant for a brief moment.
Nevermark 1y ago
Here is a slightly sideways take on the question. Since some misunderstandings crop up over and over. Things that are not true about neural networks and LLMs: 1. They are not stochastic parrots. Statistics can be used to judge model performances, as with any type of model whatsoever. But that doesn’t make a model statistical. Neural network components are not statistically driven, they are gradient driven. They don’t calculate statistics, they learn topological representations of relationships. 2. LLMs are not just doing word prediction or text completion. Yes, that is the basic task they are trained on, but the “just” (that is often stated or implied) trivializes what the network actually has to learn to perform well. Task type, and what must be learned to achieve success at that task, are two entirely different things. To predict the kinds of human reasoning documented in writing in training sets, requires that kind of reasoning be learned. Not just some compressed generalization of people’s particular responses. Simple proof that LLMs are not just compressing a lot of human behavior comes easily. Just ask an LLM to do something involving several topics unlikely to have been encountered together before, and their one-shot answer might not be God’s word on the issue, but is a far cry from the stumbling that good mimic could ever do. (Example task, ask for a Supreme Court brief to argue for rabbits rights based on sentient animal and native rights, with serious arguments, but written in Dr. Seuss prose by James Bond.) 3. LLMs do reason. Not like us or as well as us. But also, actually better than us in some ways. LLMs are far superior to us at very wide, somewhat shallow reasoning. They are able to one-shot weave together information from tens of thousands of disparate topics and ideas, on demand. But they don’t notice their own blind spots. They miss implications. Those are things humans do quickly in continuous narrower deeper reasoning cycles. They are getting better. And some of their failures should be attributed to the strict constraint of one/few-shot(s) to well written responses, that we put on them. We don’t hold humans to such strict standards. And if we did, we would also make a lot more obvious errors. Wide & shallow reasoning in ways humans can’t match, is not a trivial success. 4. LLMs are very creative. As with their reasoning, not like us: very wide in terms of instantly and fluidly weaving highly disparate information together. Somewhat shallow (but gaining) in terms of iteratively recognizing and self-correcting their own mistakes and oversights. See random original disparate topic task above. First, spend several hours performing that task oneself. Or take a day or two if need be. Then, give the task to an LLM and wait a few seconds. Compare. Ouch. — TLDR; somewhat off topic and high level, but when trying to understand models it helps to avoid seemingly endlessly repeated misunderstandings.
d4rkn0d3z 1y ago
Of course we understand how they work, don't panic. Some of us are not even surprised at the results thus far, the good, the bad, and the ugly. We have understood how neural networks function for 50 years or more. I think the interdisciplinary nature of the technology is significant, NN draw from many areas of scientific study and if you are missing any piece of the puzzle then some mysteriousness arises for you. But take heart, as I said, there are people who are not surprised one iota by what is happening. Dozens or perhaps hundreds of people have considered embarking on the brute force approach being pursued, nobody decided to try until a few years back because it is mathematically limited, costly, and might just make our problems much worse before it can make them better in any important way. What is very salient is that 10 of the most powerful corporations and the leaders of the largest countries have decided on this route and they could force all software engineers to wear extra long, wide red ties if they wanted to so buckle up. It looks like we are riding this train all the way in the hope that cash + time will solve the truncation of the science.
- hodgehog11 1y ago
  As someone who works in the theory of deep learning, I'm curious to know what you mean by "understand how they work". Sure, we understand the broad strokes, and those tend to be good enough to chip away at making improvements and vague predictions of what certain things will do, but we still have a severe lack of understanding of their implicit biases which allow them to perform well and give rise to the scaling laws. We know a lot more about basically every other machine learning approach.
  - d4rkn0d3z 1y ago
    Well, I'm thumb bashing on my phone so I can't get too far into the details as I see them. I'd read and respond to any papers you have?
    - hodgehog11 1y ago
      Are you asking about open problems? Here are a couple: There are strong arguments that deep learning generalization (and robustness in a slightly different sense) can be successfully explained in terms of PAC-Bayes theory, for example: https://arxiv.org/abs/2503.02113 https://arxiv.org/abs/2503.02113 This requires a strong understanding of the appropriate prior, which effectively quantifies implicit bias in the training setup / architecture setup. Early analyses suggest the bias is that of a minimal gradient norm (https://arxiv.org/pdf/2101.12176 https://arxiv.org/pdf/2101.12176) but this is almost certainly not the case, e.g.(https://arxiv.org/abs/2005.06398 https://arxiv.org/abs/2005.06398). There are good empirical arguments that models are trained to achieve maximal compression (https://arxiv.org/abs/2211.13609 https://arxiv.org/abs/2211.13609). But we only know this in a vague sense. How are deep learning training procedures and architectures implicitly exhibiting this form of sparsity and compressing information? Why is this more effective for some optimizers/architectures than others? Why would minimum description length circuits be so effective at mimicking human intelligence? How does this form of regularisation induce the exponents in the neural scaling laws (so we can figure out how to best optimize for these exponents)? If we don't know the implicit bias and how it relates to the training procedure, we're just doing alchemy and effectively got lucky. Change the standard approach even slightly and things easily break. We only really know what to do by process of elimination. Then there are the LLM specific questions related to in-context learning (https://arxiv.org/abs/2306.00297 https://arxiv.org/abs/2306.00297), to help explain why next token prediction through transformer designs works so well. Does it implicitly act as an optimization procedure (with bias) relative to the context? What are the biases here? What is the problem being solved? These may sound specific, but they are just examples of some approaches to answer the big question of "why does this standard procedure work so well, and what can we do to make it work better". Without this, we're just following the heuristics of what came before without a guiding framework. Post-hoc, we can do a lot more with interpretability frameworks and linear probes and the like. But without the implicit biases, we are only getting a local picture, which doesn't help us understand what a model is likely to do before we run it. We need global information. These are just a few, and don't include the philosophical and theory of mind questions, and others from neuroscience...

32 points by Philpax 1y ago

I've been saying this to people. Tons of people don't realize that we have no idea how these things work. So questions like is it AGI? It is conscious? Where the questions themselves contain words with fuzzy and ill defined meanings are pointless. The core of the matter is: We don't know! So when someone says chatGPT could be conscious. Or when someone says it can't be conscious. We don't actually know! And we aren't even fully sure about the definition of consciousness. My problem with HN is that a ton of people here take the stance that we know how LLMs work and that we know definitively they aren't conscious and the people who say otherwise are alarmist and stupid. I think the fact of the matter is, if you're putting your foot down and saying LLMs aren't intelligent... you're wildly illogical and misinformed about the status quo of Artificial intelligence and a huge portion of the mob mentality on HN thinks this way. It's like there's these buzz phrases getting thrown around saying that the LLM is just a glorified auto complete (which it is) and people latch on to this buzz phrases and their entire understanding of LLMs becomes basically a composition of these buzz concepts like "transformers" and "LLMs" and "chain of thought" when in actuality real critical thinking about what's going on in these networks tells you we don't UNDERSTAND anything. Also the characterization in the article is mistaken. It says we understand LLMs in a limited way. Yeah sure. It's as limited as our understanding of the human brain. You know scientists found a way to stimulate the reward centers of the brain using electrodes and were able to cause a person to feel the utmost pleasure? The whole golden gate bridge thing is exactly that. You perturb something in the network and causes a semi predictable output. In the end we still generally don't get wtf is going on. We literally understand nothing. What we do understand is so minuscule compared with what we don't understand that it's negligible.

sirwhinesalot 1y ago
This is such a weird take to me. We know exactly how LLMs and neural networks in general work. They're just highly scaled up versions of their smaller curve fitting cousins. For those we can even make pretty visualizations that show exactly what is happening as the network "learns". I don't mean "we can see parts of the brain light up", I mean "we can see the cuts and bends each ReLU is doing to the input". We built these things, we know exactly how they work. There's no surprise beyond just how good prediction accuracy gets with a big enough model. Deep Neural Networks are also a very different architecture from what is going on in our brains (which work more like Spiking Neural Networks) and our brains don't do backpropagation, so you can't even make direct comparisons.
- SEGyges 1y ago
  fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to make sense of and it was posted on hackernews
  - sirwhinesalot 1y ago
    Your article, unlike the bizarre desperate take from the poster above, is actually very good. We do not understand the features the neural net learns, that's 100% true (and really the whole point of them in the first place). For small image recognition models we can visualize them and get an intuition for what they are doing, but it doesn't really matter. For even smaller models, we can translate them to a classical AI model (like a mixed integer program as an example) and actually do various "queries" on the model itself to, e.g., learn that the network recognizes the number "8" by just checking 2 pixels in the image. None of this changes the fact that we know what these things are and how they work, because we built them. Any comparisons to our lack of knowledge of the human brain are ridiculous. LLMs are obviously not conscious, they don't even have real "state", they're an approximated pure function f(context: List<Token>) -> Token, that's run in a loop. The only valid alarmist take is that we're using black box algorithms to make decisions with serious real-world impact, but this is true of any black box algorithm, not just the latest and greatest ML models.
    - dpoloncsak 1y ago
      Its a complex adaptive system, right? Isn't that the whole idea? We know how each part of the system works by itself. We know all inputs, and can measure outputs. I still (even if I actually understood the math) cannot tell you 'If you prompt 'x', the model will return 'y' with 100% confidence.
      - sirwhinesalot 1y ago
        > If you prompt 'x', the model will return 'y' with 100% confidence. We can do this for smaller models. Which means it's a problem of scale/computing power rather than a fundamental limitation. The situation with the human brain is completely different. We know neurons exchange information and how that works, and we have a pretty good understanding of the architecture of parts of the brain like the visual cortex, but we have no idea of the architecture as a whole. We know the architecture of an LLM. We know how the data flows. We know what it is the individual neurons are learning (cuts and bends of a plane in a hyperdimensional space). We know how the weights are learned (backpropagation). We know the "algorithm" the LLM as a whole is approximating (List<Token> -> Token). Yes there are emergent properties we don't understand but the same is true of a spam filter. Comparing this to our lack of understanding of the human brain and discussing how these things might be "conscious" is silly.

markisus 1y ago

I’m not sure that LLMs are solely autocomplete. The next token prediction task is only for pretraining. After that I thought you apply reinforcement learning.

porridgeraisin 1y ago
Nevertheless it is next token prediction. Each token it predicts is an action for the RL setup, and context += that_token is the new state. Solutions are either human labelled (RLHF) or to math/code problems (deterministic answer) and prediction == solution is used as the reward signal. Policy gradient approaches in RL are just supervised learning, when viewed through some lenses. You can search for karpathys more fleshed out argument for the same, I'm on mobile jow.
- SEGyges 1y ago
  My short explanation would be that even for RL, you are training on a next token objective; but the next token is something that has been selected very very carefully for solving the problem, and was generated by the model itself. So you're amplifying existing trajectories in the model by feeding the model's outputs back to itself, but only when those outputs solve a problem. This elides the kl penalty and the odd group scoring, which are the same in the limit but vastly more efficient in practice.
- astrange 1y ago
  Inference often isn't next token prediction though, either weakly (because of speculative decoding/multiple token outputs) or strongly (because of tool usage like web search).
  - porridgeraisin 1y ago
    Well I may be misunderstanfing you, but speculative decoding is just using next token predictions from a few models(or many samples from one model) instead of just one sample. It is still next token prediction. Tool usage is also just next token prediction. You have it predict the next token of the syntax needed for tool use, and then it is fed the result of that in context which it then predicts the next token of.

KingMob 1y ago

As a former neuroscientist, it's both amusing and sad to see AI people run into the same issues neuroscientists have grappled with for decades, as LLMs start to approach biological systems in complexity. (And mind you, LLMs are still smaller/simpler than the human brain.) My hope is that the greater accessibility of neural networks over actual neurons will lead to some insights into biology. But I suspect they're still different enough, both in size, but more importantly in organization, that it won't be realized anytime soon. The Golden Gate Claude is reminiscent of searches for the "grandmother" neuron. We know from direct brain stimulation that certain neurons are strongly associated with certain outputs...but we also know that a lot of knowledge representation is widely distributed. The whole thing is a mix of specialization and redundancy.

nickm12 1y ago
I was, for a time, a neuroscience major and have had this same thought. I'm concerned that we're treating these systems as engineered systems when they are closer to evolved or biological systems. They at at least different in that we can study them much more precisely than biological systems because their entire state is visible and we can run them on arbitrary inputs and measure responses.
- SEGyges 1y ago
  I agree, what we do is much closer to growing them than to engineering them. We basically engineer the conditions for growth, and then check the results and try again. My best argument that insights from neuroscience will transfer to neural networks, and vice versa: For sufficiently complex phenomena (e.g., language), there should only be one reasonably efficient solution to the problem, and small variations on that solution. So there should be some reversible mapping between any two tractable solutions to the problem that is pretty close to lossless, provided both solutions actually solve the problem. And, yeah, the main advantage of neural networks is that they're white-box. You can also control your experiments in a way you can't in the real world.

bob1029 1y ago

> You are training the neural network (a big block of matrices) using gradient descent (a trick that modifies the big block of matrices) to do something. Things get even more mysterious (and efficient) if you introduce non-linearities, recurrence and a time domain. These mainstream ANN architectures are fairly pedestrian by comparison. You can visualize and walk through the statistical distributions. Everything is effectively instantaneous within each next token generation. It's hard to follow an action potential around in time because it's constantly being integrated with other potentials and its influence is only relevant for a brief moment.

Nevermark 1y ago

Here is a slightly sideways take on the question. Since some misunderstandings crop up over and over. Things that are not true about neural networks and LLMs: 1. They are not stochastic parrots. Statistics can be used to judge model performances, as with any type of model whatsoever. But that doesn’t make a model statistical. Neural network components are not statistically driven, they are gradient driven. They don’t calculate statistics, they learn topological representations of relationships. 2. LLMs are not just doing word prediction or text completion. Yes, that is the basic task they are trained on, but the “just” (that is often stated or implied) trivializes what the network actually has to learn to perform well. Task type, and what must be learned to achieve success at that task, are two entirely different things. To predict the kinds of human reasoning documented in writing in training sets, requires that kind of reasoning be learned. Not just some compressed generalization of people’s particular responses. Simple proof that LLMs are not just compressing a lot of human behavior comes easily. Just ask an LLM to do something involving several topics unlikely to have been encountered together before, and their one-shot answer might not be God’s word on the issue, but is a far cry from the stumbling that good mimic could ever do. (Example task, ask for a Supreme Court brief to argue for rabbits rights based on sentient animal and native rights, with serious arguments, but written in Dr. Seuss prose by James Bond.) 3. LLMs do reason. Not like us or as well as us. But also, actually better than us in some ways. LLMs are far superior to us at very wide, somewhat shallow reasoning. They are able to one-shot weave together information from tens of thousands of disparate topics and ideas, on demand. But they don’t notice their own blind spots. They miss implications. Those are things humans do quickly in continuous narrower deeper reasoning cycles. They are getting better. And some of their failures should be attributed to the strict constraint of one/few-shot(s) to well written responses, that we put on them. We don’t hold humans to such strict standards. And if we did, we would also make a lot more obvious errors. Wide & shallow reasoning in ways humans can’t match, is not a trivial success. 4. LLMs are very creative. As with their reasoning, not like us: very wide in terms of instantly and fluidly weaving highly disparate information together. Somewhat shallow (but gaining) in terms of iteratively recognizing and self-correcting their own mistakes and oversights. See random original disparate topic task above. First, spend several hours performing that task oneself. Or take a day or two if need be. Then, give the task to an LLM and wait a few seconds. Compare. Ouch. — TLDR; somewhat off topic and high level, but when trying to understand models it helps to avoid seemingly endlessly repeated misunderstandings.

d4rkn0d3z 1y ago

Of course we understand how they work, don't panic. Some of us are not even surprised at the results thus far, the good, the bad, and the ugly. We have understood how neural networks function for 50 years or more. I think the interdisciplinary nature of the technology is significant, NN draw from many areas of scientific study and if you are missing any piece of the puzzle then some mysteriousness arises for you. But take heart, as I said, there are people who are not surprised one iota by what is happening. Dozens or perhaps hundreds of people have considered embarking on the brute force approach being pursued, nobody decided to try until a few years back because it is mathematically limited, costly, and might just make our problems much worse before it can make them better in any important way. What is very salient is that 10 of the most powerful corporations and the leaders of the largest countries have decided on this route and they could force all software engineers to wear extra long, wide red ties if they wanted to so buckle up. It looks like we are riding this train all the way in the hope that cash + time will solve the truncation of the science.

hodgehog11 1y ago
As someone who works in the theory of deep learning, I'm curious to know what you mean by "understand how they work". Sure, we understand the broad strokes, and those tend to be good enough to chip away at making improvements and vague predictions of what certain things will do, but we still have a severe lack of understanding of their implicit biases which allow them to perform well and give rise to the scaling laws. We know a lot more about basically every other machine learning approach.
- d4rkn0d3z 1y ago
  Well, I'm thumb bashing on my phone so I can't get too far into the details as I see them. I'd read and respond to any papers you have?
  - hodgehog11 1y ago
    Are you asking about open problems? Here are a couple: There are strong arguments that deep learning generalization (and robustness in a slightly different sense) can be successfully explained in terms of PAC-Bayes theory, for example: https://arxiv.org/abs/2503.02113 https://arxiv.org/abs/2503.02113 This requires a strong understanding of the appropriate prior, which effectively quantifies implicit bias in the training setup / architecture setup. Early analyses suggest the bias is that of a minimal gradient norm (https://arxiv.org/pdf/2101.12176 https://arxiv.org/pdf/2101.12176) but this is almost certainly not the case, e.g.(https://arxiv.org/abs/2005.06398 https://arxiv.org/abs/2005.06398). There are good empirical arguments that models are trained to achieve maximal compression (https://arxiv.org/abs/2211.13609 https://arxiv.org/abs/2211.13609). But we only know this in a vague sense. How are deep learning training procedures and architectures implicitly exhibiting this form of sparsity and compressing information? Why is this more effective for some optimizers/architectures than others? Why would minimum description length circuits be so effective at mimicking human intelligence? How does this form of regularisation induce the exponents in the neural scaling laws (so we can figure out how to best optimize for these exponents)? If we don't know the implicit bias and how it relates to the training procedure, we're just doing alchemy and effectively got lucky. Change the standard approach even slightly and things easily break. We only really know what to do by process of elimination. Then there are the LLM specific questions related to in-context learning (https://arxiv.org/abs/2306.00297 https://arxiv.org/abs/2306.00297), to help explain why next token prediction through transformer designs works so well. Does it implicitly act as an optimization procedure (with bias) relative to the context? What are the biases here? What is the problem being solved? These may sound specific, but they are just examples of some approaches to answer the big question of "why does this standard procedure work so well, and what can we do to make it work better". Without this, we're just following the heuristics of what came before without a guiding framework. Post-hoc, we can do a lot more with interpretability frameworks and linear probes and the like. But without the implicit biases, we are only getting a local picture, which doesn't help us understand what a model is likely to do before we run it. We need global information. These are just a few, and don't include the philosophical and theory of mind questions, and others from neuroscience...