4 ms·

Confusing to me actually. Are you saying coding doesn't require understanding ideas?

xanderlewis 3y ago
Not at all; just that the ‘understanding’-related content of pure mathematics is much less evident on the page. It mostly lives in the heads of mathematicians and is largely independent of notation, whereas writing blocks of code is a task that is literally about using notation — something for which a huge amount of high-quality training data exists. Of course, the higher level ideas in programming and software development are not just about ‘writing code’ itself, but I suspect this is where current models begin to show their weakness.
- Davidzheng 3y ago
  Still struggling to understand. You're saying that most coding has no high level ideas and is just boilerplate, and the ones that aren't are the ones LLM's struggle with? This could be true I can see it. WRT the understanding not being shown on the page in math, I guess I tend to agree(?). But I think good mathematical papers show understanding of the ideas too more than just the proofs which result from the understanding. The problem (probably you know this but just for the benefit of whoever is reading) is that "understanding" in mathematics, at least with respect to producing proofs, often rely on mental models and analogies which are WRONG. Not like vague but often straight up incorrect. And you understand also the limitations of where the model goes wrong. And it's kind of embarrassing (I assume) for most people to write wrong statements into papers even with caveats. For simple examples there's a meme right where to visualize n-dimensional space, you visualize R^3 and say (n-dimensional) in your head. In this sense I think it's possibly straight-up unhelpful for the authors to impose their mental models on the reader as well (for example if the reader can actually visualize R^n without this crutch it would be unhelpful). But I'm not sure if this is what distinguishes math and programming. There's also the alternative hypothesis that the mental work to generate each additional line of proof is just order of magnitude higher than the average for code. Just meaning that it usually requires more thought to produce a line of math proof. In this possibility, we would expect it to be solved by scaling alone. One thing it reminds of, which is quite different admittedly, is the training of leela-zero on go. There was a period of time where it would struggle on long ladders. And eventually it was overcome with training along (despite people not believing it would be resolved at first). I think in that situation, people summarized afterwards the situation as, in particular situations, humans can search much deeper than other places, and therefore requiring more training for the machine to match the humans' ability.
  - xanderlewis 3y ago
    I’ll start with a disclaimer that I don’t know for sure (no one really does) what the difference between solving programming problems and solving pure mathematics problems is (and certainly if you stretch the domains you can emulate each within the other, thereby showing their equivalence… if you like). I’m just speculating, as usual. So if you’re confused, maybe that’s just because I’m wrong. > most coding has no high level ideas and is just boilerplate, and the ones that aren't are the ones LLM's struggle with? Pretty much, although calling it boilerplate might be going a bit far. I’m not here to claim something like ‘mathematicians think and programmers do not’ because that is clearly not the case (and sounds like a mathematician with a complex of some kind). But it is empirically the case that so far GPT-4 and the like are much better at programming than maths. Why? I think the reason is that whilst the best programmers have a deep understanding of the tools and concepts they use, it’s not necessary to get things to work. You can probably get an away without it (I have ideas about why, but for now that’s not the point). And given the amount of data available on basic programming questions (much more than there is of mathematics) if you’re an LLM it’s quite possible to fake it. I guess one could also make the point that the space of possible questions in any given programming situation, however large, is still fairly constrained. At least the questions will always be ‘compute this’ or ‘generate one of these’ or something. Whereas you can pick up any undergraduate maths textbook, choose a topic, and if you know what you’re doing it’s easy to ask a question of the form ‘describe what I get if I do this’ or ‘is it true that xyz’ that will trip ChatGPT up because it just generates something that matches the form implied by the question: ‘a mathematical-looking answer’, but doesn’t seem to actually ask itself the question first. It just writes. In perfect Mathematical English. I guess in programming it turns out that ‘a code-looking answer’ for some reason often gives something quite useful. Another difference that occurs to me is that what is considered a fixable syntax error in programming when done in the context of maths leads to complete nonsense because the output is supposed to describe rather than do. The answers are somehow much more sensitive to corruption, which perhaps says something about the data itself.

Not at all; just that the ‘understanding’-related content of pure mathematics is much less evident on the page. It mostly lives in the heads of mathematicians and is largely independent of notation, whereas writing blocks of code is a task that is literally about using notation — something for which a huge amount of high-quality training data exists. Of course, the higher level ideas in programming and software development are not just about ‘writing code’ itself, but I suspect this is where current models begin to show their weakness.

Davidzheng 3y ago
Still struggling to understand. You're saying that most coding has no high level ideas and is just boilerplate, and the ones that aren't are the ones LLM's struggle with? This could be true I can see it. WRT the understanding not being shown on the page in math, I guess I tend to agree(?). But I think good mathematical papers show understanding of the ideas too more than just the proofs which result from the understanding. The problem (probably you know this but just for the benefit of whoever is reading) is that "understanding" in mathematics, at least with respect to producing proofs, often rely on mental models and analogies which are WRONG. Not like vague but often straight up incorrect. And you understand also the limitations of where the model goes wrong. And it's kind of embarrassing (I assume) for most people to write wrong statements into papers even with caveats. For simple examples there's a meme right where to visualize n-dimensional space, you visualize R^3 and say (n-dimensional) in your head. In this sense I think it's possibly straight-up unhelpful for the authors to impose their mental models on the reader as well (for example if the reader can actually visualize R^n without this crutch it would be unhelpful). But I'm not sure if this is what distinguishes math and programming. There's also the alternative hypothesis that the mental work to generate each additional line of proof is just order of magnitude higher than the average for code. Just meaning that it usually requires more thought to produce a line of math proof. In this possibility, we would expect it to be solved by scaling alone. One thing it reminds of, which is quite different admittedly, is the training of leela-zero on go. There was a period of time where it would struggle on long ladders. And eventually it was overcome with training along (despite people not believing it would be resolved at first). I think in that situation, people summarized afterwards the situation as, in particular situations, humans can search much deeper than other places, and therefore requiring more training for the machine to match the humans' ability.
- xanderlewis 3y ago
  I’ll start with a disclaimer that I don’t know for sure (no one really does) what the difference between solving programming problems and solving pure mathematics problems is (and certainly if you stretch the domains you can emulate each within the other, thereby showing their equivalence… if you like). I’m just speculating, as usual. So if you’re confused, maybe that’s just because I’m wrong. > most coding has no high level ideas and is just boilerplate, and the ones that aren't are the ones LLM's struggle with? Pretty much, although calling it boilerplate might be going a bit far. I’m not here to claim something like ‘mathematicians think and programmers do not’ because that is clearly not the case (and sounds like a mathematician with a complex of some kind). But it is empirically the case that so far GPT-4 and the like are much better at programming than maths. Why? I think the reason is that whilst the best programmers have a deep understanding of the tools and concepts they use, it’s not necessary to get things to work. You can probably get an away without it (I have ideas about why, but for now that’s not the point). And given the amount of data available on basic programming questions (much more than there is of mathematics) if you’re an LLM it’s quite possible to fake it. I guess one could also make the point that the space of possible questions in any given programming situation, however large, is still fairly constrained. At least the questions will always be ‘compute this’ or ‘generate one of these’ or something. Whereas you can pick up any undergraduate maths textbook, choose a topic, and if you know what you’re doing it’s easy to ask a question of the form ‘describe what I get if I do this’ or ‘is it true that xyz’ that will trip ChatGPT up because it just generates something that matches the form implied by the question: ‘a mathematical-looking answer’, but doesn’t seem to actually ask itself the question first. It just writes. In perfect Mathematical English. I guess in programming it turns out that ‘a code-looking answer’ for some reason often gives something quite useful. Another difference that occurs to me is that what is considered a fixable syntax error in programming when done in the context of maths leads to complete nonsense because the output is supposed to describe rather than do. The answers are somehow much more sensitive to corruption, which perhaps says something about the data itself.