2 ms·

Neither "variable names contain valuable information" nor "AIs are finite" are anthropomorphization. That variable names contain information is not only relativ

by jerf 1y ago

Neither "variable names contain valuable information" nor "AIs are finite" are anthropomorphization. That variable names contain information is not only relative to some sort of human cognition; they objectively, mathematically, contain information. Remove that information and the AI is going to have to reconstruct it (or at least most of it), and as finite things, that's going to cost them because nothing like that is free for finite things. None of my assertions here depend on the humanity, the AI-ness, the Vulcan-ness, or any other conceivable finite intellectual architecture of any coding agent. It only depends on them being finite.

amelius 1y ago
Let's stop with the comparison to humans, I'm more interested in why it would hurt to train LLMs with harder puzzles. Isn't that what we're doing all the time when training llms? I'm just suggesting an easy way to construct new puzzles: just randomize the varnames.
- recursive 1y ago
  An even easier way to construct new puzzles is to fully randomize the problem statements and intended solutions. When you take out the information from the variable names, you're making the training data farther from real-world data. Practicing walking on your hands, while harder than walking on your feet, won't make you better at hiking. In fact, if you spend your limited training resources on it, the opportunity cost might make you worse.
JambalayaJimbo 1y ago
But we know that variable names do not matter whatsoever to a compiler. Now I do agree with you intuitively that LLMs perform better on meaningful variable names, without looking at hard data - but I don't think it has anything to do with "brainpower" - I just think your input data is more likely to resemble training data with meaningful variable names.
- socalgal2 1y ago
  I think you're just arguing semantics. It seems intuitively obvious that if I have some simple physics code newPosition = currentPos + velocity * deltaTime and change it to addressInChina = weightByGold + numberOfDogsOwned * birdPopulationInFrance that both a human and likely an LLM will struggle to understand the code and do the the right thing. The thing we're discussion is does the LLM struggle. No one cares if that's not literally "brain" power. All they care about is does the LLM do a better, worse, or the same > I just think your input data is more likely to resemble training data with meaningful variable names. Based on giving job interviews, cryptic names are common.
_0ffh 1y ago
Exactly because the task is harder if the variable name does not contain any information is what makes training like that a good idea. It forces the LLM to pay attention to the actual code to get it right, which in training is a Good Thing (TM).

by jerf 1y ago

amelius 1y ago

Let's stop with the comparison to humans, I'm more interested in why it would hurt to train LLMs with harder puzzles. Isn't that what we're doing all the time when training llms? I'm just suggesting an easy way to construct new puzzles: just randomize the varnames.

recursive 1y ago
An even easier way to construct new puzzles is to fully randomize the problem statements and intended solutions. When you take out the information from the variable names, you're making the training data farther from real-world data. Practicing walking on your hands, while harder than walking on your feet, won't make you better at hiking. In fact, if you spend your limited training resources on it, the opportunity cost might make you worse.

JambalayaJimbo 1y ago

But we know that variable names do not matter whatsoever to a compiler. Now I do agree with you intuitively that LLMs perform better on meaningful variable names, without looking at hard data - but I don't think it has anything to do with "brainpower" - I just think your input data is more likely to resemble training data with meaningful variable names.

socalgal2 1y ago
I think you're just arguing semantics. It seems intuitively obvious that if I have some simple physics code newPosition = currentPos + velocity * deltaTime and change it to addressInChina = weightByGold + numberOfDogsOwned * birdPopulationInFrance that both a human and likely an LLM will struggle to understand the code and do the the right thing. The thing we're discussion is does the LLM struggle. No one cares if that's not literally "brain" power. All they care about is does the LLM do a better, worse, or the same > I just think your input data is more likely to resemble training data with meaningful variable names. Based on giving job interviews, cryptic names are common.

_0ffh 1y ago

Exactly because the task is harder if the variable name does not contain any information is what makes training like that a good idea. It forces the LLM to pay attention to the actual code to get it right, which in training is a Good Thing (TM).