5 ms·

i would be may more impressed if LLMs could do code compression. more code == more things that can break, and when llms can generate boatloads of it with a clic

by ausbah 2y ago

i would be may more impressed if LLMs could do code compression. more code == more things that can break, and when llms can generate boatloads of it with a click you can imagine what might happen

Scene_Cast2 2y ago
This actually sparked an idea for me. Could code complexity be measured as cumulative entropy as measured by running LLM token predictions on a codebase? Notably, verbose boilerplate would be pretty low entropy, and straightforward code should be decently low as well.
- jeffparsons 2y ago
  Not quite, I think. Some kinds of redundancy are good, and some are bad. Good redundancy tends to reduce mistakes rather than introduce them. E.g. there's lots of redundancy in natural languages, and it helps resolve ambiguity and fill in blanks or corruption if you didn't hear something properly. Similarly, a lot of "entropy" in code could be reduced by shortening names, deleting types, etc., but all those things were helping to clarify intent to other humans, thereby reducing mistakes. But some is copy+paste of rules that should be enforce in one place. Teaching a computer to understand the difference is... hard. Although, if we were to ignore all this for a second, you could also make similar estimates with, e.g., gzip: the higher the compression ratio attained, the more "verbose"/"fluffy" the code is. Fun tangent: there are a lot of researchers who believe that compression and intelligence are equivalent or at least very tightly linked.
  - 8note 2y ago
    Interpreting this comment, it would predict low complexity for code copied unnecessarily. I'm not sure though. If it's copied a bunch of times, and it actually doesn't matter because each usecase of the copying is linearly independent, does it matter that it was copied? Over time, you'd still see copies being changed by themselves show up as increased entropy
- malfist 2y ago
  Code complexity can already be measured deterministically with cyclomatic complexity. No need to use an AI fuzzy logic at this. Especially when they're bad at math.
  - contravariant 2y ago
    There's nothing fuzzy about letting an LLM determine the probability of a particular piece of text. In fact it's the one thing they are explicitly designed to do, the rest is more or less a side-effect.
- david-gpu 2y ago
  > Could code complexity be measured as cumulative entropy as measured by running LLM token predictions on a codebase? Notably, verbose boilerplate would be pretty low entropy, and straightforward code should be decently low as well. WinRAR can do that for you quite effectively.
ks2048 2y ago
I agree. It seems like counting lines of generated code is like counting bytes/instructions of compiled code - who cares? If “code” becomes prompts, then AI should lead to much smaller code than before. I’m aware that the difference is that AI-generated code can be read and modified by humans. But that quantity is bad because humans have to understand it to read or modify it.
- deleted 2y ago
  [deleted]
- TZubiri 2y ago
  What's that line about accounting for lines of code on the wrong side of the balance sheet?
- latexr 2y ago
  > If “code” becomes prompts, then AI should lead to much smaller code than before. What’s the point of shorter code if you can’t trust it to do what it’s supposed to? I’ll take 20 lines of code that do what they should consistently over 1 line that may or may not do the task depending on the direction of the wind.
asah 2y ago
meh - the LLM code I'm seeing isn't particularly more verbose. And as others have said, if you want tighter code, just add that to the prompt. fun story: today I had an LLM write me a non-trivial perl one-liner. It tried to be verbose but I insisted and it gave me one tight line.
AlexandrB 2y ago
Exactly this. Code is a liability, if you can do the same thing with less code you're often better off.
- EasyMark 2y ago
  Not if it’s already stable and has been running for years. Legacy doesn’t necessarily mean “need replacement because of technical debt”. I’ve seen lots of people want to replace code that has been running basically bug free for years because “there are better coding styles and practices now”
8note 2y ago
How would it know which edge cases are being useful and which ones aren't? I understand more code as being more edge cases
- wvenable 2y ago
  More code could just be useless code that no longer serves any purpose but still looks reasonable to the naked eye. An LLM can certainly figure out and suggest maybe some conditional is impossible given the rest of the code. I can also suggest alternatives, like using existing library functions for things that might have been coded manually.
  - ekwav 2y ago
    Or just refactor to use early returns

by ausbah 2y ago

i would be may more impressed if LLMs could do code compression. more code == more things that can break, and when llms can generate boatloads of it with a click you can imagine what might happen

Scene_Cast2 2y ago

This actually sparked an idea for me. Could code complexity be measured as cumulative entropy as measured by running LLM token predictions on a codebase? Notably, verbose boilerplate would be pretty low entropy, and straightforward code should be decently low as well.

jeffparsons 2y ago
Not quite, I think. Some kinds of redundancy are good, and some are bad. Good redundancy tends to reduce mistakes rather than introduce them. E.g. there's lots of redundancy in natural languages, and it helps resolve ambiguity and fill in blanks or corruption if you didn't hear something properly. Similarly, a lot of "entropy" in code could be reduced by shortening names, deleting types, etc., but all those things were helping to clarify intent to other humans, thereby reducing mistakes. But some is copy+paste of rules that should be enforce in one place. Teaching a computer to understand the difference is... hard. Although, if we were to ignore all this for a second, you could also make similar estimates with, e.g., gzip: the higher the compression ratio attained, the more "verbose"/"fluffy" the code is. Fun tangent: there are a lot of researchers who believe that compression and intelligence are equivalent or at least very tightly linked.
- 8note 2y ago
  Interpreting this comment, it would predict low complexity for code copied unnecessarily. I'm not sure though. If it's copied a bunch of times, and it actually doesn't matter because each usecase of the copying is linearly independent, does it matter that it was copied? Over time, you'd still see copies being changed by themselves show up as increased entropy
malfist 2y ago
Code complexity can already be measured deterministically with cyclomatic complexity. No need to use an AI fuzzy logic at this. Especially when they're bad at math.
- contravariant 2y ago
  There's nothing fuzzy about letting an LLM determine the probability of a particular piece of text. In fact it's the one thing they are explicitly designed to do, the rest is more or less a side-effect.
david-gpu 2y ago
> Could code complexity be measured as cumulative entropy as measured by running LLM token predictions on a codebase? Notably, verbose boilerplate would be pretty low entropy, and straightforward code should be decently low as well. WinRAR can do that for you quite effectively.

ks2048 2y ago

I agree. It seems like counting lines of generated code is like counting bytes/instructions of compiled code - who cares? If “code” becomes prompts, then AI should lead to much smaller code than before. I’m aware that the difference is that AI-generated code can be read and modified by humans. But that quantity is bad because humans have to understand it to read or modify it.

deleted 2y ago
[deleted]
TZubiri 2y ago
What's that line about accounting for lines of code on the wrong side of the balance sheet?
latexr 2y ago
> If “code” becomes prompts, then AI should lead to much smaller code than before. What’s the point of shorter code if you can’t trust it to do what it’s supposed to? I’ll take 20 lines of code that do what they should consistently over 1 line that may or may not do the task depending on the direction of the wind.

asah 2y ago

meh - the LLM code I'm seeing isn't particularly more verbose. And as others have said, if you want tighter code, just add that to the prompt. fun story: today I had an LLM write me a non-trivial perl one-liner. It tried to be verbose but I insisted and it gave me one tight line.

AlexandrB 2y ago

Exactly this. Code is a liability, if you can do the same thing with less code you're often better off.

EasyMark 2y ago
Not if it’s already stable and has been running for years. Legacy doesn’t necessarily mean “need replacement because of technical debt”. I’ve seen lots of people want to replace code that has been running basically bug free for years because “there are better coding styles and practices now”

8note 2y ago

How would it know which edge cases are being useful and which ones aren't? I understand more code as being more edge cases

wvenable 2y ago
More code could just be useless code that no longer serves any purpose but still looks reasonable to the naked eye. An LLM can certainly figure out and suggest maybe some conditional is impossible given the rest of the code. I can also suggest alternatives, like using existing library functions for things that might have been coded manually.
- ekwav 2y ago
  Or just refactor to use early returns