3 ms·
The fact that GPT3 works at all for coding indicates that our programming languages are too low level and force a lot of redundancy (low entropy). From a progra
by nradov 5y ago
The fact that GPT3 works at all for coding indicates that our programming languages are too low level and force a lot of redundancy (low entropy). From a programmer productivity optimization perspective it should be impossible to reliably predict the next statement. Of course there might be trade offs. Some of that redundancy might be helping maintenance programmers to understand the code.
- hans1729 5y ago>From a programmer productivity optimization perspective it should be impossible to reliably predict the next statement Why? 99.9% of programming being done is composition of trivial logical propositions, in some semantic context. The things we implement are trivial, unless you're thinking about symbolic proofs etc
- tshaddox 5y agoI think that’s precisely the problem the parent commenter is describing.
- alephaleph 5y agoThat would only follow if we were trying to optimize code for brevity, and I have no clue why that would be your top priority.
- nradov 5y agoNot at all. Brevity (or verbosity) is largely orthogonal to level of entropy or redundancy. In principle it ought to be possible to code at a higher level of abstraction while still using understandable names and control flow constructs.
- crubier 5y agoI’m really not sure about that. By definition, high entropy means high information density (information per character). So with the same amount of information you would have less characters.
- nradov 5y agoThink in terms of information density per symbol on the abstract syntax tree, not per character in the source code.
- mpoteat 5y agoI have indeed seen codebases where it seems like the programmer was being charged per source code byte. Full of single letter methods and such - it takes a large confusion of ideas to motivate such a philosophy.
- mpoteat 5y agoIndeed, in the limit of maximal abstraction, i.e. semantic compression, code becomes unreadable by humans in practice. We can see this in code golf competitions.
- dharmaturtle 5y agoLet me rephrase: > The fact that GPT3 works at all for English indicates that English is too low level and forces a lot of redundancy (low entropy). I don't think the goal is to compress information/language and maximize "surprise".
- Traster 5y agoI think this is kind of true, but also kind of not true. Programming, like all writing, is the physical manifestation of a much more complex mental process. By the time that I know what I want to write, the hardwork is done. In that way, you can think of co-pilot as a way of increasing the WPM of an average coder. But the WPM isn't the bit that matters. In fact almost the onlt thing that matters are hte bits you won't predict.
- brokensegue 5y agoI write code directly as gziped text. Saves lots of key strokes and you can use off the shelf compilers by piping through zcat first