3 ms·

The issue is that often you do want heavy repetition when programming. Think about a list of strings where they mostly have a common prefix. Or JSON, or a bunch

The issue is that often you do want heavy repetition when programming. Think about a list of strings where they mostly have a common prefix. Or JSON, or a bunch of imports and exports. Good code often has these low entropy sections.

skissane 3y ago
> The issue is that often you do want heavy repetition when programming. Up to a certain number of tokens, yes. But, I doubt any high quality code would have the exact same sequence of N tokens repeated 25 times consecutively. There's a certain heaviness of repetition at which it is unlikely to be genuinely useful. > Or JSON, or a bunch of imports and exports A human programmer, when evaluating whether code is repetitive, doesn't treat all tokens as equal – they ignore "expected"/"necessary" repetitions, and focus on the "unexpected"/"unnecessary" ones. So, penalising repetitions in sampling doesn't have to treat all tokens equally either. For example, in a JSON document, one might choose to ignore the tokens required by JSON syntax. In Java, one might penalise repetitions less in the import block than in a method body. Of course, this means the sampling actually has to be aware of the syntax of the language being generated – which is possible, and can have some other advantages (e.g. if sampling only samples tokens which are allowed by the language grammar, you can eliminate many possibilities of generating syntactically invalid code.)
- teaearlgraycold 3y ago
  All true. The raw transformer architecture isn’t enough to write sane code. I’d love to see changes made to have them guided by the compiler, customer linter rules, etc. I still like them as is. I don’t let them write too much code for me. They’re really good translators (JSON to TypeScript interface definitions, shell command to Python string list) and quick documentation lookups (I like to write quick one-line comments for something short that I need that I would previously have looked up).

skissane 3y ago

> The issue is that often you do want heavy repetition when programming. Up to a certain number of tokens, yes. But, I doubt any high quality code would have the exact same sequence of N tokens repeated 25 times consecutively. There's a certain heaviness of repetition at which it is unlikely to be genuinely useful. > Or JSON, or a bunch of imports and exports A human programmer, when evaluating whether code is repetitive, doesn't treat all tokens as equal – they ignore "expected"/"necessary" repetitions, and focus on the "unexpected"/"unnecessary" ones. So, penalising repetitions in sampling doesn't have to treat all tokens equally either. For example, in a JSON document, one might choose to ignore the tokens required by JSON syntax. In Java, one might penalise repetitions less in the import block than in a method body. Of course, this means the sampling actually has to be aware of the syntax of the language being generated – which is possible, and can have some other advantages (e.g. if sampling only samples tokens which are allowed by the language grammar, you can eliminate many possibilities of generating syntactically invalid code.)

teaearlgraycold 3y ago
All true. The raw transformer architecture isn’t enough to write sane code. I’d love to see changes made to have them guided by the compiler, customer linter rules, etc. I still like them as is. I don’t let them write too much code for me. They’re really good translators (JSON to TypeScript interface definitions, shell command to Python string list) and quick documentation lookups (I like to write quick one-line comments for something short that I need that I would previously have looked up).