4 ms·

> Copilot loves suggesting about 25 nested divs as a starting point. > I assume this is because of a flaw in how LLMs work. I know with some LLM implementatio

> Copilot loves suggesting about 25 nested divs as a starting point. > I assume this is because of a flaw in how LLMs work. I know with some LLM implementations, you can configure the sampling to penalise repetitions – this is making me wonder if Copilot might benefit from that? > What does it say about Copilot’s knowledge of accessibility when it will hand us code even basic checking tools would flag? Maybe it could do with some fine-tuning based on those checking tools? e.g. sample many answers to same prompt, run them through checking tool, and then fine-tune it to prefer the answers which caused the least warnings? Or: run the suggestion through checking tools, and if it triggers warnings, sample a new suggestion, and see if the new one doesn't. This could be done on the client side in a loop – run suggestion through checks, if it fails, ask the LLM for a new and different suggestion, repeat until we get one which passes checks, or we give up.

teaearlgraycold 3y ago
The issue is that often you do want heavy repetition when programming. Think about a list of strings where they mostly have a common prefix. Or JSON, or a bunch of imports and exports. Good code often has these low entropy sections.
- skissane 3y ago
  > The issue is that often you do want heavy repetition when programming. Up to a certain number of tokens, yes. But, I doubt any high quality code would have the exact same sequence of N tokens repeated 25 times consecutively. There's a certain heaviness of repetition at which it is unlikely to be genuinely useful. > Or JSON, or a bunch of imports and exports A human programmer, when evaluating whether code is repetitive, doesn't treat all tokens as equal – they ignore "expected"/"necessary" repetitions, and focus on the "unexpected"/"unnecessary" ones. So, penalising repetitions in sampling doesn't have to treat all tokens equally either. For example, in a JSON document, one might choose to ignore the tokens required by JSON syntax. In Java, one might penalise repetitions less in the import block than in a method body. Of course, this means the sampling actually has to be aware of the syntax of the language being generated – which is possible, and can have some other advantages (e.g. if sampling only samples tokens which are allowed by the language grammar, you can eliminate many possibilities of generating syntactically invalid code.)
  - teaearlgraycold 3y ago
    All true. The raw transformer architecture isn’t enough to write sane code. I’d love to see changes made to have them guided by the compiler, customer linter rules, etc. I still like them as is. I don’t let them write too much code for me. They’re really good translators (JSON to TypeScript interface definitions, shell command to Python string list) and quick documentation lookups (I like to write quick one-line comments for something short that I need that I would previously have looked up).
pclmulqdq 3y ago
If you look at the code for many sites with high "production value" today, 25 nested divs is about right. As a non-web-dev, I have always been surprised at how often you need to throw in a new layer of divs to get some simple visual thing to work across device sizes.
- wwweston 3y ago
  As a web dev, I honestly can’t figure out the value here. It makes things considerably less legible from an inspection standpoint. And while yes each div provides a handle by which to independently control some layout property the law of diminishing returns kicks in hard by about three layers deep (three block level elements deep will cover 99.9% of things you might want to do with a given visual-block element). I can only imagine this kind of markup is tooling generated and assumes that no human eyes will ever have to review/write it.
  - pclmulqdq 3y ago
    It's a good point that the HTML and CSS of all public websites was probably fed into copilot, and I assume a lot of that came from template engines or other tools rather than being handwritten. I guess it would be like teaching a computer C++ by looking at the output of Cython.

teaearlgraycold 3y ago

The issue is that often you do want heavy repetition when programming. Think about a list of strings where they mostly have a common prefix. Or JSON, or a bunch of imports and exports. Good code often has these low entropy sections.

skissane 3y ago
> The issue is that often you do want heavy repetition when programming. Up to a certain number of tokens, yes. But, I doubt any high quality code would have the exact same sequence of N tokens repeated 25 times consecutively. There's a certain heaviness of repetition at which it is unlikely to be genuinely useful. > Or JSON, or a bunch of imports and exports A human programmer, when evaluating whether code is repetitive, doesn't treat all tokens as equal – they ignore "expected"/"necessary" repetitions, and focus on the "unexpected"/"unnecessary" ones. So, penalising repetitions in sampling doesn't have to treat all tokens equally either. For example, in a JSON document, one might choose to ignore the tokens required by JSON syntax. In Java, one might penalise repetitions less in the import block than in a method body. Of course, this means the sampling actually has to be aware of the syntax of the language being generated – which is possible, and can have some other advantages (e.g. if sampling only samples tokens which are allowed by the language grammar, you can eliminate many possibilities of generating syntactically invalid code.)
- teaearlgraycold 3y ago
  All true. The raw transformer architecture isn’t enough to write sane code. I’d love to see changes made to have them guided by the compiler, customer linter rules, etc. I still like them as is. I don’t let them write too much code for me. They’re really good translators (JSON to TypeScript interface definitions, shell command to Python string list) and quick documentation lookups (I like to write quick one-line comments for something short that I need that I would previously have looked up).

pclmulqdq 3y ago

If you look at the code for many sites with high "production value" today, 25 nested divs is about right. As a non-web-dev, I have always been surprised at how often you need to throw in a new layer of divs to get some simple visual thing to work across device sizes.

wwweston 3y ago
As a web dev, I honestly can’t figure out the value here. It makes things considerably less legible from an inspection standpoint. And while yes each div provides a handle by which to independently control some layout property the law of diminishing returns kicks in hard by about three layers deep (three block level elements deep will cover 99.9% of things you might want to do with a given visual-block element). I can only imagine this kind of markup is tooling generated and assumes that no human eyes will ever have to review/write it.
- pclmulqdq 3y ago
  It's a good point that the HTML and CSS of all public websites was probably fed into copilot, and I assume a lot of that came from template engines or other tools rather than being handwritten. I guess it would be like teaching a computer C++ by looking at the output of Cython.