4 ms·

The paper actually explains that it targets at the API misuse problem, not the semantic alignment or bugs. Semantic bugs are difficult to detect, which is alrea

by 13yuan 3y ago

The paper actually explains that it targets at the API misuse problem, not the semantic alignment or bugs. Semantic bugs are difficult to detect, which is already a consensus in software engineering field and still many ongoing work on it. And when we say 'semantic' in this special problem it means more than 'semantic' in programs but also how developers express their semantics, which is definitely a problem bigger than checking the code itself. The API usage patterns are created to help check the code snippets given by LLMs, while the race condition you mentioned could exist but not checkable unless given other components of the programs. If adding the bugs you mentioned, the buggy code generated by LLMs could even exceed the number claimed in the paper.

teraflop 3y ago
But what is "misuse", then, if not something that causes a bug? "Misuse" is not a formally defined thing in Java, and the authors never define what they mean by it. > If adding the bugs you mentioned, the buggy code generated by LLMs could even exceed the number claimed in the paper. Or it could be much fewer. Aside from a few scarce examples, the paper gives no evidence that most of the patterns that were selected are actually associated with "misuse". They just declare it to be so. (The 2018 paper they cite for their dataset also provides little such evidence. It just says that the authors reviewed the patterns, reviewed documentation, and decided which ones they considered to be poor code quality.) It is easy to come up with valid situations where a piece of code violates those patterns, but behaves as intended and is not misusing the API. So why should I assign any meaning to the fact that some percentage of code snippets violate the patterns? I could make a list of adjectives that frequently appear in comments near buggy code, and then count how many LLM outputs contain those outputs, and then say that means the LLM output is buggy. But I would not be saying anything meaningful about the LLM's quality by doing so.

by 13yuan 3y ago

teraflop 3y ago

But what is "misuse", then, if not something that causes a bug? "Misuse" is not a formally defined thing in Java, and the authors never define what they mean by it. > If adding the bugs you mentioned, the buggy code generated by LLMs could even exceed the number claimed in the paper. Or it could be much fewer. Aside from a few scarce examples, the paper gives no evidence that most of the patterns that were selected are actually associated with "misuse". They just declare it to be so. (The 2018 paper they cite for their dataset also provides little such evidence. It just says that the authors reviewed the patterns, reviewed documentation, and decided which ones they considered to be poor code quality.) It is easy to come up with valid situations where a piece of code violates those patterns, but behaves as intended and is not misusing the API. So why should I assign any meaning to the fact that some percentage of code snippets violate the patterns? I could make a list of adjectives that frequently appear in comments near buggy code, and then count how many LLM outputs contain those outputs, and then say that means the LLM output is buggy. But I would not be saying anything meaningful about the LLM's quality by doing so.