4 ms·
> what's the evidence What’s the evidence for anything software engineers use? Tests, type checkers, syntax highlighting, IDEs, code review, pair programming,
by jaredklewis 6mo ago
> what's the evidence
What’s the evidence for anything software engineers use? Tests, type checkers, syntax highlighting, IDEs, code review, pair programming, and so on.
In my experience, evidence for the efficacy of software engineering practices falls into two categories:
- the intuitions of developers, based in their experiences.
- scientific studies, which are unconvincing. Some are unconvincing because they attempt to measure the productivity of working software engineers, which is difficult; you have to rely on qualitative measures like manager evaluations or quantitative but meaningless measures like LOC or tickets closed. Others are unconvincing because they instead measure the practice against some well defined task (like a coding puzzle) that is totally unlike actual software engineering.
Evidence for this LLM pattern is the same. Some developers have an intuition it works better.
- codemog 6mo agoMy friend, there’s tons of evidence of all that stuff you talked about in hundreds of papers on arxiv. But you dismiss it entirely in your second bullet point, so I’m not entirely sure what you expect.
- ChrisGreenHeur 6mo ago[dead]
- jaredklewis 6mo agoI’ve read dozens of them and find them unconvincing for the reasons outlined. If you want a more specific critique, link a paper. I personally like and use tests, formal verification, and so on. But the evidence for these methods are weak. edit: To be clear, I am not ragging on the researchers. I think it's just kind of an inherently messy field with pretty much endless variables to control for and not a lot of good quantifiable metrics to rely on.
- thesz 6mo agoYou can measure customer facing defects. Also, lines of code is not completely meaningless metric. What one should measure is lines of code that is not verified by compiler. E.g., in C++ you cannot have unbalanced brackets or use incorrectly typed value, but you still may have off-by-one error. Given all that, you can measure customer facing defect density and compare different tools, whether they are programming languages, IDEs or LLM-supported workflow.
- codeflo 6mo ago> Also, lines of code is not completely meaningless metric. Comparing lines of code can be meaningful, mostly if you can keep a lot of other things constant, like coding style, developer experience, domain, tech stack. There are many style differences between LLM and human generated code, so that I expect 1000 lines of LLM code do a lot less than 1000 lines of human code, even in the exact same codebase.
- jaredklewis 6mo agoIt's not the worst metric ever, but is the study in question observational or controlling the problem? My issue with the observational studies is that basically everything is uncontrolled. Maybe the IDEs are causing less defects. Or maybe some problems are just harder and more defect prone than others. Maybe some teams are better managed or get clearer specifications and so on. Maybe some organizations are better at recording defects and can't be fairly compared with organizations that just report less. The studies don't ever reach the scale where you become confident these things wash out. If they control the problem, most of those issues are eliminated (though not all, for example the experience and education of the participants still needs to be controlled), but now you are left wondering how well the findings transfer from the toy projects in the experiment into real life. But finally, it's still not a perfect metric because not all defects are equal, right? What if some tool/process helped you reduce a large number of mostly cosmetic defects, but increases the occurrence of catastrophic defects? re: LoC, there's some signal here, but it's such a noisy channel, I've never read a study that I thought put it to good use. Happy to have my mind changed if you have a link to one.
- jacquesm 6mo agoThe proper metric is the defect escape rate.
- exidex 6mo agoNow you have to count defects
- jacquesm 6mo agoYou have to do that anyway, and in fact you probably were already doing that. If you do not track this then you are leaving a lot on the table.
- exidex 6mo agoI was more thinking in terms of creating a benchmark which would optimized during training. For regular projects, I agree, you have to count that anyway
- slopinthebag 6mo agoMost developer intuitions are wrong. See: OOP
- deleted 6mo ago[deleted]