4 ms·

He talks about that in the end. Here’s the relevant section: Is this even a good idea? We’ve just spent several thousand words on methodology to show how the

by yaacov 7y ago

He talks about that in the end. Here’s the relevant section: Is this even a good idea? We’ve just spent several thousand words on methodology to show how the original FSE paper was flawed. But was that all really necessary? Most people point to a much bigger issue: the entire idea of measuring a language’s defect rate via GitHub commits just doesn’t make any sense. How the heck do you go from “these people have more DFCs” to “this language is more buggy”? I saw this complaint raised a lot, and if we just stop there I could have skipped rereading all the papers a dozen times over. Jan Vitek emphasizes this in his talk: if they just said “the premise is nonsense”, nobody would have listened to them. It’s only by showing that the FSE authors messed up methodologically that TOPLAS could establish “sufficient cred” to attack the premise itself. The FSE authors put in a lot of effort into their paper. Responding to that all is “nuh-uh your premise is nonsense” is bad form, even if the premise is nonsense. Who is more trustworthy: the people who spent years getting their paper through peer review, or a random internet commenter who probably stopped at the abstract? It all comes back to science as a social enterprise. We use markers of trust and integrity as heuristics on whether or not to devote time to understanding the claim itself. This is the same reason people are more likely to listen to a claim coming from a respected scientist than from a crackpot.12 Pointing out a potential threat to validity isn’t nearly as powerful as showing how that threat actually undermines the paper, and that’s what FSE had to do to be taken seriously.

goto11 7y ago
Interesting, but as a layman this seems totally backwards. If the underlying premise is flawed, who cares if the methodology is correct? If the original study hadn't made errors like misclassifying some projects or commit messages, the result would still be based on an unfounded premise.
- SiempreViernes 7y ago
  To put it simply: if the methodology is sound then the premise can't be unfounded, because the premise is the source of the methodology.
  - smt88 7y ago
    I think you may misunderstand what others mean when they say "premise". For this paper, the premise is that GitHub commit history indicates language quality. The methodology is then the implementation of the study: how do we classify each commit? How do we identify the base language in a repository? What would lead us to discard a repository as a data source? The methodology may be perfect, but you may still end up with nonsense because GitHub is not a comprehensive or representative source of data for studying programming.
    - SiempreViernes 7y ago
      No, pretty obviously the claim is that language properties will affect the type of commits being made, this is the premise they start from. The CACM paper then did a bad job of testing if this is true by not being vary careful with the labelling and cleaning of their data, a point on which they were then attacked. Please notice that they were critiqued on their methodology, not for thinking that commits are casually influenced by the design of the language they modify.
      - mumblemumble 7y ago
        Their premise is even more deeply flawed than that. AFAICT, they didn't consider selection bias seriously enough. I'm not sure if it's politically correct to say this, but, e.g., it's entirely possible that particularly meticulous coders are both more likely to select a strong, statically typed functional language and generate bugs at a lower rate. And that the effect turns out to be almost entirely due to the person and not the programming language. There are also some fundamentally unanswerable questions. Like, would dynamic typing come off looking better if the selection of dynamic languages weren't dominated by warty messes like php and ecmascript?
        zzzcpan 7y ago
        > Like, would dynamic typing come off looking better if the selection of dynamic languages weren't dominated by warty messes like php and ecmascript? It already looks better if you consider that dynamic typing is more expressive and such languages take much less code to do the same thing and less time. This is not something that can be researched via commit messages, but it is rather obviously true, so the claim from the paper that "static typing is better" is also obviously false and is way out of paper's reach to even claim. But then again this is CS and such claims that go way beyond data and author's expertise are super common in CS papers. Personally I stopped taking CS papers seriously a long time ago.
- syrak 7y ago
  > If the underlying premise is flawed, who cares if the methodology is correct? How do we know the premise is flawed? > The first step should be to show that the github dataset can be used to say anything about quality and productivity of things like language choice. How do you concretely do that? The initial paper, even if flawed, did a lot of work to try to control for confounding factors. To do what you said, you would have to put as much effort, if not more, to convince people that the results can be falsified to claim anything, with the same level of control in place. It's not even clear that counting commits is entirely meaningless. Getting some signal out of it does seem tremendously difficult, but not impossible. So questioning the methodology is really the only way to rebut the experiment.
- mumblemumble 7y ago
  The factor you're missing is the potential for a criticism to advance the discussion. Everyone believes that they occupy the intellectual high ground, and that it's everyone else who's guilty of confirmation bias and wooly thinking. Therefore, anybody else's accusations of confirmation bias and wooly thinking are merely a product of their own confirmation bias and wooly thinking, and can safely be ignored. Attacking the methodology mitigates this effect. By demonstrating that they spent serious time digging into it and trying to replicate it, the authors give a strong signal that they're not just exercising in armchair skepticism. This, in turn, lends rhetorical weight to their attack on the premise. Without it, it would be as easy to ignore, and as quickly forgotten, as your average comment on Hacker News.
- scott_s 7y ago
  > If the underlying premise is flawed, who cares if the methodology is correct? The post you're replying to includes an explanation addressing that point.

by yaacov 7y ago

goto11 7y ago

Interesting, but as a layman this seems totally backwards. If the underlying premise is flawed, who cares if the methodology is correct? If the original study hadn't made errors like misclassifying some projects or commit messages, the result would still be based on an unfounded premise.

SiempreViernes 7y ago
To put it simply: if the methodology is sound then the premise can't be unfounded, because the premise is the source of the methodology.
- smt88 7y ago
  I think you may misunderstand what others mean when they say "premise". For this paper, the premise is that GitHub commit history indicates language quality. The methodology is then the implementation of the study: how do we classify each commit? How do we identify the base language in a repository? What would lead us to discard a repository as a data source? The methodology may be perfect, but you may still end up with nonsense because GitHub is not a comprehensive or representative source of data for studying programming.
  - SiempreViernes 7y ago
    No, pretty obviously the claim is that language properties will affect the type of commits being made, this is the premise they start from. The CACM paper then did a bad job of testing if this is true by not being vary careful with the labelling and cleaning of their data, a point on which they were then attacked. Please notice that they were critiqued on their methodology, not for thinking that commits are casually influenced by the design of the language they modify.
    - mumblemumble 7y ago
      Their premise is even more deeply flawed than that. AFAICT, they didn't consider selection bias seriously enough. I'm not sure if it's politically correct to say this, but, e.g., it's entirely possible that particularly meticulous coders are both more likely to select a strong, statically typed functional language and generate bugs at a lower rate. And that the effect turns out to be almost entirely due to the person and not the programming language. There are also some fundamentally unanswerable questions. Like, would dynamic typing come off looking better if the selection of dynamic languages weren't dominated by warty messes like php and ecmascript?
      - zzzcpan 7y ago
        > Like, would dynamic typing come off looking better if the selection of dynamic languages weren't dominated by warty messes like php and ecmascript? It already looks better if you consider that dynamic typing is more expressive and such languages take much less code to do the same thing and less time. This is not something that can be researched via commit messages, but it is rather obviously true, so the claim from the paper that "static typing is better" is also obviously false and is way out of paper's reach to even claim. But then again this is CS and such claims that go way beyond data and author's expertise are super common in CS papers. Personally I stopped taking CS papers seriously a long time ago.
syrak 7y ago
> If the underlying premise is flawed, who cares if the methodology is correct? How do we know the premise is flawed? > The first step should be to show that the github dataset can be used to say anything about quality and productivity of things like language choice. How do you concretely do that? The initial paper, even if flawed, did a lot of work to try to control for confounding factors. To do what you said, you would have to put as much effort, if not more, to convince people that the results can be falsified to claim anything, with the same level of control in place. It's not even clear that counting commits is entirely meaningless. Getting some signal out of it does seem tremendously difficult, but not impossible. So questioning the methodology is really the only way to rebut the experiment.
mumblemumble 7y ago
The factor you're missing is the potential for a criticism to advance the discussion. Everyone believes that they occupy the intellectual high ground, and that it's everyone else who's guilty of confirmation bias and wooly thinking. Therefore, anybody else's accusations of confirmation bias and wooly thinking are merely a product of their own confirmation bias and wooly thinking, and can safely be ignored. Attacking the methodology mitigates this effect. By demonstrating that they spent serious time digging into it and trying to replicate it, the authors give a strong signal that they're not just exercising in armchair skepticism. This, in turn, lends rhetorical weight to their attack on the premise. Without it, it would be as easy to ignore, and as quickly forgotten, as your average comment on Hacker News.
scott_s 7y ago
> If the underlying premise is flawed, who cares if the methodology is correct? The post you're replying to includes an explanation addressing that point.