11 ms·

The social side of science seen in the research on programming language quality

137 points by hwayne 7y ago

dang 7y ago
Related from a few months ago: https://news.ycombinator.com/item?id=21637411 https://news.ycombinator.com/item?id=21637411
muglug 7y ago
It's amazing the paper got published in the first place – the idea that comparing open source codebases could tell you all that much about the underlying language is sort of bizarre when comparing such different projects to begin with.
- pron 7y ago
  It is not the role of peer-review to do this kind of gatekeeping (although it would be nice if it could filter out bad methodology and data errors). This is the role of exchanges just like the one discussed here.
  - muglug 7y ago
    Journals can reject a paper on editorial grounds – for example, Nature doesn't publish every valid paper submitted to it.
    - pron 7y ago
      True, but it is not the role of peer-review as a whole to prevent papers with whose premise you can argue from being published anywhere. The goal of peer-review is to filter out papers that use bad methodology. Debates over premises is carried out in other papers.
goto11 7y ago
I'm confused. So the original paper was shown to be flawed, but then the rebuttal was also shown to be flawed. But this article argues the flaws in the rebuttal was not as glaring as the flaws in the original paper? But the approach of counting commits on github just seems fundamentally flawed. The first step should be to show that the github dataset can be used to say anything about quality and productivity of things like language choice. Arguing about how to classify commit messages seem pointless without this foundation.
- yaacov 7y ago
  He talks about that in the end. Here’s the relevant section: Is this even a good idea? We’ve just spent several thousand words on methodology to show how the original FSE paper was flawed. But was that all really necessary? Most people point to a much bigger issue: the entire idea of measuring a language’s defect rate via GitHub commits just doesn’t make any sense. How the heck do you go from “these people have more DFCs” to “this language is more buggy”? I saw this complaint raised a lot, and if we just stop there I could have skipped rereading all the papers a dozen times over. Jan Vitek emphasizes this in his talk: if they just said “the premise is nonsense”, nobody would have listened to them. It’s only by showing that the FSE authors messed up methodologically that TOPLAS could establish “sufficient cred” to attack the premise itself. The FSE authors put in a lot of effort into their paper. Responding to that all is “nuh-uh your premise is nonsense” is bad form, even if the premise is nonsense. Who is more trustworthy: the people who spent years getting their paper through peer review, or a random internet commenter who probably stopped at the abstract? It all comes back to science as a social enterprise. We use markers of trust and integrity as heuristics on whether or not to devote time to understanding the claim itself. This is the same reason people are more likely to listen to a claim coming from a respected scientist than from a crackpot.12 Pointing out a potential threat to validity isn’t nearly as powerful as showing how that threat actually undermines the paper, and that’s what FSE had to do to be taken seriously.
  - goto11 7y ago
    Interesting, but as a layman this seems totally backwards. If the underlying premise is flawed, who cares if the methodology is correct? If the original study hadn't made errors like misclassifying some projects or commit messages, the result would still be based on an unfounded premise.
    - SiempreViernes 7y ago
      To put it simply: if the methodology is sound then the premise can't be unfounded, because the premise is the source of the methodology.
mihaaly 7y ago
Programming and programming languages are in a big portion subject of social sciences anyway. Made for the needs of humans used by humans in groups with practices and methods addressing the particularities of humans and human groups. Not ordinary humans but humans indeed. Probably it is more of social science than mathematics or other human independent matters. Social science that is using other disciplines, similar to economics, but at least an interdisciplinary subject.
- sgt101 7y ago
  This is a great take, and one that I wish framed more of the research in programming languages than is currently the case.
- raxxorrax 7y ago
  Where I see social sciences is on the formalization of ontologies for specific applications or laying the ground work for general AI. Perhaps some thought will help us go from higher concepts of knowledge to imperative languages, which most programming languages still are. We often talk about learning systems that gain knowledge and improve themselves. But there is probably more to it than adding another table row. Given, that is probably very hard. That said, I think the article mentions science being "social" in the context of how we weight scientific research depending on the authors standing and expression, not the absolute truth behind it. Which actually is probably quite a shame.
- throwaway17_17 7y ago
  I just can not agree with this concept. Using the standard you propose, architecture for instance is a social science, and I’d argue most non-computer engineering fields would be classified as social sciences as well. Computer Science is by traditional applied mathematics, at least the theoretical courses of study treat it so. But there is a reason Computer Engineering is the predominantly understood title of the overarching software field. Programming languages do not exist in some philosophical vacuum, they are created to be implemented in the real world, on real hardware, and deal with the very real physical capabilities and drawbacks of that real world hardware. While programming, and software design as it is generally practiceD, is not as rigorous as other engineering fields, it is the application of the sciences to create or implement technology. So I just can not agree that programming languages and programming are in any way a social science. Also, just to be clear, there can be social science fields which inquire into aspects of the engineering field of Computer Science, but that doesn’t effect the underlying character of the field. Also, also, I would strenuously argue against the proposition that mathematics is in any way ‘human independent.’ But that argument is not entirely relevant.
WalterBright 7y ago
Number of commits as a metric is a poor idea, because it will immediately be gamed. I constantly advocate with D that people break up larger PRs into smaller self-contained ones, for several reasons. Undermining this with a metric that draws unfounded conclusions isn't helpful.
- goto11 7y ago
  Just make sure you don't use the word "fix" in the commit messages, then the reliability of D will be through the roof!
  - eternalban 7y ago
    That was funny, but in fairness can you imagine how expensive and long drawn a real study into this question would be?
    - zzzcpan 7y ago
      All it takes to avoid that is looking into commit histories of projects in each language and figuring out how they mark bugfixes. If it's suspiciously reliable you would know you made a mistake classifying bugfixes somewhere or your sample is too small or the programming practice in the language is unusual and needs closer look.
rossdavidh 7y ago
Wow, what a bad idea. Do Haskell and PHP get used for the same sort of projects? Or Erlang and Ruby? No, they get used for very different sorts of projects. You would need to only compare similar kinds of projects (e.g. web frameworks, or statistical analysis, or device drivers) to get anything useful out of this data source. But, that's probably not possible, because in most cases there aren't enough projects in very many languages. This also assumes that bugs are all equally likely to get FOUND in all languages. What if bugs are easier to find in language X than in language Y? Then you will get MORE, not fewer, commits that mention fixing them in language X, which is the opposite of what you would want the signal to indicate. I could go on, but there's no need, as even these points are enough to totally invalidate the premise of this research. Getting into the weeds of the analysis of the original study or the rebuttal, is not going to give you anything useful, if the original premise that you are measuring how language compares to bugs is wrong, and it almost certainly is.
- collyw 7y ago
  > What if bugs are easier to find in language X than in language Y? I am half with you on this but thinking to SQL, its probably harder to find bugs in SQL than in an imperative language language, but its also a lot harder to create bugs - its a declarative language so doesn't allow you to create whole classes of bugs that you will see in an imperative language. Personally I try to push logic towards the database for the very reason that it's less buggy in general in my experience.
  - jkaptur 7y ago
    Logic in the database has been less buggy in my experience too... except the time when someone had the idea to push a lot more logic into that layer, at which point it became equally buggy, and much harder to debug.
- Tarq0n 7y ago
  Confounding variables don't invalidate research, they merely inform us on how to weigh the evidence, and shape future research. Studies can have interesting conclusions even in the presence of confounding variables. Sometimes the state of the art in a certain hypothesis space hasn't progressed past of of these yet, but that's still better than gut feeling. In some areas like public policy research you will never completely eliminate all of these factors, but that's no reason not to attempt to create knowledge.
z3t4 7y ago
If all you cared about was fibonacci, or some other "quality", then fine, use that as benchmark. But others might care more about productivity, cost, etc. Then it's also a matter of taste and what is currently fashionable.
hcs 7y ago
Interesting rundown, thanks! I'm a bit puzzled with the "not a dox" tweet, what is going on here? I know that this is probably the least interesting part of the article, but I'm confused and curious.
- hwayne 7y ago
  Ooops, didn't quite explain that properly! The tinyurl links to Berger's Facebook page.
6510 7y ago
I'm surprised someone takes my github saved games seriously. It certainly wasn't me. I think this science project would be more fun if assumptions were discarded. Just gather the data, do it without a hypothesis. Something interesting might come up worthy of one. I liked the bit of crankology in the article. If you sound like a crank no one will take you seriously? I had one day pondered what if the cranks are right? How would we know? It follows that we can't know sounding crank equals being wrong and that our idea of "sounding serious" is based on a flawed data set, if it isn't noise entirely. The topic of the study here certainly doesn't inspire my confidence.
- nxpnsv 7y ago
  Such post hoc hypothesis trawling essentially is p hacking. Try a large enough set of explanations and your data will show something significant. Might be hard to replicate.
  - hwayne 7y ago
    One of my favorite examples is this tour de force on "proving" that chocolate helps with weight loss: https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...
    - nxpnsv 7y ago
      or the forever classic: https://www.xkcd.com/882/ https://www.xkcd.com/882/
  - 6510 7y ago
    Replication would indeed be necessary. Objectivity is the stuff to make the science!
pron 7y ago
This is a truly terrific post, but it says: > They’re underselling the effect here. While the dominant factor is number of commits, language still matters a lot, too. If you choose C++ over TypeScript, you’re going to have twice as many DFCs! That doesn’t necessarily mean twice as many bugs, but it is suggestive. Further, while they say the effect is “overwhelmingly dominated by […] project size, team size, and commit size”, that doesn’t actually bear out. Only the number of commits is a bigger factor in language choice. This is inaccurate. "Effect" is not the expected difference (i.e. difference between the means) but, roughly, the expected difference divided by variance. Just looking at the expected difference is insufficient to determine if the effect is large (and undersold) or small. Expected difference is not an interesting statistical property, just as the mean isn't (by itself). If you're looking at ten Clojure projects and ten C++ projects, and all ten Clojure projects have 10 DFCs, while eight C++ projects have 8 DFCs and two have 500, then the expected difference is huge, but the effect is small. Indeed, when looking at the variance, Clojure, the "best"-performing language in this dataset, and C++, the "worst"-performing language in this dataset, the two were not very distinguishable, supporting everyone's finding of a very small effect.
- afandian 7y ago
  DFC = "Defect-Fixing Commits"
- closed 7y ago
  > Expected difference is not an interesting statistical property, just as the mean isn't (by itself). Difference in means can meet the definition of effect size, and are listed as an example right away in the wikipedia article on it [0]! The key is that it capture the phenomena of interest (and generally, the magnitude shouldn't be a function of number of observations). In psychology, where scales often have arbitrarily defined ranges (eg IQ scales), means usually are not useful effect sizes. The case you list is a separate issue from effect size (probably the violation of distributional assumptions in some implicit or explicit model). Even using difference in means / variance (say, cohen's d), two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious (a problem not solved by dividing them). https://en.wikipedia.org/wiki/Effect_size https://en.wikipedia.org/wiki/Effect_size
d0mine 7y ago
> the entire idea of measuring a language’s defect rate via GitHub commits just doesn’t make any sense. How the heck do you go from “these people have more DFCs” to “this language is more buggy”? If Github is used by ordinary programmers (what are the reasons to assume otherwise?) i.e., if the sample is unbiased then what is the issue with going from "these [random] people" to "this language"?
ken 7y ago
At the very end of this grand tour: > Science has its problems, but it’s still the best we got. I would agree if you're doing particle physics and looking for p < 0.0000003 then absolutely. Science is a great tool for the kinds of investigations of the natural world which can be understood by science. If you're doing social science by scraping some data from the internet and looking for p < 0.05, that's a very different story. I'm not actually sure what aspect of this qualifies as "science", besides using p-values to analyze the data. Even if all the categorization and sampling and analyses were perfect, it's using commit messages and known bugs with known fixes, all of which are essentially self-reported.
- carapace 7y ago
  From my POV the real problem is that there are phenomenon that apparently cannot be studied scientifically. Like Reiki. Now I'm not a scientist but I am a huge proponent of science. Carl Sagan is among my personal heroes, okay? I am in favor of science over superstition and irrationality. One day I encountered a phenomenon that is as real and physically dramatic as, say, the electric field of a Van de Graff generator. Being a fan of science I went looking for the math/science/engineering for this phenomenon. Long story short, after a few years of research, I discovered that: 1) this phenomenon has been known since antiquity in a pre-scientific way; 2) several researchers had independently (re-)discovered it multiple times going back about 2.5 centuries; 3) each time it had achieved a certain notoriety but then been debunked and subsequently ignored or forgotten. The first was Mesmer & "animal magnetism", but there are several others including Baron Von Reichenbach & the "Odic force", and Wilhelm Reich & "Orgone". This is repeating today with several kinds of "energy healing" including Reiki (which is distinguished by being Japanese and unabashedly theistic (Rei means Divine)) where practice is widespread, but lacking scientific validation. As I say, I've felt Reiki, so I know it's real, but so far I've personally never been able to interest any scientists in studying it, despite offering to demonstrate it. And the studies that have been done haven't helped. The Wikipedia article for Reiki just straight up calls it pseudoscience.
  - Analemma_ 7y ago
    To the extent that Reiki or any other alternative medicine technique actually does anything, it's easily explainable by the placebo effect. You know what we call alternative medicine that works? Medicine.
mcguire 7y ago
Two comments: "FSE was a preprint: a paper that hadn’t necessarily gone through peer-review, editing, and publication in a journal. Academics share preprints because 1) the process of getting a paper journal-ready can take years, so this gets it out faster, and 2) academics can share their research with people who aren’t able to afford the ludicrous journal fees." To my knowledge, all ACM conferences including the "ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering" are peer-reviewed. In CS, in my experience, peer-review for conferences is more thorough than journal review. Conferences are more important. The idea that they are preprints is something from linguistics or some other field. "How did FSE take the rebuttal rebuttal? Not well. "So that’s technically not a dox, because he didn’t publish Berger’s private information, but still. That’s a really asshole thing to do!" I'm not sure I would call linking to someone's Facebook page from Twitter to be "doxxing" in any sense. Calling a fellow researcher a "donkey" is a bit of an asshole thing, too. (I have no link to said comment, but I know Emery Berger. He's very positive he's right, the smartest in the room, and very abrasive.) Tl;dr: Software engineering research is a garbage fire. But we knew that.
0xdeadbeefbabe 7y ago
> But it still seems intuitive that language should matter. Am I the only one that thinks this is a naive assumption? They aren't really languages like English is a language--so much for intuition. Don't definitions matter to science? Does science carry a selfie stick? I'm suddenly keen on the antithesis of science or a less social science.
fulafel 7y ago
There's talk and comments about the measure ("DFCs"). What would be better? And is it possible to use this one to eke out a signal even though the measurement has some known problems?
luord 7y ago
It's astounding how discussing programming languages can devolve into flamewars, even when the discussion starts as actual academic research and the participants are dedicated scientific researchers.
jdm2212 7y ago
Every project grows until it becomes too complex and hence unmanageably buggy. Some languages/static analyzers/linters/architectures/frameworks let a project become bigger and more complex before becoming unmanageably buggy. So if you look at two projects, and both have a bunch of bugs, that doesn't tell you much. You have to know bugs-per-unit-complexity, which is pretty damn close to unmeasurable. It's telling, though, that the companies managing the world's most complex systems run them in C++, Java, and TypeScript. And more recently, Go and Rust. With much finance research, if the results were real, the researchers would be making billions on Wall Street rather than be academics. So, too, with software research. If the results were real, the researchers could become gazillionaires advising FAANG instead of being academics.
- westoncb 7y ago
  This is something I come back to repeatedly when attempting to evaluate claims about certain technologies (e.g. programming languages) being extravagantly effective: people's sentiments about how great some technology is often seems out of proportion to what their gains were through using it. Of course that's difficult to evaluate. But if their gains were in proportion, I think you'd see these crazy spikes like what you're saying about these researchers becoming gazillionaires. And it just doesn't seem to happen. Which is not to say there's no difference—but 1) the improvements seem to be by smaller factors than what engineers believe 2) they don't seem to generalize as much as is often claimed: afaict, a significant aspect of the gains has to do with reducing impedance mismatch between problem domain and language (but advocates will typically adopt a stance that the language is just intrinsically superior). I would be very interested to hear counter-examples if people have them (I know the PG/Viaweb/Lisp story).
  - jdm2212 7y ago
    I think the folks really fascinated with which language is intrinsically superior are missing out on the fact that the language itself is <10% of the software development process. There's tooling, architecture, libraries, monitoring, etc. The language's feature set or verbosity is rarely the bottleneck. I'm skeptical that the PG/Viaweb/Lisp story generalizes at all to non-geniuses. Like, sure, if you gave Paul Graham raw assembly he could probably do some pretty impressive stuff even so. Doesn't mean anyone else can. He's a genius. So is Robert Morris.

137 points by hwayne 7y ago

dang 7y ago

Related from a few months ago: https://news.ycombinator.com/item?id=21637411 https://news.ycombinator.com/item?id=21637411

muglug 7y ago

It's amazing the paper got published in the first place – the idea that comparing open source codebases could tell you all that much about the underlying language is sort of bizarre when comparing such different projects to begin with.

pron 7y ago
It is not the role of peer-review to do this kind of gatekeeping (although it would be nice if it could filter out bad methodology and data errors). This is the role of exchanges just like the one discussed here.
- muglug 7y ago
  Journals can reject a paper on editorial grounds – for example, Nature doesn't publish every valid paper submitted to it.
  - pron 7y ago
    True, but it is not the role of peer-review as a whole to prevent papers with whose premise you can argue from being published anywhere. The goal of peer-review is to filter out papers that use bad methodology. Debates over premises is carried out in other papers.

goto11 7y ago

I'm confused. So the original paper was shown to be flawed, but then the rebuttal was also shown to be flawed. But this article argues the flaws in the rebuttal was not as glaring as the flaws in the original paper? But the approach of counting commits on github just seems fundamentally flawed. The first step should be to show that the github dataset can be used to say anything about quality and productivity of things like language choice. Arguing about how to classify commit messages seem pointless without this foundation.

yaacov 7y ago
He talks about that in the end. Here’s the relevant section: Is this even a good idea? We’ve just spent several thousand words on methodology to show how the original FSE paper was flawed. But was that all really necessary? Most people point to a much bigger issue: the entire idea of measuring a language’s defect rate via GitHub commits just doesn’t make any sense. How the heck do you go from “these people have more DFCs” to “this language is more buggy”? I saw this complaint raised a lot, and if we just stop there I could have skipped rereading all the papers a dozen times over. Jan Vitek emphasizes this in his talk: if they just said “the premise is nonsense”, nobody would have listened to them. It’s only by showing that the FSE authors messed up methodologically that TOPLAS could establish “sufficient cred” to attack the premise itself. The FSE authors put in a lot of effort into their paper. Responding to that all is “nuh-uh your premise is nonsense” is bad form, even if the premise is nonsense. Who is more trustworthy: the people who spent years getting their paper through peer review, or a random internet commenter who probably stopped at the abstract? It all comes back to science as a social enterprise. We use markers of trust and integrity as heuristics on whether or not to devote time to understanding the claim itself. This is the same reason people are more likely to listen to a claim coming from a respected scientist than from a crackpot.12 Pointing out a potential threat to validity isn’t nearly as powerful as showing how that threat actually undermines the paper, and that’s what FSE had to do to be taken seriously.
- goto11 7y ago
  Interesting, but as a layman this seems totally backwards. If the underlying premise is flawed, who cares if the methodology is correct? If the original study hadn't made errors like misclassifying some projects or commit messages, the result would still be based on an unfounded premise.
  - SiempreViernes 7y ago
    To put it simply: if the methodology is sound then the premise can't be unfounded, because the premise is the source of the methodology.

mihaaly 7y ago

Programming and programming languages are in a big portion subject of social sciences anyway. Made for the needs of humans used by humans in groups with practices and methods addressing the particularities of humans and human groups. Not ordinary humans but humans indeed. Probably it is more of social science than mathematics or other human independent matters. Social science that is using other disciplines, similar to economics, but at least an interdisciplinary subject.

sgt101 7y ago
This is a great take, and one that I wish framed more of the research in programming languages than is currently the case.
raxxorrax 7y ago
Where I see social sciences is on the formalization of ontologies for specific applications or laying the ground work for general AI. Perhaps some thought will help us go from higher concepts of knowledge to imperative languages, which most programming languages still are. We often talk about learning systems that gain knowledge and improve themselves. But there is probably more to it than adding another table row. Given, that is probably very hard. That said, I think the article mentions science being "social" in the context of how we weight scientific research depending on the authors standing and expression, not the absolute truth behind it. Which actually is probably quite a shame.
throwaway17_17 7y ago
I just can not agree with this concept. Using the standard you propose, architecture for instance is a social science, and I’d argue most non-computer engineering fields would be classified as social sciences as well. Computer Science is by traditional applied mathematics, at least the theoretical courses of study treat it so. But there is a reason Computer Engineering is the predominantly understood title of the overarching software field. Programming languages do not exist in some philosophical vacuum, they are created to be implemented in the real world, on real hardware, and deal with the very real physical capabilities and drawbacks of that real world hardware. While programming, and software design as it is generally practiceD, is not as rigorous as other engineering fields, it is the application of the sciences to create or implement technology. So I just can not agree that programming languages and programming are in any way a social science. Also, just to be clear, there can be social science fields which inquire into aspects of the engineering field of Computer Science, but that doesn’t effect the underlying character of the field. Also, also, I would strenuously argue against the proposition that mathematics is in any way ‘human independent.’ But that argument is not entirely relevant.

WalterBright 7y ago

Number of commits as a metric is a poor idea, because it will immediately be gamed. I constantly advocate with D that people break up larger PRs into smaller self-contained ones, for several reasons. Undermining this with a metric that draws unfounded conclusions isn't helpful.

goto11 7y ago
Just make sure you don't use the word "fix" in the commit messages, then the reliability of D will be through the roof!
- eternalban 7y ago
  That was funny, but in fairness can you imagine how expensive and long drawn a real study into this question would be?
  - zzzcpan 7y ago
    All it takes to avoid that is looking into commit histories of projects in each language and figuring out how they mark bugfixes. If it's suspiciously reliable you would know you made a mistake classifying bugfixes somewhere or your sample is too small or the programming practice in the language is unusual and needs closer look.

rossdavidh 7y ago

Wow, what a bad idea. Do Haskell and PHP get used for the same sort of projects? Or Erlang and Ruby? No, they get used for very different sorts of projects. You would need to only compare similar kinds of projects (e.g. web frameworks, or statistical analysis, or device drivers) to get anything useful out of this data source. But, that's probably not possible, because in most cases there aren't enough projects in very many languages. This also assumes that bugs are all equally likely to get FOUND in all languages. What if bugs are easier to find in language X than in language Y? Then you will get MORE, not fewer, commits that mention fixing them in language X, which is the opposite of what you would want the signal to indicate. I could go on, but there's no need, as even these points are enough to totally invalidate the premise of this research. Getting into the weeds of the analysis of the original study or the rebuttal, is not going to give you anything useful, if the original premise that you are measuring how language compares to bugs is wrong, and it almost certainly is.

collyw 7y ago
> What if bugs are easier to find in language X than in language Y? I am half with you on this but thinking to SQL, its probably harder to find bugs in SQL than in an imperative language language, but its also a lot harder to create bugs - its a declarative language so doesn't allow you to create whole classes of bugs that you will see in an imperative language. Personally I try to push logic towards the database for the very reason that it's less buggy in general in my experience.
- jkaptur 7y ago
  Logic in the database has been less buggy in my experience too... except the time when someone had the idea to push a lot more logic into that layer, at which point it became equally buggy, and much harder to debug.
Tarq0n 7y ago
Confounding variables don't invalidate research, they merely inform us on how to weigh the evidence, and shape future research. Studies can have interesting conclusions even in the presence of confounding variables. Sometimes the state of the art in a certain hypothesis space hasn't progressed past of of these yet, but that's still better than gut feeling. In some areas like public policy research you will never completely eliminate all of these factors, but that's no reason not to attempt to create knowledge.

z3t4 7y ago

If all you cared about was fibonacci, or some other "quality", then fine, use that as benchmark. But others might care more about productivity, cost, etc. Then it's also a matter of taste and what is currently fashionable.

hcs 7y ago

Interesting rundown, thanks! I'm a bit puzzled with the "not a dox" tweet, what is going on here? I know that this is probably the least interesting part of the article, but I'm confused and curious.

hwayne 7y ago
Ooops, didn't quite explain that properly! The tinyurl links to Berger's Facebook page.

6510 7y ago

I'm surprised someone takes my github saved games seriously. It certainly wasn't me. I think this science project would be more fun if assumptions were discarded. Just gather the data, do it without a hypothesis. Something interesting might come up worthy of one. I liked the bit of crankology in the article. If you sound like a crank no one will take you seriously? I had one day pondered what if the cranks are right? How would we know? It follows that we can't know sounding crank equals being wrong and that our idea of "sounding serious" is based on a flawed data set, if it isn't noise entirely. The topic of the study here certainly doesn't inspire my confidence.

nxpnsv 7y ago
Such post hoc hypothesis trawling essentially is p hacking. Try a large enough set of explanations and your data will show something significant. Might be hard to replicate.
- hwayne 7y ago
  One of my favorite examples is this tour de force on "proving" that chocolate helps with weight loss: https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...
  - nxpnsv 7y ago
    or the forever classic: https://www.xkcd.com/882/ https://www.xkcd.com/882/
- 6510 7y ago
  Replication would indeed be necessary. Objectivity is the stuff to make the science!

pron 7y ago

This is a truly terrific post, but it says: > They’re underselling the effect here. While the dominant factor is number of commits, language still matters a lot, too. If you choose C++ over TypeScript, you’re going to have twice as many DFCs! That doesn’t necessarily mean twice as many bugs, but it is suggestive. Further, while they say the effect is “overwhelmingly dominated by […] project size, team size, and commit size”, that doesn’t actually bear out. Only the number of commits is a bigger factor in language choice. This is inaccurate. "Effect" is not the expected difference (i.e. difference between the means) but, roughly, the expected difference divided by variance. Just looking at the expected difference is insufficient to determine if the effect is large (and undersold) or small. Expected difference is not an interesting statistical property, just as the mean isn't (by itself). If you're looking at ten Clojure projects and ten C++ projects, and all ten Clojure projects have 10 DFCs, while eight C++ projects have 8 DFCs and two have 500, then the expected difference is huge, but the effect is small. Indeed, when looking at the variance, Clojure, the "best"-performing language in this dataset, and C++, the "worst"-performing language in this dataset, the two were not very distinguishable, supporting everyone's finding of a very small effect.

afandian 7y ago
DFC = "Defect-Fixing Commits"
closed 7y ago
> Expected difference is not an interesting statistical property, just as the mean isn't (by itself). Difference in means can meet the definition of effect size, and are listed as an example right away in the wikipedia article on it [0]! The key is that it capture the phenomena of interest (and generally, the magnitude shouldn't be a function of number of observations). In psychology, where scales often have arbitrarily defined ranges (eg IQ scales), means usually are not useful effect sizes. The case you list is a separate issue from effect size (probably the violation of distributional assumptions in some implicit or explicit model). Even using difference in means / variance (say, cohen's d), two observations that extreme would make the interpretation of both mean and variance calculations pretty dubious (a problem not solved by dividing them). https://en.wikipedia.org/wiki/Effect_size https://en.wikipedia.org/wiki/Effect_size

d0mine 7y ago

> the entire idea of measuring a language’s defect rate via GitHub commits just doesn’t make any sense. How the heck do you go from “these people have more DFCs” to “this language is more buggy”? If Github is used by ordinary programmers (what are the reasons to assume otherwise?) i.e., if the sample is unbiased then what is the issue with going from "these [random] people" to "this language"?

ken 7y ago

At the very end of this grand tour: > Science has its problems, but it’s still the best we got. I would agree if you're doing particle physics and looking for p < 0.0000003 then absolutely. Science is a great tool for the kinds of investigations of the natural world which can be understood by science. If you're doing social science by scraping some data from the internet and looking for p < 0.05, that's a very different story. I'm not actually sure what aspect of this qualifies as "science", besides using p-values to analyze the data. Even if all the categorization and sampling and analyses were perfect, it's using commit messages and known bugs with known fixes, all of which are essentially self-reported.

carapace 7y ago
From my POV the real problem is that there are phenomenon that apparently cannot be studied scientifically. Like Reiki. Now I'm not a scientist but I am a huge proponent of science. Carl Sagan is among my personal heroes, okay? I am in favor of science over superstition and irrationality. One day I encountered a phenomenon that is as real and physically dramatic as, say, the electric field of a Van de Graff generator. Being a fan of science I went looking for the math/science/engineering for this phenomenon. Long story short, after a few years of research, I discovered that: 1) this phenomenon has been known since antiquity in a pre-scientific way; 2) several researchers had independently (re-)discovered it multiple times going back about 2.5 centuries; 3) each time it had achieved a certain notoriety but then been debunked and subsequently ignored or forgotten. The first was Mesmer & "animal magnetism", but there are several others including Baron Von Reichenbach & the "Odic force", and Wilhelm Reich & "Orgone". This is repeating today with several kinds of "energy healing" including Reiki (which is distinguished by being Japanese and unabashedly theistic (Rei means Divine)) where practice is widespread, but lacking scientific validation. As I say, I've felt Reiki, so I know it's real, but so far I've personally never been able to interest any scientists in studying it, despite offering to demonstrate it. And the studies that have been done haven't helped. The Wikipedia article for Reiki just straight up calls it pseudoscience.
- Analemma_ 7y ago
  To the extent that Reiki or any other alternative medicine technique actually does anything, it's easily explainable by the placebo effect. You know what we call alternative medicine that works? Medicine.

mcguire 7y ago

Two comments: "FSE was a preprint: a paper that hadn’t necessarily gone through peer-review, editing, and publication in a journal. Academics share preprints because 1) the process of getting a paper journal-ready can take years, so this gets it out faster, and 2) academics can share their research with people who aren’t able to afford the ludicrous journal fees." To my knowledge, all ACM conferences including the "ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering" are peer-reviewed. In CS, in my experience, peer-review for conferences is more thorough than journal review. Conferences are more important. The idea that they are preprints is something from linguistics or some other field. "How did FSE take the rebuttal rebuttal? Not well. "So that’s technically not a dox, because he didn’t publish Berger’s private information, but still. That’s a really asshole thing to do!" I'm not sure I would call linking to someone's Facebook page from Twitter to be "doxxing" in any sense. Calling a fellow researcher a "donkey" is a bit of an asshole thing, too. (I have no link to said comment, but I know Emery Berger. He's very positive he's right, the smartest in the room, and very abrasive.) Tl;dr: Software engineering research is a garbage fire. But we knew that.

0xdeadbeefbabe 7y ago

> But it still seems intuitive that language should matter. Am I the only one that thinks this is a naive assumption? They aren't really languages like English is a language--so much for intuition. Don't definitions matter to science? Does science carry a selfie stick? I'm suddenly keen on the antithesis of science or a less social science.

fulafel 7y ago

There's talk and comments about the measure ("DFCs"). What would be better? And is it possible to use this one to eke out a signal even though the measurement has some known problems?

luord 7y ago

It's astounding how discussing programming languages can devolve into flamewars, even when the discussion starts as actual academic research and the participants are dedicated scientific researchers.

jdm2212 7y ago

Every project grows until it becomes too complex and hence unmanageably buggy. Some languages/static analyzers/linters/architectures/frameworks let a project become bigger and more complex before becoming unmanageably buggy. So if you look at two projects, and both have a bunch of bugs, that doesn't tell you much. You have to know bugs-per-unit-complexity, which is pretty damn close to unmeasurable. It's telling, though, that the companies managing the world's most complex systems run them in C++, Java, and TypeScript. And more recently, Go and Rust. With much finance research, if the results were real, the researchers would be making billions on Wall Street rather than be academics. So, too, with software research. If the results were real, the researchers could become gazillionaires advising FAANG instead of being academics.

westoncb 7y ago
This is something I come back to repeatedly when attempting to evaluate claims about certain technologies (e.g. programming languages) being extravagantly effective: people's sentiments about how great some technology is often seems out of proportion to what their gains were through using it. Of course that's difficult to evaluate. But if their gains were in proportion, I think you'd see these crazy spikes like what you're saying about these researchers becoming gazillionaires. And it just doesn't seem to happen. Which is not to say there's no difference—but 1) the improvements seem to be by smaller factors than what engineers believe 2) they don't seem to generalize as much as is often claimed: afaict, a significant aspect of the gains has to do with reducing impedance mismatch between problem domain and language (but advocates will typically adopt a stance that the language is just intrinsically superior). I would be very interested to hear counter-examples if people have them (I know the PG/Viaweb/Lisp story).
- jdm2212 7y ago
  I think the folks really fascinated with which language is intrinsically superior are missing out on the fact that the language itself is <10% of the software development process. There's tooling, architecture, libraries, monitoring, etc. The language's feature set or verbosity is rarely the bottleneck. I'm skeptical that the PG/Viaweb/Lisp story generalizes at all to non-geniuses. Like, sure, if you gave Paul Graham raw assembly he could probably do some pretty impressive stuff even so. Doesn't mean anyone else can. He's a genius. So is Robert Morris.