5 ms·

Wow, what a bad idea. Do Haskell and PHP get used for the same sort of projects? Or Erlang and Ruby? No, they get used for very different sorts of projects.

Wow, what a bad idea. Do Haskell and PHP get used for the same sort of projects? Or Erlang and Ruby? No, they get used for very different sorts of projects. You would need to only compare similar kinds of projects (e.g. web frameworks, or statistical analysis, or device drivers) to get anything useful out of this data source. But, that's probably not possible, because in most cases there aren't enough projects in very many languages. This also assumes that bugs are all equally likely to get FOUND in all languages. What if bugs are easier to find in language X than in language Y? Then you will get MORE, not fewer, commits that mention fixing them in language X, which is the opposite of what you would want the signal to indicate. I could go on, but there's no need, as even these points are enough to totally invalidate the premise of this research. Getting into the weeds of the analysis of the original study or the rebuttal, is not going to give you anything useful, if the original premise that you are measuring how language compares to bugs is wrong, and it almost certainly is.

collyw 7y ago
> What if bugs are easier to find in language X than in language Y? I am half with you on this but thinking to SQL, its probably harder to find bugs in SQL than in an imperative language language, but its also a lot harder to create bugs - its a declarative language so doesn't allow you to create whole classes of bugs that you will see in an imperative language. Personally I try to push logic towards the database for the very reason that it's less buggy in general in my experience.
- jkaptur 7y ago
  Logic in the database has been less buggy in my experience too... except the time when someone had the idea to push a lot more logic into that layer, at which point it became equally buggy, and much harder to debug.
Tarq0n 7y ago
Confounding variables don't invalidate research, they merely inform us on how to weigh the evidence, and shape future research. Studies can have interesting conclusions even in the presence of confounding variables. Sometimes the state of the art in a certain hypothesis space hasn't progressed past of of these yet, but that's still better than gut feeling. In some areas like public policy research you will never completely eliminate all of these factors, but that's no reason not to attempt to create knowledge.
- cdheuer 7y ago
  I don't think GitHub mining is state of the art, and there are opportunities, as the article says, to separate the signal from the noise. I agree in the general sense, but there should be some lines drawn at which point we can say research is invalid so people don't run away with the results.
  - SiempreViernes 7y ago
    Yeah, the original study is clearly so methodological unsound that it's conclusions shouldn't be repeated. But fankly most of the posts here are people almost calling the original authors heretical for even trying to data mine commit messages.
- rossdavidh 7y ago
  Sure, you will never completely eliminate all of these factors. But this is far, far short of "not completely eliminated". This is, "the confounding variables are almost certainly much larger effects than what you are trying to measure". The difference between python, ruby, and php in make a web framework, is much much smaller than the difference between web frameworks, statistical analysis packages, machine learning, and devops scripts. Python is used for the last three, ruby and php much less so. The difference between python and ruby and php is therefore going to be dominated by the kinds of tasks they are used on, and the difference between the languages is small compared to that. Smokers are routinely excluded from analysis of cancer rates, for a similar reason; the effect is so large that if (for example) you're looking at diet, and the difference in diet is correlated with a difference in smoking rates, any findings are invalid.

collyw 7y ago

> What if bugs are easier to find in language X than in language Y? I am half with you on this but thinking to SQL, its probably harder to find bugs in SQL than in an imperative language language, but its also a lot harder to create bugs - its a declarative language so doesn't allow you to create whole classes of bugs that you will see in an imperative language. Personally I try to push logic towards the database for the very reason that it's less buggy in general in my experience.

jkaptur 7y ago
Logic in the database has been less buggy in my experience too... except the time when someone had the idea to push a lot more logic into that layer, at which point it became equally buggy, and much harder to debug.

Tarq0n 7y ago

Confounding variables don't invalidate research, they merely inform us on how to weigh the evidence, and shape future research. Studies can have interesting conclusions even in the presence of confounding variables. Sometimes the state of the art in a certain hypothesis space hasn't progressed past of of these yet, but that's still better than gut feeling. In some areas like public policy research you will never completely eliminate all of these factors, but that's no reason not to attempt to create knowledge.

cdheuer 7y ago
I don't think GitHub mining is state of the art, and there are opportunities, as the article says, to separate the signal from the noise. I agree in the general sense, but there should be some lines drawn at which point we can say research is invalid so people don't run away with the results.
- SiempreViernes 7y ago
  Yeah, the original study is clearly so methodological unsound that it's conclusions shouldn't be repeated. But fankly most of the posts here are people almost calling the original authors heretical for even trying to data mine commit messages.
rossdavidh 7y ago
Sure, you will never completely eliminate all of these factors. But this is far, far short of "not completely eliminated". This is, "the confounding variables are almost certainly much larger effects than what you are trying to measure". The difference between python, ruby, and php in make a web framework, is much much smaller than the difference between web frameworks, statistical analysis packages, machine learning, and devops scripts. Python is used for the last three, ruby and php much less so. The difference between python and ruby and php is therefore going to be dominated by the kinds of tasks they are used on, and the difference between the languages is small compared to that. Smokers are routinely excluded from analysis of cancer rates, for a similar reason; the effect is so large that if (for example) you're looking at diet, and the difference in diet is correlated with a difference in smoking rates, any findings are invalid.