3 ms·

Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the

Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab. A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.

yogthos 7y ago
Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.
- bunderbunder 7y ago
  But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
  - yogthos 7y ago
    First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.
- the_why_of_y 7y ago
  > numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. You could look at commit messages, as the (original) paper does (see 2.2.1 / 2.2.2 in the reproduction paper), but the heuristic they use doesn't look very reliable to me; I've seen projects where every feature implementation had to have an issue in the bug tracker and every commit implementing that feature had to reference the issue, so almost every commit would be counted as a defect by their metric. And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? There's an underlying assumption [edit: not implying it's yogthos' assumption, of course] that you can simply compare unrelated projects with vastly different policies and processes, and I believe that a lot more manual work is required to get reliable and comparable data.
  - yogthos 7y ago
    >You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. It's a numbers game, if we look at a lot of projects written in a lot of different languages, we'll see a sample of projects with different kinds of popularity. If there is a trend associated with a particular language that's an outlier that would be an indication that the particular language may play a role. >And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? Again, it's about trends. If you look at thousands or millions of projects these things average out across them. You can even group projects into different categories based on their domain, popularity, and so on. The underlying idea is that you should be able to see measurable trends when looking at large numbers of projects across languages. If such trends are established, then we can dig deeper and make a hypothesis as to what might be the cause.

yogthos 7y ago

Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.

bunderbunder 7y ago
But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
- yogthos 7y ago
  First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.
the_why_of_y 7y ago
> numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. You could look at commit messages, as the (original) paper does (see 2.2.1 / 2.2.2 in the reproduction paper), but the heuristic they use doesn't look very reliable to me; I've seen projects where every feature implementation had to have an issue in the bug tracker and every commit implementing that feature had to reference the issue, so almost every commit would be counted as a defect by their metric. And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? There's an underlying assumption [edit: not implying it's yogthos' assumption, of course] that you can simply compare unrelated projects with vastly different policies and processes, and I believe that a lot more manual work is required to get reliable and comparable data.
- yogthos 7y ago
  >You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. It's a numbers game, if we look at a lot of projects written in a lot of different languages, we'll see a sample of projects with different kinds of popularity. If there is a trend associated with a particular language that's an outlier that would be an indication that the particular language may play a role. >And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? Again, it's about trends. If you look at thousands or millions of projects these things average out across them. You can even group projects into different categories based on their domain, popularity, and so on. The underlying idea is that you should be able to see measurable trends when looking at large numbers of projects across languages. If such trends are established, then we can dig deeper and make a hypothesis as to what might be the cause.