5 ms·

I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping doma

I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.

bunderbunder 7y ago
Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab. A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.
- yogthos 7y ago
  Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.
  - bunderbunder 7y ago
    But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
    - yogthos 7y ago
      First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.
  - the_why_of_y 7y ago
    > numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. You could look at commit messages, as the (original) paper does (see 2.2.1 / 2.2.2 in the reproduction paper), but the heuristic they use doesn't look very reliable to me; I've seen projects where every feature implementation had to have an issue in the bug tracker and every commit implementing that feature had to reference the issue, so almost every commit would be counted as a defect by their metric. And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? There's an underlying assumption [edit: not implying it's yogthos' assumption, of course] that you can simply compare unrelated projects with vastly different policies and processes, and I believe that a lot more manual work is required to get reliable and comparable data.
    - yogthos 7y ago
      >You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. It's a numbers game, if we look at a lot of projects written in a lot of different languages, we'll see a sample of projects with different kinds of popularity. If there is a trend associated with a particular language that's an outlier that would be an indication that the particular language may play a role. >And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? Again, it's about trends. If you look at thousands or millions of projects these things average out across them. You can even group projects into different categories based on their domain, popularity, and so on. The underlying idea is that you should be able to see measurable trends when looking at large numbers of projects across languages. If such trends are established, then we can dig deeper and make a hypothesis as to what might be the cause.
pron 7y ago
As someone who develops mostly in C++, I can tell you that productivity is a very small concern. We are well aware that most GCed languages are significantly more productive. Virtually the only reason people use C++ nowadays is for performance (and/or footprint) (that is partly because we have seen reports that Java delivers better productivity in the early '00s, so those who care about that have switched a long time ago; in fact, the effect was so large that they didn't have a choice). Now, if someone showed that Haskell has a significantly better performance/footprint than C++ that may cause someone to shift.
- yogthos 7y ago
  Sure, in some domains you have to use specific languages because of the constraints of the domain. However, as you yourself state, a language can play a significant impact when it comes to productivity. For example, immutability provides similar benefits to GC. Instead of the developer having to manually manage references across the project by hand, the language takes care of that work. This frees the developer from doing additional work, and removes a source of errors. This also directly leads to the ability to do local reasoning about code resulting in developers needing less context to understand code. So, if GC plays a significant enough role then it's highly likely that immutability does as well because it addresses a similar set of problems.

bunderbunder 7y ago

Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab. A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.

yogthos 7y ago
Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.
- bunderbunder 7y ago
  But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
  - yogthos 7y ago
    First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.
- the_why_of_y 7y ago
  > numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. You could look at commit messages, as the (original) paper does (see 2.2.1 / 2.2.2 in the reproduction paper), but the heuristic they use doesn't look very reliable to me; I've seen projects where every feature implementation had to have an issue in the bug tracker and every commit implementing that feature had to reference the issue, so almost every commit would be counted as a defect by their metric. And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? There's an underlying assumption [edit: not implying it's yogthos' assumption, of course] that you can simply compare unrelated projects with vastly different policies and processes, and I believe that a lot more manual work is required to get reliable and comparable data.
  - yogthos 7y ago
    >You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. It's a numbers game, if we look at a lot of projects written in a lot of different languages, we'll see a sample of projects with different kinds of popularity. If there is a trend associated with a particular language that's an outlier that would be an indication that the particular language may play a role. >And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? Again, it's about trends. If you look at thousands or millions of projects these things average out across them. You can even group projects into different categories based on their domain, popularity, and so on. The underlying idea is that you should be able to see measurable trends when looking at large numbers of projects across languages. If such trends are established, then we can dig deeper and make a hypothesis as to what might be the cause.

pron 7y ago

As someone who develops mostly in C++, I can tell you that productivity is a very small concern. We are well aware that most GCed languages are significantly more productive. Virtually the only reason people use C++ nowadays is for performance (and/or footprint) (that is partly because we have seen reports that Java delivers better productivity in the early '00s, so those who care about that have switched a long time ago; in fact, the effect was so large that they didn't have a choice). Now, if someone showed that Haskell has a significantly better performance/footprint than C++ that may cause someone to shift.

yogthos 7y ago
Sure, in some domains you have to use specific languages because of the constraints of the domain. However, as you yourself state, a language can play a significant impact when it comes to productivity. For example, immutability provides similar benefits to GC. Instead of the developer having to manually manage references across the project by hand, the language takes care of that work. This frees the developer from doing additional work, and removes a source of errors. This also directly leads to the ability to do local reasoning about code resulting in developers needing less context to understand code. So, if GC plays a significant enough role then it's highly likely that immutability does as well because it addresses a similar set of problems.