7 ms·

Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be deliv

Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be delivered faster by a smaller team than an equivalent project in C++, that would be an important factor.

bunderbunder 7y ago
There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment. I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To start with, that would make it difficult to come up with a task that wouldn't handicap one language or the other. (For example, a video game would favor C++.) Even if you found that, you'd have a hard time coming up with a good test. Under random assignment, Haskell is presumably going to be handicapped relative to C++ by the fact that most languages are imperative and use Algol-style syntax. You could try to avoid that by using participants with zero programming experience, but then you're studying learning curve for absolute beginners rather than productivity among experts. And so on and so on. Long story short, probably best to leave this one to the spittle-flecked Internet arguments.
- yogthos 7y ago
  I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.
  - bunderbunder 7y ago
    Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab. A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.
    - yogthos 7y ago
      Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.
      - bunderbunder 7y ago
        But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
        yogthos 7y ago
        First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.
deleted 7y ago
[deleted]
danielscrubs 7y ago
Important for what? They are trying to do a study, not declare a flame war.
- yogthos 7y ago
  It would be an important factor when choosing a language obviously. If language A allows you to deliver same kinds of projects as language B, but with less people that's valuable information for the business.
  - danielscrubs 7y ago
    And that's not what they are trying to study: [the] association between eleven programming languages and software defects in projects hosted on GitHub. Nothing more, nothing less. If they wanted a "x is better than y for business", it would not be a serious study at all, just a bunch of journalistic edgy writing.
    - yogthos 7y ago
      Correct, this is outside the scope of the study. My point was that there are factors other than correctness that are important. Pron's comment implies that since there is practically no difference in quality shown across languages, that the language choice does not matter. However, clearly there are other factors that are important to consider.

bunderbunder 7y ago

There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment. I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To start with, that would make it difficult to come up with a task that wouldn't handicap one language or the other. (For example, a video game would favor C++.) Even if you found that, you'd have a hard time coming up with a good test. Under random assignment, Haskell is presumably going to be handicapped relative to C++ by the fact that most languages are imperative and use Algol-style syntax. You could try to avoid that by using participants with zero programming experience, but then you're studying learning curve for absolute beginners rather than productivity among experts. And so on and so on. Long story short, probably best to leave this one to the spittle-flecked Internet arguments.

yogthos 7y ago
I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.
- bunderbunder 7y ago
  Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab. A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.
  - yogthos 7y ago
    Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.
    - bunderbunder 7y ago
      But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
      - yogthos 7y ago
        First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.

deleted 7y ago

[deleted]

danielscrubs 7y ago

Important for what? They are trying to do a study, not declare a flame war.

yogthos 7y ago
It would be an important factor when choosing a language obviously. If language A allows you to deliver same kinds of projects as language B, but with less people that's valuable information for the business.
- danielscrubs 7y ago
  And that's not what they are trying to study: [the] association between eleven programming languages and software defects in projects hosted on GitHub. Nothing more, nothing less. If they wanted a "x is better than y for business", it would not be a serious study at all, just a bunch of journalistic edgy writing.
  - yogthos 7y ago
    Correct, this is outside the scope of the study. My point was that there are factors other than correctness that are important. Pron's comment implies that since there is practically no difference in quality shown across languages, that the language choice does not matter. However, clearly there are other factors that are important to consider.