4 ms·

There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate

There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment. I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To start with, that would make it difficult to come up with a task that wouldn't handicap one language or the other. (For example, a video game would favor C++.) Even if you found that, you'd have a hard time coming up with a good test. Under random assignment, Haskell is presumably going to be handicapped relative to C++ by the fact that most languages are imperative and use Algol-style syntax. You could try to avoid that by using participants with zero programming experience, but then you're studying learning curve for absolute beginners rather than productivity among experts. And so on and so on. Long story short, probably best to leave this one to the spittle-flecked Internet arguments.

yogthos 7y ago
I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.
- bunderbunder 7y ago
  Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab. A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.
  - yogthos 7y ago
    Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.
    - bunderbunder 7y ago
      But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
      - yogthos 7y ago
        First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.
    - the_why_of_y 7y ago
      > numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. You could look at commit messages, as the (original) paper does (see 2.2.1 / 2.2.2 in the reproduction paper), but the heuristic they use doesn't look very reliable to me; I've seen projects where every feature implementation had to have an issue in the bug tracker and every commit implementing that feature had to reference the issue, so almost every commit would be counted as a defect by their metric. And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? There's an underlying assumption [edit: not implying it's yogthos' assumption, of course] that you can simply compare unrelated projects with vastly different policies and processes, and I believe that a lot more manual work is required to get reliable and comparable data.
- pron 7y ago
  As someone who develops mostly in C++, I can tell you that productivity is a very small concern. We are well aware that most GCed languages are significantly more productive. Virtually the only reason people use C++ nowadays is for performance (and/or footprint) (that is partly because we have seen reports that Java delivers better productivity in the early '00s, so those who care about that have switched a long time ago; in fact, the effect was so large that they didn't have a choice). Now, if someone showed that Haskell has a significantly better performance/footprint than C++ that may cause someone to shift.
  - yogthos 7y ago
    Sure, in some domains you have to use specific languages because of the constraints of the domain. However, as you yourself state, a language can play a significant impact when it comes to productivity. For example, immutability provides similar benefits to GC. Instead of the developer having to manually manage references across the project by hand, the language takes care of that work. This frees the developer from doing additional work, and removes a source of errors. This also directly leads to the ability to do local reasoning about code resulting in developers needing less context to understand code. So, if GC plays a significant enough role then it's highly likely that immutability does as well because it addresses a similar set of problems.

yogthos 7y ago

I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.

bunderbunder 7y ago
Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab. A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.
- yogthos 7y ago
  Measuring quality of software products is not a social science. You have clear metrics on time it takes to develop something, number of people you need to develop it, numbers of defects per project, and so on. Once you have massive amounts of data, as we now have with repositories like GitHub, it becomes possible to look for trends across languages. I'm not saying there aren't challenges associated with measuring these things, as there obviously are, but it's certainly possible to study them. And the more studies we have, the more confidence we get regarding the effects.
  - bunderbunder 7y ago
    But you're not just measuring the quality of software products in isolation. The whole point is to measure how a certain set of conditions, particularly the choice of an artifically constructed language that humans use, among other things, to communicate with each other, affects the result of a human enterprise that's typically performed socially. For these kinds of research questions, human social behavior is not a distraction from the issue being studied. It's the very locus of the issue being studied.
    - yogthos 7y ago
      First, we need to established whether there is something that needs to be studied or not. The idea is to look for the outliers by exemaning a large volume of projects in many different languages. If you see a correlation between a language and a result then you can mke a hypothesis that the language is a factor. At that point you can make a further hypothesis as to whether certain languages encourage a particular kind of social behavior, or whether particular kinds of people are drawn to particular languages. And of course it could just be the technical features of the language that end up playing a role because social aspect may be consistent across languages.
  - the_why_of_y 7y ago
    > numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. You could look at commit messages, as the (original) paper does (see 2.2.1 / 2.2.2 in the reproduction paper), but the heuristic they use doesn't look very reliable to me; I've seen projects where every feature implementation had to have an issue in the bug tracker and every commit implementing that feature had to reference the issue, so almost every commit would be counted as a defect by their metric. And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? There's an underlying assumption [edit: not implying it's yogthos' assumption, of course] that you can simply compare unrelated projects with vastly different policies and processes, and I believe that a lot more manual work is required to get reliable and comparable data.
pron 7y ago
As someone who develops mostly in C++, I can tell you that productivity is a very small concern. We are well aware that most GCed languages are significantly more productive. Virtually the only reason people use C++ nowadays is for performance (and/or footprint) (that is partly because we have seen reports that Java delivers better productivity in the early '00s, so those who care about that have switched a long time ago; in fact, the effect was so large that they didn't have a choice). Now, if someone showed that Haskell has a significantly better performance/footprint than C++ that may cause someone to shift.
- yogthos 7y ago
  Sure, in some domains you have to use specific languages because of the constraints of the domain. However, as you yourself state, a language can play a significant impact when it comes to productivity. For example, immutability provides similar benefits to GC. Instead of the developer having to manually manage references across the project by hand, the language takes care of that work. This frees the developer from doing additional work, and removes a source of errors. This also directly leads to the ability to do local reasoning about code resulting in developers needing less context to understand code. So, if GC plays a significant enough role then it's highly likely that immutability does as well because it addresses a similar set of problems.