7 ms·

The study may or may not be flawed, but what's really interesting to me is the reaction. We need more science in our computer science, which means more experime

The study may or may not be flawed, but what's really interesting to me is the reaction. We need more science in our computer science, which means more experiments and more results like this. We should also be open to the truth that we use the tools we like because we like them rather than because they're technically superior, even though we pimp them ad nauseum as though they are. I once read an article about a technique Intel had developed for improving cooling of processors by changing the shape of the fan. I related this to some of my co-workers. One of them proceeded to tell me that this can't possibly work, backing up his argument with "reasoning" based on off-the-cuff remarks about the way air and physics "must" work. The fact that Intel had actually done this seemed to have no effect on his eagerness to continue the "debate" about this scientific fact. I have a hard time believing that I get no benefit from using Haskell over Smalltalk, but if a body of science were to appear that cast doubt on that belief, the appropriate thing to do is change the belief, not stand around debating from imagined first principles why all the science is wrong and can't be so. Shut up, design an experiment and go prove it! Perhaps there's little of this kind of actual science in our computer science because it would mean asking hard questions and accepting difficult truths. "The prisoner falls in love with his chains."

aangjie 14y ago
I second that.It's one of the reasons i quit my day job to do a Master's degree...though i chose a naive enough college and course that am now back to a programming job where i am surrounded by the same type of crowd again.
- pcopley 14y ago
  I'm curious what your Master's is in and what "type" you're surrounded by.
  - aangjie 14y ago
    Oh. Sorry, for that snarky reply(tired i guess). I have a master's in cognitive science(from india). And currently a python programmer here. Mainly because there aren't that many jobs(in cog. sci.) , here and i was not focused enough to get published in a journal during my master's. As for types, well, one ex: is a situation where i was trying to defend my choice of Dvorak kb saying usability and got a dismissive, snorty laugh as a response.
gruseom 14y ago
Emphatic upvote. Software development is so complicated that there is always an endless supply of objections to fire at any study at odds with one's beliefs. And that is exactly how all these discussions go. All we're doing is repeating shibboleths. The most interesting studies would be ones that changed somebody's beliefs. That doesn't happen very often in our field. Does it ever?
tikhonj 14y ago
I've only scanned the comments, but the chief reaction seems to be exactly what you want for a scientific approach--people are arguing about the scope and methodology of the study. A paper like this is not some broad-reaching conclusion--it's very specific and based on some potentially flawed methodology. You want people to qualify exactly how specific it is and talk about potential flaws in its approach. That's how you improve the general knowledge. Also, I suspect there are a couple of reasons such studies are uncommon in computer science. For one, CS isn't really a science; programmers and computer scientists are not trained in the scientific method or experimentation (beyond their general education); almost no CS papers I've read have contained empirical studies. If anything, they are closer to math papers than science papers! Additionally, this sort of study is basically sociology. (Or something similar.) These sorts of fields are considered a little shady by hard scientists, and CS people tend to empathize more with the latter. I think this explains the immediate attack on methodology. All that said, having more studies done about these questions would be great. I'm just not sure who's the best to do them. Maybe HCI researchers? I can't help thinking that the really intense PL people I know wouldn't be very interested in doing this.
- jbranchaud 14y ago
  "Almost no CS papers I've read have contained empirical studies." You aren't reading the right papers then. At least in Software Engineering, you can't get into the main conferences (ICSE and FSE) without a pretty significant empirical study.
  - klodolph 14y ago
    Yes, but that's the main difference between computer science and software engineering.
    - barik 14y ago
      That's a broad brush, and though it's probably a good categorization, I don't find this distinction to be all that useful as a boundary. For instance, the field of artificial intelligence and cognitive psychology branched at one point, so much of my work in cognitive architecture and algorithmic modeling necessitates user studies. One would be hard-pressed to bucket AI into software engineering though. Likewise, in machine learning, I've seen a push from classical data-driven to modern "data-informed" approaches to analyzing these results. Computational linguistics (NLP) and computational narrative are yet additional examples of fields that often requires user studies or other empirical data. More to the point, the distinction of what is and isn't computer science has become even more blurry in the research community because research in itself has become more inter-disciplinary. There seems to be little to gain from attempting to "bucket" research into distinct taxonomies.
edanm 14y ago
I think your point is important, and well made. The only problem is, people have been doing these experiments for 30 years, and do you know what the net effect it's had on the world of programmers: none at all. Saying "no no, this time really listen to this study" seems to be having no effect. There are a lot of causes for this, not least of all the things you mention (nobody cares about science, people like or dislike based on non-scientific evidence). But it's also because people know that all these studies are flawed. As much as I hate to be the "nitpicker" who takes apart studies, EVERY SINGLE STUDY on programming doesn't even come close to real-life scenarios. In fact, one of the few studies that people actually believe is the "some programmers are 10x better" study, and that was actually fairly well conducted - many students were given identical tasks, and a fairly large amount of time to do them. But take a look at the Dynamic vs. Static argument. For years, the Dynamic-fans have been saying "Quicker to program, so it's better", while the Static-fans have been saying "Quicker to program, but harder to maintain, hard to use with large teams". So now we have a study that doesn't even come close to addressing most of the issues that have been argued for years! Of course this isn't going to convince anyone.
- barik 14y ago
  "In fact, one of the few studies that people actually believe is the "some programmers are 10x better" study, and that was actually fairly well conducted - many students were given identical tasks, and a fairly large amount of time to do them." Can you by chance point me to this paper? I'd like to add it to my paper collection, since most of the studies I've seen concerning programmer variability use members of the workforce. I'm not aware of the one involving students, but such replication studies are easy to miss. Thanks!
  - dalke 14y ago
    http://forums.construx.com/blogs/stevemcc/archive/2011/01/09/origins-of-10x-how-valid-is-the-underlying-research.aspx http://forums.construx.com/blogs/stevemcc/archive/2011/01/09...
    - barik 14y ago
kitsune_ 14y ago
I don't know how science can help you here. In the case of the improved fan design, well that's easily testable. However, I don't want to go down the rabbit hole and argue social vs natural sciences. I find it dumbfounding that in this day and age, people can still create rather arbitrary social experiments with only 49 people and then think they can draw grandiose conclusions from their "data".
- chillax 14y ago
  Well, in this study there are no grandiose conclusions.
- jbranchaud 14y ago
  Unfortunately, user studies are difficult to conduct in computer science mainly because it is difficult to get users that you can study. Like it or not, 49 people is actually a pretty big study relative to what is out there. For better or for worse, I have seen much smaller studies accepted/published by top tier conferences.
zoul 14y ago
The problem is that there are so many soft factors affecting the outcome that it's very hard to tell if even a great scientific study will apply to your case. The number of people on your team, their experience with the technology being used, their relationships between each other, the politics of the working space and thousands other factors may have much bigger effect on the productivity and quality then the choice of a typing system, and it may be pretty impossible to control for them in the study. I agree that it's good to approach things scientifically, but at the same time it's also good to question how much science applies to the problems being solved.
ExpiredLink 14y ago
> We need more science in our computer science, which means more experiments and more results like this. There is an ongoing experiment wrt the usability of programming languages, libraries, frameworks, concepts, ... It's called the market.
- xyzzyz 14y ago
  Market is testing for many more different features than just usability, and as such is rather useless if you want to test a single thing.
- johnchristopher 14y ago
  What are the market's conclusions regarding dynamic and static typing ?
- madhadron 14y ago
  Counterexample: JavaScript. It is the best demonstration I know of that our choice of languages and tools is mostly based on historical accident rather than any technical criteria.
sgk284 14y ago
Agreed with your overall premise, but this particular study doesn't meet the criteria to have any conclusions drawn from its results. The number of programmers is too small and skills not representative (49 undergrad programmers). The problem (writing a simple scanner / parser) isn't one that will really benefit from a decent type system. All you need need are ints, strings, and arrays and you're good to go.
nu23 14y ago
Reasoning from basic principles is a valuable tool to evaluate conclusions. Sure, it has strong weaknesses (hidden assumptions which are wrong, insufficient imagination about what could happen, rationalizing one's biases). But it is useful, when you dont have complete trust in the quality or the scope of the experiment. For instance, claims about quantum computing solving NP complete problems are legitimately held in doubt because of theoretical reasons. Also, whenever there are short term positives hiding a long term negative, like say unsustainable financial or ecological behavior, the negatives might be only seen by a chain of reasoning and not by direct experiments. I agree in the sense that we see so many cases in the other direction - reasoning full of holes being trusted over empirics. The interesting thing is in any given situation, how much trust to give to the different tools that we have to evaluate a claim.
silverlake 14y ago
No study can control all the variables enough to convince a fanboy that his favorite language isn't the greatest ever. However, the current state of PL research is as close to astrology as you can get. Why are we working on type systems and modules and concurrency primitives et. al. without a scrap of evidence that any of it contributes to programmer productivity? There's no science there. In fact, everyone here should flip this around and ask: can you design a practical experiment to compare productivity of dynamic vs. static languages? Will others find it convincing? Probably not. PL advocates are no different than religious missionaries. They have no objective proof of any of their claims. And both will murder the natives if they don't convert. edit: Fowler's view here http://martinfowler.com/bliki/CannotMeasureProductivity.html http://martinfowler.com/bliki/CannotMeasureProductivity.html
- bunderbunder 14y ago
  Well, one thought that comes to mind is that the topic is essentially a subset of ergonomics, so perhaps experimental protocols should take a few more cues from ergonomics research. For example, controlling variables has to be done by constructing artificial systems from the ground up. You can't just pull two commercial products off the shelf and then pretend you're examining the impact of only one of the hundered different things that differs between the two. Similarly, if we wanted to compare the impact of dynamic vs. static typing, we'd have to make sure that that is the only variable. Which means you basically have to construct a new programming language from the ground up, so that you can easily create new dialects of it that differ in only one very specific characteristic.
  - silverlake 14y ago
    Did you even read the linked paper? They did exactly what you described.
    - bunderbunder 14y ago
      As the linked paper points out in the very first sentence of the section describing the language, this hasn't been a very popular approach so far. I suppose I should have nodded to that. Got me there.
deleted 14y ago
[deleted]

aangjie 14y ago

I second that.It's one of the reasons i quit my day job to do a Master's degree...though i chose a naive enough college and course that am now back to a programming job where i am surrounded by the same type of crowd again.

pcopley 14y ago
I'm curious what your Master's is in and what "type" you're surrounded by.
- aangjie 14y ago
  Oh. Sorry, for that snarky reply(tired i guess). I have a master's in cognitive science(from india). And currently a python programmer here. Mainly because there aren't that many jobs(in cog. sci.) , here and i was not focused enough to get published in a journal during my master's. As for types, well, one ex: is a situation where i was trying to defend my choice of Dvorak kb saying usability and got a dismissive, snorty laugh as a response.

gruseom 14y ago

Emphatic upvote. Software development is so complicated that there is always an endless supply of objections to fire at any study at odds with one's beliefs. And that is exactly how all these discussions go. All we're doing is repeating shibboleths. The most interesting studies would be ones that changed somebody's beliefs. That doesn't happen very often in our field. Does it ever?

tikhonj 14y ago

I've only scanned the comments, but the chief reaction seems to be exactly what you want for a scientific approach--people are arguing about the scope and methodology of the study. A paper like this is not some broad-reaching conclusion--it's very specific and based on some potentially flawed methodology. You want people to qualify exactly how specific it is and talk about potential flaws in its approach. That's how you improve the general knowledge. Also, I suspect there are a couple of reasons such studies are uncommon in computer science. For one, CS isn't really a science; programmers and computer scientists are not trained in the scientific method or experimentation (beyond their general education); almost no CS papers I've read have contained empirical studies. If anything, they are closer to math papers than science papers! Additionally, this sort of study is basically sociology. (Or something similar.) These sorts of fields are considered a little shady by hard scientists, and CS people tend to empathize more with the latter. I think this explains the immediate attack on methodology. All that said, having more studies done about these questions would be great. I'm just not sure who's the best to do them. Maybe HCI researchers? I can't help thinking that the really intense PL people I know wouldn't be very interested in doing this.

jbranchaud 14y ago
"Almost no CS papers I've read have contained empirical studies." You aren't reading the right papers then. At least in Software Engineering, you can't get into the main conferences (ICSE and FSE) without a pretty significant empirical study.
- klodolph 14y ago
  Yes, but that's the main difference between computer science and software engineering.
  - barik 14y ago
    That's a broad brush, and though it's probably a good categorization, I don't find this distinction to be all that useful as a boundary. For instance, the field of artificial intelligence and cognitive psychology branched at one point, so much of my work in cognitive architecture and algorithmic modeling necessitates user studies. One would be hard-pressed to bucket AI into software engineering though. Likewise, in machine learning, I've seen a push from classical data-driven to modern "data-informed" approaches to analyzing these results. Computational linguistics (NLP) and computational narrative are yet additional examples of fields that often requires user studies or other empirical data. More to the point, the distinction of what is and isn't computer science has become even more blurry in the research community because research in itself has become more inter-disciplinary. There seems to be little to gain from attempting to "bucket" research into distinct taxonomies.

edanm 14y ago

I think your point is important, and well made. The only problem is, people have been doing these experiments for 30 years, and do you know what the net effect it's had on the world of programmers: none at all. Saying "no no, this time really listen to this study" seems to be having no effect. There are a lot of causes for this, not least of all the things you mention (nobody cares about science, people like or dislike based on non-scientific evidence). But it's also because people know that all these studies are flawed. As much as I hate to be the "nitpicker" who takes apart studies, EVERY SINGLE STUDY on programming doesn't even come close to real-life scenarios. In fact, one of the few studies that people actually believe is the "some programmers are 10x better" study, and that was actually fairly well conducted - many students were given identical tasks, and a fairly large amount of time to do them. But take a look at the Dynamic vs. Static argument. For years, the Dynamic-fans have been saying "Quicker to program, so it's better", while the Static-fans have been saying "Quicker to program, but harder to maintain, hard to use with large teams". So now we have a study that doesn't even come close to addressing most of the issues that have been argued for years! Of course this isn't going to convince anyone.

barik 14y ago
"In fact, one of the few studies that people actually believe is the "some programmers are 10x better" study, and that was actually fairly well conducted - many students were given identical tasks, and a fairly large amount of time to do them." Can you by chance point me to this paper? I'd like to add it to my paper collection, since most of the studies I've seen concerning programmer variability use members of the workforce. I'm not aware of the one involving students, but such replication studies are easy to miss. Thanks!
- dalke 14y ago
  http://forums.construx.com/blogs/stevemcc/archive/2011/01/09/origins-of-10x-how-valid-is-the-underlying-research.aspx http://forums.construx.com/blogs/stevemcc/archive/2011/01/09...
  - barik 14y ago

kitsune_ 14y ago

I don't know how science can help you here. In the case of the improved fan design, well that's easily testable. However, I don't want to go down the rabbit hole and argue social vs natural sciences. I find it dumbfounding that in this day and age, people can still create rather arbitrary social experiments with only 49 people and then think they can draw grandiose conclusions from their "data".

chillax 14y ago
Well, in this study there are no grandiose conclusions.
jbranchaud 14y ago
Unfortunately, user studies are difficult to conduct in computer science mainly because it is difficult to get users that you can study. Like it or not, 49 people is actually a pretty big study relative to what is out there. For better or for worse, I have seen much smaller studies accepted/published by top tier conferences.

zoul 14y ago

The problem is that there are so many soft factors affecting the outcome that it's very hard to tell if even a great scientific study will apply to your case. The number of people on your team, their experience with the technology being used, their relationships between each other, the politics of the working space and thousands other factors may have much bigger effect on the productivity and quality then the choice of a typing system, and it may be pretty impossible to control for them in the study. I agree that it's good to approach things scientifically, but at the same time it's also good to question how much science applies to the problems being solved.

ExpiredLink 14y ago

> We need more science in our computer science, which means more experiments and more results like this. There is an ongoing experiment wrt the usability of programming languages, libraries, frameworks, concepts, ... It's called the market.

xyzzyz 14y ago
Market is testing for many more different features than just usability, and as such is rather useless if you want to test a single thing.
johnchristopher 14y ago
What are the market's conclusions regarding dynamic and static typing ?
madhadron 14y ago
Counterexample: JavaScript. It is the best demonstration I know of that our choice of languages and tools is mostly based on historical accident rather than any technical criteria.

sgk284 14y ago

Agreed with your overall premise, but this particular study doesn't meet the criteria to have any conclusions drawn from its results. The number of programmers is too small and skills not representative (49 undergrad programmers). The problem (writing a simple scanner / parser) isn't one that will really benefit from a decent type system. All you need need are ints, strings, and arrays and you're good to go.

nu23 14y ago

Reasoning from basic principles is a valuable tool to evaluate conclusions. Sure, it has strong weaknesses (hidden assumptions which are wrong, insufficient imagination about what could happen, rationalizing one's biases). But it is useful, when you dont have complete trust in the quality or the scope of the experiment. For instance, claims about quantum computing solving NP complete problems are legitimately held in doubt because of theoretical reasons. Also, whenever there are short term positives hiding a long term negative, like say unsustainable financial or ecological behavior, the negatives might be only seen by a chain of reasoning and not by direct experiments. I agree in the sense that we see so many cases in the other direction - reasoning full of holes being trusted over empirics. The interesting thing is in any given situation, how much trust to give to the different tools that we have to evaluate a claim.

silverlake 14y ago

No study can control all the variables enough to convince a fanboy that his favorite language isn't the greatest ever. However, the current state of PL research is as close to astrology as you can get. Why are we working on type systems and modules and concurrency primitives et. al. without a scrap of evidence that any of it contributes to programmer productivity? There's no science there. In fact, everyone here should flip this around and ask: can you design a practical experiment to compare productivity of dynamic vs. static languages? Will others find it convincing? Probably not. PL advocates are no different than religious missionaries. They have no objective proof of any of their claims. And both will murder the natives if they don't convert. edit: Fowler's view here http://martinfowler.com/bliki/CannotMeasureProductivity.html http://martinfowler.com/bliki/CannotMeasureProductivity.html

bunderbunder 14y ago
Well, one thought that comes to mind is that the topic is essentially a subset of ergonomics, so perhaps experimental protocols should take a few more cues from ergonomics research. For example, controlling variables has to be done by constructing artificial systems from the ground up. You can't just pull two commercial products off the shelf and then pretend you're examining the impact of only one of the hundered different things that differs between the two. Similarly, if we wanted to compare the impact of dynamic vs. static typing, we'd have to make sure that that is the only variable. Which means you basically have to construct a new programming language from the ground up, so that you can easily create new dialects of it that differ in only one very specific characteristic.
- silverlake 14y ago
  Did you even read the linked paper? They did exactly what you described.
  - bunderbunder 14y ago
    As the linked paper points out in the very first sentence of the section describing the language, this hasn't been a very popular approach so far. I suppose I should have nodded to that. Got me there.

deleted 14y ago

[deleted]