3 ms·

The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.

vacri 10y ago
How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done. So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group? Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.
- smallnamespace 10y ago
  This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.
  - vacri 10y ago
    Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.
    - smallnamespace 10y ago
      I completely agree with you. Pointing out a study's flaws is indeed part of the process of 'caveat[ing] the knowledge gained from that study'.
- flukus 10y ago
  Maybe we have to accept that it's untestable. Though I think we could derive some more realistic scenarios, like evolving requirements and switching developers mid project that would be more enlightening.
- icebraining 10y ago
  How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? Hackatons seem to manage. Why not set one up?
  - vacri 10y ago
    Hackathons wouldn't satisfy the OP's requirements for complexity, nor demographics - you'd be looking at a self-selecting group of highly motivated people, skewing young, who would come together hackathon style. There aren't going to be many thirty- or forty-something coders with young families spending a weekend (to work on the same set problems as everyone else) at the hackathon, yet there are plenty of those in industry.
- mwat 10y ago
  We can't arrange a good study, so let's churn out deeply flawed ones instead.
- yomly 10y ago
  Whereas the cost of finding the Higgs Boson was only a mere $13.25Bn, cheap compared to all those pesky extortionate SV engineers... As others have said, good research costs money...
- vidarh 10y ago
  You could start by finding a corporate sponsor with a lot of developers and a vested interest in finding out which methods will be best for them. The advantage of that being that you could even test it on a real project (the sponsor would "just" need to be willing to dedicate twice the number of developers to a suitably small project). It'd still not be easy, and of course there'd still be issues (e.g. is there anything about the corporate culture or training in that company that would affect the result?), but it'd still be far better than a bunch of students and toy problems.
  - vacri 10y ago
    Corporate places do research like that all the time; they just don't publish them all that often. A corporation with a vested interest is a corporation with a competitive interest.

vacri 10y ago

How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done. So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group? Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.

smallnamespace 10y ago
This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.
- vacri 10y ago
  Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.
  - smallnamespace 10y ago
    I completely agree with you. Pointing out a study's flaws is indeed part of the process of 'caveat[ing] the knowledge gained from that study'.
flukus 10y ago
Maybe we have to accept that it's untestable. Though I think we could derive some more realistic scenarios, like evolving requirements and switching developers mid project that would be more enlightening.
icebraining 10y ago
How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? Hackatons seem to manage. Why not set one up?
- vacri 10y ago
  Hackathons wouldn't satisfy the OP's requirements for complexity, nor demographics - you'd be looking at a self-selecting group of highly motivated people, skewing young, who would come together hackathon style. There aren't going to be many thirty- or forty-something coders with young families spending a weekend (to work on the same set problems as everyone else) at the hackathon, yet there are plenty of those in industry.
mwat 10y ago
We can't arrange a good study, so let's churn out deeply flawed ones instead.
yomly 10y ago
Whereas the cost of finding the Higgs Boson was only a mere $13.25Bn, cheap compared to all those pesky extortionate SV engineers... As others have said, good research costs money...
vidarh 10y ago
You could start by finding a corporate sponsor with a lot of developers and a vested interest in finding out which methods will be best for them. The advantage of that being that you could even test it on a real project (the sponsor would "just" need to be willing to dedicate twice the number of developers to a suitably small project). It'd still not be easy, and of course there'd still be issues (e.g. is there anything about the corporate culture or training in that company that would affect the result?), but it'd still be far better than a bunch of students and toy problems.
- vacri 10y ago
  Corporate places do research like that all the time; they just don't publish them all that often. A corporation with a vested interest is a corporation with a competitive interest.