4 ms·

I would be very cautious in drawing any conclusions on the impact of GitHub use on student outcomes. This is a correlational study and correlation doesn’t imply

I would be very cautious in drawing any conclusions on the impact of GitHub use on student outcomes. This is a correlational study and correlation doesn’t imply causation. They dance around the issue by using “predict” in their report. A possible (and plausible?) alternative story is that professor who are “better” require their students to use git and GitHub. As a result, the study is picking up the effects of “being better” instead of GitHub use. I wonder if these results could be replicated with an experimental study. Edit: By the way, the results are pretty cool. I ran some factor analyses/structural models and I found that students in courses that use GitHub feel, on average, 0.36 standard deviations more prepared than students in classroom where GitHub was not used. You can see the analysis in my fork of the dataset on GitHub (same username as HN).

bsder 8y ago
> A possible (and plausible?) alternative story is that professor who are “better” require their students to use git and GitHub. This is almost certainly the case. I made my students use Subversion and then Mercurial along with continuous testing for homework assignments (back almost 10 years ago ...). This "simple" (not really--administering Subversion was a PITA--Mercurial was so much better it was ridiculous ...) change made my students so dramatically better that interviewers commented to the on-campus recruiting about it.
- zeta0134 8y ago
  I think there's a certain understanding of process with any version control system that's kind of hard to teach without one. Collaboration by "sneaker-net" often results in one person doing all of the computer work, or hideous passing of flash drives or emailing of documents. It's very manual and while it can work, it's a chore and prone to errors. Working with version control, suddenly you're aware that other students in your group will see your changes via the logs (adding accountability), and if you have a good professor, you're also writing descriptive commit messages for anything you do. Over time, a student becomes cognizant of the why inherent in their work, in a way that might be hard to appreciate otherwise. It's not enough just to get the solution to work, you must also explain your approach to your classmates, and sometimes just the act of explaining your method enlightens you to a flaw in its approach. You must also become aware of their group members work, because every time you pull or merge, conflicts must be taken care of, bugs introduced must be solved, and some degree of collaboration is therefore forced. It's really a lovely tool, and it adds structure and teachable moments in ways that are hard to replicate otherwise.
  - bsder 8y ago
    I love your optimism, but it wasn't anything that wonderful. It forces the students to be consistent. That's all. However, that's huge by itself. Very few of them understood the larger purpose of version control. Almost all of them simply viewed it as a robust, reliable homework submission system (too many professors had Blackboard(tm) things that sucked giant hairy donkey balls). And since I had automated tests running, they could get feedback instantly for almost a full two weeks as to whether they were passing or not. And that view as a submission system was sufficient. A couple students started figuring out merging. A couple tied themselves into knots or lost files and I had to help them out of the morass. That was fine, too. As for collaboration, I almost never allow group projects because I can't control the team dynamics of them like a real manager would control team dynamics. The only time I give a "group" assignment is when it is "the class" vs "me". One class wanted extra credit. "Fine. Here is the code I use to run automated tests against the code you all submit. Contribute some tests. Tests have to pass on correct code and fail on bad code. Whoever submits a correct test gets a point (limit 10) and the class overall gets 1/10 of a point." Not a single point given. And a couple students learned that validating that your code is correct is MUCH more difficult than simply writing the code to the specification that the professor gave you...
mynewtb 8y ago
It is also obviously in github's interest to say that their service is beneficial, so a significant bias must be assumed.

bsder 8y ago

> A possible (and plausible?) alternative story is that professor who are “better” require their students to use git and GitHub. This is almost certainly the case. I made my students use Subversion and then Mercurial along with continuous testing for homework assignments (back almost 10 years ago ...). This "simple" (not really--administering Subversion was a PITA--Mercurial was so much better it was ridiculous ...) change made my students so dramatically better that interviewers commented to the on-campus recruiting about it.

zeta0134 8y ago
I think there's a certain understanding of process with any version control system that's kind of hard to teach without one. Collaboration by "sneaker-net" often results in one person doing all of the computer work, or hideous passing of flash drives or emailing of documents. It's very manual and while it can work, it's a chore and prone to errors. Working with version control, suddenly you're aware that other students in your group will see your changes via the logs (adding accountability), and if you have a good professor, you're also writing descriptive commit messages for anything you do. Over time, a student becomes cognizant of the why inherent in their work, in a way that might be hard to appreciate otherwise. It's not enough just to get the solution to work, you must also explain your approach to your classmates, and sometimes just the act of explaining your method enlightens you to a flaw in its approach. You must also become aware of their group members work, because every time you pull or merge, conflicts must be taken care of, bugs introduced must be solved, and some degree of collaboration is therefore forced. It's really a lovely tool, and it adds structure and teachable moments in ways that are hard to replicate otherwise.
- bsder 8y ago
  I love your optimism, but it wasn't anything that wonderful. It forces the students to be consistent. That's all. However, that's huge by itself. Very few of them understood the larger purpose of version control. Almost all of them simply viewed it as a robust, reliable homework submission system (too many professors had Blackboard(tm) things that sucked giant hairy donkey balls). And since I had automated tests running, they could get feedback instantly for almost a full two weeks as to whether they were passing or not. And that view as a submission system was sufficient. A couple students started figuring out merging. A couple tied themselves into knots or lost files and I had to help them out of the morass. That was fine, too. As for collaboration, I almost never allow group projects because I can't control the team dynamics of them like a real manager would control team dynamics. The only time I give a "group" assignment is when it is "the class" vs "me". One class wanted extra credit. "Fine. Here is the code I use to run automated tests against the code you all submit. Contribute some tests. Tests have to pass on correct code and fail on bad code. Whoever submits a correct test gets a point (limit 10) and the class overall gets 1/10 of a point." Not a single point given. And a couple students learned that validating that your code is correct is MUCH more difficult than simply writing the code to the specification that the professor gave you...

mynewtb 8y ago

It is also obviously in github's interest to say that their service is beneficial, so a significant bias must be assumed.