5 ms·

There have been quite a few studies that have done this in a scientific manner. Internally we have taken sets of students that apply to our free “mini code boo

There have been quite a few studies that have done this in a scientific manner. Internally we have taken sets of students that apply to our free “mini code bootcamp” and sent half to an archived version published each night and send the other half a live link that becomes an archive each night. So we didn’t control scientifically, jsut an AB test, but the shift has been breathtaking. We can also take students that are struggling learning from our MOOCs and they finish the live course almost every time. I know the latter is unscientific, but it becomes obvious over time.

kbutler 9y ago
This Harvard study & Atlantic article by the study authors give the same 5% completion rate you cited, but give a completely opposite interpretation: basically, that the MOOC broadened access, allowing more people to get what they wanted out of the course. https://www.theatlantic.com/education/archive/2014/01/the-tricky-task-of-figuring-out-what-makes-a-mooc-successful/283274/ https://www.theatlantic.com/education/archive/2014/01/the-tr... https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263 "In these course[sic], “dropping out” is not a breach of expectations but the natural result of an open, free, and asynchronous registration process, where students get just as much as they wish out of a course and registering for a course does not imply a commitment to completing it." ... Also the discussion of the "Colbert Bump", where increased publicity to less-motivated (just curious) students doubled completion but tripled registration (thus reducing completion rate). Also the point about whether to allow or disallow registration after the course begins (late registrants are ineligible for certification, so they are automatically "failed"). "HarvardX could boost its certification rate by closing those courses to new registrants or restrict courses to those most likely to complete them. Instead, it keeps courses open to maximize the number of students who are learning something new." I absolutely agree that (at least some) users can be helped by live instruction or supplemented online instruction, but I disagree that commonly cited, low completion rates measure students with the "same level of motivation".
gliese1337 9y ago
As long which ones you send to each link is determined entirely randomly, you have indeed done a "scientific" (i.e., statistically valid) control.
- kbutler 9y ago
  That's one element. This first couple of things I'd check would be: - Double-blind: What if you tell students it is live, but it is actually pre-recorded? - Are the live+archive and archive-only videos the same? - Are the live+archive students actually watching the video when it is live, or are they using the archive feature? - Do archive-only students who consistently watch at the same time do better? - Does time/day of watching affect results? (archive-only students get the link at least [duration of class] later.) You also need to control other interactions with the participants, track the results faithfully, avoid bias because instructors are more familiar with students from "live" interactions, ensure cohort size and result size are large enough to be meaningful, that group composition isn't biased by confounding factors, etc. But random assignment is a good start!
randomdata 9y ago
'Mini code bootcamp' suggests that your target audience is people looking for careers? What are the end employment results like for all the participants? For instance, are MOOC students more likely to not complete the course because they have already found work in the meantime? Perhaps because they have more freedom to look for work during business hours, for example, rather than sitting in on a live class during the optimal employment search time? If the goal is to find work, there is no incentive to keep going in the course after you have met your goal.
- brudgers 9y ago
  The student handbook for the author's school suggests a graduate should expect to spend three to six months applying for five jobs a day. That's about 500 to a 1000 applications following six months of 45 hours a week at their online school.
  - ThrustVectoring 9y ago
    Five per day is a pretty low application rate for what it's worth. You can get through twenty in an hour if you're organized and motivated.
    - brudgers 9y ago
      That's thousands per month. See: https://www.joelonsoftware.com/2006/09/06/finding-great-developers-2/ https://www.joelonsoftware.com/2006/09/06/finding-great-deve...
- watwut 9y ago
  Classes for this crowd (older non-traditional students) tend to be in the evening or during weekend. Because they have jobs, education is not something they do while they are unemployed with intention to drop it the moment they find jobs. Through, major reason why they drop out is that handling work, family and school and harder, more demotivating and more tiresome then handling just school. Beyond broadening access, if you have to be somewhere in person you have to make it priority at that time - and you are not interrupted while you are there. In home study is very easily interrupted and easy to postpone due to other very real duties until it is too late. The motivation hit from meeting other students is also missing.
brudgers 9y ago
What studies would you cite for someone interested in learning more about the field?
- austenallred 9y ago
  I'll dig some of them up later today, but it will be a bit as I need to finish some stuff.

kbutler 9y ago

This Harvard study & Atlantic article by the study authors give the same 5% completion rate you cited, but give a completely opposite interpretation: basically, that the MOOC broadened access, allowing more people to get what they wanted out of the course. https://www.theatlantic.com/education/archive/2014/01/the-tricky-task-of-figuring-out-what-makes-a-mooc-successful/283274/ https://www.theatlantic.com/education/archive/2014/01/the-tr... https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263 "In these course[sic], “dropping out” is not a breach of expectations but the natural result of an open, free, and asynchronous registration process, where students get just as much as they wish out of a course and registering for a course does not imply a commitment to completing it." ... Also the discussion of the "Colbert Bump", where increased publicity to less-motivated (just curious) students doubled completion but tripled registration (thus reducing completion rate). Also the point about whether to allow or disallow registration after the course begins (late registrants are ineligible for certification, so they are automatically "failed"). "HarvardX could boost its certification rate by closing those courses to new registrants or restrict courses to those most likely to complete them. Instead, it keeps courses open to maximize the number of students who are learning something new." I absolutely agree that (at least some) users can be helped by live instruction or supplemented online instruction, but I disagree that commonly cited, low completion rates measure students with the "same level of motivation".

gliese1337 9y ago

As long which ones you send to each link is determined entirely randomly, you have indeed done a "scientific" (i.e., statistically valid) control.

kbutler 9y ago
That's one element. This first couple of things I'd check would be: - Double-blind: What if you tell students it is live, but it is actually pre-recorded? - Are the live+archive and archive-only videos the same? - Are the live+archive students actually watching the video when it is live, or are they using the archive feature? - Do archive-only students who consistently watch at the same time do better? - Does time/day of watching affect results? (archive-only students get the link at least [duration of class] later.) You also need to control other interactions with the participants, track the results faithfully, avoid bias because instructors are more familiar with students from "live" interactions, ensure cohort size and result size are large enough to be meaningful, that group composition isn't biased by confounding factors, etc. But random assignment is a good start!

randomdata 9y ago

'Mini code bootcamp' suggests that your target audience is people looking for careers? What are the end employment results like for all the participants? For instance, are MOOC students more likely to not complete the course because they have already found work in the meantime? Perhaps because they have more freedom to look for work during business hours, for example, rather than sitting in on a live class during the optimal employment search time? If the goal is to find work, there is no incentive to keep going in the course after you have met your goal.

brudgers 9y ago
The student handbook for the author's school suggests a graduate should expect to spend three to six months applying for five jobs a day. That's about 500 to a 1000 applications following six months of 45 hours a week at their online school.
- ThrustVectoring 9y ago
  Five per day is a pretty low application rate for what it's worth. You can get through twenty in an hour if you're organized and motivated.
  - brudgers 9y ago
    That's thousands per month. See: https://www.joelonsoftware.com/2006/09/06/finding-great-developers-2/ https://www.joelonsoftware.com/2006/09/06/finding-great-deve...
watwut 9y ago
Classes for this crowd (older non-traditional students) tend to be in the evening or during weekend. Because they have jobs, education is not something they do while they are unemployed with intention to drop it the moment they find jobs. Through, major reason why they drop out is that handling work, family and school and harder, more demotivating and more tiresome then handling just school. Beyond broadening access, if you have to be somewhere in person you have to make it priority at that time - and you are not interrupted while you are there. In home study is very easily interrupted and easy to postpone due to other very real duties until it is too late. The motivation hit from meeting other students is also missing.

brudgers 9y ago

What studies would you cite for someone interested in learning more about the field?

austenallred 9y ago
I'll dig some of them up later today, but it will be a bit as I need to finish some stuff.