6 ms·

Not knocking your own choices, but its more the backlash to the 2013 "MOOCs are the greatest thing ever! KhanAcademy/EdX/Coursera! Wooo!" Personally, I love the

Not knocking your own choices, but its more the backlash to the 2013 "MOOCs are the greatest thing ever! KhanAcademy/EdX/Coursera! Wooo!" Personally, I love them and would love to work with those companies. In fact, when I started teaching, I used many of the aspects I saw MOOCs using in my traditional classrooms. However, there is a truth to the statement. Almost no one finishes a MOOC (some never even log in to take MOOCs they register for), and predictive models can determine whether someone will complete the MOOC as early as the 2nd week (if not sooner). Some people do grab what they want and leave, but that is not the case for everyone. If we then look to justify why bother, its a little hard to warrant hosting costs, etc. if you aren't seeing anything come from it. If I ran a MOOC with a 95% fail rate, how can I label it successful. We are unable to do meaningful research on MOOC strategy effectiveness because we are too busy trying to understand why people never log in in the first place.

astrodev 9y ago
> If we then look to justify why bother, its a little hard to warrant hosting costs, etc. if you aren't seeing anything come from it > If I ran a MOOC with a 95% fail rate, how can I label it successful. How is the completion rate a better measure of value added than the absolute number of finishers?
- austenallred 9y ago
  Both are important. There are multiple questions to be asked. For extremely self motivated people, MOOCS are a win, no doubt. Again, MOOCs are definitely better than no MOOCs. But the promise was that MOOCs can just replace college to a large degree, because the content you get in college will be online. In that instance the question becomes "do MOOCs work for most people?" For what we'll call the "average" person, studies have shown that just isn't the case. Most people don't learn well from MOOCs, unfortunately. Therefore what we're learning is that a non-trivial amount of what makes a college education successful is some combination of the following things that MOOCs don't have: external pressure, scheduled courses, due dates, a community of learners, a physical campus, etc. As such, I view part of the next step to figure out which of those aspects MOOCs are missing that are vital to the mix in order to help average people learn the things they need to know. Traditional education just says all of them are necessary, but I'm not convinced that's the case.
  - noobhacker 9y ago
    Could you cite the studies that say "MOOCs don't work for the average person"? While it is certainly an intuitive statement, I find it hard to empirically verify because the population of people who take MOOCs are very different from those who attend colleges. For example, those who take MOOCs are probably working full / part-time, trying to pick up some knowledge on the side. They will obviously have less time for learning than full-time students.
    - austenallred 9y ago
      Many are behind paywalls, but... let's just say there are ways around those. http://onlinelibrary.wiley.com/doi/10.1111/jcal.12131/full http://onlinelibrary.wiley.com/doi/10.1111/jcal.12131/full http://onlinelibrary.wiley.com/doi/10.1002/aehe.20018/full http://onlinelibrary.wiley.com/doi/10.1002/aehe.20018/full https://dl.acm.org/citation.cfm?id=2835842 https://dl.acm.org/citation.cfm?id=2835842 http://www.irrodl.org/index.php/irrodl/article/view/2112 http://www.irrodl.org/index.php/irrodl/article/view/2112 https://www.openeducationeuropa.eu/sites/default/files/legacy_files/asset/From-field_37_6.pdf https://www.openeducationeuropa.eu/sites/default/files/legac... http://www.educationaldatamining.org/EDM2014/uploads/procs2014/short%20papers/273_EDM-2014-Short.pdf http://www.educationaldatamining.org/EDM2014/uploads/procs20...
    - tsumnia 9y ago
      While "average person" is not the term I'd use, the question you're asking is a bit loaded. What measure is used to say a MOOC "works"? If we are saying completion, then high drop out rates are an issue. If we say high interactivity among those that don't drop out, this is where the current field of research in MOOCs is at. While I do not study it, colleagues in my lab are. My point is that quantifying the effectiveness of a MOOC is an ill-defined domain. Instead, we acknowledge the attrition rate and do analysis on those that pursue the course. Current research looks at system interactions, the social networks of MOOC forums, etc. to identify student behaviors that show higher "gains", be it course completion or grade. If a student leaves a MOOC, there is no way to identify why they leave. Likewise, unless asked, it is hard to identify why a student joins a MOOC. Students enrolled in a MOOC can come from a variety of culture, geographical, sociological, motivation, etc. backgrounds. As such, assuming traditional student behaviors is ill advised. However, in the process of learning, access to material is not enough to learning the material and additional effort is needed by the student to build the necessary mental models in their head. If a student drops out of a MOOC, it is hard for the runners/analysts of the MOOC to appropriately say the student learned the material. As austenallred mentions, current education research is looking at the motivation of learning, as well as what constitutes motivation (and learning for that matter).
dqpb 9y ago
>some never even log in to take MOOCs they register fo So what? How is this at all relevant? If we look at how many people have walked into a bookstore or library, have browsed through a book but never bothered to actually buy and read it, would we say books are a failure? Would you say books have a 95% fail rate? This is such a pointless train of thought.
- twblalock 9y ago
  Why did you decide that MOOCs should be held to the same standard as libraries or bookstores? MOOCs are marketed as courses, and sometimes as an alternative form of higher education. Furthermore, I doubt the people who go to the trouble of designing curricula for MOOCs expect students to drop out partway through. Comparing MOOCs to college courses seems like a better match to the image the MOOC companies themselves have promoted. And if we make that comparison, completion definitely matters.
  - deleted 9y ago
    [deleted]
  - dqpb 9y ago
    The fundamental difference is that MOOCs are recorded rather than performed live. This makes both the distribution and access patterns more similar to books than to college courses. Of course any particular MOOC can be made more or less like a normal class by putting the content behind a hefty paywall, enforcing strict due dates, and adding perverse performance incentives.
artimaeis 9y ago
> Some people do grab what they want and leave, but that is not the case for everyone. Are there metrics that have measured this? What percentage of users find a MOOC useful even if it wasn't completed? What I was trying to get at is that by looking at completion rates we might not be looking at the whole picture. Especially if the question we're revolving around is whether the course is worth the creation/hosting costs. > some never even log in to take MOOCs they register for [...] > we are too busy trying to understand why people never log in in the first place So you have at least 4 classes of 'user': - did not register - registered but never logged in again - logged in at least once - completed course It seems to me that there's a different way to categorize the users in measuring course effectiveness: - did not find the course useful/helpful - did find the course useful/helpful So if you are able to craft a course that has magically high engagement, but people don't find it particularly useful - you've still failed. Maybe the value that course-makers hope to deliver through completion of the course simply isn't high enough? If a user completes 70% of a course but doesn't see the need for the rest of it, then stopping at that point is the logical move. What sets college/university apart in this is degree which has a very high social impact. The fact remains that MOOC achievements are still widely looked down upon as somehow lesser than their college equivalent. This whole discourse has encouraged me to revisit those courses I found most effective and complete them. At the very least I can help those courses stats and provide some further narrative of their utility.
- tsumnia 9y ago
  A major issue with using "useful/helpful-ness" as our metric for success is quantifying the term. A self-report Likert scale is susceptible to users just rating 1 or 7, or saying its useful even when it was not (https://en.wikipedia.org/wiki/Self-report_study#Disadvantages https://en.wikipedia.org/wiki/Self-report_study#Disadvantage..., not the greatest citation, but I am short for time). The issue then looks at what is "success" in a MOOC. If the goal is to just have videos online, I'll just watch YouTube (as I have my own lecture series there). However, observing is considered one of the lower level of learning and things like Bloom's taxonomy point out there needs to be some type of interaction for better learning gains. These interactions require the student to have a more active role in the learning process. If they are not interacting with the system (logging in, watching videos, completing exercises, etc.), then they are not taking this needed active role. This is where "drop out" begins to be quantified and where then we can measure what worked, what didn't. To address you example, if the user stops at 70%, researchers will ask "why?" From there, analysis of student behaviors, effectiveness of interface/instruction/material, etc. will arise. If the ability to study these things is confounded by the fact that the vast majority quit before completion, it makes it harder to answer the "Why" and "How do we fix it" questions. Again, what we quantify as "successful" is still up for debate; but if a student drops out, it becomes harder to tell if they learned from the course and if it was student or material that drove that decision.
ghaff 9y ago
Given the nearly non-existent barrier to entry, I'm not sure that completion/engagement percentages mean much. MOOC content can be useful for help learning certain types of things. But so can books, YouTube, etc. Useful resources for (mostly) continuing professional education has to be seen as a very large step back from changing the future of education for underserved populations.
Chathamization 9y ago
> some never even log in to take MOOCs they register for I've done this many times. Why? On some platforms (like Coursera) you retain access to the content of a course you have registered for, but that content isn't available for people who didn't register once the class is over. If I think I might ever want to view that content, my best bet is to register (there's no cost) and then have that content available when I want it. I've also done this on platforms like Udacity where that's not the issue, because by registering I can create a curated list of courses I'm interested in possibly looking at in the future. In my experience, the big failings of MOOCs is that they try to copy university classes, and university classes just aren't that good a lot of the time. Many people don't find lectures to be useful at all, but the vast majority of MOOCs focus on them. After the fifth time I'm stuck waiting for the teacher to finish their long personal anecdote or humorous story I usually give up on the MOOC and go looking for a good textbook. I don't think I've ever seen a MOOC that had written content anywhere close to the quality of Dive Into Python or Learn C the Hard Way. Also, MOOC efforts for community building are (at least were) focused on a particular class rather than a particular subject. This doesn't lend itself well to long-term community building, especially when people are going through these courses and very different paces. There seems to be lots of discussion about how MOOCs can be improved, but not much effort into actually trying different approaches.
- tsumnia 9y ago
  > the big failings of MOOCs is that they try to copy university classes, and university classes just aren't that good a lot of the time We are still trying to understand what makes for better "learning". If I practice the piano everyday, I will no doubt have a better ability to play, but I may not understand music theory. As a shameless plug, I am trying to add practice to be a core part of learning Computer Science and a link to my research platform is in my profile. This is the same separate of vocational schools and coding bootcamps to graduate schools. The traditional 4-year school sits in the middle ground between application and theory and is probably why we have such difficulty deciding "success". > to finish their long personal anecdote or humorous story I usually give up on the MOOC and go looking for a good textbook This is ultimately student preference. Some students like a more affable teacher, some don't want a social-able instructor (so long as its not interfering). I will say that your post is confusing as you say you don't want instructor anecdotes but then MOOCs should be doing more for larger-scope community building. I will argue larger-scope community building is difficult and MOOCs struggle with it; I would say because of its availability. It is harder to build community when the individuals come from such diverse backgrounds. Many factors can make it difficult to build bounds (social, cultural, temporal, etc.). Furthermore, you cannot control why the student wants to take a MOOC. They may not want to give that additional effort to community building for a subject they do not perceive as having higher priority to other life decisions. I would disagree with "actually trying different approaches". This is being done and academic journals are trying to study them (the deadline for the Journal for Education Data Mining just passed). There are also non-academic approaches like Duolingo and Khan Academy, and they are trying different techniques as well. Ultimately, there is no panacea for learning. We know engagement can lead to learning and therefore if we can maintain engagement, we might be able to teach. Community-building can be that engagement. Self motivation can be. At that point we need to research these things more to know for sure.
andreykurenkov 9y ago
I suspect something like Udacity (on which classes rae not free and there are more structures 'degree' programs) has far higher completion rates, since it's not reasonable to just impulsively sign up and then stop. That is certainly true for something like Georgia Tech's OMCS. I still think the hype was justified in that stuff like OMCS is legitimately the future of education.
John_KZ 9y ago
Why would you complete a MOOC "cover to cover" when they offer no accreditation? People just pick up what they need and move on. If they were offering a complete program and degrees, millions would complete them. MOOCs are amazing, but a simple VOD system can't replace the education system on it's own.
derefr 9y ago
> some never even log in to take MOOCs they register for > If we then look to justify why bother, its a little hard to warrant hosting costs, etc. Presumably the people who never log in aren't imposing any load on the system, no? Just rescale to 2% of the size your user-metrics say you need.

astrodev 9y ago

> If we then look to justify why bother, its a little hard to warrant hosting costs, etc. if you aren't seeing anything come from it > If I ran a MOOC with a 95% fail rate, how can I label it successful. How is the completion rate a better measure of value added than the absolute number of finishers?

austenallred 9y ago
Both are important. There are multiple questions to be asked. For extremely self motivated people, MOOCS are a win, no doubt. Again, MOOCs are definitely better than no MOOCs. But the promise was that MOOCs can just replace college to a large degree, because the content you get in college will be online. In that instance the question becomes "do MOOCs work for most people?" For what we'll call the "average" person, studies have shown that just isn't the case. Most people don't learn well from MOOCs, unfortunately. Therefore what we're learning is that a non-trivial amount of what makes a college education successful is some combination of the following things that MOOCs don't have: external pressure, scheduled courses, due dates, a community of learners, a physical campus, etc. As such, I view part of the next step to figure out which of those aspects MOOCs are missing that are vital to the mix in order to help average people learn the things they need to know. Traditional education just says all of them are necessary, but I'm not convinced that's the case.
- noobhacker 9y ago
  Could you cite the studies that say "MOOCs don't work for the average person"? While it is certainly an intuitive statement, I find it hard to empirically verify because the population of people who take MOOCs are very different from those who attend colleges. For example, those who take MOOCs are probably working full / part-time, trying to pick up some knowledge on the side. They will obviously have less time for learning than full-time students.
  - austenallred 9y ago
    Many are behind paywalls, but... let's just say there are ways around those. http://onlinelibrary.wiley.com/doi/10.1111/jcal.12131/full http://onlinelibrary.wiley.com/doi/10.1111/jcal.12131/full http://onlinelibrary.wiley.com/doi/10.1002/aehe.20018/full http://onlinelibrary.wiley.com/doi/10.1002/aehe.20018/full https://dl.acm.org/citation.cfm?id=2835842 https://dl.acm.org/citation.cfm?id=2835842 http://www.irrodl.org/index.php/irrodl/article/view/2112 http://www.irrodl.org/index.php/irrodl/article/view/2112 https://www.openeducationeuropa.eu/sites/default/files/legacy_files/asset/From-field_37_6.pdf https://www.openeducationeuropa.eu/sites/default/files/legac... http://www.educationaldatamining.org/EDM2014/uploads/procs2014/short%20papers/273_EDM-2014-Short.pdf http://www.educationaldatamining.org/EDM2014/uploads/procs20...
  - tsumnia 9y ago
    While "average person" is not the term I'd use, the question you're asking is a bit loaded. What measure is used to say a MOOC "works"? If we are saying completion, then high drop out rates are an issue. If we say high interactivity among those that don't drop out, this is where the current field of research in MOOCs is at. While I do not study it, colleagues in my lab are. My point is that quantifying the effectiveness of a MOOC is an ill-defined domain. Instead, we acknowledge the attrition rate and do analysis on those that pursue the course. Current research looks at system interactions, the social networks of MOOC forums, etc. to identify student behaviors that show higher "gains", be it course completion or grade. If a student leaves a MOOC, there is no way to identify why they leave. Likewise, unless asked, it is hard to identify why a student joins a MOOC. Students enrolled in a MOOC can come from a variety of culture, geographical, sociological, motivation, etc. backgrounds. As such, assuming traditional student behaviors is ill advised. However, in the process of learning, access to material is not enough to learning the material and additional effort is needed by the student to build the necessary mental models in their head. If a student drops out of a MOOC, it is hard for the runners/analysts of the MOOC to appropriately say the student learned the material. As austenallred mentions, current education research is looking at the motivation of learning, as well as what constitutes motivation (and learning for that matter).

dqpb 9y ago

>some never even log in to take MOOCs they register fo So what? How is this at all relevant? If we look at how many people have walked into a bookstore or library, have browsed through a book but never bothered to actually buy and read it, would we say books are a failure? Would you say books have a 95% fail rate? This is such a pointless train of thought.

twblalock 9y ago
Why did you decide that MOOCs should be held to the same standard as libraries or bookstores? MOOCs are marketed as courses, and sometimes as an alternative form of higher education. Furthermore, I doubt the people who go to the trouble of designing curricula for MOOCs expect students to drop out partway through. Comparing MOOCs to college courses seems like a better match to the image the MOOC companies themselves have promoted. And if we make that comparison, completion definitely matters.
- deleted 9y ago
  [deleted]
- dqpb 9y ago
  The fundamental difference is that MOOCs are recorded rather than performed live. This makes both the distribution and access patterns more similar to books than to college courses. Of course any particular MOOC can be made more or less like a normal class by putting the content behind a hefty paywall, enforcing strict due dates, and adding perverse performance incentives.

artimaeis 9y ago

> Some people do grab what they want and leave, but that is not the case for everyone. Are there metrics that have measured this? What percentage of users find a MOOC useful even if it wasn't completed? What I was trying to get at is that by looking at completion rates we might not be looking at the whole picture. Especially if the question we're revolving around is whether the course is worth the creation/hosting costs. > some never even log in to take MOOCs they register for [...] > we are too busy trying to understand why people never log in in the first place So you have at least 4 classes of 'user': - did not register - registered but never logged in again - logged in at least once - completed course It seems to me that there's a different way to categorize the users in measuring course effectiveness: - did not find the course useful/helpful - did find the course useful/helpful So if you are able to craft a course that has magically high engagement, but people don't find it particularly useful - you've still failed. Maybe the value that course-makers hope to deliver through completion of the course simply isn't high enough? If a user completes 70% of a course but doesn't see the need for the rest of it, then stopping at that point is the logical move. What sets college/university apart in this is degree which has a very high social impact. The fact remains that MOOC achievements are still widely looked down upon as somehow lesser than their college equivalent. This whole discourse has encouraged me to revisit those courses I found most effective and complete them. At the very least I can help those courses stats and provide some further narrative of their utility.

tsumnia 9y ago
A major issue with using "useful/helpful-ness" as our metric for success is quantifying the term. A self-report Likert scale is susceptible to users just rating 1 or 7, or saying its useful even when it was not (https://en.wikipedia.org/wiki/Self-report_study#Disadvantages https://en.wikipedia.org/wiki/Self-report_study#Disadvantage..., not the greatest citation, but I am short for time). The issue then looks at what is "success" in a MOOC. If the goal is to just have videos online, I'll just watch YouTube (as I have my own lecture series there). However, observing is considered one of the lower level of learning and things like Bloom's taxonomy point out there needs to be some type of interaction for better learning gains. These interactions require the student to have a more active role in the learning process. If they are not interacting with the system (logging in, watching videos, completing exercises, etc.), then they are not taking this needed active role. This is where "drop out" begins to be quantified and where then we can measure what worked, what didn't. To address you example, if the user stops at 70%, researchers will ask "why?" From there, analysis of student behaviors, effectiveness of interface/instruction/material, etc. will arise. If the ability to study these things is confounded by the fact that the vast majority quit before completion, it makes it harder to answer the "Why" and "How do we fix it" questions. Again, what we quantify as "successful" is still up for debate; but if a student drops out, it becomes harder to tell if they learned from the course and if it was student or material that drove that decision.

ghaff 9y ago

Given the nearly non-existent barrier to entry, I'm not sure that completion/engagement percentages mean much. MOOC content can be useful for help learning certain types of things. But so can books, YouTube, etc. Useful resources for (mostly) continuing professional education has to be seen as a very large step back from changing the future of education for underserved populations.

Chathamization 9y ago

> some never even log in to take MOOCs they register for I've done this many times. Why? On some platforms (like Coursera) you retain access to the content of a course you have registered for, but that content isn't available for people who didn't register once the class is over. If I think I might ever want to view that content, my best bet is to register (there's no cost) and then have that content available when I want it. I've also done this on platforms like Udacity where that's not the issue, because by registering I can create a curated list of courses I'm interested in possibly looking at in the future. In my experience, the big failings of MOOCs is that they try to copy university classes, and university classes just aren't that good a lot of the time. Many people don't find lectures to be useful at all, but the vast majority of MOOCs focus on them. After the fifth time I'm stuck waiting for the teacher to finish their long personal anecdote or humorous story I usually give up on the MOOC and go looking for a good textbook. I don't think I've ever seen a MOOC that had written content anywhere close to the quality of Dive Into Python or Learn C the Hard Way. Also, MOOC efforts for community building are (at least were) focused on a particular class rather than a particular subject. This doesn't lend itself well to long-term community building, especially when people are going through these courses and very different paces. There seems to be lots of discussion about how MOOCs can be improved, but not much effort into actually trying different approaches.

tsumnia 9y ago
> the big failings of MOOCs is that they try to copy university classes, and university classes just aren't that good a lot of the time We are still trying to understand what makes for better "learning". If I practice the piano everyday, I will no doubt have a better ability to play, but I may not understand music theory. As a shameless plug, I am trying to add practice to be a core part of learning Computer Science and a link to my research platform is in my profile. This is the same separate of vocational schools and coding bootcamps to graduate schools. The traditional 4-year school sits in the middle ground between application and theory and is probably why we have such difficulty deciding "success". > to finish their long personal anecdote or humorous story I usually give up on the MOOC and go looking for a good textbook This is ultimately student preference. Some students like a more affable teacher, some don't want a social-able instructor (so long as its not interfering). I will say that your post is confusing as you say you don't want instructor anecdotes but then MOOCs should be doing more for larger-scope community building. I will argue larger-scope community building is difficult and MOOCs struggle with it; I would say because of its availability. It is harder to build community when the individuals come from such diverse backgrounds. Many factors can make it difficult to build bounds (social, cultural, temporal, etc.). Furthermore, you cannot control why the student wants to take a MOOC. They may not want to give that additional effort to community building for a subject they do not perceive as having higher priority to other life decisions. I would disagree with "actually trying different approaches". This is being done and academic journals are trying to study them (the deadline for the Journal for Education Data Mining just passed). There are also non-academic approaches like Duolingo and Khan Academy, and they are trying different techniques as well. Ultimately, there is no panacea for learning. We know engagement can lead to learning and therefore if we can maintain engagement, we might be able to teach. Community-building can be that engagement. Self motivation can be. At that point we need to research these things more to know for sure.

andreykurenkov 9y ago

I suspect something like Udacity (on which classes rae not free and there are more structures 'degree' programs) has far higher completion rates, since it's not reasonable to just impulsively sign up and then stop. That is certainly true for something like Georgia Tech's OMCS. I still think the hype was justified in that stuff like OMCS is legitimately the future of education.

John_KZ 9y ago

Why would you complete a MOOC "cover to cover" when they offer no accreditation? People just pick up what they need and move on. If they were offering a complete program and degrees, millions would complete them. MOOCs are amazing, but a simple VOD system can't replace the education system on it's own.

derefr 9y ago

> some never even log in to take MOOCs they register for > If we then look to justify why bother, its a little hard to warrant hosting costs, etc. Presumably the people who never log in aren't imposing any load on the system, no? Just rescale to 2% of the size your user-metrics say you need.