10 ms·

Coursera-dl – A script for downloading course material from coursera.org

184 points by carlosgg 13y ago

emmelaich 13y ago
I have had mostly good success using wget and a cookie.txt file exported from chrome with https://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh?hl=en https://chrome.google.com/webstore/detail/cookietxt-export/l... But it's not so convenient. I'll have to try this in future.
- bloometal 13y ago
  This is awesome. But, does anyone know how to do this with `--save-cookies` command with wget?
  - codygman 13y ago
    I remember crafting something similar along the lines of this (incorrect argument names almost assuredly): curl $url_that_has_cookies -c - That will save the cookies to stdout, and IIRC you can pipe them to curl. I've done similar stuff with wget along the lines of just saving the cookies and then loading them within a single command.
- joshmn 13y ago
  Even better, Chrome's developer options allows you to right click in the HAR and do a "Copy as CURL" which includes all the bells and whistles (headers, cookies) to throw it into terminal and start hacking away.
hdx 13y ago
lol, I wrote a little Javascript thingy to do that for me, but I'll try this for sure. I hate that they don't keep the course materials open indefinitely... and I don't get it either.
- hayksaakian 13y ago
  i presume their goal is to make $
  - psbp 13y ago
    Their only current source of revenue is through a signature track tier where you get a certificate that is verifiable with your identity. I doubt it will stay that way indefinitely, but for now the course material is provided without charge.
    - ConAntonakos 13y ago
      I definitely foresee a subscription-type model or a la carte via the signature track model. Either way, I think anyone would be happy to pay for this awesome education.
      - ghaff 13y ago
        I think you'd be surprised. If you look at the stats, there's a lot of professional-level (BS/Masters/PhD degree holders) as well as a lot of students from (broadly) the developing world. I'd submit that both those groups are probably pretty price sensitive--especially the latter, many of which don't even have credit cards in many cases. Would "adult ed" students be willing to pay some nominal amount--say $25-$50 for a course? Probably yes in some cases but remember the pricing discussions that take place here all the time. Getting from free to paid (in any amount) is a big barrier to get people over. Arguably the case would be different in certifications actually meant something. After all, people and their companies spend lots of $$ on various software certifications. But that's a whole other topic.
        psbp 13y ago
        It isn't a huge barrier if the courses are as legitimate as actual university courses. As for the pricing question, the price of a can of coca-cola isn't the same in the U.S. as it is in Sudan. Once these MOOCS establish a brand, it will be easy for them to adjusts their prices to maximize profits.
calcsam 13y ago
Works great. Each Coursera class takes about 1GB of HD space. Will watch the videos on my BART commute.
- JabavuAdams 13y ago
  Yup. Subway time is super-powers time.
narrator 13y ago
Most of my free time these days is spent watching Coursera videos. It is an absolute revolution for those of us who love to learn. There is so much material though and I don't have enough time!
- psbp 13y ago
  The quality and volume is truly incredible. There are at least 4 classes that I'm interested in at any given time, and that's on coursera alone (not including udacity, edx, etc.). I wonder how much longer these services are going to stay free. I like to download all the content just to be sure that it's not taken away when they eventually go to paid models.
  - B-Con 13y ago
    > I wonder how much longer these services are going to stay free. I like to download all the content just to be sure that it's not taken away when they eventually go to paid models. I also download everything. I'd hate to want to reference a lecture later only to find it missing. Last I heard, Coursera was still searching to find a revenue model. I've noticed their "resume" section expand, and you can now opt in (or is it out, by default?) to job hunter searches. Which makes me curious as to what Coursera's revenue-generating product is going to be. Is it the database of people with certified(-ish) skills, or will it be the content they provide?
    - psbp 13y ago
      They're probably trying to find as many ways as possible to embed themselves into relevant industries before they aggressively monetize. I bet the end goal (for all these MOOCS) is to become the de facto source for web based courses offered, for pay, through the current university system. You're right, they're also trying to become a recruitment tool for various industries, but they'll make the most money by switching on the revenue steam once universities become dependent on them and there's no turning back.
    - mkartic 13y ago
      why can't they charge the universities instead to put up their own courses? It'd be a good advertisement. Like how everyone knows about Strang's linal courses. The complexity of the courses are limited by the number of people attending it, so anyway, things can get only so much advanced. And use it as a platform for getting into more advanced courses in physical classrooms.
qntmfred 13y ago
Here is something similar for Udacity https://github.com/nzmsv/udacity-dl https://github.com/nzmsv/udacity-dl but hey who needs that when there is https://www.udacity.com/wiki/downloads https://www.udacity.com/wiki/downloads
- nzmsv 13y ago
  I wrote the extension for Udacity before they offered downloads. I agree that it is pretty useless now. I should see if it even works still, thanks for reminding me :) I should probably put a note to that effect in the readme. I wrote this thing in one evening because I prefer to watch these videos on a big screen. What surprised me was how popular this quick hack became. There were people using it to download videos and share them with others in countries that blocked YouTube. I think at its peak there were a few thousand installs.
dfc 13y ago
If you use Firefox you can just use DownThemAll. To be honest DTA makes downloading anything and everything easier.
- shitlord 13y ago
  +1 for DTA If DTA can't find the link to the content for some reason (it happened to me on youtube a while ago), you can use an addon like LiveHTTPHeaders to sniff HTTP requests and figure out the link for the video. edit: I forgot to mention that DTA will lag the entire browser like hell and kill your battery, so only use it when plugged in. This happens on Windows + Linux, and idk about OSX.
  - joshmn 13y ago
    You can also use developer tools and view the requests coming in, look at the obvious file type (usually flv/mp4) and grab the URL from there.
  - dfc 13y ago
    Nice tip. I never knew DTA would work on youtube. I have always used youtube-dl.
  - dfc 13y ago
    I just saw your edit. In case you ever stumble back across this checkout the beta versions of DTA. The beta versions have been greatly improved. DTA nevers locks up the browser for me. Its an incredible improvement.
    - shitlord 13y ago
      Just tried it, and the performance is awesome! I do miss seeing the obnoxious "donate" banner on the top right though. Thanks for the tip.
- voodooChile 13y ago
  DTA fails on download for many of the courses actually. Something about "login" keep popping up.
deleted 13y ago
[deleted]
ekianjo 13y ago
This is not very new. This coursera-dl script has been around for like... a year or something ? ANyway, I have been using it and it works quite well but there are some classes where compatibility problems occur and not all material is downloaded at once. Now the next step is for someone to make a GUI for it.
plg 13y ago
I just wish one didn't have to log in with an account. If they want to give this stuff away, great! ... but please why are you making me give you my info? Even fake info
- cma 13y ago
  They obviously want to give it away for your info. Even fake info =P
larsonf 13y ago
Awesome, although I don't know why you couldn't have it automatically accept the 'honor code.' But then again, perhaps if you're using this instead of just rolling your own solution you might not really need/want that sort of automation in the first place. Also, and more importantly, isn't it a bit strange that there needs to be tool like this at all? Is it still going on where Coursera pulls old course material off when the course is finished? If it is, can we have a discussion on that?
uniclaude 13y ago
I find it too bad that we can't simply create some mirror for the material and have them available indefinitely on a crowdfunded s3 or something similar. All of this because of the terms of use. It feels like duplication of "effort" to me. Anyways, that's a great thing to be able to download simply.
sakai 13y ago
Anyone able to get scicomp-001 to work? I get this warning: "Warning: no downloadable content found for scicomp-001, did you accept the honour code?" (I have)
- ivoflipse 13y ago
  Probably University of Washington not allowing downloading of their material. Or you should just try it again a bit later.
  - sakai 13y ago
    Hmm, I'm not that familiar with BeautifulSoup, but it appears to not properly be parsing the page, rather than a download / availability issue. Specifically, there's no content in the soup object ~line 150 and thus the list of content to parse and download is just []. Will keep probing, but if anybody has experienced something similar and has a handy solution, I'd greatly appreciate the tip! (And I'm sure the author would appreciate a PR)
- haakon 13y ago
  I get the same thing, for humankind-001. Oh well.
seanhandley 13y ago
Doesn't seem to currently work for old courses: https://github.com/dgorissen/coursera-dl/issues/72 https://github.com/dgorissen/coursera-dl/issues/72
carlosgg 13y ago
Seems to work great over here, and the installation was pretty easy, too. You can even choose not to download certain types of files using the -n option. For example, if you have a large hard drive and a smaller one, you can download the whole course to the large HD: coursera-dl -u username -p password -d pathToLargeHD course_name and only download pdf lecture notes to the smaller one coursera-dl -u username -p password -d pathToSmallHD -n mp4,pptx course_name I tried that over here, worked great. Some schools prefer students don't download course materials. I succesfully downloaded Machine Learning and Algorithms courses from Stanford but could not download this one, it says "now downloadable content found": https://www.coursera.org/course/datasci https://www.coursera.org/course/datasci
- carlosgg 13y ago
  After upgrading to latest version of script, I was able to download this one, too.
ambler0 13y ago
This violates Coursera's terms of service. The relevant language: "...as a condition of accessing the Sites, you agree not to...(c) use any high-volume, automated or electronic means to access the Sites (including without limitation, robots, spiders, scripts or web-scraping tools)" https://www.coursera.org/about/terms https://www.coursera.org/about/terms I was actually going to write such a script myself as an exercise (I'm new to programming), but this language dissuaded me. EDIT: Could someone please explain to me why this simple statement of fact would be downvoted?
- idm 13y ago
  This script would appear to be at odds with the language of that clause, but I wonder if it is truly against the spirit of Coursera. I can see clear ways to automatically exploit the website for the purpose of scraping their content. However, I have had a legitimate use for this in the past and, instead of just doing the smart automatic thing, I did it manually. My take is this: for personal use, a low-volume, automated tool would not violate the spirit of online education. The presence of that clause is probably intended to protect against exploitative uses.
  - ambler0 13y ago
    I had a similar take, especially since we all know that terms of service are written very broadly. But on the other hand, I would really hate to mess up my Coursera account. After posting my initial comment, I saw further down that HN user pamelafox is a former Coursera employee. I would love to know whether she or anyone else has insight into how Coursera views tools like this.
- carlosgg 13y ago
  I upvoted it. :) Good catch! I did not know that! I really hope they mean that clause for people that would try to frivolously download 200 courses or something, just to have them...I figure that would tax their servers and cost money because of the bandwidth being consumed. Before I knew about coursera-dl, I tried to download one of the courses by hand, and it's BEYOND TEDIOUS. So far, I have downloaded 2 courses with coursera-dl, both of which I have taken and submitted lots of homework to. I think it's ok in that case...
Zoomla 13y ago
Anyone saved material from Jeff Leek's last Data Analysis class? it seemed like a good class. It was in Jan. 2013 and it's coming back in Oct. 2013...
carlosgg 13y ago
Quick FYI, there might be a problem if one tries to run Firefox concurrently: https://github.com/dgorissen/coursera-dl/issues/81 https://github.com/dgorissen/coursera-dl/issues/81

184 points by carlosgg 13y ago

I have had mostly good success using wget and a cookie.txt file exported from chrome with https://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh?hl=en https://chrome.google.com/webstore/detail/cookietxt-export/l... But it's not so convenient. I'll have to try this in future.

bloometal 13y ago
This is awesome. But, does anyone know how to do this with `--save-cookies` command with wget?
- codygman 13y ago
  I remember crafting something similar along the lines of this (incorrect argument names almost assuredly): curl $url_that_has_cookies -c - That will save the cookies to stdout, and IIRC you can pipe them to curl. I've done similar stuff with wget along the lines of just saving the cookies and then loading them within a single command.
joshmn 13y ago
Even better, Chrome's developer options allows you to right click in the HAR and do a "Copy as CURL" which includes all the bells and whistles (headers, cookies) to throw it into terminal and start hacking away.

hdx 13y ago

lol, I wrote a little Javascript thingy to do that for me, but I'll try this for sure. I hate that they don't keep the course materials open indefinitely... and I don't get it either.

hayksaakian 13y ago
i presume their goal is to make $
- psbp 13y ago
  Their only current source of revenue is through a signature track tier where you get a certificate that is verifiable with your identity. I doubt it will stay that way indefinitely, but for now the course material is provided without charge.
  - ConAntonakos 13y ago
    I definitely foresee a subscription-type model or a la carte via the signature track model. Either way, I think anyone would be happy to pay for this awesome education.
    - ghaff 13y ago
      I think you'd be surprised. If you look at the stats, there's a lot of professional-level (BS/Masters/PhD degree holders) as well as a lot of students from (broadly) the developing world. I'd submit that both those groups are probably pretty price sensitive--especially the latter, many of which don't even have credit cards in many cases. Would "adult ed" students be willing to pay some nominal amount--say $25-$50 for a course? Probably yes in some cases but remember the pricing discussions that take place here all the time. Getting from free to paid (in any amount) is a big barrier to get people over. Arguably the case would be different in certifications actually meant something. After all, people and their companies spend lots of $$ on various software certifications. But that's a whole other topic.
      - psbp 13y ago
        It isn't a huge barrier if the courses are as legitimate as actual university courses. As for the pricing question, the price of a can of coca-cola isn't the same in the U.S. as it is in Sudan. Once these MOOCS establish a brand, it will be easy for them to adjusts their prices to maximize profits.

calcsam 13y ago

Works great. Each Coursera class takes about 1GB of HD space. Will watch the videos on my BART commute.

JabavuAdams 13y ago
Yup. Subway time is super-powers time.

narrator 13y ago

Most of my free time these days is spent watching Coursera videos. It is an absolute revolution for those of us who love to learn. There is so much material though and I don't have enough time!

psbp 13y ago
The quality and volume is truly incredible. There are at least 4 classes that I'm interested in at any given time, and that's on coursera alone (not including udacity, edx, etc.). I wonder how much longer these services are going to stay free. I like to download all the content just to be sure that it's not taken away when they eventually go to paid models.
- B-Con 13y ago
  > I wonder how much longer these services are going to stay free. I like to download all the content just to be sure that it's not taken away when they eventually go to paid models. I also download everything. I'd hate to want to reference a lecture later only to find it missing. Last I heard, Coursera was still searching to find a revenue model. I've noticed their "resume" section expand, and you can now opt in (or is it out, by default?) to job hunter searches. Which makes me curious as to what Coursera's revenue-generating product is going to be. Is it the database of people with certified(-ish) skills, or will it be the content they provide?
  - psbp 13y ago
    They're probably trying to find as many ways as possible to embed themselves into relevant industries before they aggressively monetize. I bet the end goal (for all these MOOCS) is to become the de facto source for web based courses offered, for pay, through the current university system. You're right, they're also trying to become a recruitment tool for various industries, but they'll make the most money by switching on the revenue steam once universities become dependent on them and there's no turning back.
  - mkartic 13y ago
    why can't they charge the universities instead to put up their own courses? It'd be a good advertisement. Like how everyone knows about Strang's linal courses. The complexity of the courses are limited by the number of people attending it, so anyway, things can get only so much advanced. And use it as a platform for getting into more advanced courses in physical classrooms.

qntmfred 13y ago

Here is something similar for Udacity https://github.com/nzmsv/udacity-dl https://github.com/nzmsv/udacity-dl but hey who needs that when there is https://www.udacity.com/wiki/downloads https://www.udacity.com/wiki/downloads

nzmsv 13y ago
I wrote the extension for Udacity before they offered downloads. I agree that it is pretty useless now. I should see if it even works still, thanks for reminding me :) I should probably put a note to that effect in the readme. I wrote this thing in one evening because I prefer to watch these videos on a big screen. What surprised me was how popular this quick hack became. There were people using it to download videos and share them with others in countries that blocked YouTube. I think at its peak there were a few thousand installs.

dfc 13y ago

If you use Firefox you can just use DownThemAll. To be honest DTA makes downloading anything and everything easier.

shitlord 13y ago
+1 for DTA If DTA can't find the link to the content for some reason (it happened to me on youtube a while ago), you can use an addon like LiveHTTPHeaders to sniff HTTP requests and figure out the link for the video. edit: I forgot to mention that DTA will lag the entire browser like hell and kill your battery, so only use it when plugged in. This happens on Windows + Linux, and idk about OSX.
- joshmn 13y ago
  You can also use developer tools and view the requests coming in, look at the obvious file type (usually flv/mp4) and grab the URL from there.
- dfc 13y ago
  Nice tip. I never knew DTA would work on youtube. I have always used youtube-dl.
- dfc 13y ago
  I just saw your edit. In case you ever stumble back across this checkout the beta versions of DTA. The beta versions have been greatly improved. DTA nevers locks up the browser for me. Its an incredible improvement.
  - shitlord 13y ago
    Just tried it, and the performance is awesome! I do miss seeing the obnoxious "donate" banner on the top right though. Thanks for the tip.
voodooChile 13y ago
DTA fails on download for many of the courses actually. Something about "login" keep popping up.

deleted 13y ago

[deleted]

ekianjo 13y ago

This is not very new. This coursera-dl script has been around for like... a year or something ? ANyway, I have been using it and it works quite well but there are some classes where compatibility problems occur and not all material is downloaded at once. Now the next step is for someone to make a GUI for it.

plg 13y ago

I just wish one didn't have to log in with an account. If they want to give this stuff away, great! ... but please why are you making me give you my info? Even fake info

cma 13y ago
They obviously want to give it away for your info. Even fake info =P

larsonf 13y ago

Awesome, although I don't know why you couldn't have it automatically accept the 'honor code.' But then again, perhaps if you're using this instead of just rolling your own solution you might not really need/want that sort of automation in the first place. Also, and more importantly, isn't it a bit strange that there needs to be tool like this at all? Is it still going on where Coursera pulls old course material off when the course is finished? If it is, can we have a discussion on that?

uniclaude 13y ago

I find it too bad that we can't simply create some mirror for the material and have them available indefinitely on a crowdfunded s3 or something similar. All of this because of the terms of use. It feels like duplication of "effort" to me. Anyways, that's a great thing to be able to download simply.

sakai 13y ago

Anyone able to get scicomp-001 to work? I get this warning: "Warning: no downloadable content found for scicomp-001, did you accept the honour code?" (I have)

ivoflipse 13y ago
Probably University of Washington not allowing downloading of their material. Or you should just try it again a bit later.
- sakai 13y ago
  Hmm, I'm not that familiar with BeautifulSoup, but it appears to not properly be parsing the page, rather than a download / availability issue. Specifically, there's no content in the soup object ~line 150 and thus the list of content to parse and download is just []. Will keep probing, but if anybody has experienced something similar and has a handy solution, I'd greatly appreciate the tip! (And I'm sure the author would appreciate a PR)
haakon 13y ago
I get the same thing, for humankind-001. Oh well.

seanhandley 13y ago

Doesn't seem to currently work for old courses: https://github.com/dgorissen/coursera-dl/issues/72 https://github.com/dgorissen/coursera-dl/issues/72

carlosgg 13y ago

Seems to work great over here, and the installation was pretty easy, too. You can even choose not to download certain types of files using the -n option. For example, if you have a large hard drive and a smaller one, you can download the whole course to the large HD: coursera-dl -u username -p password -d pathToLargeHD course_name and only download pdf lecture notes to the smaller one coursera-dl -u username -p password -d pathToSmallHD -n mp4,pptx course_name I tried that over here, worked great. Some schools prefer students don't download course materials. I succesfully downloaded Machine Learning and Algorithms courses from Stanford but could not download this one, it says "now downloadable content found": https://www.coursera.org/course/datasci https://www.coursera.org/course/datasci

carlosgg 13y ago
After upgrading to latest version of script, I was able to download this one, too.

ambler0 13y ago

This violates Coursera's terms of service. The relevant language: "...as a condition of accessing the Sites, you agree not to...(c) use any high-volume, automated or electronic means to access the Sites (including without limitation, robots, spiders, scripts or web-scraping tools)" https://www.coursera.org/about/terms https://www.coursera.org/about/terms I was actually going to write such a script myself as an exercise (I'm new to programming), but this language dissuaded me. EDIT: Could someone please explain to me why this simple statement of fact would be downvoted?

idm 13y ago
This script would appear to be at odds with the language of that clause, but I wonder if it is truly against the spirit of Coursera. I can see clear ways to automatically exploit the website for the purpose of scraping their content. However, I have had a legitimate use for this in the past and, instead of just doing the smart automatic thing, I did it manually. My take is this: for personal use, a low-volume, automated tool would not violate the spirit of online education. The presence of that clause is probably intended to protect against exploitative uses.
- ambler0 13y ago
  I had a similar take, especially since we all know that terms of service are written very broadly. But on the other hand, I would really hate to mess up my Coursera account. After posting my initial comment, I saw further down that HN user pamelafox is a former Coursera employee. I would love to know whether she or anyone else has insight into how Coursera views tools like this.
carlosgg 13y ago
I upvoted it. :) Good catch! I did not know that! I really hope they mean that clause for people that would try to frivolously download 200 courses or something, just to have them...I figure that would tax their servers and cost money because of the bandwidth being consumed. Before I knew about coursera-dl, I tried to download one of the courses by hand, and it's BEYOND TEDIOUS. So far, I have downloaded 2 courses with coursera-dl, both of which I have taken and submitted lots of homework to. I think it's ok in that case...

Zoomla 13y ago

Anyone saved material from Jeff Leek's last Data Analysis class? it seemed like a good class. It was in Jan. 2013 and it's coming back in Oct. 2013...

carlosgg 13y ago

Quick FYI, there might be a problem if one tries to run Firefox concurrently: https://github.com/dgorissen/coursera-dl/issues/81 https://github.com/dgorissen/coursera-dl/issues/81