8 ms·

Origins of the youtube-dl project

559 points by rg3 6y ago

deleted 6y ago
[deleted]
greggman3 6y ago
I don't know if this is related or not but back when Flickr was popular in like 2005 or 2006 my friends and I were uploading pictures of our events there (they're all still there AFAIK. We'd upload them to either our own accounts or for certain events to a shared account. One shared account is here: https://www.flickr.com/photos/sourpower/albums https://www.flickr.com/photos/sourpower/albums I wanted copies of those pictures and the easiest way to get them was the write a tool to download them rather than have to coordinate with 3 to 15 friends and ask them to copy the images to a CD or USB stick or some other nonsense. Dropbox wasn't a thing and not all my friends were tech heads that would want to setup FTP servers. Flickr had also come out with an API. APIs for online services seemed kind of new at that point and Flickr was one of the first AFAIK. So I wrote the app https://blog.greggman.com/blog/flickrdown/ https://blog.greggman.com/blog/flickrdown/ and a few months later it was accused by other users of flickr of being solely for the purpose of downloading copywritten images. Not once did I ever use it for such a purpose nor, AFAIK did any of my friends. None of us had any interest in other people's images on flickr, only shared images of mutually attended parties, bbq, picnics, events. Those users reported the app to Flickr and the app was banned. It was banned by the app's id. That meant you could register your own app and then hack in your app's id and still use it. IIRC I continued to use it to download pictures from our events but it always pissed me off they banned it. It also pissed me off because it wasn't accessing anything you couldn't just scrape for. The API made it easy to get a list of URLs, search for albums or people etc but you could easily write a script that just scraped the HTML to find all the same data. Didn't matter, flickr didn't budge. It further pissed me off that over zealous flickr members accused me of lying about its purpose. Like many topics today, there is often absolutely nothing you can say that will convince someone else your intensions are not bad.
- michaelmior 6y ago
  While I can understand your frustration, I also understand the frustration of photographers and other creators trying to make a living who have their work stolen. Their anger was wrongly directed at you, but it is a real problem.
  - pjc50 6y ago
    It's basically over now; since photographers don't have a cartel backing them like film or music, and photos are so easy to copy, everything is basically everywhere. Newspapers will routinely rip off photos from social media, sometimes in the face of explicit non permission.
  - greggman3 6y ago
    While I agree with you that photographers are hurt by having their photos copied I'm not convinced my app was a problem. I personally doubt anyone was using it to bulk download copywritten photos and somehow use those photos. I could be wrong but I believe most people that steal photos still them a few at time. Basically they're making some article, they need a related photo, they search, grab a 2 to 5 and stick them in their article. They'd do this with the browser, not my app. My app didn't let you search by keyword, only by user. Further it didn't remove watermarks or do anything else. If photographers put their photos on flickr and they care about stealing they usually both watermark and only put relatively low-res versions there and you have to pay them for high res versions. If flickr provided logs that showed bulk download and further some proof that even with bulk download that it was actually affecting professional photographers and not just a few geeks collecting some pictures they liked then I'd be more inclined to buy into their ban but without that I'm pretty confident the ban had no basis in reality.
    - michaelmior 6y ago
      I wasn't intending to suggest you were contributing to the problem. If others used your tool improperly, that's not your fault. But I can understand why some photographers who have had their content stolen could be upset. As far as Flickr's response in banning your app, it does seem at least misguided. I also wonder if they were concerned about making it too easy for people to move to another platform.
jchw 6y ago
Thanks for this. It is weird how long it’s been. I totally forgot that Wireshark was once called Ethereal. I hope that the DMCA takedown issue can be resolved reasonably, but it’s starting to seem more and more like a move off of Github is overdue. Especially in a world where anyone can stand up a Gitea or Gitlab CE instance.
- nosmokewhereiam 6y ago
  My first tech book after Mike Meyers "All in one A+ 4th Ed" was a book written on Wireshark by Laura. I never got too deep as it was taken from the desk I pulled CQ duty at.
- llsf 6y ago
  What I had totally forgotten, was libflashplayer.so :) it was so flaky. Glad I do not have to worry about this anymore.
- woofie11 6y ago
  gitlab isn't evil. One of my short life lessons is that, in general, when I do business with evil, sketchy, or nasty organizations or people, I generally come out behind. If the good guys have an inferior product and charge double, I'll sometimes pick the bad guy. And more often than not, I get burned, costing me tenfold what it would have cost to just go with the high-integrity choice in the first place. I'm not leaving github over this, but I'm mostly starting new project on gitlab instead.
  - speeder 6y ago
    I've been using GitLab for a while now. GitHub did many, many decisions that I found sketchy.
    - res0nat0r 6y ago
      Gitlab enforces DMCA requests. If you're a reputable company wanting to do legit business in the USA you have to follow the process, it really isn't up to Github or Gitlab unless they want to lose safe harbor status. https://about.gitlab.com/handbook/dmca/ https://about.gitlab.com/handbook/dmca/
wpietri 6y ago
What a wonderful little retrospective. It's a nice walk down memory lane, and I really appreciate how clear and sensible his thinking was around handing off the project when the time was right.
gspr 6y ago
What a wonderful writeup that touches on both the original author's personal history, little technical tidbits, and so many ethical and societal aspects of computing!
kzrdude 6y ago
jwz also had his youtube downloader script out there - but that was apparently not the start of youtube dl
vagrantJin 6y ago
Long live Youtube-dl.
gman83 6y ago
Damn, freshmeat.net... I used to visit that site every day in the early 2000s, and I'd totally forgotten about it.
saghul 6y ago
TIL the project was started by a fellow Spaniard! Very nice write up, kudos!
humford 6y ago
Been a longtime fan of YouTube-dl, so much better and faster than any alternative. I remember being in high school in a group project with some theater kids. They needed a ton of free stock footage from Vimeo and they were going through links one-by-one to download each video individually. I just compiled a list of the links, ran youtube-dl, and had it done in 5 minutes when they'd already wasted three hours on it.
nzmsv 6y ago
Back when Udacity just launched with their first course (Thrun's self driving car intro) I wanted to watch the videos on a big screen. I had a somewhat smart TV that could play files off a USB stick but did not have Internet access. Udacity hosted all their videos on YouTube at the time and there was no convenient way to download them. So I spent an afternoon hacking together a Chrome extension that would modify the Udacity website DOM and give me download links for the lectures (it had to be an extension to get around same origin restrictions). Udacity had some weird naming convention for their videos so I had to make some calls to their API as well as YouTube and correlate videos with titles. The YouTube part was "influenced" by youtube-dl. This was much faster than reverse engineering, even though I was writing new JavaScript and not using the original Python. Anyway, trivial stuff. After watching the videos from my couch for a few days I decided to post a link to my extension on the Udacity message board... and it absolutely blew up! My dinky little extension had thousands of users all over the world seemingly overnight. But the absolute highlight was getting an email from a student from Iran. Iran just blocked YouTube because of https://en.wikipedia.org/wiki/Innocence_of_Muslims https://en.wikipedia.org/wiki/Innocence_of_Muslims and there was a whole group of students who could no longer participate in the course. Apparently they had some friends at a US university use my extension to download the videos and reupload to a VPS they ran. I was blown away - my quest to sit on a couch ended up accidentally helping fight censorship. I maintained the extension until Udacity added a native video download feature and then took it down. But it was an interesting experience and definitely shaped my perception of fair use laws. They are important. People have way more legitimate uses for information than lawyers can imagine.
- andai 6y ago
  > my quest to sit on a couch ended up accidentally helping fight censorship That's brilliant! We can never predict the impact our tools will have on people's lives. Your idea of partially porting youtube-dl to the browser gives me an idea... would it be feasible to port it fully? I think the biggest hurdle would be ffmpeg, but a few days ago I saw "A pure WebAssembly / JavaScript port of FFmpeg": https://news.ycombinator.com/item?id=24987861 https://news.ycombinator.com/item?id=24987861
  - 6y ago
AntiImperialist 6y ago
Wow, I did not know youtube-dl had existed for so long. I remember coding my own YouTube downloader because of similar reasons. My internet connection was way too slow to stream videos, even at the lowest quality, so I'd make a list of videos, download them in the background the entire day and then watch it offline at the end of the day. When I finally discovered youtube-dl, I was relieved that I no longer had to keep maintaining my script... and it supported almost every other video website. Also, I just realized that it can still be downloaded from the official website and updated using --update argument.
Sektor 6y ago
Hey rg3 your link https://yt-dl.org/supportedsites.html https://yt-dl.org/supportedsites.html is broken. Great tool btw thanks for all your efforts.
- rg3 6y ago
  Thanks, I've reported that to the current maintainers.
TeeMassive 6y ago
I've had a personal playlist of videos I maintained since 2008, a collection really, with multiple fads I once used to share with the various social circles I had during the last decade at college and university. But now more than half of the videos are gone due to the various ban episodes YouTube had (FU cancel culture, FU Carlos Maza). Fortunately with YTdl I can save at least half of what has been my good fairly big portion of my adult life.

559 points by rg3 6y ago

deleted 6y ago

[deleted]

I don't know if this is related or not but back when Flickr was popular in like 2005 or 2006 my friends and I were uploading pictures of our events there (they're all still there AFAIK. We'd upload them to either our own accounts or for certain events to a shared account. One shared account is here: https://www.flickr.com/photos/sourpower/albums https://www.flickr.com/photos/sourpower/albums I wanted copies of those pictures and the easiest way to get them was the write a tool to download them rather than have to coordinate with 3 to 15 friends and ask them to copy the images to a CD or USB stick or some other nonsense. Dropbox wasn't a thing and not all my friends were tech heads that would want to setup FTP servers. Flickr had also come out with an API. APIs for online services seemed kind of new at that point and Flickr was one of the first AFAIK. So I wrote the app https://blog.greggman.com/blog/flickrdown/ https://blog.greggman.com/blog/flickrdown/ and a few months later it was accused by other users of flickr of being solely for the purpose of downloading copywritten images. Not once did I ever use it for such a purpose nor, AFAIK did any of my friends. None of us had any interest in other people's images on flickr, only shared images of mutually attended parties, bbq, picnics, events. Those users reported the app to Flickr and the app was banned. It was banned by the app's id. That meant you could register your own app and then hack in your app's id and still use it. IIRC I continued to use it to download pictures from our events but it always pissed me off they banned it. It also pissed me off because it wasn't accessing anything you couldn't just scrape for. The API made it easy to get a list of URLs, search for albums or people etc but you could easily write a script that just scraped the HTML to find all the same data. Didn't matter, flickr didn't budge. It further pissed me off that over zealous flickr members accused me of lying about its purpose. Like many topics today, there is often absolutely nothing you can say that will convince someone else your intensions are not bad.

michaelmior 6y ago
While I can understand your frustration, I also understand the frustration of photographers and other creators trying to make a living who have their work stolen. Their anger was wrongly directed at you, but it is a real problem.
- pjc50 6y ago
  It's basically over now; since photographers don't have a cartel backing them like film or music, and photos are so easy to copy, everything is basically everywhere. Newspapers will routinely rip off photos from social media, sometimes in the face of explicit non permission.
- greggman3 6y ago
  While I agree with you that photographers are hurt by having their photos copied I'm not convinced my app was a problem. I personally doubt anyone was using it to bulk download copywritten photos and somehow use those photos. I could be wrong but I believe most people that steal photos still them a few at time. Basically they're making some article, they need a related photo, they search, grab a 2 to 5 and stick them in their article. They'd do this with the browser, not my app. My app didn't let you search by keyword, only by user. Further it didn't remove watermarks or do anything else. If photographers put their photos on flickr and they care about stealing they usually both watermark and only put relatively low-res versions there and you have to pay them for high res versions. If flickr provided logs that showed bulk download and further some proof that even with bulk download that it was actually affecting professional photographers and not just a few geeks collecting some pictures they liked then I'd be more inclined to buy into their ban but without that I'm pretty confident the ban had no basis in reality.
  - michaelmior 6y ago
    I wasn't intending to suggest you were contributing to the problem. If others used your tool improperly, that's not your fault. But I can understand why some photographers who have had their content stolen could be upset. As far as Flickr's response in banning your app, it does seem at least misguided. I also wonder if they were concerned about making it too easy for people to move to another platform.

jchw 6y ago

Thanks for this. It is weird how long it’s been. I totally forgot that Wireshark was once called Ethereal. I hope that the DMCA takedown issue can be resolved reasonably, but it’s starting to seem more and more like a move off of Github is overdue. Especially in a world where anyone can stand up a Gitea or Gitlab CE instance.

nosmokewhereiam 6y ago
My first tech book after Mike Meyers "All in one A+ 4th Ed" was a book written on Wireshark by Laura. I never got too deep as it was taken from the desk I pulled CQ duty at.
llsf 6y ago
What I had totally forgotten, was libflashplayer.so :) it was so flaky. Glad I do not have to worry about this anymore.
woofie11 6y ago
gitlab isn't evil. One of my short life lessons is that, in general, when I do business with evil, sketchy, or nasty organizations or people, I generally come out behind. If the good guys have an inferior product and charge double, I'll sometimes pick the bad guy. And more often than not, I get burned, costing me tenfold what it would have cost to just go with the high-integrity choice in the first place. I'm not leaving github over this, but I'm mostly starting new project on gitlab instead.
- speeder 6y ago
  I've been using GitLab for a while now. GitHub did many, many decisions that I found sketchy.
  - res0nat0r 6y ago
    Gitlab enforces DMCA requests. If you're a reputable company wanting to do legit business in the USA you have to follow the process, it really isn't up to Github or Gitlab unless they want to lose safe harbor status. https://about.gitlab.com/handbook/dmca/ https://about.gitlab.com/handbook/dmca/

wpietri 6y ago

What a wonderful little retrospective. It's a nice walk down memory lane, and I really appreciate how clear and sensible his thinking was around handing off the project when the time was right.

gspr 6y ago

What a wonderful writeup that touches on both the original author's personal history, little technical tidbits, and so many ethical and societal aspects of computing!

kzrdude 6y ago

jwz also had his youtube downloader script out there - but that was apparently not the start of youtube dl

vagrantJin 6y ago

Long live Youtube-dl.

gman83 6y ago

Damn, freshmeat.net... I used to visit that site every day in the early 2000s, and I'd totally forgotten about it.

saghul 6y ago

TIL the project was started by a fellow Spaniard! Very nice write up, kudos!

humford 6y ago

Been a longtime fan of YouTube-dl, so much better and faster than any alternative. I remember being in high school in a group project with some theater kids. They needed a ton of free stock footage from Vimeo and they were going through links one-by-one to download each video individually. I just compiled a list of the links, ran youtube-dl, and had it done in 5 minutes when they'd already wasted three hours on it.

nzmsv 6y ago

Back when Udacity just launched with their first course (Thrun's self driving car intro) I wanted to watch the videos on a big screen. I had a somewhat smart TV that could play files off a USB stick but did not have Internet access. Udacity hosted all their videos on YouTube at the time and there was no convenient way to download them. So I spent an afternoon hacking together a Chrome extension that would modify the Udacity website DOM and give me download links for the lectures (it had to be an extension to get around same origin restrictions). Udacity had some weird naming convention for their videos so I had to make some calls to their API as well as YouTube and correlate videos with titles. The YouTube part was "influenced" by youtube-dl. This was much faster than reverse engineering, even though I was writing new JavaScript and not using the original Python. Anyway, trivial stuff. After watching the videos from my couch for a few days I decided to post a link to my extension on the Udacity message board... and it absolutely blew up! My dinky little extension had thousands of users all over the world seemingly overnight. But the absolute highlight was getting an email from a student from Iran. Iran just blocked YouTube because of https://en.wikipedia.org/wiki/Innocence_of_Muslims https://en.wikipedia.org/wiki/Innocence_of_Muslims and there was a whole group of students who could no longer participate in the course. Apparently they had some friends at a US university use my extension to download the videos and reupload to a VPS they ran. I was blown away - my quest to sit on a couch ended up accidentally helping fight censorship. I maintained the extension until Udacity added a native video download feature and then took it down. But it was an interesting experience and definitely shaped my perception of fair use laws. They are important. People have way more legitimate uses for information than lawyers can imagine.

andai 6y ago
> my quest to sit on a couch ended up accidentally helping fight censorship That's brilliant! We can never predict the impact our tools will have on people's lives. Your idea of partially porting youtube-dl to the browser gives me an idea... would it be feasible to port it fully? I think the biggest hurdle would be ffmpeg, but a few days ago I saw "A pure WebAssembly / JavaScript port of FFmpeg": https://news.ycombinator.com/item?id=24987861 https://news.ycombinator.com/item?id=24987861
- 6y ago

AntiImperialist 6y ago

Wow, I did not know youtube-dl had existed for so long. I remember coding my own YouTube downloader because of similar reasons. My internet connection was way too slow to stream videos, even at the lowest quality, so I'd make a list of videos, download them in the background the entire day and then watch it offline at the end of the day. When I finally discovered youtube-dl, I was relieved that I no longer had to keep maintaining my script... and it supported almost every other video website. Also, I just realized that it can still be downloaded from the official website and updated using --update argument.

Sektor 6y ago

Hey rg3 your link https://yt-dl.org/supportedsites.html https://yt-dl.org/supportedsites.html is broken. Great tool btw thanks for all your efforts.

rg3 6y ago
Thanks, I've reported that to the current maintainers.

TeeMassive 6y ago

I've had a personal playlist of videos I maintained since 2008, a collection really, with multiple fads I once used to share with the various social circles I had during the last decade at college and university. But now more than half of the videos are gone due to the various ban episodes YouTube had (FU cancel culture, FU Carlos Maza). Fortunately with YTdl I can save at least half of what has been my good fairly big portion of my adult life.