7 ms·

I had never used youtube-dl until the story happened. I downloaded for windows and its speeds were throttled to around 50kB/s. Posters on Stack recommended the

I had never used youtube-dl until the story happened. I downloaded for windows and its speeds were throttled to around 50kB/s. Posters on Stack recommended the ytdlp fork which i tried and it was 5-10 MB/s. Just fyi

bspammer 5y ago
Yeah youtube-dl is missing workarounds for throttling implemented by google. youtube-dl is pretty much unmaintained compared to yt-dlp: https://github.com/ytdl-org/youtube-dl/graphs/commit-activity https://github.com/ytdl-org/youtube-dl/graphs/commit-activit... https://github.com/yt-dlp/yt-dlp/graphs/commit-activity https://github.com/yt-dlp/yt-dlp/graphs/commit-activity
- vmoore 5y ago
  And yt-dlp uses an 'Android API' to stop throttling
  - heavyset_go 5y ago
    I expect the ability to use APIs like this will be hindered once remote attestation becomes the norm.
    - cute_boi 5y ago
      This is reason why we must diversify these techs. If same company builds everything like mobile, os, search engine, Youtube, it would have debacle in future. If there is diversification, it simply takes toil and sweat to convince everyone.
  - 1vuio0pswjnm7 5y ago
    One has to execute functions from base.js^1 to modify the "n" URL parameter and the "sig" parameter to get the fastest download speeds. (One can still download videos with the original n parameter, or without the n parameter, but download speeds will be slower.) By itself, using the ANDROID API instead of the WEB API^2 does nothing to affect download speeds. I can block yt-dlp's POST request indicating the client and API name and this has no effect on download speed. A website that forces users to run Javascript in order to get faster download speeds. This is not a new idea. 1. https://www.youtube.com/s/player/{player_version}/player_ias.vflset/en_US/base.js https://www.youtube.com/s/player/{player_version}/player_ias... 2. From the YouTube video page JSON: ANDROID API KEY AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w WEB API KEY AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8
    - 1vuio0pswjnm7 5y ago
      yt-dl and yt-dlp use Python to interpet the Javascript functions that modify "n" and "sig", but a faster language could be used. For example, V8 is written in C++. One of the reasons yt-dl and yt-dlp are so slow is because they do (too) many do other things besides running the Javascript, modifying n and sig, and spitting out a fast download URL. Before they can run the JS, the video page needs to be retrieved, but waiting for Python to start up and do this is slow. A YouTube video page can be retrieved much faster using netcat, outside of Python. YouTube video can be downloaded much faster using an HTTP client like tnftp, directly, outside of Python. YouTube video can be converted much faster using ffmpeg, directly, outside of Python. And so on. These programs start instantaniously when compared to the slow start up time of Python. The Python startup latency is unbearable. At the very least yt-dl and yt-dlp should accept an already downloaded YouTube video page as input instead of forcing the user to use Python (or Python calling another program) to download the page. The recalculation of "n" and "sig" do not have to occur within seconds of retrieving the video page. There is no need to require the user to use Python for downloading webpages. The values in the YouTube video page are good for a substantial amount of time. Both scripts have an option to output the new n and sig values in a download URL or JSON containing the download URLs, so the user can use an HTTP client directly, outside of Python. But the user still has to use Python to download each video page. Using an HTTP client directly would be faster. Using yt-dl and yt-dlp just to output optimal download URLs feels like overkill.
      - Scaevolus 5y ago
        If you find yt-dlp particularly slow, you can use aria2c to perform the download itself. It doesn't solve any startup latency problems, but it generally can do the HTTP requests faster.
      - marcan_42 5y ago
        This is complete nonsense. The overwhelming majority of video download time is spent transferring the actual video data, limited by throttling or network speed. The time it takes for these tools to start up and deal with the API/JS stuff is inconsequential for any video longer than a minute or so. And even then most of the time is network latency for the API stuff, not local processing. Netcat isn't going to go any faster. Absolutely nobody thinks optimizing the meta/API processing of yt-dlp & co is worth it. This is exactly why we have high level programming languages that make all of this much easier, instead of trying to write HTML and JS parsing in plain C. Keep in mind these tools support dozens or hundreds or websites, not just YouTube. If you think rewriting yt-dlp in C is worth it, go right ahead, but you're not going to make it significantly faster; you're just going to make maintenance a much bigger pain for yourself. Pick the right tool for the job. Python is absolutely (one of) the right tools for this job. (For the record: I use C and Python on a daily basis, and will use whatever language is appropriate for each situation.)
  - deleted 5y ago
    [deleted]
- elcomet 5y ago
  This is why I come on HN, I always learn something useful.
AdmiralAsshat 5y ago
I noticed this recently, too. I frequently scrape concerts from YouTube by adding them to a private playlist and then running a script on my HTPC to pull down everything from that playlist. As recently as six months ago, pulling down a 2-3GB playlist file (maybe a 30-40 minute concert at 720P) took ten to fifteen minutes. The last time I tried it, it estimated several hours.
- soheil 5y ago
  If you're downloading a playlist you could run it in parallel to achieve your max network bandwidth speed: function ytp() { youtube-dl --get-id "$1" | xargs -I '{}' -P 200 youtube-dl -i --embed-thumbnail --add-metadata -f 'bestaudio[ext=m4a]' -o '%(title)s.%(ext)s' 'https://youtube.com/watch?v={}'; }
- rane 5y ago
  FWIW, just recently wrote a go program that watches a Youtube playlist and downloads new videos to a configurable path: https://github.com/raine/ytdlwatch https://github.com/raine/ytdlwatch I use it to download videos into a Plex library.
mgdlbp 5y ago
The workaround for vanilla youtube-dl is to use it with aria2, with options like: --external-downloader aria2c --external-downloader-args "--continue --max-concurrent-downloads=3 --max-connection-per-server=3 --split 3 --min-split-size 1M" (possibly in your config file) Also, neither --format best nor --format bestvideo chooses the best encoding in all cases; they use bitrate as a heuristic for quality, and a less efficient codec can have higher bitrate but worse quality, resolution, or framerate. The workaround for this is specifying --format with an enumeration of every combination of codec, resolution, and framerate in preferred order, which goes like this: --format "(bestvideo[vcodec^=av01][height>=4320][fps>30]/bestvideo[vcodec^=vp9.2][height>=4320] ... Here's a full example (hmm... they're using it with yt-dlp, which I thought had fixed this?): https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate-YouTube-DL-Scripts-Collection/blob/47715c877a9ac3ecca116fd5ab9693a966bd8a09/scripts/Archivist%20Scripts/Archivist%20Scripts/Playlists/Playlists.sh https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate... I think there's a bit of variation in the exact order among the config files found online. If you're goals are archival, consider also retrieving metadata, thumbnail, and subtitles in all languages; I also have in my config the options: --verbose --download-archive ./ytdl-archive.txt --cookies ./ytdl-cookies.txt --merge-output-format mkv --add-metadata --all-subs --embed-subs --write-info-json --write-thumbnail --no-overwrites --continue --force-ipv4 (the only remaining workaround for age-restricted videos is to give it cookies extracted from a browser with a real Google account logged in)
UberFly 5y ago
FireDM (https://pypi.org/project/FireDM https://pypi.org/project/FireDM) is an awesome front end for ytdlp for those interested.
- belter 5y ago
  GitHub Links from the project page all get a 404. https://github.com/firedm/ https://github.com/firedm/ shows no public repos and of course https://github.com/firedm/FireDM https://github.com/firedm/FireDM gets a 404
burnte 5y ago
I always find jwz's youtubedown works flawlessly for me. https://www.jwz.org/hacks/youtubedown https://www.jwz.org/hacks/youtubedown
- js2 5y ago
  Do not direct link to jwz.org from HN. Here's a click-safe link: https://dereferer.me/?https%3A//www.jwz.org/hacks/youtubedown https://dereferer.me/?https%3A//www.jwz.org/hacks/youtubedow...
  - Aissen 5y ago
    Does not work either because apparently bit.ly's 301 preserves the Referer.
    - js2 5y ago
      Weird, I had tested it. I’ve updated to a de-referrer site instead.
      - Aissen 5y ago
        Might be a browser difference (firefox here). Thanks for the update, it works better.
  - 77pt77 5y ago
    > Do not direct link to jwz.org from HN Why?
    - psyc 5y ago
      Because the admin detects HN referer explicitly and presents a joke page.
      - mnd999 5y ago
        Perhaps I’m in the minority, but I find this kind of subversion amusing. Like all good jokes, it’s pretty close to the mark.
altcognito 5y ago
Am I the only one totally chill with the throttling? Look, I'm not sure if it's the best idea for their own resource utilitization (there are probably peak times when more throttline is better, and quiet internet times when throttling doesn't make sense) As long as they are being gracious hosts and allowing downloads, I'm good with being throttled I guess?
- emteycz 5y ago
  50 kB/s means you'll be downloading a 10min video like 10 hours.
  - mdp2021 5y ago
    > 50 kB/s means you'll be downloading a 10min video like 10 hours Very misleading phrasing: you would download in 10 hours a ~10 hours long video, which (of course) could have been downloaded in a fraction of the time. The throttling has the user download at a speed similar to that required to viewing the video.
    - emteycz 5y ago
      I played a random 1080p YT video and the player-reported data rate is at least 2.5 MB/s, peaking to 20 MB/s (cca 10 MB/s average). 4K is of course much more than that.
      - mdp2021 5y ago
        I just took a video as a parameter - a recent engineering animation - and got the following: format 18 - 640x360 AVC1, V+A : ~50kB/s format 22 - 1280x720 AVC1, V+A : ~60kB/s format 137 - 1920x1080 AVC1, V : ~128kB/s format 400 - 2560x1440 av01, V : ~450kB/s format 401 - 3840x2160 av01, V : ~850kB/s Are you sure you are not reporting mega/bits/? When you mention the «player reported data rate», are you sure that is not the "connection speed"? 10MB/s means downloading 36GB/h (one CD per minute, dozens gigabytes per hour)...
77pt77 5y ago
I started getting systematic > Connection reset by peer in a script that I have that downloads podcasts and immediately transcodes to low bitrate opus using ffmpeg. It's a real bummer...
formerkrogemp 5y ago
I miss when youtube-dl wasn't getting so much attention and just worked. Ah well.
svnpenn 5y ago
Fast alternative: https://github.com/89z/mech https://github.com/89z/mech
WhyNotHugo 5y ago
Yeah, trying to delete `youtube-dl` from the GitHub triggered a huge Streisand Effect[1]. [1]: https://en.wikipedia.org/wiki/Streisand_effect https://en.wikipedia.org/wiki/Streisand_effect

bspammer 5y ago

Yeah youtube-dl is missing workarounds for throttling implemented by google. youtube-dl is pretty much unmaintained compared to yt-dlp: https://github.com/ytdl-org/youtube-dl/graphs/commit-activity https://github.com/ytdl-org/youtube-dl/graphs/commit-activit... https://github.com/yt-dlp/yt-dlp/graphs/commit-activity https://github.com/yt-dlp/yt-dlp/graphs/commit-activity

vmoore 5y ago
And yt-dlp uses an 'Android API' to stop throttling
- heavyset_go 5y ago
  I expect the ability to use APIs like this will be hindered once remote attestation becomes the norm.
  - cute_boi 5y ago
    This is reason why we must diversify these techs. If same company builds everything like mobile, os, search engine, Youtube, it would have debacle in future. If there is diversification, it simply takes toil and sweat to convince everyone.
- 1vuio0pswjnm7 5y ago
  One has to execute functions from base.js^1 to modify the "n" URL parameter and the "sig" parameter to get the fastest download speeds. (One can still download videos with the original n parameter, or without the n parameter, but download speeds will be slower.) By itself, using the ANDROID API instead of the WEB API^2 does nothing to affect download speeds. I can block yt-dlp's POST request indicating the client and API name and this has no effect on download speed. A website that forces users to run Javascript in order to get faster download speeds. This is not a new idea. 1. https://www.youtube.com/s/player/{player_version}/player_ias.vflset/en_US/base.js https://www.youtube.com/s/player/{player_version}/player_ias... 2. From the YouTube video page JSON: ANDROID API KEY AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w WEB API KEY AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8
  - 1vuio0pswjnm7 5y ago
    yt-dl and yt-dlp use Python to interpet the Javascript functions that modify "n" and "sig", but a faster language could be used. For example, V8 is written in C++. One of the reasons yt-dl and yt-dlp are so slow is because they do (too) many do other things besides running the Javascript, modifying n and sig, and spitting out a fast download URL. Before they can run the JS, the video page needs to be retrieved, but waiting for Python to start up and do this is slow. A YouTube video page can be retrieved much faster using netcat, outside of Python. YouTube video can be downloaded much faster using an HTTP client like tnftp, directly, outside of Python. YouTube video can be converted much faster using ffmpeg, directly, outside of Python. And so on. These programs start instantaniously when compared to the slow start up time of Python. The Python startup latency is unbearable. At the very least yt-dl and yt-dlp should accept an already downloaded YouTube video page as input instead of forcing the user to use Python (or Python calling another program) to download the page. The recalculation of "n" and "sig" do not have to occur within seconds of retrieving the video page. There is no need to require the user to use Python for downloading webpages. The values in the YouTube video page are good for a substantial amount of time. Both scripts have an option to output the new n and sig values in a download URL or JSON containing the download URLs, so the user can use an HTTP client directly, outside of Python. But the user still has to use Python to download each video page. Using an HTTP client directly would be faster. Using yt-dl and yt-dlp just to output optimal download URLs feels like overkill.
    - Scaevolus 5y ago
      If you find yt-dlp particularly slow, you can use aria2c to perform the download itself. It doesn't solve any startup latency problems, but it generally can do the HTTP requests faster.
    - marcan_42 5y ago
      This is complete nonsense. The overwhelming majority of video download time is spent transferring the actual video data, limited by throttling or network speed. The time it takes for these tools to start up and deal with the API/JS stuff is inconsequential for any video longer than a minute or so. And even then most of the time is network latency for the API stuff, not local processing. Netcat isn't going to go any faster. Absolutely nobody thinks optimizing the meta/API processing of yt-dlp & co is worth it. This is exactly why we have high level programming languages that make all of this much easier, instead of trying to write HTML and JS parsing in plain C. Keep in mind these tools support dozens or hundreds or websites, not just YouTube. If you think rewriting yt-dlp in C is worth it, go right ahead, but you're not going to make it significantly faster; you're just going to make maintenance a much bigger pain for yourself. Pick the right tool for the job. Python is absolutely (one of) the right tools for this job. (For the record: I use C and Python on a daily basis, and will use whatever language is appropriate for each situation.)
- deleted 5y ago
  [deleted]
elcomet 5y ago
This is why I come on HN, I always learn something useful.

AdmiralAsshat 5y ago

I noticed this recently, too. I frequently scrape concerts from YouTube by adding them to a private playlist and then running a script on my HTPC to pull down everything from that playlist. As recently as six months ago, pulling down a 2-3GB playlist file (maybe a 30-40 minute concert at 720P) took ten to fifteen minutes. The last time I tried it, it estimated several hours.

soheil 5y ago
If you're downloading a playlist you could run it in parallel to achieve your max network bandwidth speed: function ytp() { youtube-dl --get-id "$1" | xargs -I '{}' -P 200 youtube-dl -i --embed-thumbnail --add-metadata -f 'bestaudio[ext=m4a]' -o '%(title)s.%(ext)s' 'https://youtube.com/watch?v={}'; }
rane 5y ago
FWIW, just recently wrote a go program that watches a Youtube playlist and downloads new videos to a configurable path: https://github.com/raine/ytdlwatch https://github.com/raine/ytdlwatch I use it to download videos into a Plex library.

mgdlbp 5y ago

The workaround for vanilla youtube-dl is to use it with aria2, with options like: --external-downloader aria2c --external-downloader-args "--continue --max-concurrent-downloads=3 --max-connection-per-server=3 --split 3 --min-split-size 1M" (possibly in your config file) Also, neither --format best nor --format bestvideo chooses the best encoding in all cases; they use bitrate as a heuristic for quality, and a less efficient codec can have higher bitrate but worse quality, resolution, or framerate. The workaround for this is specifying --format with an enumeration of every combination of codec, resolution, and framerate in preferred order, which goes like this: --format "(bestvideo[vcodec^=av01][height>=4320][fps>30]/bestvideo[vcodec^=vp9.2][height>=4320] ... Here's a full example (hmm... they're using it with yt-dlp, which I thought had fixed this?): https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate-YouTube-DL-Scripts-Collection/blob/47715c877a9ac3ecca116fd5ab9693a966bd8a09/scripts/Archivist%20Scripts/Archivist%20Scripts/Playlists/Playlists.sh https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate... I think there's a bit of variation in the exact order among the config files found online. If you're goals are archival, consider also retrieving metadata, thumbnail, and subtitles in all languages; I also have in my config the options: --verbose --download-archive ./ytdl-archive.txt --cookies ./ytdl-cookies.txt --merge-output-format mkv --add-metadata --all-subs --embed-subs --write-info-json --write-thumbnail --no-overwrites --continue --force-ipv4 (the only remaining workaround for age-restricted videos is to give it cookies extracted from a browser with a real Google account logged in)

UberFly 5y ago

FireDM (https://pypi.org/project/FireDM https://pypi.org/project/FireDM) is an awesome front end for ytdlp for those interested.

belter 5y ago
GitHub Links from the project page all get a 404. https://github.com/firedm/ https://github.com/firedm/ shows no public repos and of course https://github.com/firedm/FireDM https://github.com/firedm/FireDM gets a 404

burnte 5y ago

I always find jwz's youtubedown works flawlessly for me. https://www.jwz.org/hacks/youtubedown https://www.jwz.org/hacks/youtubedown

js2 5y ago
Do not direct link to jwz.org from HN. Here's a click-safe link: https://dereferer.me/?https%3A//www.jwz.org/hacks/youtubedown https://dereferer.me/?https%3A//www.jwz.org/hacks/youtubedow...
- Aissen 5y ago
  Does not work either because apparently bit.ly's 301 preserves the Referer.
  - js2 5y ago
    Weird, I had tested it. I’ve updated to a de-referrer site instead.
    - Aissen 5y ago
      Might be a browser difference (firefox here). Thanks for the update, it works better.
- 77pt77 5y ago
  > Do not direct link to jwz.org from HN Why?
  - psyc 5y ago
    Because the admin detects HN referer explicitly and presents a joke page.
    - mnd999 5y ago
      Perhaps I’m in the minority, but I find this kind of subversion amusing. Like all good jokes, it’s pretty close to the mark.

altcognito 5y ago

Am I the only one totally chill with the throttling? Look, I'm not sure if it's the best idea for their own resource utilitization (there are probably peak times when more throttline is better, and quiet internet times when throttling doesn't make sense) As long as they are being gracious hosts and allowing downloads, I'm good with being throttled I guess?

emteycz 5y ago
50 kB/s means you'll be downloading a 10min video like 10 hours.
- mdp2021 5y ago
  > 50 kB/s means you'll be downloading a 10min video like 10 hours Very misleading phrasing: you would download in 10 hours a ~10 hours long video, which (of course) could have been downloaded in a fraction of the time. The throttling has the user download at a speed similar to that required to viewing the video.
  - emteycz 5y ago
    I played a random 1080p YT video and the player-reported data rate is at least 2.5 MB/s, peaking to 20 MB/s (cca 10 MB/s average). 4K is of course much more than that.
    - mdp2021 5y ago
      I just took a video as a parameter - a recent engineering animation - and got the following: format 18 - 640x360 AVC1, V+A : ~50kB/s format 22 - 1280x720 AVC1, V+A : ~60kB/s format 137 - 1920x1080 AVC1, V : ~128kB/s format 400 - 2560x1440 av01, V : ~450kB/s format 401 - 3840x2160 av01, V : ~850kB/s Are you sure you are not reporting mega/bits/? When you mention the «player reported data rate», are you sure that is not the "connection speed"? 10MB/s means downloading 36GB/h (one CD per minute, dozens gigabytes per hour)...

77pt77 5y ago

I started getting systematic > Connection reset by peer in a script that I have that downloads podcasts and immediately transcodes to low bitrate opus using ffmpeg. It's a real bummer...

formerkrogemp 5y ago

I miss when youtube-dl wasn't getting so much attention and just worked. Ah well.

svnpenn 5y ago

Fast alternative: https://github.com/89z/mech https://github.com/89z/mech

WhyNotHugo 5y ago

Yeah, trying to delete `youtube-dl` from the GitHub triggered a huge Streisand Effect[1]. [1]: https://en.wikipedia.org/wiki/Streisand_effect https://en.wikipedia.org/wiki/Streisand_effect