4 ms·

Apparently the trick is to pretend being an Android device, or something in that style...

by kaetemi 5y ago

kevin_thibedeau 5y ago
You used to be able to bypass throttling by watching the video in browser during the download.
- usr1106 5y ago
  I have always wondered why video sites (who don't want downloaders for various reasons) serve their content faster than required for normal speed viewing. Well, I don't complain, but I wonder.
  - gattilorenz 5y ago
    For buffering, since there might be a network hiccup sometime in the future. But generally also a better user experience if the user wants to skip a few seconds/minutes forward. And I guess the bandwidth required also depends on the video bit rate/format/compression ratio, so you need to implement a lot of smarts to make sure you don't serve something too slow or too fast, and still people would be able to download the video, albeit at a slower speed. So, basically, it's not worth it.
  - foxfluff 5y ago
    The primary reason I've used ytdl is to work around youtube's crappy buffering. In the past you used to be able to leave a youtube page open and it'd buffer the whole video but now it stops at like.. thirty seconds? Have fun watching when their servers can't serve.
  - kaetemi 5y ago
    Video streams are just a series of 2-second-long video files. It's no different from any regular CDN. (The real money is in the compression processes.)
    - cube00 5y ago
      I was surprised by just how many small chunks (as you said 2 seconds, a few hundred kilobytes) the browser has to request and string together to make a video stream that play smoothly and that I can skip around. It's kinda amazing when you think about it.
  - varenc 5y ago
    They address this by having the video player stop buffering more than 30 seconds ahead. It's also less compute resources for the servers to send all the bytes for the next 30 seconds to a client at max speed, let the client disconnect and wait 3-5 seconds, then give them the next 3-5 seconds immediately when it's requested. If they only let you download video at a rate slightly faster than video playback rate than all clients would have a constantly open connection downloading video which would tie up more resources.
varenc 5y ago
The work needed to bypass the rate limiting is pretty impressive: https://github.com/ytdl-org/youtube-dl/pull/30184#issuecomment-974180758 https://github.com/ytdl-org/youtube-dl/pull/30184#issuecomme... An excerpt: > ...the challenge algorithm is served in a mini-language within the minified player JS and therefore the specific algorithm could be extracted and executed by interpreting the mini-language without actually running the JS itself.
- skytreader 5y ago
  Quicker explanation of how Youtube is doing it is at the source code of pytube: https://github.com/tfdahlin/pytube/blob/bb890af3b271a616bbe1d08abc8c65bac9266b0a/pytube/cipher.py https://github.com/tfdahlin/pytube/blob/bb890af3b271a616bbe1... This is linked from the description of PR 30184 of youtube-dl but @varenc linked to a specific comment (talking about patching Windows builds, AFAICT) which might confuse some people. I'll even save you the click: > YouTube's strategy to restrict downloading videos is to send a ciphered version of the signature to the client, along with the decryption algorithm obfuscated in JavaScript. For the clients to play the videos, JavaScript must take the ciphered version, cycle it through a series of "transform functions," and then signs the media URL with the output. There are still a lot of things I don't understand here but gotta give Youtube a slow-clap moment for their devious ingenuity.
  - Kadin 5y ago
    > There are still a lot of things I don't understand here but gotta give Youtube a slow-clap moment for their devious ingenuity. Well, sorta. They have basically come up with a DRM scheme by another name. They're always just speed bumps.
  - varenc 5y ago
    Whoops, didn’t mean to include the link to that comment... that’s a much better explanation!
- rstuart4133 5y ago
  As someone else said here on HN, in situations like this all it amounts to "mutually assured time wasting". It reminded me ofthe old joke about two people entering an area infested by lion's, and one of them sits down on a log, and takes 5 minutes to replace his robust hiking shoes by a very expensive pair of running shows. The other says "why are you doing that - you can't our run a lion." The reply is "I don't have to be faster than the lion - just faster than you". It makes prefect sense of course, when the lion has a choice. But in this case there is no choice. There is only one youtube. No matter how much time and expense youtube putting into their running shoes, youtube-dl / yt-dlp is always going to chase after them. They are always going to win that race. While Google may be paying 5 or 10 of the cream of worlds programmers, those two open source projects ultimately how pools of 100's of equally talented people who occasionally want to download a video, and will happily contribute back a few hours of their time to do it. Which is probably how yt-dlp came about - youtube-dl slowed down, and someone said "fuck it - I'll do it myself". Worse from google's point of view, unlike splitinering a commercial endeavours, once those two projects are already contributing code to each other and may well merge again.

Apparently the trick is to pretend being an Android device, or something in that style...

kevin_thibedeau 5y ago

You used to be able to bypass throttling by watching the video in browser during the download.

usr1106 5y ago
I have always wondered why video sites (who don't want downloaders for various reasons) serve their content faster than required for normal speed viewing. Well, I don't complain, but I wonder.
- gattilorenz 5y ago
  For buffering, since there might be a network hiccup sometime in the future. But generally also a better user experience if the user wants to skip a few seconds/minutes forward. And I guess the bandwidth required also depends on the video bit rate/format/compression ratio, so you need to implement a lot of smarts to make sure you don't serve something too slow or too fast, and still people would be able to download the video, albeit at a slower speed. So, basically, it's not worth it.
- foxfluff 5y ago
  The primary reason I've used ytdl is to work around youtube's crappy buffering. In the past you used to be able to leave a youtube page open and it'd buffer the whole video but now it stops at like.. thirty seconds? Have fun watching when their servers can't serve.
- kaetemi 5y ago
  Video streams are just a series of 2-second-long video files. It's no different from any regular CDN. (The real money is in the compression processes.)
  - cube00 5y ago
    I was surprised by just how many small chunks (as you said 2 seconds, a few hundred kilobytes) the browser has to request and string together to make a video stream that play smoothly and that I can skip around. It's kinda amazing when you think about it.
- varenc 5y ago
  They address this by having the video player stop buffering more than 30 seconds ahead. It's also less compute resources for the servers to send all the bytes for the next 30 seconds to a client at max speed, let the client disconnect and wait 3-5 seconds, then give them the next 3-5 seconds immediately when it's requested. If they only let you download video at a rate slightly faster than video playback rate than all clients would have a constantly open connection downloading video which would tie up more resources.

varenc 5y ago

The work needed to bypass the rate limiting is pretty impressive: https://github.com/ytdl-org/youtube-dl/pull/30184#issuecomment-974180758 https://github.com/ytdl-org/youtube-dl/pull/30184#issuecomme... An excerpt: > ...the challenge algorithm is served in a mini-language within the minified player JS and therefore the specific algorithm could be extracted and executed by interpreting the mini-language without actually running the JS itself.

skytreader 5y ago
Quicker explanation of how Youtube is doing it is at the source code of pytube: https://github.com/tfdahlin/pytube/blob/bb890af3b271a616bbe1d08abc8c65bac9266b0a/pytube/cipher.py https://github.com/tfdahlin/pytube/blob/bb890af3b271a616bbe1... This is linked from the description of PR 30184 of youtube-dl but @varenc linked to a specific comment (talking about patching Windows builds, AFAICT) which might confuse some people. I'll even save you the click: > YouTube's strategy to restrict downloading videos is to send a ciphered version of the signature to the client, along with the decryption algorithm obfuscated in JavaScript. For the clients to play the videos, JavaScript must take the ciphered version, cycle it through a series of "transform functions," and then signs the media URL with the output. There are still a lot of things I don't understand here but gotta give Youtube a slow-clap moment for their devious ingenuity.
- Kadin 5y ago
  > There are still a lot of things I don't understand here but gotta give Youtube a slow-clap moment for their devious ingenuity. Well, sorta. They have basically come up with a DRM scheme by another name. They're always just speed bumps.
- varenc 5y ago
  Whoops, didn’t mean to include the link to that comment... that’s a much better explanation!
rstuart4133 5y ago
As someone else said here on HN, in situations like this all it amounts to "mutually assured time wasting". It reminded me ofthe old joke about two people entering an area infested by lion's, and one of them sits down on a log, and takes 5 minutes to replace his robust hiking shoes by a very expensive pair of running shows. The other says "why are you doing that - you can't our run a lion." The reply is "I don't have to be faster than the lion - just faster than you". It makes prefect sense of course, when the lion has a choice. But in this case there is no choice. There is only one youtube. No matter how much time and expense youtube putting into their running shoes, youtube-dl / yt-dlp is always going to chase after them. They are always going to win that race. While Google may be paying 5 or 10 of the cream of worlds programmers, those two open source projects ultimately how pools of 100's of equally talented people who occasionally want to download a video, and will happily contribute back a few hours of their time to do it. Which is probably how yt-dlp came about - youtube-dl slowed down, and someone said "fuck it - I'll do it myself". Worse from google's point of view, unlike splitinering a commercial endeavours, once those two projects are already contributing code to each other and may well merge again.