10 ms·

What purpose does it serve?

by anony23 4y ago

deleted 4y ago
[deleted]
throwaway0984 4y ago
IIRC it's used to extract/generate the signatures needed for YouTube media URLs
oynqr 4y ago
You need to run some obscured JS to get decent download speeds from Youtube. Something along the lines of PoW.
- db48x 4y ago
  It’s not like proof of work at all. It’s just a challenge and response; youtube includes a random number in the webpage for each video, and expects to see a request parameter with a particular value calculated from that random number when you request the video. If you don’t do the arithmetic it throttles you to 50kb/s. Since the calculation of the response is done in JS, and they occasionally change the formula, some download programs are moving towards running the JS rather than trying to keep up with the changes. It’s really just bullshit to make people’s lives harder.
  - xg15 4y ago
    Next step will probably be moving the calculation to webassembly or requiring the script to fetch the result via websocket or webrtc...
  - mistrial9 4y ago
    .. pirate determination is a thing to behold, as is crazed-repetitive digital grabs.. Its not a fair or accurate characterization to dismiss it as "making people's lives harder" .. it is remarkable that the Debian distros now include ytdl; lets do what is reasonable to make it continue
    - db48x 4y ago
      You can’t exactly pirate a youtube video, since they’re all publicly available.
      - MiguelX413 4y ago
        That's not really how piracy works. I say this as an advocate of it.
  - dannyw 4y ago
    YouTube PM: We need to stop youtube-dl. Engineers: make half arsed attempt.
rany_ 4y ago
They need to run a JavaScript function to download YouTube videos at normal speeds. Edit: it's also required to download music, otherwise it will just fail Source: - https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-894619419 https://github.com/ytdl-org/youtube-dl/issues/29326#issuecom... - https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1964f8fdde9ceea514a5e581d/youtube_dl/extractor/youtube.py#L1484-L1500 https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1... - https://github.com/ytdl-org/youtube-dl/commit/cf001636600430b4b03963238c14a11c33d6d013 https://github.com/ytdl-org/youtube-dl/commit/cf001636600430...
- ajkjk 4y ago
  Wow: Overview of the control flow (already known): The Youtube API provides you with n - your video access token If their new changes apply to your client (they do for "web") then it is expected your client will modify n based on internal logic. This logic is inside player...base.js n is modified by a cryptic function Modified n is sent back to server as proof that we're an official client. If you send n unmodified, the server will eventually throttle you. So they can always change the function to keep you on your toes, hence you need to be able to run semi-arbitrary JS in order to keep using the API. Waste of human brainpower but I guess that energy is better spent imagining a world where Google isn't in charge instead of kvetching about what they're doing with their influence.
  - isatty 4y ago
    There is a reason Google is able to serve the amount of video bandwidth, and also a reason why there are no worthwhile youtube clones. Some amount of scrape protection is absolutely essential.
    - Aperocky 4y ago
      Seems like they ultimately failed, youtube-dl is available freely as a pip package, anyone with scraping intent would have been able to use it.
elaus 4y ago
I'd have to read up on the specifics as well, but I think basically Youtube uses a lot of obfuscated, rapidly and automatically changing Javascript code to fetch the video data. A project like youtube-dl has to run this code to be able to download videos, because that's what's happening in the browser as well.
- temp_account_32 4y ago
  For those interested further, in some of the past few weeks youtube-dl had stopped working intermittently for multiple hours at a time, and it was precisely related to this code. We have a custom-made Discord music bot on our server which uses ytdl to stream songs so we can listen together, and at one point we were listening and suddenly got some obscure JavaScript error. We began joking that there's some bug in the code which breaks it after 6PM, but later found out that Google had changed some of the obfuscated JS and this basically broke this part of code, which prevented us from fetching the song information.
- bitexploder 4y ago
  What is interesting is it seems to be constant cat and mouse. I download a YT vid. It crawls. Update yt-dlp, it flies again. I love yt-dlp and use it a lot.
- lupire 4y ago
  But why not just use a normal JS engine called from Python?
- londons_explore 4y ago
  If you start a youtube video and then pause it and resume a few days later, you'll notice that the youtube page plays for ~30 seconds (ie. whats buffered) and then the page refreshes. I'd guess this refresh is to pick up the new javascript and any updates to the HTML code. It's kinda annoying if you have a lot of youtube tabs open for a long time and come back to them.
hadrien01 4y ago
It's used in the YouTube extractor: https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1964f8fdde9ceea514a5e581d/youtube_dl/extractor/youtube.py https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1... I believe YouTube limits your bitrate if you don't pass a specific calculated value; it's possible youtube-dl has to parse and eval JS to get it.
- RicoElectrico 4y ago
  > I believe YouTube limits your bitrate if you don't pass a specific calculated value It's starting to become Widevine bullshit all over again.
  - kevin_thibedeau 4y ago
    It's their platform. They can do with it what they want.
    - uwuemu 4y ago
      - MiguelX413 4y ago
        There's a difference between combatting entitlement to a platform and complaining about something not serving a greater good. Leftists are also censored either way. Those companies censor or don't censor according to what would maximize profit. Monetization and reach are probably mutually exclusive with freedom.
      - tsukikage 4y ago
        > you also think billionares should be taxed more That's... quite a response in defense of a tool intended for breaching TOS and performing copyright infringement. Can you clarify exactly who it is and isn't OK to steal from, again? I'm struggling here.
        4y ago

What purpose does it serve?

deleted 4y ago

[deleted]

throwaway0984 4y ago

IIRC it's used to extract/generate the signatures needed for YouTube media URLs

oynqr 4y ago

You need to run some obscured JS to get decent download speeds from Youtube. Something along the lines of PoW.

db48x 4y ago
It’s not like proof of work at all. It’s just a challenge and response; youtube includes a random number in the webpage for each video, and expects to see a request parameter with a particular value calculated from that random number when you request the video. If you don’t do the arithmetic it throttles you to 50kb/s. Since the calculation of the response is done in JS, and they occasionally change the formula, some download programs are moving towards running the JS rather than trying to keep up with the changes. It’s really just bullshit to make people’s lives harder.
- xg15 4y ago
  Next step will probably be moving the calculation to webassembly or requiring the script to fetch the result via websocket or webrtc...
- mistrial9 4y ago
  .. pirate determination is a thing to behold, as is crazed-repetitive digital grabs.. Its not a fair or accurate characterization to dismiss it as "making people's lives harder" .. it is remarkable that the Debian distros now include ytdl; lets do what is reasonable to make it continue
  - db48x 4y ago
    You can’t exactly pirate a youtube video, since they’re all publicly available.
    - MiguelX413 4y ago
      That's not really how piracy works. I say this as an advocate of it.
- dannyw 4y ago
  YouTube PM: We need to stop youtube-dl. Engineers: make half arsed attempt.

rany_ 4y ago

They need to run a JavaScript function to download YouTube videos at normal speeds. Edit: it's also required to download music, otherwise it will just fail Source: - https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-894619419 https://github.com/ytdl-org/youtube-dl/issues/29326#issuecom... - https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1964f8fdde9ceea514a5e581d/youtube_dl/extractor/youtube.py#L1484-L1500 https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1... - https://github.com/ytdl-org/youtube-dl/commit/cf001636600430b4b03963238c14a11c33d6d013 https://github.com/ytdl-org/youtube-dl/commit/cf001636600430...

ajkjk 4y ago
Wow: Overview of the control flow (already known): The Youtube API provides you with n - your video access token If their new changes apply to your client (they do for "web") then it is expected your client will modify n based on internal logic. This logic is inside player...base.js n is modified by a cryptic function Modified n is sent back to server as proof that we're an official client. If you send n unmodified, the server will eventually throttle you. So they can always change the function to keep you on your toes, hence you need to be able to run semi-arbitrary JS in order to keep using the API. Waste of human brainpower but I guess that energy is better spent imagining a world where Google isn't in charge instead of kvetching about what they're doing with their influence.
- isatty 4y ago
  There is a reason Google is able to serve the amount of video bandwidth, and also a reason why there are no worthwhile youtube clones. Some amount of scrape protection is absolutely essential.
  - Aperocky 4y ago
    Seems like they ultimately failed, youtube-dl is available freely as a pip package, anyone with scraping intent would have been able to use it.

elaus 4y ago

I'd have to read up on the specifics as well, but I think basically Youtube uses a lot of obfuscated, rapidly and automatically changing Javascript code to fetch the video data. A project like youtube-dl has to run this code to be able to download videos, because that's what's happening in the browser as well.

temp_account_32 4y ago
For those interested further, in some of the past few weeks youtube-dl had stopped working intermittently for multiple hours at a time, and it was precisely related to this code. We have a custom-made Discord music bot on our server which uses ytdl to stream songs so we can listen together, and at one point we were listening and suddenly got some obscure JavaScript error. We began joking that there's some bug in the code which breaks it after 6PM, but later found out that Google had changed some of the obfuscated JS and this basically broke this part of code, which prevented us from fetching the song information.
bitexploder 4y ago
What is interesting is it seems to be constant cat and mouse. I download a YT vid. It crawls. Update yt-dlp, it flies again. I love yt-dlp and use it a lot.
lupire 4y ago
But why not just use a normal JS engine called from Python?
londons_explore 4y ago
If you start a youtube video and then pause it and resume a few days later, you'll notice that the youtube page plays for ~30 seconds (ie. whats buffered) and then the page refreshes. I'd guess this refresh is to pick up the new javascript and any updates to the HTML code. It's kinda annoying if you have a lot of youtube tabs open for a long time and come back to them.

hadrien01 4y ago

It's used in the YouTube extractor: https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1964f8fdde9ceea514a5e581d/youtube_dl/extractor/youtube.py https://github.com/ytdl-org/youtube-dl/blob/d619dd712f63aab1... I believe YouTube limits your bitrate if you don't pass a specific calculated value; it's possible youtube-dl has to parse and eval JS to get it.

RicoElectrico 4y ago
> I believe YouTube limits your bitrate if you don't pass a specific calculated value It's starting to become Widevine bullshit all over again.
- kevin_thibedeau 4y ago
  It's their platform. They can do with it what they want.
  - uwuemu 4y ago
    - MiguelX413 4y ago
      There's a difference between combatting entitlement to a platform and complaining about something not serving a greater good. Leftists are also censored either way. Those companies censor or don't censor according to what would maximize profit. Monetization and reach are probably mutually exclusive with freedom.
    - tsukikage 4y ago
      > you also think billionares should be taxed more That's... quite a response in defense of a tool intended for breaching TOS and performing copyright infringement. Can you clarify exactly who it is and isn't OK to steal from, again? I'm struggling here.
      - 4y ago