18 ms·

Bypassing YouTube video download throttling

587 points by 0x7d0 3y ago

0x7d0 3y ago
Have you ever tried to download videos from YouTube? I mean manually without relying on software like youtube-dl, yt-dlp or one of “these” websites. It’s much more complicated than you might think.
- rasz 3y ago
  Yes, by injecting my own userscript using my (judging by WEI not for long) USER AGENT. I dont even screw around reimplementing their signature/n decoding/throttling functions, I grep for player.js match(/(?:player\/([a-zA-Z0-9_-]+)\/)?(?:html5player|(?:www|player(?:_ias)?))[-\.]([^/]+?)(?:(?:\/html5player(?:-new)?)?|(?:\/[a-z]{2,3}_[A-Z]{2})?\/base)\.js/), then grep in that for relevant functions and call those directly. You could say that from YT perspective everything is 100% kosher, its their own DRM functions unlocking .mp4 link for me :)
- NoZebra120vClip 3y ago
  Like, press the "Download" button and pay for Premium?
  - mynameisvlad 3y ago
    YouTube Premium doesn’t let you download the video file. Even if you do it from a browser, it is only accessible from the YouTube website run offline as a SPA. Even then, downloaded videos can only be played for 29 days before having to reconnect most of the time, with some regions restricting it to 48 hours.
    - G3rn0ti 3y ago
      I think parent means the YouTube app. On iOS (and android/chromos?) you can actually download videos and watch them offline if you got a premium membership. But then the videos are under control by the app.
      - leshenka 3y ago
        You can't get .mp4 though. It's there somewhere inside YT app's persistent storage but there is no "share" button for it.
      - mynameisvlad 3y ago
        Yes, and I replied to that. It’s not really an alternative to an actual video download. It’s not the raw video file, you can only access it from the app or website, and there’s restrictions on how long you can be offline before the app/website will stop you from watching it.
    - js8 3y ago
      I would totally buy Premium subscription if they let me download the videos, and I told Google that. The way it is now - no deal.
      - Mindwipe 3y ago
        Meh, Google know that nobody would pay for that as a selling point when it only applied to a third of videos, and they will never get copyright clearance from third parties for more than that.
        js8 3y ago
        I think you misunderstood what I said. I would gladly pay premium/subscription for videos that are OK with it. I don't care about those that don't, they just won't get my subscription.
  - leshenka 3y ago
    Premium is not available worldwide. I tried to trick google by creating an account using VPN in Europe. I even managed to subscribe for Premium. Even with an active Premium subscription Youtube app won't let me use its features such as downloading, background play and picture in picture (you know, basically everything) Because "you live in the wrong part of the world". Creating a problem, selling the solution is what they do. Like, I don't even care about downloading, but Google intentionally cripples their mobile website experience by suppressing pip and background play. It would cost zero dollars not to do this but they did.
    - abwizz 3y ago
      [flagged]
      - leshenka 3y ago
        Most of my grievances are related to mobile experience so I'm not sure why are you talking about mpv. I barely have any problems in desktop because issues like no pip and background play just don't exist there.
      - Aachen 3y ago
        Not sure what part of not using your suggested command line tools but wanting to just use the official version like everyone else warrants calling someone openly stupid
    - whywhywhywhy 3y ago
      Baffling they enforce those fake restrictions in parts of the world the product to enable them doesn't exist. The fact Apple allows Google to resell their multitasking and PIP features as part of their own subscription is pretty un-Apple.
  - Geisterde 3y ago
    [dead]
- wazoox 3y ago
  A very long time ago, I've worked on a Perl script to do just that ( https://www.perlmonks.org/?node_id=636777 https://www.perlmonks.org/?node_id=636777 ). Of course the problem with this sort of scripts is that they keep chasing behind changes youtube makes precisely to prevent video downloading.
hruzgar 3y ago
So make a nice, good documented blog post for google engineers to understand and fix this issue?? Whyy
- mynameisvlad 3y ago
  I’m almost positive that the engineers already know how the throttling is applied and how it could be circumvented. Downloading the same file used in the official UI and applying/reverse engineering the function is not exactly rocket science.
- yMMe2WYE_D 3y ago
  The only real thing they could try to do is to try enforcing video/IP speed cap. They probably don't due to false positives - for example people jumping around a long video also make may "range" requests. And the other approach is currently in progress - soon logged users won't be able to view videos without watching(or at least displaying/downloading) adds, so the logical next step is to nerf anonymous(as is without google account) viewing. No matter how I look YT has exact problems (and solutions) as all file locker sites. The only difference is that YT is not at mercy of Ad companies, it is the Ad company(at least Google is). So they might try some more aggressive measures, that normally would get a site banned from publishing ads.
  - deno 3y ago
    There’s plenty they could do. They could flip a switch tomorrow and limit access to only signed-in users, and they could further enable DRM, as pretty much by now the majority of users are already on DRM-handicapped platforms. Any will-be-called "legacy" users would just get limited to 480p.
    - adhvaryu 3y ago
      They already lowered bitrates for non premium members.
      - noisem4ker 3y ago
hknmtt 3y ago
a good read on HN after a very long time, for me.
- adhvaryu 3y ago
  I have to agree, it's an interesting topic with a bit of "hacking" masala and just very well written. Can't remember the last time I read a full article here.
jgtrosh 3y ago
Technically, all interesting. Ethically, if you don't only think “fuck Google”, I feel like it's reasonable to stop after the first optimization (“pass the real browser test to get regular browser speeds”). There you're not “wasting” any more of YouTube's resources than a browser user with ad-block. Getting full Gb/s without paying anything feels to me like you're pushing all the ad-blocked users' luck. But then again, fuck Google I guess?
- planede 3y ago
  I wonder how abusive this is though. In the end you are downloading the same amount of data, but in a shorter time. You are utilizing more bandwidth but you go away earlier. I think the original browser use case is tuned for the common occurrence of not watching the whole video. But if you intend to watch (and archive) the whole video to begin with then I don't think this eats away Google's bandwidth more. OTOH it probably has more overhead due to the amount of connections.
  - abwizz 3y ago
    yea, i also don't think that the increased rate is a problem per se, but i also doubt that a majority of youtube views cover the entire length of the video, hence downloaders do probably use more data.
    - Applejinx 3y ago
      Any youtuber, myself included, can attest that a majority of youtube views demonstrably don't. A key factor in finding success as a youtuber is getting better at retention, hence all the ridiculousness and MrBeastness: some people specialize in retention, and they do better. It's weird from the standpoint of someone who sets out to watch an entire thing, but almost nobody sticks around while watching videos. It seems like the mass of youtube viewerdom are bouncing around like mad, all the time.
      - abwizz 3y ago
        not unlike "zapping" tv channels
ngc6677 3y ago
Super cool breakdown, gg!
albert_e 3y ago
Some videos offer multiple audio channels for different languages? Why have I never come across such videos before / missed somehow?
- reggegg 3y ago
  Mr Beast's videos have at least a spanish audio track, which funnily enough NewPipe defaults to (or did the last I checked) and NewPipe doesn't support changing the track as far as I can tell
  - BlueGh0st 3y ago
    Just checked NewPipe on a MrBeast video. There is an option now to select the audio track and MrBeast uploads dubs in more than a dozen languages.
    - extraduder_ire 3y ago
      When I first noticed this, I thought it was cool that dub-spiderman[0] migrated to using that right away, since he already went so hard with the mrbeast spanish and other dub channels. I assume it's preferable to have all of your subscribers on the one channel. 0: Jimmy's voice in the spanish dub of his channel is the same actor who dubs spiderman.
- Eavolution 3y ago
  Chubbyemu videos often have it, and James Hoffman once used it to provide an audio channel with slurping sounds (he's a coffee channel, that's not as dodgy as it sounds) and one without.
- Knee_Pain 3y ago
  Because it's an extremely new feature and honestly only a few channels can afford to make use of it
  - hombre_fatal 3y ago
no_time 3y ago
I'm constantly suprised when YT deploys another half measure against downloaders when GOOG also owns widevine. I wonder what is their reasons for not using it.
- londons_explore 3y ago
  Youtube is compatible with a lot of platforms, old browsers, ancient smart TV's, ancient android, etc - I would guess widevine isn't.
  - Mindwipe 3y ago
    Widevine has been a mandatory requirement for any OEM pre-installing YouTube for something like seven years now. There is not much out there that Google would care about EOL access for.
- scrollaway 3y ago
  Widevine is used in some youtube videos. Not all, though; not even a high percentage -- I've only seen it in certain music videos. I'm guessing it's on a paid license basis…
- Gigachad 3y ago
  Maybe they care about not cutting off devices that don’t support it. TVs, ARM Linux, etc. While making downloading videos just annoying enough that people don’t bother.
- londons_explore 3y ago
  I don't think youtube really cares if you pirate their content or use a 3rd party client. What they care about is you wasting their bandwidth. For an ad-supported video streaming site, bandwidth is normally more expensive than revenue - Google only manages to make it just about work because they have probably the worlds cheapest bandwidth due to being able to bully ISP's into peering with them for free. (they don't let you peer with Google for just Google Search but not youtube). All these throttling measures are simply trying to reserve most of the bandwidth for real users, not people scraping all the content.
wodenokoto 3y ago
Any guides to learn how to do a similar analysis on other websites?
- hk__2 3y ago
  Which websites?
  - wodenokoto 3y ago
    No, the broader concept of how to dissect the behaviour to figure out the api endpoints called, what was sent to them, what was returned, etc.
    - tamimio 3y ago
      MITM proxy, ZAP, fiddler, and Burp suite are some tools to start with, and wireshark/postman just in case you needed it. Rest is just your knowledge in JavaScript mostly.
      - xchkr1337 3y ago
        I don't think these tools are particularly suitable for reverse engineering websites, it's much easier to use devtools and userscripts
- JKCalhoun 3y ago
  I assumed, perhaps incorrectly, that the author is a Google engineer.
probably_wrong 3y ago
I've never tried YouTube, but I have downloaded videos from sketchier streaming websites using the web developer tools. Almost all of them have the same protection: some code that triggers only when you open the tools and stops the video by creating a debugger statement you cannot skip and triggering some cpu-heavy code (probably an infinite loop, although I wouldn't discard cryptominers). More importantly this code also clears the network request information, making it more difficult to analyze the traffic sent so far. Note to Firefox devs: enabling "persist logs" should persist the logs. Don't clear them! None of this is perfect and I never found a video I couldn't eventually download (timing attacks ftw), but I do wish I could find a deeper explanation on how this all works.
- no_time 3y ago
  >Almost all of them have the same protection: some code that triggers only when you open the tools and stops the video by creating a debugger statement you cannot skip If you missed it, not so long ago there was a submission that evaded exactly this. Their solution is so simple yet effective: Recompiling the browser with the debugger keyword renamed. Made me smile. https://news.ycombinator.com/item?id=36961445 https://news.ycombinator.com/item?id=36961445
  - deleted 3y ago
    [deleted]
  - matheusmoreira 3y ago
    Honestly recompilation shouldn't even be necessary, browsers should just let us disable the debugger statement when "clever" sites start taking advantage of it. This usage of debuggers to circumvent our tools is abuse and should be literally impossible unless we consent to it. Our computers are our realms. God giveth and god taketh away.
    - rivo 3y ago
      Chrome Developer Tools allows this. There's a button in the Source tab to deactivate breakpoints from debug statements. The button looks like an arrow cut in half.
1vuio0pswjnm7 3y ago
"Have you ever tried to download videos from YouTube? I mean manually without relying on software like youtube-dl, yt-dlp or one of "these" websites. It's much more complicated than you might think." This reminds me of some sort of fizzbuzz test. This is not complicated at all. There is no need to use the Range header or run Javascript. The short script below does not download anything because there is no need. It does not use Range headers, it does not run Javascript and it makes only one TCP connection. With the JSON it fetches, one can simply extract the videoplayback URLs and put them in a locally-hosted HTML page with no Javascript. #!/bin/sh # usage: echo videoId | $0 <-- this will indicate len to use # usage: echo videoId | $0 len | openssl s_client -connect www.youtube.com:443 -ign_eof # usage: $0 len < videoId-list | openssl s_client -connect www.youtube.com:443 -ign_eof ( while read x;do test ${#x} -eq 11||continue if test $# -ne 1;then len=${#x};x=$(grep -m1 ^\{ $0|sed 's/\$x//'|wc -c);exec echo usage: ${0##*/} $((x+len));fi cr=$(printf '\r'); sed "/^[a-zA-Z].*: /s/$/$cr/;s/^$/$cr/" << eof POST /youtubei/v1/player?key=AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w HTTP/1.1 Host: www.youtube.com Content-Type: application/json Content-Length: $1 Connection: keep-alive {"context": {"client": {"clientName": "IOS", "clientVersion": "17.33.2" }}, "videoId": "$x", "params": "CgIQBg==", "playbackContext": {"contentPlaybackContext": {"html5Preference": "HTML5_PREF_WANTS"}}, "contentCheckOk": true, "racyCheckOk": true} eof done printf '\r\n' printf 'GET /robots.txt HTTP/1.0\r\nHost: www.youtube.com\r\nConnection: close\r\n\r\n'; ) For processing the JSON I wrote custom utilities in C that (a) extract videoIds and other useful strings, (b) generate HTTP similar to above, and (c) filter the returned JSON into CSV, SQL or HTML. For me, these run faster than Python and jq and are easier to edit. Using these utilities I can also do full searches that return hundreds to thousands of results and I can easily exclude all "suggested" or "recommended" videos. CSV output 1666520150,23 Oct 2022 10:15:50 UTC,22,aqz-KE-bpKQ,"Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film",00:10:35,635,UCSMOQeBJ2RAnuFungnQOxLg,19211597,"Blender" SQL output INSERT INTO t1(ts,utc,itag,vid,title,dur,len,cid,views,author) VALUES(1666520150,'23 Oct 2022 10:15:50 UTC',22,'aqz-KE-bpKQ','Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film','00:10:35',635,'UCSMOQeBJ2RAnuFungnQOxLg',19211597,'Blender') ON CONFLICT(vid) DO UPDATE SET views=excluded.views; HTML output Looks just like CSV except vid is a hyperlink
- 3y ago
AltruisticGapHN 3y ago
On a sidenote am I imagining things or do videos actually look a tiny bit better in YouTube? This really has puzzled me. I downloaded a few favourites and watch them on VLC or Infuse on my AppleTV. In the YouTube app I can use the "nerd stats" to confirm I am viewing the exact same video/audio streams... ... it could be my imagination but it seems like YouTube does a really subtle kind of filter that makes the "blocky" compression artifacts smoother. It doesnt ehance edges or anything - my guess is it looks for areas WITHOUT edges where there are sublte shfts of colour, and it makes the blocky artifacts less prominent. ... it's really subtle and I still cant tell if its just my imagination, like my OCD thinking that my downloaded video doesnt look as good and yet, I noticed on YouTube the video feels more vibrant and solid. When I watch my downloaded vid there are these really sublte, but noticable artifacts often in the background , in the shadows and these constant tiny little jitters even on a 1440p video - make the final picture look not as good. Am I making this up? Audio wise there is definitely a change as well. YouTube audio is always more or less level for me, while a downloaded video always needs to crank up the volume which is annoying. I wish players like VLC or Infuse did whatever YouTube does to make videos just more pleasant to look at. I dont think YoUTube changes the colours or does any kind of vibrancy filter though I may be wrong, but it does things to "level" audio so that you have a more consistent experience going from one video/channel to another.
- AltruisticGapHN 3y ago
  If anyone's still reading this or anyone cares here is what I found out: - the issue I was experiencing is with a lower quality encoding vp9 stream - from a video that was recently uploaded (explained below) - though I didn't trust smaller filesize VP9 streams initially, they look in fact noticably better - the picture is smoother, cleaner, the artifacts of compression are less visible. Where AAC can have jittery/glittery distracting dots moving in the background in areas where you have subtle gradients (eg. a plain wall) - Vp9 has none of these, those areas look smoother and cleaner and it gives an overall nicer looking picture without compromising the detail as far I can tell - Opus audio stream appears to have less of the ReplayGain issue, I'm not sure - but since I downloaded Opus instead of the 140 m4a stream I notice I dont need to adjust the volume compared to viewing same video in YouTube - and since the codec is newer anyway and the filesize is relatively the same or a tad smaller - also it is in 48k not 44k, I am going to download Opus from here on - a very confusing thing is it appears ; for a recent upload ; you can have an initial VP9 stream of say 500mb which is in fact no better than the AAC and havs the grainy artifacts - and the vp9 stream gets replaced weeks later by one significantly smaller like 400mb vs 500 mb !! and looks way better . whic hsuggst there was a first pass with low quality encoding, replaced by a higher quality encoding later - therefore my assumption that larger filesize is better was wrong
thrdbndndn 3y ago
> To bypass this limitation, we can break the download into several smaller parts using the HTTP Range header. This header allows you to specify which part of the file you want to download with each request (eg: Range bytes=2000-3000). The following code implements this logic. Last time I read discussion about it in yt-dlp repo [1], you can actually bypass it by just adding range=xxx query parameter (not header), and it will return to full speed even if your range is just the whole thing. And IIRC YouTube have already lifted this restriction. Edit: find the ref [1] https://github.com/yt-dlp/yt-dlp/issues/6400 https://github.com/yt-dlp/yt-dlp/issues/6400
kelvinjps 3y ago
I always wondered How YouTube distribute videos l, it's the smoothest video platform, even on when I had crappy internet it worked fine also not all.platform works well in south America.The closest to YouTube it's Netflix but lacks behinds a lot.
- majewsky 3y ago
  They have cache servers deployed into the ISP networks close to your home. The ISPs allow them to do this because they also benefit from it: Their bottleneck is the connection from their own network into the wider internet backbone, and having cache servers for the big CDNs takes a huge chunk of load away from that bottleneck. Here's documentation about a similar setup for Netflix: https://openconnect.netflix.com/ https://openconnect.netflix.com/
  - simonjgreen 3y ago
    This is not a universal answer and I think deserves some correction. 1. Majority of ISPs do not host any cache for Google content 2. Credible ISPs do not have bottlenecks at the transit or peering level 3. Netflix makes use of much more local caching but their model works very differently to Youtubes 4. The concept of "internet backbone" does not really translate to reality. Peering is significantly more mesh-like than that, and transit more diverse. Source: I have owned multiple ISPs, and still do.
bayesianbot 3y ago
Anyone else having throttling problems with yt-dlp lately? I always watch youtube through mpv that uses yt-dlp on the background, but last week it's been terrible. It starts quickly (I've throttled it to 500kBps so that's the starting speed) but then after a while I'm getting a second of stream for three seconds of download, so I gotta queue up the video long time before playing. I'm using git version of yt-dlp and haven't noticed anything related in the git issues.
- jorams 3y ago
  mpv only uses yt-dlp to get a video URL, then passes that URL to ffmpeg. ffmpeg doesn't implement the workarounds with range headers, so you get throttled. It's possible to have yt-dlp perform the download, pipe it to mpv and make mpv play from stdin, but it breaks seeking to parts that haven't been downloaded yet. There are many issues about this in the mpv issue tracker.
ck2 3y ago
I mean you don't actually think this will continue working a week after it's widely shared, right?
- antiloper 3y ago
  yt-dlp is open source and I'm sure the Google engineers have been aware of it ever since it or it's ancestors were released.
abalashov 3y ago
Not trying to be a sketchy contrarian, but why would you do this with JavaScript? It just doesn't seem very fit for purpose...
- benregenspan 3y ago
  The most interesting part of this (the bypass itself) involves executing a Javascript challenge. It's very convenient to do that from Javascript (the author mentions that Python implementations need to add a Javascript interpreter). Besides that, it's mostly async I/O which: 1) might be annoying in earlier versions of Javascript, but with Promises and async/await, it's very clear and readable (to me) in Javascript; 2) is the exact case (I/O-bound, not CPU-bound) where Node.js performs efficiently.
- hombre_fatal 3y ago
  You should at least say why you don't think Javascript is fit for the trivial task of downloading files. Especially in an article that already had to unwind and explain Youtube's Javascript code.
Ianpett 3y ago
[dead]
rickreynoldssf 3y ago
YouTube changes small things in this process all the time. I used to work on an internal editing tool for YouTube videos that needed the MP4 files. Every month or so the editor would break because of a YouTube change and I needed to dive into the debugger to see what they changed and adjust to that.
swyx 3y ago
> he most popular one is yt-dlp (a fork of youtube-dl) programmed in Python, but it includes its own custom JavaScript interpreter to transform the n parameter. ah i remember this one: https://news.ycombinator.com/item?id=32793061 https://news.ycombinator.com/item?id=32793061 i confess i still dont really understand why they had to make this but i'd love to hear the story behind it
causi 3y ago
I highly recommend anybody who cares about content on Youtube to download the videos you like and maintain local copies. Material is being deleted or hidden faster than ever and Youtube is only going to get more user-hostile over time.
PeterStuer 3y ago
In the endgame you will have to pay in WorldCoin to keep watching your screen, and you can only earn that untradable WorldCoin by viewing the ads, monitored through the mini Orb embedded in the screen. You didn't truly believe this was about UBI did you? We solved that one ages ago with bank accounts and KYC.
- wdb 3y ago
  Nothing with the eye?

587 points by 0x7d0 3y ago

Have you ever tried to download videos from YouTube? I mean manually without relying on software like youtube-dl, yt-dlp or one of “these” websites. It’s much more complicated than you might think.

rasz 3y ago
Yes, by injecting my own userscript using my (judging by WEI not for long) USER AGENT. I dont even screw around reimplementing their signature/n decoding/throttling functions, I grep for player.js match(/(?:player\/([a-zA-Z0-9_-]+)\/)?(?:html5player|(?:www|player(?:_ias)?))[-\.]([^/]+?)(?:(?:\/html5player(?:-new)?)?|(?:\/[a-z]{2,3}_[A-Z]{2})?\/base)\.js/), then grep in that for relevant functions and call those directly. You could say that from YT perspective everything is 100% kosher, its their own DRM functions unlocking .mp4 link for me :)
NoZebra120vClip 3y ago
Like, press the "Download" button and pay for Premium?
- mynameisvlad 3y ago
  YouTube Premium doesn’t let you download the video file. Even if you do it from a browser, it is only accessible from the YouTube website run offline as a SPA. Even then, downloaded videos can only be played for 29 days before having to reconnect most of the time, with some regions restricting it to 48 hours.
  - G3rn0ti 3y ago
    I think parent means the YouTube app. On iOS (and android/chromos?) you can actually download videos and watch them offline if you got a premium membership. But then the videos are under control by the app.
    - leshenka 3y ago
      You can't get .mp4 though. It's there somewhere inside YT app's persistent storage but there is no "share" button for it.
    - mynameisvlad 3y ago
      Yes, and I replied to that. It’s not really an alternative to an actual video download. It’s not the raw video file, you can only access it from the app or website, and there’s restrictions on how long you can be offline before the app/website will stop you from watching it.
  - js8 3y ago
    I would totally buy Premium subscription if they let me download the videos, and I told Google that. The way it is now - no deal.
    - Mindwipe 3y ago
      Meh, Google know that nobody would pay for that as a selling point when it only applied to a third of videos, and they will never get copyright clearance from third parties for more than that.
      - js8 3y ago
        I think you misunderstood what I said. I would gladly pay premium/subscription for videos that are OK with it. I don't care about those that don't, they just won't get my subscription.
- leshenka 3y ago
  Premium is not available worldwide. I tried to trick google by creating an account using VPN in Europe. I even managed to subscribe for Premium. Even with an active Premium subscription Youtube app won't let me use its features such as downloading, background play and picture in picture (you know, basically everything) Because "you live in the wrong part of the world". Creating a problem, selling the solution is what they do. Like, I don't even care about downloading, but Google intentionally cripples their mobile website experience by suppressing pip and background play. It would cost zero dollars not to do this but they did.
  - abwizz 3y ago
    [flagged]
    - leshenka 3y ago
      Most of my grievances are related to mobile experience so I'm not sure why are you talking about mpv. I barely have any problems in desktop because issues like no pip and background play just don't exist there.
    - Aachen 3y ago
      Not sure what part of not using your suggested command line tools but wanting to just use the official version like everyone else warrants calling someone openly stupid
  - whywhywhywhy 3y ago
    Baffling they enforce those fake restrictions in parts of the world the product to enable them doesn't exist. The fact Apple allows Google to resell their multitasking and PIP features as part of their own subscription is pretty un-Apple.
- Geisterde 3y ago
  [dead]
wazoox 3y ago
A very long time ago, I've worked on a Perl script to do just that ( https://www.perlmonks.org/?node_id=636777 https://www.perlmonks.org/?node_id=636777 ). Of course the problem with this sort of scripts is that they keep chasing behind changes youtube makes precisely to prevent video downloading.

hruzgar 3y ago

So make a nice, good documented blog post for google engineers to understand and fix this issue?? Whyy

mynameisvlad 3y ago
I’m almost positive that the engineers already know how the throttling is applied and how it could be circumvented. Downloading the same file used in the official UI and applying/reverse engineering the function is not exactly rocket science.
yMMe2WYE_D 3y ago
The only real thing they could try to do is to try enforcing video/IP speed cap. They probably don't due to false positives - for example people jumping around a long video also make may "range" requests. And the other approach is currently in progress - soon logged users won't be able to view videos without watching(or at least displaying/downloading) adds, so the logical next step is to nerf anonymous(as is without google account) viewing. No matter how I look YT has exact problems (and solutions) as all file locker sites. The only difference is that YT is not at mercy of Ad companies, it is the Ad company(at least Google is). So they might try some more aggressive measures, that normally would get a site banned from publishing ads.
- deno 3y ago
  There’s plenty they could do. They could flip a switch tomorrow and limit access to only signed-in users, and they could further enable DRM, as pretty much by now the majority of users are already on DRM-handicapped platforms. Any will-be-called "legacy" users would just get limited to 480p.
  - adhvaryu 3y ago
    They already lowered bitrates for non premium members.
    - noisem4ker 3y ago

hknmtt 3y ago

a good read on HN after a very long time, for me.

adhvaryu 3y ago
I have to agree, it's an interesting topic with a bit of "hacking" masala and just very well written. Can't remember the last time I read a full article here.

jgtrosh 3y ago

Technically, all interesting. Ethically, if you don't only think “fuck Google”, I feel like it's reasonable to stop after the first optimization (“pass the real browser test to get regular browser speeds”). There you're not “wasting” any more of YouTube's resources than a browser user with ad-block. Getting full Gb/s without paying anything feels to me like you're pushing all the ad-blocked users' luck. But then again, fuck Google I guess?

planede 3y ago
I wonder how abusive this is though. In the end you are downloading the same amount of data, but in a shorter time. You are utilizing more bandwidth but you go away earlier. I think the original browser use case is tuned for the common occurrence of not watching the whole video. But if you intend to watch (and archive) the whole video to begin with then I don't think this eats away Google's bandwidth more. OTOH it probably has more overhead due to the amount of connections.
- abwizz 3y ago
  yea, i also don't think that the increased rate is a problem per se, but i also doubt that a majority of youtube views cover the entire length of the video, hence downloaders do probably use more data.
  - Applejinx 3y ago
    Any youtuber, myself included, can attest that a majority of youtube views demonstrably don't. A key factor in finding success as a youtuber is getting better at retention, hence all the ridiculousness and MrBeastness: some people specialize in retention, and they do better. It's weird from the standpoint of someone who sets out to watch an entire thing, but almost nobody sticks around while watching videos. It seems like the mass of youtube viewerdom are bouncing around like mad, all the time.
    - abwizz 3y ago
      not unlike "zapping" tv channels

ngc6677 3y ago

Super cool breakdown, gg!

albert_e 3y ago

Some videos offer multiple audio channels for different languages? Why have I never come across such videos before / missed somehow?

reggegg 3y ago
Mr Beast's videos have at least a spanish audio track, which funnily enough NewPipe defaults to (or did the last I checked) and NewPipe doesn't support changing the track as far as I can tell
- BlueGh0st 3y ago
  Just checked NewPipe on a MrBeast video. There is an option now to select the audio track and MrBeast uploads dubs in more than a dozen languages.
  - extraduder_ire 3y ago
    When I first noticed this, I thought it was cool that dub-spiderman[0] migrated to using that right away, since he already went so hard with the mrbeast spanish and other dub channels. I assume it's preferable to have all of your subscribers on the one channel. 0: Jimmy's voice in the spanish dub of his channel is the same actor who dubs spiderman.
Eavolution 3y ago
Chubbyemu videos often have it, and James Hoffman once used it to provide an audio channel with slurping sounds (he's a coffee channel, that's not as dodgy as it sounds) and one without.
Knee_Pain 3y ago
Because it's an extremely new feature and honestly only a few channels can afford to make use of it
- hombre_fatal 3y ago

no_time 3y ago

I'm constantly suprised when YT deploys another half measure against downloaders when GOOG also owns widevine. I wonder what is their reasons for not using it.

londons_explore 3y ago
Youtube is compatible with a lot of platforms, old browsers, ancient smart TV's, ancient android, etc - I would guess widevine isn't.
- Mindwipe 3y ago
  Widevine has been a mandatory requirement for any OEM pre-installing YouTube for something like seven years now. There is not much out there that Google would care about EOL access for.
scrollaway 3y ago
Widevine is used in some youtube videos. Not all, though; not even a high percentage -- I've only seen it in certain music videos. I'm guessing it's on a paid license basis…
Gigachad 3y ago
Maybe they care about not cutting off devices that don’t support it. TVs, ARM Linux, etc. While making downloading videos just annoying enough that people don’t bother.
londons_explore 3y ago
I don't think youtube really cares if you pirate their content or use a 3rd party client. What they care about is you wasting their bandwidth. For an ad-supported video streaming site, bandwidth is normally more expensive than revenue - Google only manages to make it just about work because they have probably the worlds cheapest bandwidth due to being able to bully ISP's into peering with them for free. (they don't let you peer with Google for just Google Search but not youtube). All these throttling measures are simply trying to reserve most of the bandwidth for real users, not people scraping all the content.

wodenokoto 3y ago

Any guides to learn how to do a similar analysis on other websites?

hk__2 3y ago
Which websites?
- wodenokoto 3y ago
  No, the broader concept of how to dissect the behaviour to figure out the api endpoints called, what was sent to them, what was returned, etc.
  - tamimio 3y ago
    MITM proxy, ZAP, fiddler, and Burp suite are some tools to start with, and wireshark/postman just in case you needed it. Rest is just your knowledge in JavaScript mostly.
    - xchkr1337 3y ago
      I don't think these tools are particularly suitable for reverse engineering websites, it's much easier to use devtools and userscripts
JKCalhoun 3y ago
I assumed, perhaps incorrectly, that the author is a Google engineer.

probably_wrong 3y ago

I've never tried YouTube, but I have downloaded videos from sketchier streaming websites using the web developer tools. Almost all of them have the same protection: some code that triggers only when you open the tools and stops the video by creating a debugger statement you cannot skip and triggering some cpu-heavy code (probably an infinite loop, although I wouldn't discard cryptominers). More importantly this code also clears the network request information, making it more difficult to analyze the traffic sent so far. Note to Firefox devs: enabling "persist logs" should persist the logs. Don't clear them! None of this is perfect and I never found a video I couldn't eventually download (timing attacks ftw), but I do wish I could find a deeper explanation on how this all works.

no_time 3y ago
>Almost all of them have the same protection: some code that triggers only when you open the tools and stops the video by creating a debugger statement you cannot skip If you missed it, not so long ago there was a submission that evaded exactly this. Their solution is so simple yet effective: Recompiling the browser with the debugger keyword renamed. Made me smile. https://news.ycombinator.com/item?id=36961445 https://news.ycombinator.com/item?id=36961445
- deleted 3y ago
  [deleted]
- matheusmoreira 3y ago
  Honestly recompilation shouldn't even be necessary, browsers should just let us disable the debugger statement when "clever" sites start taking advantage of it. This usage of debuggers to circumvent our tools is abuse and should be literally impossible unless we consent to it. Our computers are our realms. God giveth and god taketh away.
  - rivo 3y ago
    Chrome Developer Tools allows this. There's a button in the Source tab to deactivate breakpoints from debug statements. The button looks like an arrow cut in half.

1vuio0pswjnm7 3y ago

"Have you ever tried to download videos from YouTube? I mean manually without relying on software like youtube-dl, yt-dlp or one of "these" websites. It's much more complicated than you might think." This reminds me of some sort of fizzbuzz test. This is not complicated at all. There is no need to use the Range header or run Javascript. The short script below does not download anything because there is no need. It does not use Range headers, it does not run Javascript and it makes only one TCP connection. With the JSON it fetches, one can simply extract the videoplayback URLs and put them in a locally-hosted HTML page with no Javascript. #!/bin/sh # usage: echo videoId | $0 <-- this will indicate len to use # usage: echo videoId | $0 len | openssl s_client -connect www.youtube.com:443 -ign_eof # usage: $0 len < videoId-list | openssl s_client -connect www.youtube.com:443 -ign_eof ( while read x;do test ${#x} -eq 11||continue if test $# -ne 1;then len=${#x};x=$(grep -m1 ^\{ $0|sed 's/\$x//'|wc -c);exec echo usage: ${0##*/} $((x+len));fi cr=$(printf '\r'); sed "/^[a-zA-Z].*: /s/$/$cr/;s/^$/$cr/" << eof POST /youtubei/v1/player?key=AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w HTTP/1.1 Host: www.youtube.com Content-Type: application/json Content-Length: $1 Connection: keep-alive {"context": {"client": {"clientName": "IOS", "clientVersion": "17.33.2" }}, "videoId": "$x", "params": "CgIQBg==", "playbackContext": {"contentPlaybackContext": {"html5Preference": "HTML5_PREF_WANTS"}}, "contentCheckOk": true, "racyCheckOk": true} eof done printf '\r\n' printf 'GET /robots.txt HTTP/1.0\r\nHost: www.youtube.com\r\nConnection: close\r\n\r\n'; ) For processing the JSON I wrote custom utilities in C that (a) extract videoIds and other useful strings, (b) generate HTTP similar to above, and (c) filter the returned JSON into CSV, SQL or HTML. For me, these run faster than Python and jq and are easier to edit. Using these utilities I can also do full searches that return hundreds to thousands of results and I can easily exclude all "suggested" or "recommended" videos. CSV output 1666520150,23 Oct 2022 10:15:50 UTC,22,aqz-KE-bpKQ,"Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film",00:10:35,635,UCSMOQeBJ2RAnuFungnQOxLg,19211597,"Blender" SQL output INSERT INTO t1(ts,utc,itag,vid,title,dur,len,cid,views,author) VALUES(1666520150,'23 Oct 2022 10:15:50 UTC',22,'aqz-KE-bpKQ','Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film','00:10:35',635,'UCSMOQeBJ2RAnuFungnQOxLg',19211597,'Blender') ON CONFLICT(vid) DO UPDATE SET views=excluded.views; HTML output Looks just like CSV except vid is a hyperlink

3y ago

AltruisticGapHN 3y ago

On a sidenote am I imagining things or do videos actually look a tiny bit better in YouTube? This really has puzzled me. I downloaded a few favourites and watch them on VLC or Infuse on my AppleTV. In the YouTube app I can use the "nerd stats" to confirm I am viewing the exact same video/audio streams... ... it could be my imagination but it seems like YouTube does a really subtle kind of filter that makes the "blocky" compression artifacts smoother. It doesnt ehance edges or anything - my guess is it looks for areas WITHOUT edges where there are sublte shfts of colour, and it makes the blocky artifacts less prominent. ... it's really subtle and I still cant tell if its just my imagination, like my OCD thinking that my downloaded video doesnt look as good and yet, I noticed on YouTube the video feels more vibrant and solid. When I watch my downloaded vid there are these really sublte, but noticable artifacts often in the background , in the shadows and these constant tiny little jitters even on a 1440p video - make the final picture look not as good. Am I making this up? Audio wise there is definitely a change as well. YouTube audio is always more or less level for me, while a downloaded video always needs to crank up the volume which is annoying. I wish players like VLC or Infuse did whatever YouTube does to make videos just more pleasant to look at. I dont think YoUTube changes the colours or does any kind of vibrancy filter though I may be wrong, but it does things to "level" audio so that you have a more consistent experience going from one video/channel to another.

AltruisticGapHN 3y ago
If anyone's still reading this or anyone cares here is what I found out: - the issue I was experiencing is with a lower quality encoding vp9 stream - from a video that was recently uploaded (explained below) - though I didn't trust smaller filesize VP9 streams initially, they look in fact noticably better - the picture is smoother, cleaner, the artifacts of compression are less visible. Where AAC can have jittery/glittery distracting dots moving in the background in areas where you have subtle gradients (eg. a plain wall) - Vp9 has none of these, those areas look smoother and cleaner and it gives an overall nicer looking picture without compromising the detail as far I can tell - Opus audio stream appears to have less of the ReplayGain issue, I'm not sure - but since I downloaded Opus instead of the 140 m4a stream I notice I dont need to adjust the volume compared to viewing same video in YouTube - and since the codec is newer anyway and the filesize is relatively the same or a tad smaller - also it is in 48k not 44k, I am going to download Opus from here on - a very confusing thing is it appears ; for a recent upload ; you can have an initial VP9 stream of say 500mb which is in fact no better than the AAC and havs the grainy artifacts - and the vp9 stream gets replaced weeks later by one significantly smaller like 400mb vs 500 mb !! and looks way better . whic hsuggst there was a first pass with low quality encoding, replaced by a higher quality encoding later - therefore my assumption that larger filesize is better was wrong

thrdbndndn 3y ago

> To bypass this limitation, we can break the download into several smaller parts using the HTTP Range header. This header allows you to specify which part of the file you want to download with each request (eg: Range bytes=2000-3000). The following code implements this logic. Last time I read discussion about it in yt-dlp repo [1], you can actually bypass it by just adding range=xxx query parameter (not header), and it will return to full speed even if your range is just the whole thing. And IIRC YouTube have already lifted this restriction. Edit: find the ref [1] https://github.com/yt-dlp/yt-dlp/issues/6400 https://github.com/yt-dlp/yt-dlp/issues/6400

kelvinjps 3y ago

I always wondered How YouTube distribute videos l, it's the smoothest video platform, even on when I had crappy internet it worked fine also not all.platform works well in south America.The closest to YouTube it's Netflix but lacks behinds a lot.

majewsky 3y ago
They have cache servers deployed into the ISP networks close to your home. The ISPs allow them to do this because they also benefit from it: Their bottleneck is the connection from their own network into the wider internet backbone, and having cache servers for the big CDNs takes a huge chunk of load away from that bottleneck. Here's documentation about a similar setup for Netflix: https://openconnect.netflix.com/ https://openconnect.netflix.com/
- simonjgreen 3y ago
  This is not a universal answer and I think deserves some correction. 1. Majority of ISPs do not host any cache for Google content 2. Credible ISPs do not have bottlenecks at the transit or peering level 3. Netflix makes use of much more local caching but their model works very differently to Youtubes 4. The concept of "internet backbone" does not really translate to reality. Peering is significantly more mesh-like than that, and transit more diverse. Source: I have owned multiple ISPs, and still do.

bayesianbot 3y ago

Anyone else having throttling problems with yt-dlp lately? I always watch youtube through mpv that uses yt-dlp on the background, but last week it's been terrible. It starts quickly (I've throttled it to 500kBps so that's the starting speed) but then after a while I'm getting a second of stream for three seconds of download, so I gotta queue up the video long time before playing. I'm using git version of yt-dlp and haven't noticed anything related in the git issues.

jorams 3y ago
mpv only uses yt-dlp to get a video URL, then passes that URL to ffmpeg. ffmpeg doesn't implement the workarounds with range headers, so you get throttled. It's possible to have yt-dlp perform the download, pipe it to mpv and make mpv play from stdin, but it breaks seeking to parts that haven't been downloaded yet. There are many issues about this in the mpv issue tracker.

ck2 3y ago

I mean you don't actually think this will continue working a week after it's widely shared, right?

antiloper 3y ago
yt-dlp is open source and I'm sure the Google engineers have been aware of it ever since it or it's ancestors were released.

abalashov 3y ago

Not trying to be a sketchy contrarian, but why would you do this with JavaScript? It just doesn't seem very fit for purpose...

benregenspan 3y ago
The most interesting part of this (the bypass itself) involves executing a Javascript challenge. It's very convenient to do that from Javascript (the author mentions that Python implementations need to add a Javascript interpreter). Besides that, it's mostly async I/O which: 1) might be annoying in earlier versions of Javascript, but with Promises and async/await, it's very clear and readable (to me) in Javascript; 2) is the exact case (I/O-bound, not CPU-bound) where Node.js performs efficiently.
hombre_fatal 3y ago
You should at least say why you don't think Javascript is fit for the trivial task of downloading files. Especially in an article that already had to unwind and explain Youtube's Javascript code.

Ianpett 3y ago

[dead]

rickreynoldssf 3y ago

YouTube changes small things in this process all the time. I used to work on an internal editing tool for YouTube videos that needed the MP4 files. Every month or so the editor would break because of a YouTube change and I needed to dive into the debugger to see what they changed and adjust to that.

swyx 3y ago

> he most popular one is yt-dlp (a fork of youtube-dl) programmed in Python, but it includes its own custom JavaScript interpreter to transform the n parameter. ah i remember this one: https://news.ycombinator.com/item?id=32793061 https://news.ycombinator.com/item?id=32793061 i confess i still dont really understand why they had to make this but i'd love to hear the story behind it

causi 3y ago

I highly recommend anybody who cares about content on Youtube to download the videos you like and maintain local copies. Material is being deleted or hidden faster than ever and Youtube is only going to get more user-hostile over time.

PeterStuer 3y ago

In the endgame you will have to pay in WorldCoin to keep watching your screen, and you can only earn that untradable WorldCoin by viewing the ads, monitored through the mini Orb embedded in the screen. You didn't truly believe this was about UBI did you? We solved that one ages ago with bank accounts and KYC.

wdb 3y ago
Nothing with the eye?