3 ms·
"Have you ever tried to download videos from YouTube? I mean manually without relying on software like youtube-dl, yt-dlp or one of "these" websites. It's much
by 1vuio0pswjnm7 3y ago
"Have you ever tried to download videos from YouTube? I mean manually without relying on software like youtube-dl, yt-dlp or one of "these" websites. It's much more complicated than you might think."
This reminds me of some sort of fizzbuzz test. This is not complicated at all. There is no need to use the Range header or run Javascript.
The short script below does not download anything because there is no need. It does not use Range headers, it does not run Javascript and it makes only one TCP connection. With the JSON it fetches, one can simply extract the videoplayback URLs and put them in a locally-hosted HTML page with no Javascript.
#!/bin/sh
# usage: echo videoId | $0 <-- this will indicate len to use
# usage: echo videoId | $0 len | openssl s_client -connect www.youtube.com:443 -ign_eof
# usage: $0 len < videoId-list | openssl s_client -connect www.youtube.com:443 -ign_eof
(
while read x;do
test ${#x} -eq 11||continue
if test $# -ne 1;then len=${#x};x=$(grep -m1 ^\{ $0|sed 's/\$x//'|wc -c);exec echo usage: ${0##*/} $((x+len));fi
cr=$(printf '\r');
sed "/^[a-zA-Z].*: /s/$/$cr/;s/^$/$cr/" << eof
POST /youtubei/v1/player?key=AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w HTTP/1.1
Host: www.youtube.com
Content-Type: application/json
Content-Length: $1
Connection: keep-alive
{"context": {"client": {"clientName": "IOS", "clientVersion": "17.33.2" }}, "videoId": "$x", "params": "CgIQBg==", "playbackContext": {"contentPlaybackContext": {"html5Preference": "HTML5_PREF_WANTS"}}, "contentCheckOk": true, "racyCheckOk": true}
eof
done
printf '\r\n'
printf 'GET /robots.txt HTTP/1.0\r\nHost: www.youtube.com\r\nConnection: close\r\n\r\n';
)
For processing the JSON I wrote custom utilities in C that (a) extract videoIds and other useful strings, (b) generate HTTP similar to above, and (c) filter the returned JSON into CSV, SQL or HTML. For me, these run faster than Python and jq and are easier to edit. Using these utilities I can also do full searches that return hundreds to thousands of results and I can easily exclude all "suggested" or "recommended" videos.
CSV output
1666520150,23 Oct 2022 10:15:50 UTC,22,aqz-KE-bpKQ,"Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film",00:10:35,635,UCSMOQeBJ2RAnuFungnQOxLg,19211597,"Blender"
SQL output
INSERT INTO t1(ts,utc,itag,vid,title,dur,len,cid,views,author) VALUES(1666520150,'23 Oct 2022 10:15:50 UTC',22,'aqz-KE-bpKQ','Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film','00:10:35',635,'UCSMOQeBJ2RAnuFungnQOxLg',19211597,'Blender') ON CONFLICT(vid) DO UPDATE SET views=excluded.views;
HTML output
Looks just like CSV except vid is a hyperlink
- 1vuio0pswjnm7 3y agoThis is demonstration of W3C Ethical Web Principles 6.11 and 6.12 https://www.w3.org/TR/ethical-web-principles/ https://www.w3.org/TR/ethical-web-principles/
- SuperCuber 3y agoLooks interesting. In the post, the author does echo -n '{"videoId":"aqz-KE-bpKQ","context":{"client":{"clientName":"WEB","clientVersion":"2.20230810.05.00"}}}' | http post 'https://www.youtube.com/youtubei/v1/player' | jq -r '.streamingData.adaptiveFormats[0].url' which is very similar to what you do, but runs into an issue of throttling to ~70Kbps. Is the difference just the "key" parameter? Do you get no throttling?
- 1vuio0pswjnm7 3y agoThere is no throttling when using the JSON returned by the HTTP request in the shell script or generated by the utilities I wrote. it's not the key the author is using, it's the post-data. Moreover, to get throttled videoplayback URLs with the "WEB" key and client info like the author is using, one does not need to make POST requests to /youtubei/v1/player. There are throttled videoplayback URLs in the HTML of the /watch?v= page. For example, curl -A "" -40s https://www.youtube.com/watch?v=aqz-KE-bpKQ|grep -o https://rr[^\"]*|sed -n 's/\\u0026/\&/g;/itag=22/p' It's ironic how the author is claiming this is complicated. That's his own doing.
- SuperCuber 3y agoInteresting. I ran the script to extract the json - that part was almost instant, then i used the first `url` field of `streamingData.adaptiveFormats`. I then ran curl 'https://...googlevideo.com...' --output video.mp4 for me the download is throttled to "768k", i assume thats in bits per second and not bytes which is very low: the random video i tried would take 8 minutes. on the other hand, yt-dlp videoIdHere does its processing then downloads the whole thing in about 5 seconds. Does that curl command run much faster for you? Or do you do something else?