2 ms·
YouTube Continuation Searches
#!/bin/sh
# 1.sh
# usage: 1.sh query
# usage: 1.sh < query.yts >> query.yts
_x2_(){ local x=$(printf '\r\n');sed "s/ *//;s/$/$x/";};
case $# in :-)
;;0) # continuation search
KEY=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8;
MIN=\"context\":{\"client\":{\"hl\":\"en-US\",\"gl\":\"US\",\"clientName\":\"MWEB\",\"clientVersion\":\"2.20210916.06.01\"}},\"continuation\":;
TOKEN=$(sed -n "s/\"token\":/$MIN/p"|sed -n \$p);
_x2_ <<eof|nc -vvn 127.8 80|yy045|yy059
POST /youtubei/v1/search?key=$KEY HTTP/1.1
Host: www.youtube.com
Content-Length: ${#TOKEN}
Content-Type: application/json
Connection: close
$TOKEN
eof
;;*) # initial search
S=$(echo "$@"|sed 's/ /+/g');
_x2_ <<eof|nc -vvn 127.8 80|yy045|yy059|sed -n "/\"responseContext\":/,/\"searchSubMenu\":/w$S.yts"
GET /results?search_query="$S" HTTP/1.1
Host: www.youtube.com
Connection: close
eof
esac;
For an alternative way to search YouTube from the command line and create TSV tables of results, see, e.g.,
https://github.com/MarcoLucidi01/bin/blob/master/ytsearch
No support for continuation searches unfortunately.
- 1vuio0pswjnm7 5y ago1.sh uses netcat with stunnel listening on 127.0.0.8:80 stunnel stunnel.cfg #kill $(cat /tmp/1.pid) stunnel.cfg: debug=debug pid=/tmp/1.pid foreground=no [ youtube ] accept=127.0.0.8:80 client=yes connect=64.233.177.100:443 options=NO_TICKET options=NO_RENEGOTIATION renegotiation=no sni= sslVersion=TLSv1.3
- 1vuio0pswjnm7 5y ago1.sh uses two custom filters called yy045 and yy059. These are created with flex. flex -8iCrf 045.l cc -std=c89 -Wall -pedantic -pipe lex.yy.c -static -o yy045 flex -8iCrf 059.l cc -std=c89 -Wall -pedantic -pipe lex.yy.c -static -o yy059 045.l: /* decode chunked transfer */ int fileno (FILE *); xa "\15"|"\12" xb "\15\12" %option noyywrap nounput noinput %% ^[A-Fa-f0-9]+{xa} {xa}+[A-Fa-f0-9]+{xa} {xb}[A-Fa-f0-9]+{xb} %% int main(){ yylex();exit(0);} 059.l: /* reformat json */ int fileno (FILE *); #define echo do{if(fwrite(yytext,(size_t)yyleng,1,yyout)){}}while(0) #define jmp (yy_start) = 1 + 2 * #define pc(x) putchar(x) %option nounput noinput noyywrap %s xa xb %% \173 pc(10);pc(123);jmp xa; <xa>\175 echo;jmp 0; <xa>\40/[^\40] echo; <xa>[^\175\n\40]+ echo; \n|. %% int main(){ yylex();exit(0);}
- 1vuio0pswjnm7 5y agoExample: Turning the YouTube JSON into a TSV table using UNIX utilities. The initial YouTube web search returns a JSON schema that differs from one that is returned from a continuation search. 2.sh 0 is for the first 15 results 2.sh 1 is for the rest of the results #!/bin/sh # 2.sh # usage: 2.sh 0 < query.yts # usage: 2.sh 1 < query.yts x=$(echo x|tr x '\34'); tr -cd '[ -~\12]' \ |sed -n " /\"text\": \"YouTube Home\"/d; /canonicalBaseUrl/p; s/\\\\.//g; s/.*videoId/$x{ \"videoId/;/thumbnail/{/videoId/p;}; /\"text\"/p; /simpleText/{/views/p; }" \ |tr -d '\12'|tr '\34' '\12' \ |case $1 in :) ;;0)cut -d\" -f4,22,28,42,64,174 ;;1)cut -d\" -f4,10,14,24,32,36 esac \ |tr '\42' '\11' \ |sed 's/snippetText$/0 seconds/;/primaryContents/d;/canonicalBaseUrl/d;/commandMetadata/d' TODO: 2.sh does not handle entries for "YouTube Movies" correctly.