3 ms·
"I build this bot to automate the process of going to wayback machine (or archive.ph) to find the archive link for posts with articles that have subscription bl
by 1vuio0pswjnm7 4y ago
"I build this bot to automate the process of going to wayback machine (or archive.ph) to find the archive link for posts with articles that have subscription blocks"
It is easy to automate this using a forward proxy. I use one on a loopback address. The way I have it set up, I add a given domain to a text file, reload,^1 and from then on, the proxy will rewrite the URLs using that domain to point to archive.org or whatever.
I have never encountered an archive.ph URL on HN where the site proxied, purportedly to avoid a "subscription block",^2 was not accessible without using the archive.ph URL. For example, sometimes the issue is simply that Javascript needs to be disabled, or one needs to use an AMP URL without Javascript. (The later looks great in a text-only browsersuch as links.) Whereas submissions for articles on sites that are password-protected, e.g., ft.com, rarely or never have archive.ph URLs.
This notion of "bots" is a never-ending source of amusement.
Everyone thinks they know what a "bot" is, but IMO that term alone tells us next to nothing. For example, one can write scripts, here using Python, that are disrespectful of server resources or whatever "rules" are in place and, as a result, get "banned". One can also write scripts that are respectful of server resources and any rules and never get banned. Automation is not per se "bots" and "bots" are not per se "bad". Computers and automation are everywhere. We do not label all automation using the internet as "bots". Whether something is "good" or "bad" depends on what a script is doing and, sometimes, who is using it.^3
I have been using automation for reading HN for over 10 years. Yes, it is possible to get banned but it depends on what a script does. The HN software does have limitations and one has to learn to work within those constraints.
1. Something like
echo reload |socat stdio /path/to/control/socket
2. IMO a true "subscription" would require a password. That is, non-subscribers would not have access to the content. Why do sites make content avaiable to to non-subscribers. Is it only to try to attract new subscribers. The site is gathering data about computer users and serving advertising. The site operator profits from the traffic from non-subscribers. If a site operator wants to make content only accesible to sucbscribers it can make that content password-protected. If the operator chooses not to use password protection then she may have a difficult time convincing a court the content published was not public. Annoying people with Javascript is not access control.
3. Corporations performing telemetry and remotely installing and running "updates", automatically, without explicit consent on computers that do not belong to them generally do not get labeled as "bots". Whereas using automation to "subscribe" to "tech" company websites that aim to collect data about users in order to target advertising might be labeled as "bots" in a perjorative sense, if one is in the business of online advertising.
- heywoodlh 4y ago> It is easy to automate this using a forward proxy. I use one on a loopback address. The way I have it set up, I add a given domain to a text file, reload,^1 and from then on, the proxy will rewrite the URLs using that domain to point to archive.org or whatever. This sounds super useful to me. Would you be able to point me to some resources to set up a workflow for something similar?
- 1vuio0pswjnm7 4y agoUsing haproxy^1 as an example To redirect to the most recent archive.org capture, I use a date in the distant future such as 203001234567890. This will automatically redirect to the latest capture. http-request set-uri https://web.archive.org/web/20301234567890if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f hosts-archive.txt } To redirect to the earliest archive.org capture, I use 1. This will automatically redirect to the earliest capture. http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f hosts-archive.txt } If I want to all requests for example.org files to retrieve from archive.org instead of example.org, I do something like echo example.org >> hosts-archive.txt echo reload|socat stdio /path/to/control/socket 1. https://www.haproxy.org https://www.haproxy.org for documentation.
- 1vuio0pswjnm7 4y ago"I have never encountered an archive.ph URL on HN where the site proxied, purportedly to avoid a "subscription block",^2 was not accessible without using the archive.ph URL. For example, sometimes the issue is simply that Javascript needs to be disabled, or one needs to use an AMP URL without Javascript. (The later looks great in a text-only browsersuch as links.) Whereas submissions for articles on sites that are password-protected, e.g., ft.com, rarely or never have archive.ph URLs." Correction: FT was wrong example I had forgotten ft.com like many other sites sends the plain text of the article to non-subscribers. It can be read in the page HTML. There is no way for non-subscribers to "opt-out" of receiving the article text when they follow ("click") a URL for a FT article. It is in fact common to find archive.ph URLs for ft.com submissions. Here is short program to make reading the article in the ft.com HTML page easier. It discards everything but the article text and replaces "\n" with newline. cat > 071.l /* ft.com */ int fileno(FILE *); #define jmp (yy_start) = 1 + 2 * #define echo do{if(fwrite(yytext,(size_t)yyleng,1,yyout)){}}while(0) %option noyywrap noinput nounput %s xb %% \"articleBody\":\" jmp xb; <xb>\",\"wordCount\" jmp 0; <xb>\134\156 putchar(10); <xb>\40\* putchar(32); <xb>. echo; \n|. %% int main(){ yylex();putchar(10);exit(0);} ^D flex -8iCrf 071.l cc-O3 -std=c89 -W -Wall -pedantic -pipe lex.yy.c -o yy071 #example usage curl -4s https://www.ft.com/content/fa956177-5daa-43fd-8a64-79f5eebcc309|yy071 curl -4o 1.htm https://www.ft.com/content/fa956177-5daa-43fd-8a64-79f5eebcc309 yy071 < 1.htm