4 ms·
This is from 2020. Besides a small change to the "Introduction to the Command Line" section, it has not been updated. Back in 2015, the author reported using
by 1vuio0pswjnm7 4y ago
This is from 2020. Besides a small change to the "Introduction to the Command Line" section, it has not been updated.
Back in 2015, the author reported using CasperJS to scrape public LinkedIn profiles. The author reported this was a PITA.
Here the author recommends using WebDriver implementations, e.g., chromedriver or geckodriver, in addition to scripting language frameworks such as Puppeteer and Selenium. Is scraping LinkedIn still a PITA.
Because the examples given are always relatively simple, i.e., not LinkedIn, I am skeptical when I see "web scraping" tutorials using Python frameworks and cURL as the only recommended option for automated public data/information retrieval from the www.[FN1,2] I use none of the above. For small tasks like the examples given in these tutorials, the approaches I use are not nearly as sophisticated/complicated and yet they are faster and use fewer resources than using Python and/or cURL. They are also easier to update if something changes. That is in part because (1) the binaries I use are smaller, (2) I do not rely on scripting languages[FN3] and third party libraries (and so much less code involved), (3) the programs I use start working immediately whereas Python takes seconds to start up and (4) compared to the programs I use, cURL as a means of sending HTTP requests is inflexible, e.g., one is limited to what "options" cURL provides and cURL has no option for HTTP/1.1 pipelining.
1. LinkedIn's so-called "technological measures" to prevent retrieval of public information have failed. Similarly, its attempts to prevent retrieval of public information through intimidation, e.g., cease-and-desist letters and threats of CFAA claims, have failed. Tutorials on "web scraping" that extol Python frameworks should use LinkedIn as an example instead of trivial examples for which using Python is, IMHO, overkill.
2. What would be more interesting is a Rosetta Code for "web scraping" tasks. There are many, many ways to do public data/information retrieval from the www. Using scripting languages such as Python, Ruby, NodeJS, etc. and frameworks are one way. That approach may be ideally suited for large scale jobs, like those undertaken by what the author calls "internet companies". But for smaller tasks undertaken by individual www users for noncommercial purposes, e.g., this author's concept of "scrapism", there are also faster, less complicated and more efficient options.
3. Other than the Almquist shell
- saaaam 4y agoHi - I'd be interested to hear more details about what approaches you suggest!
- 1vuio0pswjnm7 4y agoTaking the examples from https://www.youtube-nocookie.com/embed/hA1ZsxE8VJg https://www.youtube-nocookie.com/embed/hA1ZsxE8VJg I am sharing how I approach the simple problems in the video without using Python or having any knowledge of CSS selectors. Retrieving the HTML echo https://www.nytimes.com|yy025|nc -vv proxy 80 > 1.htm yy025 is a flexible utility I wrote to generate custom HTTP from URLs. It is controlled through environmental variables. nc is a tcpclient, such as netcat. proxy is a HOSTS file entry for a localhost TLS proxy. The sequence "yy025|tcpclient" is normally contained in a shell script that adds a <base href> tag, something like #! /bin/sh yy025 5>.1 >.2 read x < .1; echo "<base href=https://$x />"; nc -vv proxy 80 < .2|yy045; yy045 is a utility that removes chunked transfer encoding. The benefit of using separate, small programs that do one thing will be illustrated in the solution for Problem 3.
- 1vuio0pswjnm7 4y agoProblem 1 - Extract the values of <h2> tags from NYT front page NB. In 1.htm, NYT is using the <h3> tag for headlines, not <h2> as in the 2020 video. Solution A - Use UNIX utilties grep -o "<h3[^\>]*>[^\<]*" 1.htm |sed -n '/indicate-hover/s/.*\">//p' The grep utility is ubiquitous, but the -o option is not. https://web.archive.org/web/20201202103125/https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html https://web.archive.org/web/20201202103125/https://pubs.open... For example, Plan9 grep does not have an -o option. This solution is fast and flexible, but not portable. There are myriad other portable solutions using POSIX UNIX utilities such as sh, tr and sed. For small tasks like those in "web scraping" tutorials these can still be faster than Python (due to Python start up time alone). Solution B - Use flex to make small, fast, custom utilities Create a file called 1.l that contains int fileno(FILE *); #define jmp (yy_start) = 1 + 2 * #define echo do {if(fwrite(yytext,(size_t)yyleng,1,yyout)){}}while(0) %s xa xb %option noyywrap noinput nounput %% \<h3 jmp xa; <xa>\> jmp xb; <xb>\< jmp 0; <xb>[^<]* echo;putchar(10); .|\n %% int main(){ yylex();exit(0);} Then compile with something like flex -8iCrf 1.l cc -std=c89 -Wall -pedantic -I$HOME -pipe lex.yy.c -static -o yy1 And finally, yy1 < 1.htm This is faster than Python. Solution C - Extract values from JSON instead of HTML The file 1.htm contains a large proportion of what appears to be JSON. I wrote a quick and dirty WIP JSON reformatter that takes web pages as input called yy059. https://news.ycombinator.com/item?id=31174088 https://news.ycombinator.com/item?id=31174088 yy059 < 1.htm|sed -n '/promotionalHeadline\":\"[^\"]/p'|cut -d\" -f4 Sure enough, the JSON contains the headlines. One could rewrite Solution B to extract from the JSON instead of the HTML.
- is_true 4y agoSorry if I missed something, but, Which programs do you use?