3 ms·

I've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real wor

I've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real world example, I was looking at scraping unjobs.org for a friend the other day. The need for JS rendering turned the job from 15 minutes of requests and beautifulsoup into a full-blown session with selenium, geckodriver etc. I'm not saying the linked framework isn't nice, I've not looked at it, but tools for simple scraping are plenty and easy to use. There's a lot more that a new framework needs to do to distinguish itself. I'd love something that makes JS rendering, proxy-rotation and catchpa solving easier, in a nice package I can deploy myself.

1vuio0pswjnm7 3y ago
Thank you for providing an example. As expected, retrieving the jobs listings from unjobs.org is trivial. Below is a simple demonstration using only common UNIX utilties and minimising the number of TCP conections. No browser. No Javascript. No Selenium. No Geckodriver. No proxies. Step 1 requires 40 TCP connections and completes in under a minute. Step 2 requires one TCP connection and completes in 10min. (NB. The connection minimisation used here, what RFCs used to call "web etiquette", is not possible using a popular headless graphical browser.) 1.htm is 1.5M, 2.htm is 40M # step 1 x="some user-agent string" # e.g., https://raw.githubusercontent.com/51Degrees/Device-Detection/master/data/20000%20User%20Agents.csv n=1;while true;do test $n -le 40||break # unjobs.org only shows listings 1-1000 case $n in 9|17|25|33)sleep 10;esac # need a delay after every 8 requests echo "GET /new/$n HTTP/1.0@Host: unjobs.org@User-Agent: $x@" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof -servername unjobs.org n=$((n+1)); done > 1.htm # step 2 n=1000; grep -Eo -m$n /vacancies/1[0-9]\{12\} 1.htm \ |tail -$n \ |sed "\$!s>.*>GET & HTTP/1.1@Host: $x@Connection: keep-alive@>; \$s>.*>GET & HTTP/1.1@Host: $x@Connection: close@>" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof > 2.htm If provided with an example of what the formatted output should loook like, I will demonstrate how to do it quickly and easily, without Python. Quite sure the text processing methods I use to extract data from HTML are faster than Python.

1vuio0pswjnm7 3y ago

Thank you for providing an example. As expected, retrieving the jobs listings from unjobs.org is trivial. Below is a simple demonstration using only common UNIX utilties and minimising the number of TCP conections. No browser. No Javascript. No Selenium. No Geckodriver. No proxies. Step 1 requires 40 TCP connections and completes in under a minute. Step 2 requires one TCP connection and completes in 10min. (NB. The connection minimisation used here, what RFCs used to call "web etiquette", is not possible using a popular headless graphical browser.) 1.htm is 1.5M, 2.htm is 40M # step 1 x="some user-agent string" # e.g., https://raw.githubusercontent.com/51Degrees/Device-Detection/master/data/20000%20User%20Agents.csv n=1;while true;do test $n -le 40||break # unjobs.org only shows listings 1-1000 case $n in 9|17|25|33)sleep 10;esac # need a delay after every 8 requests echo "GET /new/$n HTTP/1.0@Host: unjobs.org@User-Agent: $x@" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof -servername unjobs.org n=$((n+1)); done > 1.htm # step 2 n=1000; grep -Eo -m$n /vacancies/1[0-9]\{12\} 1.htm \ |tail -$n \ |sed "\$!s>.*>GET & HTTP/1.1@Host: $x@Connection: keep-alive@>; \$s>.*>GET & HTTP/1.1@Host: $x@Connection: close@>" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof > 2.htm If provided with an example of what the formatted output should loook like, I will demonstrate how to do it quickly and easily, without Python. Quite sure the text processing methods I use to extract data from HTML are faster than Python.