3 ms·

I had a look at https://github.com/philippta/flyscrape/blob/master/scrape.go https://github.com/philippta/flyscrape/blob/master/scrape.go. It’s just using the b

by oefrha 3y ago

I had a look at https://github.com/philippta/flyscrape/blob/master/scrape.go https://github.com/philippta/flyscrape/blob/master/scrape.go. It’s just using the builtin HTTP client to fire off requests, with an identifying user agent. Which means it’s useless for scraping most real world sites you may want to scrape, unfortunately. You’ll get served a JavaScrpt challenge, or not even that (many sites will refuse to serve anything if they see a random user agent like flyscrape/1.0).

1vuio0pswjnm7 3y ago
What are some examples of "most real world sites". What are some examples of sites that are not "most real world sites". Is HN a "real world site". What percentage of sites submitted to HN are "most real world sites". (IME, it's a minority fraction.) Why not just delete or change the user-agent line in scrape.go before compiling. (Personal experience: I have been successfully retrieving information from the www for decades without including a user-agent header. The number of sites I have found that require this header is relatively small. It does not rise to the level of "most".) HN replies usually fail to include even a single example. Similarly, do examples provided for scraper programs and libraries ever include "most real world sites".
- specproc 3y ago
  I've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real world example, I was looking at scraping unjobs.org for a friend the other day. The need for JS rendering turned the job from 15 minutes of requests and beautifulsoup into a full-blown session with selenium, geckodriver etc. I'm not saying the linked framework isn't nice, I've not looked at it, but tools for simple scraping are plenty and easy to use. There's a lot more that a new framework needs to do to distinguish itself. I'd love something that makes JS rendering, proxy-rotation and catchpa solving easier, in a nice package I can deploy myself.
  - 1vuio0pswjnm7 3y ago
    Thank you for providing an example. As expected, retrieving the jobs listings from unjobs.org is trivial. Below is a simple demonstration using only common UNIX utilties and minimising the number of TCP conections. No browser. No Javascript. No Selenium. No Geckodriver. No proxies. Step 1 requires 40 TCP connections and completes in under a minute. Step 2 requires one TCP connection and completes in 10min. (NB. The connection minimisation used here, what RFCs used to call "web etiquette", is not possible using a popular headless graphical browser.) 1.htm is 1.5M, 2.htm is 40M # step 1 x="some user-agent string" # e.g., https://raw.githubusercontent.com/51Degrees/Device-Detection/master/data/20000%20User%20Agents.csv n=1;while true;do test $n -le 40||break # unjobs.org only shows listings 1-1000 case $n in 9|17|25|33)sleep 10;esac # need a delay after every 8 requests echo "GET /new/$n HTTP/1.0@Host: unjobs.org@User-Agent: $x@" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof -servername unjobs.org n=$((n+1)); done > 1.htm # step 2 n=1000; grep -Eo -m$n /vacancies/1[0-9]\{12\} 1.htm \ |tail -$n \ |sed "\$!s>.*>GET & HTTP/1.1@Host: $x@Connection: keep-alive@>; \$s>.*>GET & HTTP/1.1@Host: $x@Connection: close@>" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof > 2.htm If provided with an example of what the formatted output should loook like, I will demonstrate how to do it quickly and easily, without Python. Quite sure the text processing methods I use to extract data from HTML are faster than Python.
- rmilejczz 3y ago
  I was scraping a Wordpress site a few months ago using Go and I had to spoof my user agent to get results. So it definitely happens
  - 1vuio0pswjnm7 3y ago
    But the site remains unnamed. To prove, why not let others test the theory. Tell us the Wordpress site. IME, Wordpress sites do not require a user agent header. Contrast, for example, with Squarespace sites which do require a user agent header. IME, if send a user agent header with a particular value, then some sites will block, depending on the value. Whereas if _do not send_ a user agent header, then almost all sites will accept. The so-called "developers" who publish code and commentary about "scraping" almost always include a user agent header. Usually they use fake values. They try to guess the "correct" values to send. Im not referring to sending fake values. Im referring to not sending the header at all. No "spoofing" is involved. This works for me, for decades, across thousands of websites.
- Klonoar 3y ago
  Yeah, nobody cares that most sites will work with it. The number of sites that require it might be small, but that number tends to include significant and notable ones - the ones that most people actually wind up wanting to crawl. Cloudflare & co make this more difficult with each passing year. OP is making a very well known day-one point about this topic. It is somewhat surprising that the library doesn't offer a way to dynamically set it. Edit: and this isn't even getting into stuff like Tls fingerprinting, header order, etc
- oefrha 3y ago
  “Most real world sites” and “most real world sites you may want to scrape” are different, especially when weighted by information volume, so your selective quoting doesn’t help. Alexa top 100 probably contain more information, especially new information, than the rest of Alexa top 10000 combined (random guess I pulled out of my ass, don’t quote me on that), so the overwhelmingly scraping-resistant Alexa top 100 is what most scraping effort is directed against. Anyway, I’m not interested in a pedantic debate. Most (but not all) people who have attempted to scrape any popular site, or unpopular site behind Cloudflare at above-minimum protection level, in recent years should know exactly what I’m taking about. Edit: In case you’re not aware, many of the sites I have in mind are/were capable of defeating puppeteer-extra-plugin-stealth. User agent is only the most basic thing, like level 1 in a hundred-level dungeon.
- deleted 3y ago
  [deleted]

by oefrha 3y ago

1vuio0pswjnm7 3y ago

What are some examples of "most real world sites". What are some examples of sites that are not "most real world sites". Is HN a "real world site". What percentage of sites submitted to HN are "most real world sites". (IME, it's a minority fraction.) Why not just delete or change the user-agent line in scrape.go before compiling. (Personal experience: I have been successfully retrieving information from the www for decades without including a user-agent header. The number of sites I have found that require this header is relatively small. It does not rise to the level of "most".) HN replies usually fail to include even a single example. Similarly, do examples provided for scraper programs and libraries ever include "most real world sites".

specproc 3y ago
I've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real world example, I was looking at scraping unjobs.org for a friend the other day. The need for JS rendering turned the job from 15 minutes of requests and beautifulsoup into a full-blown session with selenium, geckodriver etc. I'm not saying the linked framework isn't nice, I've not looked at it, but tools for simple scraping are plenty and easy to use. There's a lot more that a new framework needs to do to distinguish itself. I'd love something that makes JS rendering, proxy-rotation and catchpa solving easier, in a nice package I can deploy myself.
- 1vuio0pswjnm7 3y ago
  Thank you for providing an example. As expected, retrieving the jobs listings from unjobs.org is trivial. Below is a simple demonstration using only common UNIX utilties and minimising the number of TCP conections. No browser. No Javascript. No Selenium. No Geckodriver. No proxies. Step 1 requires 40 TCP connections and completes in under a minute. Step 2 requires one TCP connection and completes in 10min. (NB. The connection minimisation used here, what RFCs used to call "web etiquette", is not possible using a popular headless graphical browser.) 1.htm is 1.5M, 2.htm is 40M # step 1 x="some user-agent string" # e.g., https://raw.githubusercontent.com/51Degrees/Device-Detection/master/data/20000%20User%20Agents.csv n=1;while true;do test $n -le 40||break # unjobs.org only shows listings 1-1000 case $n in 9|17|25|33)sleep 10;esac # need a delay after every 8 requests echo "GET /new/$n HTTP/1.0@Host: unjobs.org@User-Agent: $x@" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof -servername unjobs.org n=$((n+1)); done > 1.htm # step 2 n=1000; grep -Eo -m$n /vacancies/1[0-9]\{12\} 1.htm \ |tail -$n \ |sed "\$!s>.*>GET & HTTP/1.1@Host: $x@Connection: keep-alive@>; \$s>.*>GET & HTTP/1.1@Host: $x@Connection: close@>" \ |tr @@ '\r\n' \ |openssl s_client -connect unjobs.org:443 -ign_eof > 2.htm If provided with an example of what the formatted output should loook like, I will demonstrate how to do it quickly and easily, without Python. Quite sure the text processing methods I use to extract data from HTML are faster than Python.
rmilejczz 3y ago
I was scraping a Wordpress site a few months ago using Go and I had to spoof my user agent to get results. So it definitely happens
- 1vuio0pswjnm7 3y ago
  But the site remains unnamed. To prove, why not let others test the theory. Tell us the Wordpress site. IME, Wordpress sites do not require a user agent header. Contrast, for example, with Squarespace sites which do require a user agent header. IME, if send a user agent header with a particular value, then some sites will block, depending on the value. Whereas if _do not send_ a user agent header, then almost all sites will accept. The so-called "developers" who publish code and commentary about "scraping" almost always include a user agent header. Usually they use fake values. They try to guess the "correct" values to send. Im not referring to sending fake values. Im referring to not sending the header at all. No "spoofing" is involved. This works for me, for decades, across thousands of websites.
Klonoar 3y ago
Yeah, nobody cares that most sites will work with it. The number of sites that require it might be small, but that number tends to include significant and notable ones - the ones that most people actually wind up wanting to crawl. Cloudflare & co make this more difficult with each passing year. OP is making a very well known day-one point about this topic. It is somewhat surprising that the library doesn't offer a way to dynamically set it. Edit: and this isn't even getting into stuff like Tls fingerprinting, header order, etc
oefrha 3y ago
“Most real world sites” and “most real world sites you may want to scrape” are different, especially when weighted by information volume, so your selective quoting doesn’t help. Alexa top 100 probably contain more information, especially new information, than the rest of Alexa top 10000 combined (random guess I pulled out of my ass, don’t quote me on that), so the overwhelmingly scraping-resistant Alexa top 100 is what most scraping effort is directed against. Anyway, I’m not interested in a pedantic debate. Most (but not all) people who have attempted to scrape any popular site, or unpopular site behind Cloudflare at above-minimum protection level, in recent years should know exactly what I’m taking about. Edit: In case you’re not aware, many of the sites I have in mind are/were capable of defeating puppeteer-extra-plugin-stealth. User agent is only the most basic thing, like level 1 in a hundred-level dungeon.
deleted 3y ago
[deleted]