5 ms·

One thing I'm interested in is scraping those annoying sites that require JavaScript execution. More and more webpages are requiring JS even to display anything

One thing I'm interested in is scraping those annoying sites that require JavaScript execution. More and more webpages are requiring JS even to display anything beyond a blank page. These sites self-select themselves for exclusion in scraping like this. I've looked into Headless Chrome, but I'd be interested to see a 'scraping framework' level abstraction for those sites.

deleted 9y ago
[deleted]
spiderfarmer 9y ago
You could look into PhantomJS. Also effectively a headless browser, but it's scriptable. http://phantomjs.org http://phantomjs.org
- afandian 9y ago
  I started with that. Last time I looked, the maintainer had stepped down because of Chrome Headless. And it's not a "scraping framework" any more than Chrome Headless is. edit: Headless Chrome has very rich scripting integration with e.g. https://github.com/webfolderio/cdp4j https://github.com/webfolderio/cdp4j
tym0 9y ago
You might have missed puppeteer[1], if you don't mind writing Javascript it seems to provide a simple interface for scraping. [1] https://github.com/GoogleChrome/puppeteer https://github.com/GoogleChrome/puppeteer
asciimoo 9y ago
Scraping JS-only sites is also possible without a headless browser, but requires a bit more debugging of the internal structure of these sites. Most of JS-only websites have API endpoints with JSON responses, which can make scraping more reliable than parsing custom (and sometimes invalid) HTML. The drawback of headless browser based scraping is that it requires significant amount of cpu time and memory compared to "static" scraping frameworks.
wenc 9y ago
As mentioned, Puppeteer was used in this project, which is Chrome based. I've also used Selenium (via Python -- I used BeautifulSoup to parse the resulting HTML) in the past for precisely for the reasons you stated. Selenium uses "web drivers", which lets you start other headless browsers as well (Firefox, Opera, IE, etc.). http://www.seleniumhq.org/ http://www.seleniumhq.org/ http://selenium-python.readthedocs.io/ http://selenium-python.readthedocs.io/ All it took was a couple of lines of Python code...
- webfolder 9y ago
  You should try cdp4j - https://webfolderio.github.io/cdp4j/ https://webfolderio.github.io/cdp4j/
bryanrasmussen 9y ago
yes, this would be capable of waiting, but it doesn't have the abstraction built in so it looks like it would have the same problem I often run into with scapers - you go through page to get links to follow/markup to act on and then have to wait a reasonable amount of time / catch some browser based event to determine if you need to look at content one more time before continuing.
feelin_googley 9y ago
"More and more webpages are requiring JS even to display anything beyond a blank page." Can you provide some example webpages so we can take a look? Also, I agree with @asciimoo's point about endpoints. One could make an argument that compared to the website design of the 1990's and 2000's, retrieving structured data from websites is actually getting easier not more difficult. I recall one period of time where the trend was to design websites entirely in Macromedia/Adobe Flash. Here is what is missing from this project (and many others like it): when providing software that performs text processing one needs to not only provide example code but also example output. This enables a user to quickly compare her current text-processing solution with the software being provided without having to install, review and run the unfamiliar software. For example, without some sample output she cannot test, e.g., whether her current solution produces the same output faster or using less code.
- afandian 9y ago
 With a minute's searching, here's a webpage that's meaningless without JavaScript: https://blogger.googleblog.com/ https://blogger.googleblog.com/ It's arguably a page of text not a web app that needs user interaction. I'm not trying to start an argument about JS: the consensus on HN seems to be that if you don't execute JavaScript you don't deserve to read webpages. I'm just saying that your website's clients are more diverse than 'normal' well-sighted humans. There may be machines reading the site, for all kinds of reasons. And regarding endpoints. One could make the argument that with AJAX we now have richer APIs. I disagree. We have a well-understood API for getting hypertext (HTTP) in a well-understood format (HTML) that works/worked across all websites. Replacing that with a custom-built API for every website isn't apples-for-apples.
 - feelin_googley 9y ago
 The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript. Blogger has a feed. Here is one way to retrieve it, in two steps. 1. get TargetBlogID 2. retrieve data optional: 3. format data for viewing Example: 1. x=$(exec curl https://blogger.googleblog.com \ |exec sed ' s/\\046/\&/g; s/\\46/\&/g; s/\\075/=/g; s/\\75/=/g; /targetBlogID/!d; s/.*targetBlogID=//; s/&.*//; '); 2. curl -o y.htm https://www.blogger.com/feeds/$x/posts/default 3. exec sed ' # ^M is "\r" s/^[0-9a-f]*^M//; s/</</g; s/>/>/g; s/&/\&/g; s/"/\"/g; 1i\ s/<name>/ name &/g; s/<uri>/ uri &/g; s/<generator>/ generator &/g; s/Blogger//; s/<id>/ id &/g; s/<published>/ published &/g; s/<email>/ email &/g; s/<title type=.text.>/ &/g; s/<openSearch:totalResults>/ total results &/g; s/<openSearch:startIndex>/ start index &/g; s/<openSearch:itemsPerPage>/ items per page &/g; s/<updated>/ updated &/g; s/<thr:total>/ thr:total &/g; s/<\/feed>/& /; s/^M*/ /; ' y.htm \ |exec tr -cd '\12\40-\176'
weberc2 9y ago
You'd need to parse and run the JavaScript, including a virtual DOM. You'd also need to support any browser events that the page depends on. It's conceptually simple, and getting support for 80-95% of cases is probably easily doable, but I imagine a long tail of cases where your browser emulator isn't quite compatible in an important way. I know Google and others do this to index the web, but in not sure about any existing open source projects.
chatmasta 9y ago
If you're targeting a specific site (as opposed to blindly spidering multiple), all JavaScript is actually better for scraping, in my experience. JavaScript apps communicate with an API 99% of the time, so to scrape them you can just replicate the API requests. And as a bonus you'll get nicely formatted JSON responses; no need to parse fragile HTML. Similarly, I've found that most sites worth scraping also have a mobile app, which you can run through a MITM proxy, and then simply write a scraper to call the API endpoints directly.
- assafmo 9y ago
  This is absolutely true. This is why SPAs are awesome to scrape
laktek 9y ago
I released Page.REST(https://page.rest#prerender https://page.rest#prerender) couple of weeks ago. It will pre-render JS based sites - so you can then extract rendered content using CSS selectors.
- jotto 9y ago
  Ditto: https://www.prerender.cloud/docs/api https://www.prerender.cloud/docs/api // URL to screenshot service.prerender.cloud/screenshot/https://www.google.com/ // URL to pdf service.prerender.cloud/pdf/https://www.google.com/ // URL to html (prerender) service.prerender.cloud/https://www.google.com/

deleted 9y ago

[deleted]

spiderfarmer 9y ago

You could look into PhantomJS. Also effectively a headless browser, but it's scriptable. http://phantomjs.org http://phantomjs.org

afandian 9y ago
I started with that. Last time I looked, the maintainer had stepped down because of Chrome Headless. And it's not a "scraping framework" any more than Chrome Headless is. edit: Headless Chrome has very rich scripting integration with e.g. https://github.com/webfolderio/cdp4j https://github.com/webfolderio/cdp4j

tym0 9y ago

You might have missed puppeteer[1], if you don't mind writing Javascript it seems to provide a simple interface for scraping. [1] https://github.com/GoogleChrome/puppeteer https://github.com/GoogleChrome/puppeteer

asciimoo 9y ago

Scraping JS-only sites is also possible without a headless browser, but requires a bit more debugging of the internal structure of these sites. Most of JS-only websites have API endpoints with JSON responses, which can make scraping more reliable than parsing custom (and sometimes invalid) HTML. The drawback of headless browser based scraping is that it requires significant amount of cpu time and memory compared to "static" scraping frameworks.

wenc 9y ago

As mentioned, Puppeteer was used in this project, which is Chrome based. I've also used Selenium (via Python -- I used BeautifulSoup to parse the resulting HTML) in the past for precisely for the reasons you stated. Selenium uses "web drivers", which lets you start other headless browsers as well (Firefox, Opera, IE, etc.). http://www.seleniumhq.org/ http://www.seleniumhq.org/ http://selenium-python.readthedocs.io/ http://selenium-python.readthedocs.io/ All it took was a couple of lines of Python code...

webfolder 9y ago
You should try cdp4j - https://webfolderio.github.io/cdp4j/ https://webfolderio.github.io/cdp4j/

bryanrasmussen 9y ago

yes, this would be capable of waiting, but it doesn't have the abstraction built in so it looks like it would have the same problem I often run into with scapers - you go through page to get links to follow/markup to act on and then have to wait a reasonable amount of time / catch some browser based event to determine if you need to look at content one more time before continuing.

feelin_googley 9y ago

"More and more webpages are requiring JS even to display anything beyond a blank page." Can you provide some example webpages so we can take a look? Also, I agree with @asciimoo's point about endpoints. One could make an argument that compared to the website design of the 1990's and 2000's, retrieving structured data from websites is actually getting easier not more difficult. I recall one period of time where the trend was to design websites entirely in Macromedia/Adobe Flash. Here is what is missing from this project (and many others like it): when providing software that performs text processing one needs to not only provide example code but also example output. This enables a user to quickly compare her current text-processing solution with the software being provided without having to install, review and run the unfamiliar software. For example, without some sample output she cannot test, e.g., whether her current solution produces the same output faster or using less code.

afandian 9y ago
With a minute's searching, here's a webpage that's meaningless without JavaScript: https://blogger.googleblog.com/ https://blogger.googleblog.com/ It's arguably a page of text not a web app that needs user interaction. I'm not trying to start an argument about JS: the consensus on HN seems to be that if you don't execute JavaScript you don't deserve to read webpages. I'm just saying that your website's clients are more diverse than 'normal' well-sighted humans. There may be machines reading the site, for all kinds of reasons. And regarding endpoints. One could make the argument that with AJAX we now have richer APIs. I disagree. We have a well-understood API for getting hypertext (HTTP) in a well-understood format (HTML) that works/worked across all websites. Replacing that with a custom-built API for every website isn't apples-for-apples.
- feelin_googley 9y ago
 The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript. Blogger has a feed. Here is one way to retrieve it, in two steps. 1. get TargetBlogID 2. retrieve data optional: 3. format data for viewing Example: 1. x=$(exec curl https://blogger.googleblog.com \ |exec sed ' s/\\046/\&/g; s/\\46/\&/g; s/\\075/=/g; s/\\75/=/g; /targetBlogID/!d; s/.*targetBlogID=//; s/&.*//; '); 2. curl -o y.htm https://www.blogger.com/feeds/$x/posts/default 3. exec sed ' # ^M is "\r" s/^[0-9a-f]*^M//; s/</</g; s/>/>/g; s/&/\&/g; s/"/\"/g; 1i\ s/<name>/ name &/g; s/<uri>/ uri &/g; s/<generator>/ generator &/g; s/Blogger//; s/<id>/ id &/g; s/<published>/ published &/g; s/<email>/ email &/g; s/<title type=.text.>/ &/g; s/<openSearch:totalResults>/ total results &/g; s/<openSearch:startIndex>/ start index &/g; s/<openSearch:itemsPerPage>/ items per page &/g; s/<updated>/ updated &/g; s/<thr:total>/ thr:total &/g; s/<\/feed>/& /; s/^M*/ /; ' y.htm \ |exec tr -cd '\12\40-\176'

weberc2 9y ago

You'd need to parse and run the JavaScript, including a virtual DOM. You'd also need to support any browser events that the page depends on. It's conceptually simple, and getting support for 80-95% of cases is probably easily doable, but I imagine a long tail of cases where your browser emulator isn't quite compatible in an important way. I know Google and others do this to index the web, but in not sure about any existing open source projects.

chatmasta 9y ago

If you're targeting a specific site (as opposed to blindly spidering multiple), all JavaScript is actually better for scraping, in my experience. JavaScript apps communicate with an API 99% of the time, so to scrape them you can just replicate the API requests. And as a bonus you'll get nicely formatted JSON responses; no need to parse fragile HTML. Similarly, I've found that most sites worth scraping also have a mobile app, which you can run through a MITM proxy, and then simply write a scraper to call the API endpoints directly.

assafmo 9y ago
This is absolutely true. This is why SPAs are awesome to scrape

laktek 9y ago

I released Page.REST(https://page.rest#prerender https://page.rest#prerender) couple of weeks ago. It will pre-render JS based sites - so you can then extract rendered content using CSS selectors.

jotto 9y ago
Ditto: https://www.prerender.cloud/docs/api https://www.prerender.cloud/docs/api // URL to screenshot service.prerender.cloud/screenshot/https://www.google.com/ // URL to pdf service.prerender.cloud/pdf/https://www.google.com/ // URL to html (prerender) service.prerender.cloud/https://www.google.com/