3 ms·
This is perhaps the fastest way to screenscrape a dynamically executed website. 1. First go get and run this code, which allows immediate gathering of all text
by austincheney 9y ago
This is perhaps the fastest way to screenscrape a dynamically executed website.
1. First go get and run this code, which allows immediate gathering of all text nodes from the DOM: https://github.com/prettydiff/getNodesByType/blob/master/getNodesByType.js https://github.com/prettydiff/getNodesByType/blob/master/get...
2. Extract the text content from the text nodes and ignore nodes that contain only white space:
let text = document.getNodesByType(3), a = 0, b = text.length, output = []; do { if ((/^(\s+)$/).test(text[a].textContent) === false) { output.push(text[a].textContent); } a = a + 1; } while (a < b); output;
That will gather ALL text from the page. Since you are working from the DOM directly you can filter your results by various contextual and stylistic factors. Since this code is small and executes stupid fast it can be executed by bots easily.
Test this out in your browser console.
- AznHisoka 9y agoAnd how do you do #1? Node, I presume?
- austincheney 9y agoNo, manually go there and copy/paste the code. Then when building your scraper bot use that code.
- AznHisoka 9y agobut how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?
- austincheney 9y agoYou could write a crawler in any language. Crawling is easy as you are listening for HTTP traffic and analyzing the HTML in the response. To accurately get the content in dynamically executed pages you need to interact with the DOM. This is the reason Google updated its crawler to execute JavaScript.
- AznHisoka 9y agoYep, I know, but that means if I am writing the crawler in Ruby/Python, this is not something I can do, right?
- austincheney 9y agoYes. The crawler can be written in nearly any language. The actual scraper probably has to be written in JavaScript in order to access and interact with the DOM as the user would and thereby gain access to content that is not present by default.