3 ms·
This is perhaps the fastest way to screenscrape a dynamically executed website. 1. First go get and run this code, which allows immediate gathering of all text
by austincheney 9y ago
This is perhaps the fastest way to screenscrape a dynamically executed website.
1. First go get and run this code, which allows immediate gathering of all text nodes from the DOM: https://github.com/prettydiff/getNodesByType/blob/master/getNodesByType.js https://github.com/prettydiff/getNodesByType/blob/master/get...
2. Extract the text content from the text nodes and ignore nodes that contain only white space:
let text = document.getNodesByType(3),
a = 0,
b = text.length,
output = [];
do {
if ((/^(\s+)$/).test(text[a].textContent) === false) {
output.push(text[a].textContent);
}
a = a + 1;
} while (a < b);
output;
That will gather ALL text from the page. Since you are working from the DOM directly you can filter your results by various contextual and stylistic factors. Since this code is small and executes stupid fast it can be executed by bots easily.