3 ms·
but how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?
by AznHisoka 9y ago
but how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?
- austincheney 9y agoYou could write a crawler in any language. Crawling is easy as you are listening for HTTP traffic and analyzing the HTML in the response. To accurately get the content in dynamically executed pages you need to interact with the DOM. This is the reason Google updated its crawler to execute JavaScript.
- AznHisoka 9y agoYep, I know, but that means if I am writing the crawler in Ruby/Python, this is not something I can do, right?
- austincheney 9y agoYes. The crawler can be written in nearly any language. The actual scraper probably has to be written in JavaScript in order to access and interact with the DOM as the user would and thereby gain access to content that is not present by default.