3 ms·
And how do you do #1? Node, I presume?
by AznHisoka 9y ago
And how do you do #1? Node, I presume?
- austincheney 9y agoNo, manually go there and copy/paste the code. Then when building your scraper bot use that code.
- AznHisoka 9y agobut how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?
- austincheney 9y agoYou could write a crawler in any language. Crawling is easy as you are listening for HTTP traffic and analyzing the HTML in the response. To accurately get the content in dynamically executed pages you need to interact with the DOM. This is the reason Google updated its crawler to execute JavaScript.
- AznHisoka 9y agoYep, I know, but that means if I am writing the crawler in Ruby/Python, this is not something I can do, right?
- austincheney 9y agoYes. The crawler can be written in nearly any language. The actual scraper probably has to be written in JavaScript in order to access and interact with the DOM as the user would and thereby gain access to content that is not present by default.