3 ms·
Interesting. Do you have an example of the latter approach? I have been trying to build a web scraper, but many times after the HTML is returned, the javascript
by lambtron 13y ago
Interesting. Do you have an example of the latter approach? I have been trying to build a web scraper, but many times after the HTML is returned, the javascript then loads the data dynamically. I'd love to learn of a few techniques to fetch the AJAX data.
- HarryRotha 13y agoGet firebug, or use developer tools in chrome and look at the requests the browser is making when you load the page, and when you scroll down and it loads more data. Then you just implement that in whatever language you are scraping with. I usually use pythons requests module.. That's about as easy as it gets for something like this.
- malandrew 13y agoNot an example I can share, but you can approach this starting out looking at the global variable which basic bootstrap data is assigned to on initial payload. From this variable you should be able to work backwards to the function that reads this data on page ready and instantiates some models and collections. From there, I'd study the model/collection code to see how they are configured for fetching data. If you tap in at this point, you can just request data directly via new models or collections.
- foxwoods 13y agoIf the site use jQuery, maybe you can hijack Ajax requests by using [jQuery.ajaxPrefilter](http://api.jquery.com/jQuery.ajaxPrefilter/ http://api.jquery.com/jQuery.ajaxPrefilter/).