4 ms·
A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr: $.ajaxSetup({
by prashnts 10y ago
A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr:
$.ajaxSetup({
dataFilter: function (data, type) {
if (this.url === 'some url that you want to watch!') {
// Do anything with the data here
awesomeMethod(this.data)
}
return data
}
})
I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page down every 2 seconds, and the `awesomeMethod` just initiated the download. Pasted it all in dev-tools console, and the cheap "scraper" was ready!
- franciskim 10y agoYup I've been known to do this as well :) I'd have a Node.js + Mongo endpoint ready on the other side.
- zappo2938 10y agoWhy not user Nightmare with Node.js + Mongo? Here is an example of injecting a jQuery script into a page with jQuery loaded and getting nicely formatted information returned. [1] [1]https://github.com/adam-s/playboy-fm/blob/master/server/scraper/monthNames.js https://github.com/adam-s/playboy-fm/blob/master/server/scra...
- macromaniac 10y agoAnother trick: You can hover over elements in chrome with F12 and the inspect tool, then right click > copy > copy selector and chrome will generate one for you that way you don't have to actually do any work. With a selector it's easy to grab data, here's a linux command that gets every user that posted in this thread: lynx -base -source 'https://news.ycombinator.com/item?id=12345693' | hxnormalize -x | \ hxselect -c -s '\n' "td > table > tbody > tr > td.default > div:nth-child(1) > span > a.hnuser" Here are the most frequent commenters: 27 cookiecaper 22 franciskim 6 fake-name 4 niftich 4 flukus 4 elmigranto 4 downandout 3 tedunangst 3 siegecraft 3 muglug 3 minimaxir 3 madamelic
- pault 10y agoYou can also build a chrome extension if you need to navigate to multiple pages and use a long-running scraping process. I've done this several times and it's really easy to get one up and running if you use an extension boilerplate (30 minutes tops).
- esac 10y agodo you have something? i was going to write the very same extension (but distributed so i could add it to my pc and my friends) but never did that