7 ms·
What I need is an API scraper, scrapy seems to be mostly for HTML. I know how to look at network requests in Chrome dev tools and JS function to understand the
by hokkos 11y ago
What I need is an API scraper, scrapy seems to be mostly for HTML. I know how to look at network requests in Chrome dev tools and JS function to understand the shape of REST API, so I need something to plan the exploration of the arguments space. For example if you want to scrap airbnb, you look at their API, find there is a REST call with a lat, long box, I need something to automatically explore an area, if the api only give the 50 first results and you hit this number of calls it should schedule 4 calls with half the size boxes and so on. If the request has cursors, you should be able to indicate to the scraper how to follow it. I don't know what is the best tool for that.
- ddebernardy 11y agoIt's not entirely clear what you're up to, but FYI Scrapy works fine when scraping JSON data: http://stackoverflow.com/a/18172776/417194 http://stackoverflow.com/a/18172776/417194
- xrjn 11y agoAre you talking about something that would map out the API? The closest analogy to what I understood from your text was http://nodered.org/ http://nodered.org/ in terms of design, I haven't seen anything like this for scraping yet. Definitely an interesting product to be made here if it doesn't exist yet.