3 ms·

There are services that cover at least part of what you mentioned. These effectively provide you a tool to visually build a scraper and then they automate the s

There are services that cover at least part of what you mentioned. These effectively provide you a tool to visually build a scraper and then they automate the scraping in the background, creating an API or spreadsheet of the data. Import.io is one example, and I think there's another more recent YC-backed one. However, I tried using import.io a little while back but without much joy.

lumpypua 12y ago
We have a non-tech intern and import.io looks like like a great tool to get him chewing up data. I'm playing with it now. Why didn't it work out for you? Beyond the wrapped browser interface being a little funky lol. (Edit: eugh, selecting data for import is really clunky.) Ask HN: Anybody got a visual scraping service they like?
- samcrawford 12y ago
  It was the data extraction and selection process I couldn't get to work. I was trying to scrape a particular search on autotrader.co.uk (I wanted more up to date results than their daily emails provide, and I wanted to filter out cars that had been written off). I don't remember all the details, but I followed the tutorial video and got to the stage where you select a single item that matches your criteria and it's supposed to extrapolate from there. However I just seemed to be stuck in an infinite loop of it asking me to do this.
  - mattmanser 12y ago
    I found you often have to select two, then it figures it out. I assumed it was probably because of alternating odd/even row CSS classes.
vantagepoint 12y ago
I think import.io is buggy to say the least, having used it in the past to scrape some websites, it was pain to work with. Kimonolabs is still very lacking in terms of ability to handle different websites, it is very much limited to a certain portion of the web, it looks like they are more about creating APIs...APIs that people are supposed to find interesting and valuable but like the topic of this question, it seems like it's only valuable to someone who has a direct need for that dataset, by itsef would serve no interest to say. Having private access to Scrape.it, I can say that it focuses strictly on making a great tool and the ability to scrape websites without costing a fortune. I know the founders and they are extremely dedicated to making a tool that can pretty much handle anything you throw at it like AJAX, Single page apps, crawling selected links and all sub links after the page. They've just begun adding login and form support so should be able to play with those as well very soon. It only supports csv output at the moment but hopefully they can make available like API output.

lumpypua 12y ago

We have a non-tech intern and import.io looks like like a great tool to get him chewing up data. I'm playing with it now. Why didn't it work out for you? Beyond the wrapped browser interface being a little funky lol. (Edit: eugh, selecting data for import is really clunky.) Ask HN: Anybody got a visual scraping service they like?

samcrawford 12y ago
It was the data extraction and selection process I couldn't get to work. I was trying to scrape a particular search on autotrader.co.uk (I wanted more up to date results than their daily emails provide, and I wanted to filter out cars that had been written off). I don't remember all the details, but I followed the tutorial video and got to the stage where you select a single item that matches your criteria and it's supposed to extrapolate from there. However I just seemed to be stuck in an infinite loop of it asking me to do this.
- mattmanser 12y ago
  I found you often have to select two, then it figures it out. I assumed it was probably because of alternating odd/even row CSS classes.

vantagepoint 12y ago

I think import.io is buggy to say the least, having used it in the past to scrape some websites, it was pain to work with. Kimonolabs is still very lacking in terms of ability to handle different websites, it is very much limited to a certain portion of the web, it looks like they are more about creating APIs...APIs that people are supposed to find interesting and valuable but like the topic of this question, it seems like it's only valuable to someone who has a direct need for that dataset, by itsef would serve no interest to say. Having private access to Scrape.it, I can say that it focuses strictly on making a great tool and the ability to scrape websites without costing a fortune. I know the founders and they are extremely dedicated to making a tool that can pretty much handle anything you throw at it like AJAX, Single page apps, crawling selected links and all sub links after the page. They've just begun adding login and form support so should be able to play with those as well very soon. It only supports csv output at the moment but hopefully they can make available like API output.