3 ms·
It sometimes depends on what sort of data you are trying to get. Hipmunk could be crawling travel sites. Mint couldn't have been crawling banks as those are pro
by pedalpete 16y ago
It sometimes depends on what sort of data you are trying to get. Hipmunk could be crawling travel sites. Mint couldn't have been crawling banks as those are protected, unless people were giving them the passwords to their accounts.
Are you familiar with web scraping? Depending on the volume, and type of data you are trying to get, it might work for you. It should at least get you through proof of concept.
- dglassan 16y agoI'm familiar with web scraping...hasn't it always been looked down upon though? It just seems like it's got a negative connotation or something.
- pedalpete 16y agoI think it depends on the application. Though I asked the question below as ffumarola said it was 'bad practice'. Why scraping is any worse than crawling is beyond me. But again, it depends on the application. I built a concert search engine which scraped data from multiple sites. It is my understanding that all the concert sites are getting a bunch of data from published apis and then topping up with scraped data.