6 ms·
Ask HN: Has anyone used web-scraping-as-a-service that didn't suck?
I'm trying to be a little more commercially minded, and avoid writing my own scraping infrastructure and maintaining a bunch of scraping integrations for specific sites I want data from.
There seems to be a large number of web-scraping-as-a-service products out there, and very little to recommend between them. Has anyone used one which didn't suck, or am I likely to need to write my own scraping tools?
- t_liu 11y agoit depends, tools like Kimono can gather data on a topical layer, but cant really interact with the page(via js). I wanted to scrape some data on a site I was interested in, but I need to interact with the site to get the data, so I ended up writing my own crawler
- mariocesar 11y agoI got great experiences with http://scrapinghub.com/ http://scrapinghub.com/, you can hire them as consultants also and short time contract is enough to get in the right direction. really smart people
- renaudg 11y agoI heard good things about https://import.io/ https://import.io/
- Roger_Archive 11y agoMy colleagues at https://www.archive-it.org https://www.archive-it.org do a pretty darn good job
- ccarter84 11y agoSeconding this question... I've wanted a way to get HN top posts each hour, aggregated (and deduped) by day/week, so that I dont have to feel the need to check as often. Thanks for this links, hopefully one of these will get me there!
- dangrossman 11y agoYou don't have to scrape HN, it has an API. It's linked to at the bottom of each page.
- mswen 11y agowork with a group that use Mozenda - I haven't worked much with it directly. They made it through a couple years of scraping with Mozenda. They recently decided to spend more and installed Kapow. More mature platform - lots of features but also a lot more expensive if I understand correctly.
- nreece 11y agoGive Feedity - https://feedity.com https://feedity.com a try. Drop us a message https://feedity.com/contact.aspx https://feedity.com/contact.aspx if you have any questions or concerns.
- TomBeckman 11y agoI've used http://www.mozenda.com http://www.mozenda.com in the past. Worked on some fairly complex Web directories with multiple levels.