3 ms·
I tested a lot of these services and libraries a while ago as part of developing a product that required extracting article text and metadata from a URL. The b
by nikcub 15y ago
I tested a lot of these services and libraries a while ago as part of developing a product that required extracting article text and metadata from a URL.
The best service, and it won by some margin, was Diffbot (www.diffbot.com). I ran comparisons between approx 20 different services and libraries and it won by some margin. It uses machine learning rather than regular expressions or per-site filters, and the engine has been extensively trained (I threw a lot of edge cases at it, which improved it). There seem to be a lot of similar services that do well with common cases but completely fall apart when applied broadly.
So to the author of this service - what features or examples do you have that distinguish your implementation from others? What is the technique being used here?
- mmackh 15y agoI set out to build my own interpretation and that's what I did. It does have an automatic extraction pattern, but also uses per side rules, if available. I'd say the main distinguishing factor is the price point: free.
- kolektiv 15y agoWhich if you want people to use it with confidence probably isn't a selling point: http://blog.pinboard.in/2011/12/don_t_be_a_free_user/ http://blog.pinboard.in/2011/12/don_t_be_a_free_user/
- mmackh 15y agoI cannot, in good conscience, charge for just scraped content. I mentioned it before, but once I complete testing, I'll release the full source on GitHub
- rudasn 15y agoThen figure out why developers would use your service and charge them for making it easier to achieve that goal. I'm sure that scraping content is just a means to and end.
- mrleinad 15y agoI used to work for an online editorial in Spain, which did exactly that: charge for scraped content. You can't imagine how many customers there are for this kind of thing.
- eli 15y agoFair enough, but I'm a whole lot less likely to integrate an API that could break or disappear with no recourse.
- bravura 15y agoWould you care to write a blog post about your comparison? Or, at least, list the 20 services + libs you used?