4 ms·

Interesting you say this, since it seems like DiffBot is doing something similar and received positive media recently: http://techcrunch.com/2016/02/11/diffbot-

Interesting you say this, since it seems like DiffBot is doing something similar and received positive media recently: http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-intel-of-data-for-artificial-intelligence/ http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-i...

minimaxir 11y ago
DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues. I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
- mistermann 11y ago
  Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is: http://www.stockhouse.com/companies/bullboard/t.y/yellow-media-limited http://www.stockhouse.com/companies/bullboard/t.y/yellow-med... Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
  - dwynings 11y ago
    Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/ http://www.diffbot.com/products/crawlbot/ If you'd like me to set you up with a demo, feel free to email me: dru@diffbot.com
- dwynings 11y ago
  Hi Max, All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3A%2F%2Fnews.ycombinator.com%2Fitem%3Fid%3D11105652 http://www.diffbot.com/testdrive/?api=discussion&url=https%3... If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/ http://www.diffbot.com/products/custom/
- Bammybums 11y ago
  if you run HN front page through import.io's magic, it works pretty well: https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.com https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.c...

minimaxir 11y ago

DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues. I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.

mistermann 11y ago
Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is: http://www.stockhouse.com/companies/bullboard/t.y/yellow-media-limited http://www.stockhouse.com/companies/bullboard/t.y/yellow-med... Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
- dwynings 11y ago
  Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/ http://www.diffbot.com/products/crawlbot/ If you'd like me to set you up with a demo, feel free to email me: dru@diffbot.com
dwynings 11y ago
Hi Max, All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3A%2F%2Fnews.ycombinator.com%2Fitem%3Fid%3D11105652 http://www.diffbot.com/testdrive/?api=discussion&url=https%3... If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/ http://www.diffbot.com/products/custom/
Bammybums 11y ago
if you run HN front page through import.io's magic, it works pretty well: https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.com https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.c...