8 ms·

Spidermon: Scrapinghub’s Now Open Sourced Spider Monitoring Library

• hartator 8y ago
  That's awesome, great work from the ScrapingHub team. We had a similar approach at SerpApi. We write open source rspec tests and run them daily. e.g.: - https://github.com/serpapi/test-knowledge-graph-desktop/tree/master/spec https://github.com/serpapi/test-knowledge-graph-desktop/tree... - https://travis-ci.org/serpapi/test-knowledge-graph-desktop https://travis-ci.org/serpapi/test-knowledge-graph-desktop - https://github.com/serpapi/test-organic-results-desktop/tree/master/spec https://github.com/serpapi/test-organic-results-desktop/tree... - https://travis-ci.org/serpapi/test-organic-results-desktop https://travis-ci.org/serpapi/test-organic-results-desktop - https://github.com/serpapi/test-news-results-desktop/blob/master/spec/news_results_trump_spec.rb https://github.com/serpapi/test-news-results-desktop/blob/ma... - https://travis-ci.org/serpapi/test-news-results-desktop https://travis-ci.org/serpapi/test-news-results-desktop Producing reliable scrapers and parsers is very hard. Testing as much possible is the only way to go. Smart use also of JSON schema on Spidermon.
• martinlaz 8y ago
  Cool name :)
• Excluse 8y ago
  As a Jamaican, I got a unique chuckle out of this. Clever branding!
  • robk 8y ago
    Sim simma!
• bryanrasmussen 8y ago
  How does this handle sites that are rendered entirely or nearly entirely by JavaScript client side? My only real concern when looking at a new scraping solution, almost everything else can be worked with.
  • landyman 8y ago
    I haven't used Spidermon yet, but Scrapinghub has a Scrapy plugin called Splash which can be used to render JS pages within a Scrapy spider.
• appsonify 8y ago
  How the fuck are you not sued yet by Google? Scraping their SERP and profiteering off of is clearly against their ToS and will alert their legal team. edit: dude, you need to stop, or you are going to get sued, I'm serious. https://www.reddit.com/r/webdev/comments/8a2zx9/weve_just_launch_serpapi_a_google_search_results/?st=jsw38bhf&sh=550f8e7c https://www.reddit.com/r/webdev/comments/8a2zx9/weve_just_la... I've alerted legal@google.com about SerpAPI Thanks for your feedback. We appreciate your bringing this to our attention, and we'll investigate. We'll follow up with you only if we require more info or we have additional info to share.
  • deleted 8y ago
    [deleted]
  • TheCowboy 8y ago
    I don't think an aggressive litigious attitude like this is constructive. It also ignores the fact that Google has scraped many sites to build its own profitable products and features over the years.
    • appsonify 8y ago
      that's totally not the same thing.
      • TheCowboy 8y ago
        "Online-reviews firm Yelp Inc. alleged that Google is breaking a promise it made as part of a 2012 regulatory settlement to not scrape content from certain third-party sites including Yelp, escalating its yearslong battle against the search giant." https://www.wsj.com/articles/google-rival-yelp-claims-search-giant-broke-promise-made-to-regulators-1505167498 https://www.wsj.com/articles/google-rival-yelp-claims-search...
        • appsonify 8y ago
          your point being? that has nothing to do with bypassing IP bans and defeating captcha. see 3taps vs craigslist ruling from EFF. https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-settle-suit-craigslist-over-use-real-estate-facts https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
• faitswulff 8y ago
  Great, now to figure out some way to use this with Batman.js ...