4 ms·
What Are the Latest Scraping APIs or Services/Websites?
I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.
Anyone have recommendations for up-to-date scraping APIs or platforms that work well with a variety of websites?
- gbertb 2y agotry https://spider.cloud https://spider.cloud
- jdcampolargo 2y agothanks!
- ipaddr 2y agoThey are powered by: https://github.com/spider-rs/spider https://github.com/spider-rs/spider For those who want to self host.
- mike97 2y agoI am actually working on a scraping API for serps, currently it only supports Google, may add more search engines if it gets any traction, it isn't ready yet, but hopefully it will be in a couple of days, if you want to give it a go let me know and I can give you a subscription in exchange for some feedback.
- nreece 2y ago* Shameless plug * Our feed builder combines a web scraper, AI-powered content auto-detection and data transformer to generate custom JSON and RSS feeds for any public webpage. https://newsloth.com https://newsloth.com
- davidsojevic 2y agoIf you're looking to scrape any of the major search engines, eBay, Yelp, and a few others, then SerpApi has a massive number of up-to-date APIs to choose from: https://serpapi.com/search-api https://serpapi.com/search-api Disclaimer: I work there!
- bootstrpppin 2y agoWe use Bright Data - so far we've found it to be pretty great, and very very cheap. Would highly recommend