Incrementing the numeric value in url makes web scraping so much more reliable. Reliability and maintenance is a trade off in scraping. The big pain comes from

Incrementing the numeric value in url makes web scraping so much more reliable. Reliability and maintenance is a trade off in scraping. The big pain comes from when websites change designs, imagine Buzzfeed from a few quarters now, it would surely change some parts requiring you to relabel the data fields. https://scrape.it https://scrape.it automatically adapts to page layout changes and continues scraping without interruption. It requires no maintenance work from the user point of view. The other thing I'd like to see Scrape.it do is the ability to scrape by incrementing the page url but unlike Kimono, Scrape.it is designed to work on dynamic web pages as the primary goal (sites requiring ajax POST to increment page), it's quite amazing to see how many sites out there simply use page number in their urls.