It looks based on rules. There are Python libraries which try to solve similar tasks using Machine Learning: https://pypi.python.org/pypi/autopager https://pypi

It looks based on rules. There are Python libraries which try to solve similar tasks using Machine Learning: https://pypi.python.org/pypi/autopager https://pypi.python.org/pypi/autopager, http://formasaurus.readthedocs.io/en/latest/ http://formasaurus.readthedocs.io/en/latest/, https://github.com/scrapinghub/page_finder https://github.com/scrapinghub/page_finder. I wonder how the quality compares. It is hard to make rules work reliably on thousands of unseen websites.