A good way to find new pages on a site is to use the site map feed in xml or text format. Then look for open graph tags in the page for titles descriptions, ca

A good way to find new pages on a site is to use the site map feed in xml or text format. Then look for open graph tags in the page for titles descriptions, canonical urls and images. Most modern websites have all of those these days and they are automatically updates. Making an RSS feed reader is thus easier and less prone to errors from analyzing the markup of pages. Neat product though.