Ask HN: Has anyone tried to crawl all possible /newest links?

`https://news.ycombinator.com/newest` is one of the best real-time reliable intellectual curation. But how much interesting links not posted does someone reading them all miss? Is there an existing crawler that index all the interesting websites that could be posted in /new? Is there a natural and algorithmic definition for "interesting" URLs?