I built something with an API that uses Selenium to image a site. It works on a large percentage of the sites I feed it. I don't recursively call links found i

I built something with an API that uses Selenium to image a site. It works on a large percentage of the sites I feed it. I don't recursively call links found in the pages. I expect the user to give me the URLs to crawl and save. In order to "find" new content, I let the user specify where they want to search for things the engine hasn't "crawled" yet. So, a search for scooters to buy might end up searching Amazon directly, then lets the user "save" the site by passing the Amazon URL for the scooter they like to the system for imaging. I use GPT-3 or other ML models to do some of the heavy lifting for adding labels to the pages or documents the user uploads. This ends up being a "curated" list of documents important to the individual user, not an exhaustive crawl of all things which are important to all users.