Allowing anyone to define the crawling policy and snapshot any site is in the plan. I need to build up the UI for the backend process to crawl and transform th

Allowing anyone to define the crawling policy and snapshot any site is in the plan. I need to build up the UI for the backend process to crawl and transform the webpages. I'm building the backend tools as I go along. So far it's just a proof of concept with couple sites defined. Once the tools are ready, I'll open the backend up.