Here's a project that describes the use of llm to generate crawling rules and then capture them, but it looks like it's still in the early stages of research.

Here's a project that describes the use of llm to generate crawling rules and then capture them, but it looks like it's still in the early stages of research. https://github.com/EZ-hwh/AutoCrawler https://github.com/EZ-hwh/AutoCrawler