I haven't done this, but take a peek at these. Perhaps you can have your own linkextractor that rejects certain items, or perhaps a custom scheduler: https://d

I haven't done this, but take a peek at these. Perhaps you can have your own linkextractor that rejects certain items, or perhaps a custom scheduler: https://doc.scrapy.org/en/latest/topics/link-extractors.html https://doc.scrapy.org/en/latest/topics/link-extractors.html https://doc.scrapy.org/en/0.14/topics/architecture.html?highlight=scheduler https://doc.scrapy.org/en/0.14/topics/architecture.html?high... https://doc.scrapy.org/en/0.9/experimental/scheduler-middleware.html https://doc.scrapy.org/en/0.9/experimental/scheduler-middlew... https://doc.scrapy.org/en/latest/topics/settings.html#scheduler https://doc.scrapy.org/en/latest/topics/settings.html#schedu... https://scrapy.readthedocs.io/en/0.14/faq.html#does-scrapy-crawl-in-breath-first-or-depth-first-order https://scrapy.readthedocs.io/en/0.14/faq.html#does-scrapy-c...