I was investigating how to add asyncio / async def support to Scrapy (see https://github.com/scrapy/scrapy/issues/1144#issuecomment-141843616 https://github.com

I was investigating how to add asyncio / async def support to Scrapy (see https://github.com/scrapy/scrapy/issues/1144#issuecomment-141843616 https://github.com/scrapy/scrapy/issues/1144#issuecomment-14...). Small examples like the one at the link look neat, but it is not all roses as you go further. The problems are not specific to Scrapy; I think any advanced `async def` based crawler will face them. There are 2 challenges with async def I don't know how to solve elegantly: 1. how to integrate coroutine-based scraping code with on-disk persistent request queues; 2. how to deallocate resources without boilerplate in coroutine-based scraping code. (1) is easier with callbacks-as-methods because this way state is passed explicitly (it is not in local variables), so Scrapy can choose to save it to disk. Example of (2) is this code: async def parse(self, response): resp = await self.fetch(url) # ... find another URL to follow # Here we have the problem: # response object is kept in memory # until second response is fully received. # This is a problem if 10s and 100s # of requests are processed in parallel # and responses are large. # Because of refcounting, with callbacks # response would have been kept in # memory only until second request # starts - callbacks+refcounting provide # an elegant way for resource deallocation. resp = await self.fetch(url2) If anyone has suggestions please comment on https://github.com/scrapy/scrapy/issues/1144#issuecomment-141843616 https://github.com/scrapy/scrapy/issues/1144#issuecomment-14....