> The example I'm thinking of is when I was trying to create a pipeline that would output a skeleton configuration file when you passed one switch and would pro

> The example I'm thinking of is when I was trying to create a pipeline that would output a skeleton configuration file when you passed one switch and would process and serialize the data parsed when you passed another. It was possible but kludgy. I don't get it - how is creating a configuration file related to the processing of the items, why would you do it in items pipeline? > It's more the fact that it has its own weird configuration/setup quirks (e.g. its own settings.py, reliance on environment variables, executables). It is possible to create a Crawler from any settings object (not just a module), and Scrapy does not rely on executables AFAIK. But this all is poorly documented. Also, there is an ongoing GSoC project to make settings easier and more "official". > I don't really want scrapy doing process handling at all. Scrapy doesn't handle processes, it is single-threaded and uses a single process. This means that you can use e.g. a shared in-process state. > Using other code to do serialization also doesn't necessitate running it on the same process. You can import the django ORM wherever you want and use it to save to the DB. I know you can do that - but, again, kludgy. You can't move Python objects between processes without serialization. Why is using django ORM kludgy in Scrapy but not in Celery? > The problem it is solving (distributed task processing) is not an easy problem. Celery is not simple, but it is GOOD. You don't necessarily need distributed task processing to do web crawling. Celery is a great piece of software, and it is developing nicely, but you always pay for complexity. For example, I faced the following problems when I was using Celery: * When redis was used as a broker its memory usage was growing infinitely. Lots of debugging, found a reason and a hacky way to overcome it (https://github.com/celery/celery/issues/436 https://github.com/celery/celery/issues/436). The issue was fixed, but apparently there is still a similar issue when MongoDB is used as a broker. * Celery stopped processing without anything useful in logs (and of course Celery error sending facilities failed and I didn't have external monitoring) - it turns out an unicode exception was eaten. A couple of days of nightmarish debugging; see https://github.com/celery/celery/issues/92 https://github.com/celery/celery/issues/92. * I implemented an email sender using Celery + RabbitMQ once. I think I was sending email text to tasks as parameters. Never do that (just use MTA:)! When a large batch of emails was sent at once RabbitMQ used all memory, corrupted its database, dropped the queue; I haven't found a way to check which emails were sent and which were not. This was 100% my fault, but it shows that complex setup is not your friend. Crawling tasks differ - e.g. if you need to crawl many different websites (which is not uncommon) you will almost certainly be IO and CPU limited. Scrapy is not a system for distributed task processing, it is just an event-loop based crawler. I'm not saying your way to solve the problem is wrong; if you already use celery it makes a lot of sense to use it for crawling as well. But I don't agree that going distributed turtles all the way down with celery+redis+DB for storage+... is easier or more efficient than using plain Scrapy. A lot of tasks can be solved by writing a spider, getting a json file with data and doing whatever one wants with it (upload to DB, etc).