3 ms·
Hey, A good feedback, thanks! > XPath selectors just plain suck. Scrapy supports CSS selectors. > The item pipeline is way too straitjacketed. You have to d
by kmike84 12y ago
Hey,
A good feedback, thanks!
> XPath selectors just plain suck.
Scrapy supports CSS selectors.
> The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it.
Could you plese give an example?
> Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so.
This is true. But it is a pain to integrate any even-loop based app with another app that is not event-loop based. It is also true that Scrapy is not easy to plug into existing event loop (e.g. if you already have twisted or tornado-based service), but it should be fixed soon.
> Tons of useless features. Telnet console? wtf?
Telnet console is a Twisted feature; it came almost for free, and it is useful to debug long-running spiders (which can run hours and days).
> It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself.
If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization. If you just want to do some post-processing then I don't see why putting code to Scrapy spider is worse than putting it to other script and calling Scrapy from this script.
> For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up).
This may be true.. I don't quite get what is it for :)
> IMO this is a classic case of "framework that should have been a library".
It can't be a library like requests or mechanize for technical reasons - to make crawling efficient Scrapy uses event loop. It can (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified.
> * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining.
Celery is also not the easiest piece of software. Scrapy is just a single Python process that doesn't require any databases, etc.; Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks.
- crdoconnor 12y ago>Scrapy supports CSS selectors. Still far inferior to JQuery selectors. >Could you plese give an example? The example I'm thinking of is when I was trying to create a pipeline that would output a skeleton configuration file when you passed one switch and would process and serialize the data parsed when you passed another. It was possible but kludgy. >But it is a pain to integrate any even-loop based app with another app that is not event-loop based. That's not where the pain lies. It's more the fact that it has its own weird configuration/setup quirks (e.g. its own settings.py, reliance on environment variables, executables). >If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization. I don't really want scrapy doing process handling at all. It's not particularly good at it. Celery is much better. Using other code to do serialization also doesn't necessitate running it on the same process. You can import the django ORM wherever you want and use it to save to the DB. I know you can do that - but, again, kludgy. >It can't be a library like requests or mechanize for technical reasons - to make crawling efficient Scrapy uses event loop. I get that. It should have been more like twisted from the outset though. The developers were clearly inspired from django and that led them down a treacherous path. >It can (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified. Well, that's good I suppose. I still think that it focuses on bringing together a bunch of mediocre modules for which, individually, you can find much better equivalents. Also, (unlike django) tight, seamless integration between those modules doesn't really gain you much. >Celery is also not the easiest piece of software. The problem it is solving (distributed task processing) is not an easy problem. Celery is not simple, but it is GOOD. >Scrapy is just a single Python process that doesn't require any databases, etc. Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks. A) You can use redis as a broker and that's trivial to set up. I always have a redis available anyway because I always need a cache of some kind (even when crawling!). B) My crawling tasks are never I/O bound or CPU bound. They're bound by the rate limiting imposed upon me by the websites I'm trying to crawl. C) I'm usually using celery anyway. I still have to do task processing that DOESN'T involve crawling. Where do I put that code when I'm using scrapy?