10 ms·

Python 3 comes to Scrapy

249 points by ddebernardy 11y ago

djm_ 11y ago
Many congrats on the release & also thanks to the Scrapy team for the effort involved. As far as I am concerned, this was the last package I used heavily that still had not made the upgrade. For the Python community: which packages are you still waiting for/working on?
- vially 11y ago
  I've been waiting for Fabric to make the migration to Python 3. Fortunately I no longer use Python that much (I jumped on the hype train to Go and I've been very satisfied with it) so I no longer need Fabric nowadays.
- crdoconnor 11y ago
  Mechanize would be nice, but it's practically abandonware at this point.
  - kevinwang 11y ago
    same
- random_upvoter 11y ago
  A wrapper for the exiv2 library (for editing metadata of picture files) that works on Windows
- xemoka 11y ago
  Esri's ArcPy module and ArcGIS Desktop's "built-in" python environment. :( ArcGIS Pro is using 3.4, so the wrappers for ArcObjects are updated, but who knows if they will put that work back into ArcGIS Desktop...
- craigds 11y ago
  Still waiting on Ansible to make the jump. As far as I can tell it's not even on their roadmap, which is a bit crazy given that _almost_ everyone else has solved this by now. This may not be a blocker for the rest of our stuff though, since ansible can install its own python 2 on servers automatically.
bbayer 11y ago
It is a major change to ask but I am wondering if they consider to switch to asyncio instead of using Twisted. Twisted is great library but it is a huge dependency to maintain.
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
- kmike84 11y ago
  We're seriously considering this option (maybe for scrapy 2.0?), but no concrete plans yet. It can't be just asyncio - we'll need e.g. aiohttp and other packages. Some links: * POC for aiohttp as a http handler: https://github.com/scrapy/scrapy/pull/1455 https://github.com/scrapy/scrapy/pull/1455 * some thought about how to make async/await API for Scrapy; it is not all roses: https://github.com/scrapy/scrapy/issues/1144#issuecomment-141843616 https://github.com/scrapy/scrapy/issues/1144#issuecomment-14...
- eliasdorneles 11y ago
  There has been some discussions and a proof-of-concept with asyncio + aiohttp: https://github.com/scrapy/scrapy/pull/1455 https://github.com/scrapy/scrapy/pull/1455 However, as you said, it'd be a major change and it would affect the whole ecosystem (plugins and extensions), so it's complicated. We'll see what happens. :)
- glyph 11y ago
  Can you elaborate? What makes it a "huge dependency to maintain"? Is there anything that the Twisted project can do to make it easier? If this is actually a problem I'd really like to hear from users on the Twisted mailing list and bug tracker.
  - bbayer 11y ago
    Twisted is a general purpose library/framework with lots of features. This is the "huge" part. In my previous projects I have used it a lot and appreciated it. What I was trying to tell is if Scrapy uses only small part of library, it may be possible for developers to use similar constructs from Python's standard library. In any case dependency is dependency and it is always better to minimize code footprint.
engi_nerd 11y ago
I've seen developers refuse to use Python 3 because of not being able to use Scrapy. Hopefully this gets some more devs to finally make the switch.
- squeaky-clean 11y ago
  I've been refusing to use Scrapy because of not being able to use Python 3 :P So this is a very exciting announcement for me.
  - ddebernardy 11y ago
    Believe me, we've been hearing this too. We're super excited as well. :-)
jstoiko 11y ago
This is great news for Python 3. We're almost there! -> http://py3readiness.org http://py3readiness.org
- pen2l 11y ago
  Hm, why is opencv not there? I was playing around with it last week, and it seems it works well for only py2 right now (from what I could gather). OpenCV should be there in that list, a lot of people really need it.
  - rspeer 11y ago
    My guess: because OpenCV is so unlikely to be installable through pip, its download statistics on PyPI end up artificially low.
  - orf 11y ago
    It's only the 360 most downloaded packages, perhaps OpenCV is not one of them?
  - jritchie 11y ago
    OpenCV 3 supports Python 3[1], but it's not distributed through PyPI. [1] http://opencv.org/opencv-3-0.html http://opencv.org/opencv-3-0.html
- daveguy 11y ago
  Hahahaha. 7 years later. Python 3 is almost there!
  - rspeer 11y ago
    Your response to lots of people's programming effort is "Hahahaha"? Get lost.
    - pen2l 11y ago
      I agree he's not being a great sport, but I sort of see where he's coming from. Having worked with python in the last few months for computer vision problems, I've been really frustrated with how things are. I was expecting better. At this point I don't use python for my work anymore, I use matlab. It's superbly well-documented, and things just /work/. And of course, nothing like the mess you see with this py2/py3 problem is to be seen anywhere near matlab. Yes, it's not free, but I've decided that the amount of time I had to spend, the headaches I had to endure, they were not worth the free price of python (and matlab for its great support was in fact worth its price). ymmv.
njharman 11y ago
Minor point of clarification re title; Python 3 has been here, it's Scrapy that has (finally and good for them) come to Python 3.
- kmike84 11y ago
  that's an example of https://en.wikipedia.org/wiki/Galilean_invariance https://en.wikipedia.org/wiki/Galilean_invariance
penetrarthur 11y ago
Not related to Scrapy, but what are some things you scrape the web for?
- kevin_thibedeau 11y ago
  I once scraped every October posting from Slashdot to see long term trends. Short story: it's dying. I project the active userbase will be gone by 2020. Curiously the bulk of the posters were in the 100k to 300k UID range. There was also evidence of shenanigans with UID assignment where they were skipping even numbers and odd numbers at various times possibly to inflate their numbers. This will get you IP banned BTW but I did get a full data set before their script caught me.
  - gst 11y ago
    I wonder if Netcraft already confirmed that Slashdot is dying ;)
  - hatchoo 11y ago
    ScrapingHub (the guys behind Scrapy) offers Crawlera which provides some sort of automatic proxying and throttling so you can scrape away avoiding getting banned.
Keats 11y ago
Nice! Everytime I create a new Scrapy project I keep forgetting it was not python3 and had to recreate the virtualenv. Great news!
erroneousfunk 11y ago
This is great! I wrote "Web Scraping with Python" (O'Reilly) and did everything with Python 3... except for the oddball section on Scrapy. Glad to know I can update that for the second edition!
bobby_9x 11y ago
I'm glad Python 3 supports so many libraries now. I just made the switch a couple of months ago (from 2).
ceronman 11y ago
The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic way ever again. I'm glad that we're starting to see light at the end of the tunnel. I find myself using Python 3 in most of my projects. Some times I still have to resort to Python 2 when some dependency is not ready but those cases are more rare every day. Also it's frustrating to use Python 2 when many cool features are now Python 3 only. It will take some more time, but I'm sure that the transition will eventually be completed.
- thearn4 11y ago
  I'm happy that we finally updated a lot of our code at work to be Python 3 compatible last year. The conversion was not that bad in the end. However, under Python 3 a number of benchmarks are slower, which we do need to look into some more. Has anyone else experienced this, specifically with scientific computing code migrating from 2 to 3? Ultimately, we test in 2 and 3 via CI, but still primarily develop using 2.
  - kmike84 11y ago
    A data point: `scrapy bench` is about 1.5x - 2x slower in Python 3.5 than in Python 2.7. This command starts a simple HTTP server which serves HTML pages linked in arbitrary depth, and crawls it using Scrapy. Profiler shows URL parsing is much slower in Python 3 for some reason, I'm not sure why. I haven't digged further. Python 3 code also have to encode/decode strings more often, but the overhead was not significant for `scrapy bench`.
    - ogrisel 11y ago
      I think you should google the python.org bug tracker with the function names that seems to exhibit a performance regression according to your profiling session. If don't find any existing issue, please open a new one with a minimalist reproduction script that demonstrates the performance regression.
      - 11y ago
bootload 11y ago
This is great news. I often read the stick developers give to Py3, not wanting to upgrade from 2->3 citing code bases still using Py2.
BuckRogers 11y ago
I've been holding out for a while on moving to 3. I have tried every 3 release and always found issues or performance regressions. Like others have said here it's a concern about the performance issues, Python is already slow. It's an even harder sell when - all major (and equally important, the long tail of minor) libraries support 2 - CPython2 has the performance advantage in most (if not all) applications - virtually every 3rd party implementation supports it very well (PyPy in particular) - Python2 already supported unicode, so that gets old to hear about - Most of the new features are available as backports - Some new features are absurd, like the new 4th string formatting method in 3.6 People shouldn't openly wonder why someone uses 2 instead of 3 if you really look at it. I just started a new 3.5 project because while I gave 3.0-3.4 a shot, 3.5 hasn't had its runthrough yet. Most people in my shoes have more than likely moved on from Python to Go. I'd like to have this be the one and stop going back to 2.7. Admittedly patience is running on fumes after ~8 years of testing CPython3 releases. It wasn't just a bad break, it seems like it was a sloppy break. Instead of feature bloat, I'd like to see Python3 focus on performance.
- Chris2048 11y ago
  Also, Jython py3 dev lagged far behind py2.
Chris2048 11y ago
I mentioned some of the issues I had with scrapy ages ago on reddit: http://www.reddit.com/r/Python/comments/g112q/installing_and_using_scrapy_web_crawler_to_search/ http://www.reddit.com/r/Python/comments/g112q/installing_and... Never got a reply, but I'll reproduce here; I wonder how much of this is still true? " The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started. When I used it, I wanted to scrape a site until certain conditions were met; when the last page scraped returned no objects. I wanted all results initially returned from a page to be dropped if they were older than a certain date; Thus I wanted Scrapy to keep scraping until no new items were found. Also, I wanted the latest date of the items returned so I could use this the next time I scrape. I created the 'DropElderMiddleware' middleware to do this. I couldn't see any other way of making calculations based on items returned from a particular page. I could never figure out what the difference between input and output processors where, or when I should use one or the other. The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain. The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can't remember the details of why I found this hard, but I can try to come up with an example if you like? In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy's features I didn't need e.g. I didn't want command-line control (I would actually prefer not to use the interface, though didn't discover how I could write a python script to apply the spider directly). Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy's object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what's going on in my scripts. "
- kmike84 11y ago
  > The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started. Yeah, docs used to be a problem; they improved a lot in 1.0 and 1.1 releases though. > When I used it, I wanted to scrape a site until certain conditions were met; when the last page scraped returned no objects. I wanted all results initially returned from a page to be dropped if they were older than a certain date; Thus I wanted Scrapy to keep scraping until no new items were found. Also, I wanted the latest date of the items returned so I could use this the next time I scrape. The easies way is to raise CloseSpider exception in a callback if no new items are scraped - see http://doc.scrapy.org/en/1.0/topics/exceptions.html#closespider http://doc.scrapy.org/en/1.0/topics/exceptions.html#closespi... > I could never figure out what the difference between input and output processors where, or when I should use one or the other. > The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain. I also have troubles understanding ItemLoader details. They are totally optional though, and they are no longer in Scrapy tutorial (http://doc.scrapy.org/en/latest/intro/tutorial.html http://doc.scrapy.org/en/latest/intro/tutorial.html). Item loaders provide features very similar to https://github.com/Suor/funcy https://github.com/Suor/funcy or https://github.com/kachayev/fn.py https://github.com/kachayev/fn.py. > The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can't remember the details of why I found this hard, but I can try to come up with an example if you like? I'm not sure what problems did you have. Scrapy selectors library (https://github.com/scrapy/parsel https://github.com/scrapy/parsel) is quite similar to PyQuery (esp. when CSS selectors are used), and nothing prevents you from using PyQuery with Scrapy. In future we may add PyQuery (and BeautifulSoup?) support to parsel and provide PyQuery selectors as response.pq (like response.css and response.xpath), +1 to do that. > In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy's features I didn't need e.g. I didn't want command-line control (I would actually prefer not to use the interface, though didn't discover how I could write a python script to apply the spider directly). Yeah, library interface used to be a problem. It was improved in 1.0 release (there is an official API for integrating Scrapy with Twisted apps and running spiders from user scripts), but there is still more to go. See http://doc.scrapy.org/en/1.0/topics/practices.html#run-scrapy-from-a-script http://doc.scrapy.org/en/1.0/topics/practices.html#run-scrap.... It probably won't be as easy to integrate with regular Python scripts as mechanize because Scrapy is async. On the other hand, Scrapy is easier to integrate with async servers like Twisted or Tornado. > Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy's object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what's going on in my scripts. You may want to check the new 'Scrapy at glance' page (http://doc.scrapy.org/en/latest/intro/overview.html http://doc.scrapy.org/en/latest/intro/overview.html). The main advantage of Scrapy over mechanize is that it handles parallel downloads and have a wide range of built-in extensions you won't have to implement yourselves.
darkrho 11y ago
Try it out via conda! conda install -c scrapinghub/label/dev scrapy

249 points by ddebernardy 11y ago

djm_ 11y ago

Many congrats on the release & also thanks to the Scrapy team for the effort involved. As far as I am concerned, this was the last package I used heavily that still had not made the upgrade. For the Python community: which packages are you still waiting for/working on?

vially 11y ago
I've been waiting for Fabric to make the migration to Python 3. Fortunately I no longer use Python that much (I jumped on the hype train to Go and I've been very satisfied with it) so I no longer need Fabric nowadays.
crdoconnor 11y ago
Mechanize would be nice, but it's practically abandonware at this point.
- kevinwang 11y ago
  same
random_upvoter 11y ago
A wrapper for the exiv2 library (for editing metadata of picture files) that works on Windows
xemoka 11y ago
Esri's ArcPy module and ArcGIS Desktop's "built-in" python environment. :( ArcGIS Pro is using 3.4, so the wrappers for ArcObjects are updated, but who knows if they will put that work back into ArcGIS Desktop...
craigds 11y ago
Still waiting on Ansible to make the jump. As far as I can tell it's not even on their roadmap, which is a bit crazy given that _almost_ everyone else has solved this by now. This may not be a blocker for the rest of our stuff though, since ansible can install its own python 2 on servers automatically.

bbayer 11y ago

It is a major change to ask but I am wondering if they consider to switch to asyncio instead of using Twisted. Twisted is great library but it is a huge dependency to maintain.

deleted 11y ago
[deleted]
deleted 11y ago
[deleted]
kmike84 11y ago
We're seriously considering this option (maybe for scrapy 2.0?), but no concrete plans yet. It can't be just asyncio - we'll need e.g. aiohttp and other packages. Some links: * POC for aiohttp as a http handler: https://github.com/scrapy/scrapy/pull/1455 https://github.com/scrapy/scrapy/pull/1455 * some thought about how to make async/await API for Scrapy; it is not all roses: https://github.com/scrapy/scrapy/issues/1144#issuecomment-141843616 https://github.com/scrapy/scrapy/issues/1144#issuecomment-14...
eliasdorneles 11y ago
There has been some discussions and a proof-of-concept with asyncio + aiohttp: https://github.com/scrapy/scrapy/pull/1455 https://github.com/scrapy/scrapy/pull/1455 However, as you said, it'd be a major change and it would affect the whole ecosystem (plugins and extensions), so it's complicated. We'll see what happens. :)
glyph 11y ago
Can you elaborate? What makes it a "huge dependency to maintain"? Is there anything that the Twisted project can do to make it easier? If this is actually a problem I'd really like to hear from users on the Twisted mailing list and bug tracker.
- bbayer 11y ago
  Twisted is a general purpose library/framework with lots of features. This is the "huge" part. In my previous projects I have used it a lot and appreciated it. What I was trying to tell is if Scrapy uses only small part of library, it may be possible for developers to use similar constructs from Python's standard library. In any case dependency is dependency and it is always better to minimize code footprint.

engi_nerd 11y ago

I've seen developers refuse to use Python 3 because of not being able to use Scrapy. Hopefully this gets some more devs to finally make the switch.

squeaky-clean 11y ago
I've been refusing to use Scrapy because of not being able to use Python 3 :P So this is a very exciting announcement for me.
- ddebernardy 11y ago
  Believe me, we've been hearing this too. We're super excited as well. :-)

jstoiko 11y ago

This is great news for Python 3. We're almost there! -> http://py3readiness.org http://py3readiness.org

pen2l 11y ago
Hm, why is opencv not there? I was playing around with it last week, and it seems it works well for only py2 right now (from what I could gather). OpenCV should be there in that list, a lot of people really need it.
- rspeer 11y ago
  My guess: because OpenCV is so unlikely to be installable through pip, its download statistics on PyPI end up artificially low.
- orf 11y ago
  It's only the 360 most downloaded packages, perhaps OpenCV is not one of them?
- jritchie 11y ago
  OpenCV 3 supports Python 3[1], but it's not distributed through PyPI. [1] http://opencv.org/opencv-3-0.html http://opencv.org/opencv-3-0.html
daveguy 11y ago
Hahahaha. 7 years later. Python 3 is almost there!
- rspeer 11y ago
  Your response to lots of people's programming effort is "Hahahaha"? Get lost.
  - pen2l 11y ago
    I agree he's not being a great sport, but I sort of see where he's coming from. Having worked with python in the last few months for computer vision problems, I've been really frustrated with how things are. I was expecting better. At this point I don't use python for my work anymore, I use matlab. It's superbly well-documented, and things just /work/. And of course, nothing like the mess you see with this py2/py3 problem is to be seen anywhere near matlab. Yes, it's not free, but I've decided that the amount of time I had to spend, the headaches I had to endure, they were not worth the free price of python (and matlab for its great support was in fact worth its price). ymmv.

njharman 11y ago

Minor point of clarification re title; Python 3 has been here, it's Scrapy that has (finally and good for them) come to Python 3.

kmike84 11y ago
that's an example of https://en.wikipedia.org/wiki/Galilean_invariance https://en.wikipedia.org/wiki/Galilean_invariance

penetrarthur 11y ago

Not related to Scrapy, but what are some things you scrape the web for?

kevin_thibedeau 11y ago
I once scraped every October posting from Slashdot to see long term trends. Short story: it's dying. I project the active userbase will be gone by 2020. Curiously the bulk of the posters were in the 100k to 300k UID range. There was also evidence of shenanigans with UID assignment where they were skipping even numbers and odd numbers at various times possibly to inflate their numbers. This will get you IP banned BTW but I did get a full data set before their script caught me.
- gst 11y ago
  I wonder if Netcraft already confirmed that Slashdot is dying ;)
- hatchoo 11y ago
  ScrapingHub (the guys behind Scrapy) offers Crawlera which provides some sort of automatic proxying and throttling so you can scrape away avoiding getting banned.

Keats 11y ago

Nice! Everytime I create a new Scrapy project I keep forgetting it was not python3 and had to recreate the virtualenv. Great news!

erroneousfunk 11y ago

This is great! I wrote "Web Scraping with Python" (O'Reilly) and did everything with Python 3... except for the oddball section on Scrapy. Glad to know I can update that for the second edition!

bobby_9x 11y ago

I'm glad Python 3 supports so many libraries now. I just made the switch a couple of months ago (from 2).

ceronman 11y ago

The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic way ever again. I'm glad that we're starting to see light at the end of the tunnel. I find myself using Python 3 in most of my projects. Some times I still have to resort to Python 2 when some dependency is not ready but those cases are more rare every day. Also it's frustrating to use Python 2 when many cool features are now Python 3 only. It will take some more time, but I'm sure that the transition will eventually be completed.

thearn4 11y ago
I'm happy that we finally updated a lot of our code at work to be Python 3 compatible last year. The conversion was not that bad in the end. However, under Python 3 a number of benchmarks are slower, which we do need to look into some more. Has anyone else experienced this, specifically with scientific computing code migrating from 2 to 3? Ultimately, we test in 2 and 3 via CI, but still primarily develop using 2.
- kmike84 11y ago
  A data point: `scrapy bench` is about 1.5x - 2x slower in Python 3.5 than in Python 2.7. This command starts a simple HTTP server which serves HTML pages linked in arbitrary depth, and crawls it using Scrapy. Profiler shows URL parsing is much slower in Python 3 for some reason, I'm not sure why. I haven't digged further. Python 3 code also have to encode/decode strings more often, but the overhead was not significant for `scrapy bench`.
  - ogrisel 11y ago
    I think you should google the python.org bug tracker with the function names that seems to exhibit a performance regression according to your profiling session. If don't find any existing issue, please open a new one with a minimalist reproduction script that demonstrates the performance regression.
    - 11y ago

bootload 11y ago

This is great news. I often read the stick developers give to Py3, not wanting to upgrade from 2->3 citing code bases still using Py2.

BuckRogers 11y ago

I've been holding out for a while on moving to 3. I have tried every 3 release and always found issues or performance regressions. Like others have said here it's a concern about the performance issues, Python is already slow. It's an even harder sell when - all major (and equally important, the long tail of minor) libraries support 2 - CPython2 has the performance advantage in most (if not all) applications - virtually every 3rd party implementation supports it very well (PyPy in particular) - Python2 already supported unicode, so that gets old to hear about - Most of the new features are available as backports - Some new features are absurd, like the new 4th string formatting method in 3.6 People shouldn't openly wonder why someone uses 2 instead of 3 if you really look at it. I just started a new 3.5 project because while I gave 3.0-3.4 a shot, 3.5 hasn't had its runthrough yet. Most people in my shoes have more than likely moved on from Python to Go. I'd like to have this be the one and stop going back to 2.7. Admittedly patience is running on fumes after ~8 years of testing CPython3 releases. It wasn't just a bad break, it seems like it was a sloppy break. Instead of feature bloat, I'd like to see Python3 focus on performance.

Chris2048 11y ago
Also, Jython py3 dev lagged far behind py2.

Chris2048 11y ago

I mentioned some of the issues I had with scrapy ages ago on reddit: http://www.reddit.com/r/Python/comments/g112q/installing_and_using_scrapy_web_crawler_to_search/ http://www.reddit.com/r/Python/comments/g112q/installing_and... Never got a reply, but I'll reproduce here; I wonder how much of this is still true? " The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started. When I used it, I wanted to scrape a site until certain conditions were met; when the last page scraped returned no objects. I wanted all results initially returned from a page to be dropped if they were older than a certain date; Thus I wanted Scrapy to keep scraping until no new items were found. Also, I wanted the latest date of the items returned so I could use this the next time I scrape. I created the 'DropElderMiddleware' middleware to do this. I couldn't see any other way of making calculations based on items returned from a particular page. I could never figure out what the difference between input and output processors where, or when I should use one or the other. The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain. The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can't remember the details of why I found this hard, but I can try to come up with an example if you like? In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy's features I didn't need e.g. I didn't want command-line control (I would actually prefer not to use the interface, though didn't discover how I could write a python script to apply the spider directly). Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy's object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what's going on in my scripts. "

kmike84 11y ago
> The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started. Yeah, docs used to be a problem; they improved a lot in 1.0 and 1.1 releases though. > When I used it, I wanted to scrape a site until certain conditions were met; when the last page scraped returned no objects. I wanted all results initially returned from a page to be dropped if they were older than a certain date; Thus I wanted Scrapy to keep scraping until no new items were found. Also, I wanted the latest date of the items returned so I could use this the next time I scrape. The easies way is to raise CloseSpider exception in a callback if no new items are scraped - see http://doc.scrapy.org/en/1.0/topics/exceptions.html#closespider http://doc.scrapy.org/en/1.0/topics/exceptions.html#closespi... > I could never figure out what the difference between input and output processors where, or when I should use one or the other. > The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain. I also have troubles understanding ItemLoader details. They are totally optional though, and they are no longer in Scrapy tutorial (http://doc.scrapy.org/en/latest/intro/tutorial.html http://doc.scrapy.org/en/latest/intro/tutorial.html). Item loaders provide features very similar to https://github.com/Suor/funcy https://github.com/Suor/funcy or https://github.com/kachayev/fn.py https://github.com/kachayev/fn.py. > The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can't remember the details of why I found this hard, but I can try to come up with an example if you like? I'm not sure what problems did you have. Scrapy selectors library (https://github.com/scrapy/parsel https://github.com/scrapy/parsel) is quite similar to PyQuery (esp. when CSS selectors are used), and nothing prevents you from using PyQuery with Scrapy. In future we may add PyQuery (and BeautifulSoup?) support to parsel and provide PyQuery selectors as response.pq (like response.css and response.xpath), +1 to do that. > In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy's features I didn't need e.g. I didn't want command-line control (I would actually prefer not to use the interface, though didn't discover how I could write a python script to apply the spider directly). Yeah, library interface used to be a problem. It was improved in 1.0 release (there is an official API for integrating Scrapy with Twisted apps and running spiders from user scripts), but there is still more to go. See http://doc.scrapy.org/en/1.0/topics/practices.html#run-scrapy-from-a-script http://doc.scrapy.org/en/1.0/topics/practices.html#run-scrap.... It probably won't be as easy to integrate with regular Python scripts as mechanize because Scrapy is async. On the other hand, Scrapy is easier to integrate with async servers like Twisted or Tornado. > Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy's object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what's going on in my scripts. You may want to check the new 'Scrapy at glance' page (http://doc.scrapy.org/en/latest/intro/overview.html http://doc.scrapy.org/en/latest/intro/overview.html). The main advantage of Scrapy over mechanize is that it handles parallel downloads and have a wide range of built-in extensions you won't have to implement yourselves.

darkrho 11y ago

Try it out via conda! conda install -c scrapinghub/label/dev scrapy