6 ms·
Crawl a website with scrapy and store extracted results with MongoDB
- hack_edu 14y agoI really want to read. Topic is right down my alley. Unfortunately, the page is literally broken and unreadable on Android ICS with Chrome :(
- mumphster 14y agoSame using mobile safari
- jordanmessina 14y agoSame using chrome on windows unless I resize the browser and make the width about 1000px
- deleted 14y ago[deleted]
- martius 14y agoYes, it's true for any viewport with a width of less than 940px. I'll do my best to fix this too.
- noinput 14y agohttp://www.readability.com/mobile/articles/yfhqwo0t http://www.readability.com/mobile/articles/yfhqwo0t cleans it right up.
- martius 14y agoHi all, thank you for reporting ! This is a known issue and I'm working on it! I'll try to push a "responsive" version today or tomorrow.
- joshu 14y agoCompletely unreadable on iPhone too.
- jat1 14y agoAlso check this out for a pretty good discussion on scraping http://pyvideo.org/video/609/web-scraping-reliably-and-efficiently-pull-data http://pyvideo.org/video/609/web-scraping-reliably-and-effic...
- BaltoRouberol 14y agoYeah, I actually learnt scraping from Asheesh :) He's awesome.
- jat1 14y agoI have been playing with scraping for quite some time now and have my own scripts and stuff, but I found that video informing and there were a few useful snippets I had missed. Keep meaning to check out more of the Pycon vids
- ananthrk 14y agoCool. BTW, is there a reason for naming the file "isullshit_spiders.py" and not as "isbullshit_spiders.py"? :)
- BaltoRouberol 14y agoOh, that's just a typo. My bad. Edit: there, corrected.
- JackC 14y agoFor really quick one-off scraping, httplib2+lxml+PyQuery is a pretty neat combination: import httplib2, lxml, pyquery h = httplib2.Http(".cache") def get(url): resp, content = h.request( url, headers={'cache-control':'max-age=3600'}) return pyquery.PyQuery( lxml.etree.HTML(content) ) This gives you a little function that fetches any URL as a jquery-like object: pq = get("http://foo.com/bar") checkboxes = pq('form input[type=checkbox]') nextpage = pq('a.next').attr('href') And of course all of the requests are cached using whatever cache headers you want, so repeated requests will load instantly as you iterate. Just something else to throw in the toolbelt ...
- the_cat_kittles 14y agoHave checked out kenneth reitz's requests? Its fantastic, you might like it
- codehenge 14y agoLink, for the interested: https://github.com/kennethreitz/requests https://github.com/kennethreitz/requests
- the_cat_kittles 14y agothanks, i should have included a code sample too: import requests from lxml import etree jquery_like_page = etree.HTML(requests.get('url').text)
- deleted 14y ago[deleted]
- danneu 14y agoHere's the same functionality written in Ruby using Chris Kite's crawler called Anemone[1]. Gist: https://gist.github.com/2475824 https://gist.github.com/2475824. Screenshot: http://i.imgur.com/cbv9A.png http://i.imgur.com/cbv9A.png [1]: http://anemone.rubyforge.org/doc/index.html http://anemone.rubyforge.org/doc/index.html