4 ms·

I am using Scrapy a lot. http://scrapy.org/ http://scrapy.org/ It is very well designed web crawling library.

by bbayer 11y ago

vaibhavkul 11y ago
No Python 3 support yet?
- elyase 11y ago
  unfortunately not yet, just tried it yesterday. There has been some progress but it looks like it might still take a while [1]. [1] https://github.com/scrapy/scrapy/issues/263 https://github.com/scrapy/scrapy/issues/263
IanCal 11y ago
Another vote for scrapy, massively useful for web crawling. Particularly love the shell for debugging things.
scrollaway 11y ago
I found out, through reddit a couple of days ago, about Pomp: https://bitbucket.org/estin/pomp https://bitbucket.org/estin/pomp It looks like a much cleaner Scrapy-inspired spider framework, without the twisted dependency. And it's python 2+3 compatible. I'm very excited to try it out.
- bbayer 11y ago
  It looks very promising. I will give it a try.
daturkel 11y ago
Just seeing this for the first time. Got a proof-of-concept demo working in no time (after fussing with install requirements...). Looks like a great tool for non-devs like me who still need to scrape things occasionally for data collection and analysis.
steinsgate 11y ago
I love that the Python community is coming up with so many robust frameworks. I have used Scrapy, and it provides most boilerplate functionality out of the box. Gone are the days I would use wget for my scraping tasks. This being said, it's a bit disappointing that Scrapy still doesn't have native support for dynamic pages. I am hoping to see this feature in the upcoming releases. With more and more of the web becoming dynamic, this should be priority feature. Other than that, I have nothing but praise for Scrapy. Pomp is apparently a simplistic take on Scraping, but it doesn't handle redirects, caching, cookies, authentication etc. I wonder if it provides parallel processing out of the box; most likely not. This is a bit strange, because these are the features for which I would prefer using a framework over writing my own code, which makes me less inclined to try Pomp. Scrapy for the win :D
tuyguntn 11y ago
My vote goes to pyspider[1] [1] - https://github.com/binux/pyspider https://github.com/binux/pyspider

I am using Scrapy a lot. http://scrapy.org/ http://scrapy.org/ It is very well designed web crawling library.

by bbayer 11y ago

I am using Scrapy a lot. http://scrapy.org/ http://scrapy.org/ It is very well designed web crawling library.

vaibhavkul 11y ago

No Python 3 support yet?

elyase 11y ago
unfortunately not yet, just tried it yesterday. There has been some progress but it looks like it might still take a while [1]. [1] https://github.com/scrapy/scrapy/issues/263 https://github.com/scrapy/scrapy/issues/263

IanCal 11y ago

Another vote for scrapy, massively useful for web crawling. Particularly love the shell for debugging things.

scrollaway 11y ago

I found out, through reddit a couple of days ago, about Pomp: https://bitbucket.org/estin/pomp https://bitbucket.org/estin/pomp It looks like a much cleaner Scrapy-inspired spider framework, without the twisted dependency. And it's python 2+3 compatible. I'm very excited to try it out.

bbayer 11y ago
It looks very promising. I will give it a try.

daturkel 11y ago

Just seeing this for the first time. Got a proof-of-concept demo working in no time (after fussing with install requirements...). Looks like a great tool for non-devs like me who still need to scrape things occasionally for data collection and analysis.

steinsgate 11y ago

I love that the Python community is coming up with so many robust frameworks. I have used Scrapy, and it provides most boilerplate functionality out of the box. Gone are the days I would use wget for my scraping tasks. This being said, it's a bit disappointing that Scrapy still doesn't have native support for dynamic pages. I am hoping to see this feature in the upcoming releases. With more and more of the web becoming dynamic, this should be priority feature. Other than that, I have nothing but praise for Scrapy. Pomp is apparently a simplistic take on Scraping, but it doesn't handle redirects, caching, cookies, authentication etc. I wonder if it provides parallel processing out of the box; most likely not. This is a bit strange, because these are the features for which I would prefer using a framework over writing my own code, which makes me less inclined to try Pomp. Scrapy for the win :D

tuyguntn 11y ago

My vote goes to pyspider[1] [1] - https://github.com/binux/pyspider https://github.com/binux/pyspider