6 ms·
Dead simple Python crawler for extracting structured data from a website to CSV
- sethherr 11y agoThis is why I recommend Ruby over Python for web development. The difference between a blog posted code snippet and the wealth of mature, multiple contributor projects like https://github.com/propublica/upton https://github.com/propublica/upton
- orf 11y agoThe author has just decided to roll their own half baked script for some reason, there are plenty of libraries to help. Python is good for scraping, I released my own library recently[1] that I think hits a sweet spot but there are many others. Python and Ruby have their places, I wouldnt say one is clearly better than the other. 1. http://tomforb.es/scraping-websites-with-cyborg http://tomforb.es/scraping-websites-with-cyborg
- rangeva 11y agoThe script is intended to be a copy/paste script, with no imports and just "Fill in the blanks" parameters. There A LOT of other options that are much better, but require more setup: http://scrapy.org/ http://scrapy.org/ https://pypi.python.org/pypi/spider.py https://pypi.python.org/pypi/spider.py
- aerioux 11y agoupvote - python has ridiculously good (sometimes slowish) scraping libraries, unfortunately the author has written a python wrapper over a regex which completely throws any half-sane crawling out the window. The except: and continues is enough to cause a heartattack
- brachi 11y agoProbably you didn't hear of scrapy https://github.com/scrapy/scrapy https://github.com/scrapy/scrapy
- catmanjan 11y ago+1 for scrapy, very easy to set up while remaining powerful enough to actually use!
- simonw 11y agoPython has an enormous quantity of mature, multiple contributor projects.
- pinewurst 11y agoLike the very usable Beautiful Soup: http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/
- sethherr 11y agoI came across as aggressively flamey - but what I intended to communicate was the difference in the communities. In the Ruby community, if somebody posted a really long regex driven scraper, people would ask why it was written and recommend looking at libraries/gems instead of writing your own. In Python, it ends up on the front page of HN. Ruby's community is very pragmatic and Python's is more scientific. Most people I'm discussing learning programming with are interested in making something, and pragmatism is far better suited to their goals. e.g. from my perspective, as someone who uses programming as a tool, writing my own scraper is boring (other scrapers work well enough for me). From my friend's perspective, writing a scraper is a fun experiment.
- brachi 11y agourlparse().netloc, not netlock. Would be nice to clarify is for Python 2, even better to just do it in 3. Maybe not as dead simple for beginners if you use regular expressions.
- blacksqr 11y agoUses one weird trick. Ruby users HATE this!
- echochar 11y ago"No external imports required..." I convert HTML to CSV regularly (multiple times a day). But I do not use Python; I use C. Actually I use flex to make filters which I compile as static binaries that read from stdin. This is in fact how I read HN. The HTML is converted to CSV and then the CSV is imported into a database. Prior to using flex I primarily used sed. For many sites I still do; it's faster than having to compile, test, recompile. If anyone has a website they want in CSV, and need something faster than Python or Ruby, just post the url. I like to think I am reasonably good at this, but I only do it for personal use on sites I'm interested in so who knows. For me HTML conversion to CSV and plain text is an art - I practice it every day.
- catmanjan 11y ago>This is in fact how I read HN. Out of curiosity, why?
- echochar 11y ago1. Practice with new programming language and database. 2. Turn unstructured, difficult to parse data adorned with HTML, and other window dressing into structured data that is easier to parse.
- random778 11y agoI'd like to see HN, actually.