3 ms·
There's no single answer that will stand the test of time, because the structure of Web pages changes over time, to some extent because of scrapers. This gener
by lutusp 12y ago
There's no single answer that will stand the test of time, because the structure of Web pages changes over time, to some extent because of scrapers.
This general advice should serve:
1. Get the page content the simplest way possible.
2. Use regular expressions to extract the desired content.
For both the above goals, and because so much revision is needed over time, I recommend Python.
- webmaven 12y agoWell, sure, but even if your recommendation is to do it yourself with Python (rather than using a service) and regexes rather than BeautifulSoup, then the question becomes mechanize vs. scrapy vs. robobrowser, etc. etc.
- nreece 12y agoUse regular expressions to extract the desired content. You shouldn't use regex for parsing HTML. It's not reliable, nor suitable. Use a proper DOM parser. Read http://blog.codinghorror.com/parsing-html-the-cthulhu-way/ http://blog.codinghorror.com/parsing-html-the-cthulhu-way/