3 ms·
I use YQL to do web scraping. It lets me do something like this: `select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"` Could you ela
by hobonumber1 13y ago
I use YQL to do web scraping. It lets me do something like this:
`select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"`
Could you elaborate on the benefits of using Upton instead of this?
- joshaidan 13y agoI've always used this method: http://railscasts.com/episodes/190-screen-scraping-with-nokogiri http://railscasts.com/episodes/190-screen-scraping-with-noko... As well as Mechanize when working with sites that require session cookies and all that. I'm wondering too what the advantages of Upton are?
- jeremybmerrill 13y agoAFAICT, YQL can only handle scraping individual pages that way. Upton can scrape a whole set of pages. If you have a page that lists the pages you're interested in; suppose you're interested in HN commenters on front page posts, you could specify the front page URL and a selector for links to comment pages, and Upton would automatically scrape those pages and return them to you. Upton could even write the commenter names to a CSV for you with just a filename and a CSS selector/XPath expression. It's not stuff you couldn't do with YQL or Python/BeautifulSoup. But it's stuff that I didn't want to have to write over and over each time I wrote a new scraper.
- hobonumber1 13y agoMakes sense! Thanks for clarifying that.