3 ms·
Just write yourself a Ruby script that downloads a webpage. Then parse the downloaded content with Ruby's equivalent of BeautifulSoup (a forgiving HTML parsing
by tocomment 16y ago
Just write yourself a Ruby script that downloads a webpage. Then parse the downloaded content with Ruby's equivalent of BeautifulSoup (a forgiving HTML parsing library).
That's pretty much all there is to it. You'll want to manually inspect the HTML from the page you're scraping to find patterns. For example all courses are in <li> tags, etc.
Let me know if you need more help. But I think as a first step write a program to download a web page.
- bdclimber14 16y agoI've never written anything like this... Is there a function like CURL that downloads the page as a string from a URL?
- chrisa 16y agorequire 'net/http' require 'uri' output = Net::HTTP.get(URI.parse("http://www.google.com"))
- seanmccann 16y agocurb is a lot faster than net/http require 'curb' c = Curl::Easy.perform("http://www.google.com http://www.google.com) puts c.body_str
- nc 16y agoPremature optimization much?
- bdclimber14 16y agoSo apparently accessing a secure URL isn't straightforward... No login is required, its just an HTTPS but it won't go in curb.
- bdclimber14 16y agoWasn't an SSL problem but a cookie problem...
- bdclimber14 16y agoI wanted to thank you for your help. I wrote a script to scrape over 30,000 courses and textbooks from Arizona State University successfully in a couple days.