3 ms·
Thanks! I've been doing some scraping projects lately and really like it a lot. There's a pretty steep learning curve, but it gets easier and easier as you go a
by bgraves 15y ago
Thanks! I've been doing some scraping projects lately and really like it a lot. There's a pretty steep learning curve, but it gets easier and easier as you go along, I think.
1. Caching pages is definitely a great idea while debugging. Especially if the data source has a request limit :)
2. I've never heard of Nokogiri, but it looks like BeautifulSoup for Ruby. I've found that Python has worked for everything I need so far, but thanks for the reference.
3. I suck so bad at regex, but using it more will help me climb that mountain.
4. One tip I've used is writing out the "INSERT INTO TABLE..." statements along with the scraped results. I definitely use CSV (and Google Refine) for general clean up and spot checking.
5. You should write a 'Data Scraping One-liners Explained' ebook :)
- berntb 15y ago>>I suck so bad at regex Then you are the opposite of where I used to be -- I thought I understood and could use regexps. Hell, I do Perl for fun. :-) Something like when I first sat down with Photoshop -- "Hey, I know how to program Macs [before MacOs X]. This is just using a Macintosh program, so I should have no problems"... :-) Read "Mastering Regular Expressions". It made me feel embarrassed about my previous stupidity [Edit: Embarrassment, your name is Dunning–Kruger :-) ]. Just the first few chapters are enough to change your world. Edit: I might add, I still can't use Photoshop.
- semanticist 15y agoI spent the first half of this year writing scrapers for every newspaper in the UK. My Top Regex Tip is http://rubular.com/ http://rubular.com/ - this thing saved me HOURS of my life.