3 ms·
When people think about web scraping in Python, they usually think BeautifulSoup. When I think about web scraping in Python, I think Mechanize. BeautifulSoup i
by lunchbox 18y ago
When people think about web scraping in Python, they usually think BeautifulSoup.
When I think about web scraping in Python, I think Mechanize. BeautifulSoup is great for parsing already downloaded HTML files, but it doesn't have the same web-navigating features Mechanize does such as stateful web browsing and easy form filling (unless I'm missing something).
- utnick 18y ago+1 for mechanize, been doin alot of scraping in Ruby lately and the ruby mechanize ( i'm assuming its a port ? ) is quite nice
- ianb 18y agoPerl's Mechanize was probably the basis for both.
- kaens 18y agoI've been thinking about checking out Mechanize, but I've been using BeautifulSoup and a small library of helper functions for a while and haven't found time to familiarize myself with Mechanize. BeautifulSoup is lacking easy form-filling functions, but they're pretty easy to write, assuming that the site doesn't do really funky stuff with their parameters. Basically, just grab all the input and select elements from a given form and put them in a dict, update values, and urlencode.