4 ms·
Most "screen scraping" these days is just extracting content from web pages. 1) Write a program that can load webpages as if it was a user of the site. 2) Hav
by Sukotto 16y ago
Most "screen scraping" these days is just extracting content from web pages.
1) Write a program that can load webpages as if it was a user of the site.
2) Have it save everything it loads.
3) Write a program that can extract the data you care about out of the html and put it into a more useful format (or into a database or something)
Many languages have libraries for this or you can use a tool like cURL or wget. I do this a lot with perl and the LWP family of modules but the sites I work on don't use javascript or dom manipulation... There's so much javascript and ajax out there now though that I'm not sure if you can scrape those kinds of sites with perl.