My first scraping project was well over 10 years ago in college. I was a member of the education club and we wanted to get funding so I convinced the college o

My first scraping project was well over 10 years ago in college. I was a member of the education club and we wanted to get funding so I convinced the college of education to allow us to charge $10 automatically to students in their school. But then the administration dragged their feet to give us a list to submit to the accounting office for billing. 1 (of the many professors) submitted a list to me via outlook that they copied off the site so I was able to look at the HTML structure of their list. The university used basic security (htaccess) and didn't verify that you had permission for a task once you were in. I had access because I worked for the dean of men. So I scraped all the faculties student lists and then used another system behind the htaccess point to get all the relevant information on each student. Compiled a list of 300 students and submitted it getting the club $3,000 in funding. The college of ed office staff were freaked because they had no clue how I came up with the student roster (no one in their office gave it to me) but nothing came of it. Been scraping a lot lately but mostly: - government website for license holders - creating lists of businesses for different segments (market research/analysis) - using those lists to scrape individual sites and make analysis (how many use facebook/youtube/etc)