5 ms·
Ask HN: What do you scrape the web for?
Hey folks,
It's not rare for me to see myself scraping the web for both side projects and personal needs - checking whether the prices of the groceries are good or not based on the local supermarket's website; gathering some emails to cold-email about ideas ...
It seems to me that just by having the ability to quickly write some code to gather information from a government web page or anything that shows stuff on the web is a huge power.
What about you? Any great web scraping stories? Have you ever got in trouble by doing so? Made an entire company of it?
- beld_pro 9y agoI think some of the most real-world use cases are in the realestate/hotels/coworking-spaces sector. The second space, email gathering (guess what, all the time people have to use <blabla> at <something> dot com to try to not have their emails scraped).
- cirowrc 9y agototally agree! I've never been exposed to realestate stuff but it definitely seems a place where that makes a lot of sense. Did you have some experiences with it yourself?
- byoung2 9y agoI just wrote a crawler for a client who needs to check vendor licenses (plumber, electrician, etc). Given a license number, state, and trade, it looks up the appropriate licensing agency in that state and pulls the license info (issue/expiration dates, biographical information, infractions, etc). Luckily most of these are old school sites so they're easy to crawl, but a few have captcha (easily solved with OCR) or are single page apps.
- cirowrc 9y agoyeah, that's been my experience too: old (or in my case, gov websites) are much much easier but at least here I never needed to solve captchas - maybe it's just that scraping is not a big thing where I live (Brazil).
- PaulHoule 9y agoI made the State of Delaware change its search form so that you can only look up a company if you know both the name and the number.