4 ms·
I don't know why, but I love web scraping. Somehow it's fun to me to be able to grab data and organize it in a DB in a meaningful way. Even better is using th
by paulhauggis 15y ago
I don't know why, but I love web scraping. Somehow it's fun to me to be able to grab data and organize it in a DB in a meaningful way. Even better is using the data to make money.
I wrote my own scraper framework for various page types (I don't want to go into too many details) and my latest uses approx 500GB of bandwidth/month. I run it on one VPS for around $50/month.
- Tichy 15y agoWhere do you store the data? Most hosts don't seem to offer a lot of storage on their servers.
- paulhauggis 15y agomy DBs aren't that huge because I don't store all of the data I scrape, just the important stuff. I used Godaddy for awhile, but I now use 1and1. Out of all the hosts I've used over the years, 1and1 ha been the best in terms of support and up time. They have a VPS plan with 2TB of bandwidth and you can upgrade the storage. I think my Mysql DB is currently around 20GB.
- mdaniel 15y agoIn that case, sign up at https://ScraperWiki.com/ https://ScraperWiki.com/ and put your passion to good use!
- sente 15y agoThat's awesome. I also love scraping, there's just something about which I find really satisfying.