4 ms·
Ask HN: How to Make Money from Web Scraping?
What are some ways to make money from web scraping?
Few options that I can think of, are:
1. Scrape and sell data
2. Scrape public data and publish as a website. Make money with ads / subscriptions
3. Build price comparison websites
4. Offer web scraping services.
Any real life examples / use-cases out there?
- jppope 7y agoSo not to be decisive... but if you have a skill such as a talent for scraping why not look for a value-add for the world compared to primarily "rent-seeking" activities like the above. (Advertising is never a good way to make money) Why not set up a scraping system that helps people improve accessibility by alerting them to flaws in their sites?
- pythonbase 7y agoGood idea. Can you elaborate this a bit?
- rdbell 7y agoIf any HN community members are doing web scraping that may benefit from residential proxies rather than datacenter IPs, I can get you set up with some free test credits on our residential proxy network: https://packetstream.io https://packetstream.io We tunnel requests through real opt-in residential ISP users. Send us a message via our contact form and we'll load up your account with some test credits. Happy scraping!
- nickthemagicman 7y agoThat's badass but how do they stop from being banned? Like if I scrape Craigslist enough and get that IP banned the resident can't visit Craigslist? I don't know enough about this to understand. Or would it cycle through the proxy IPs real fast?
- surewhynat 7y agousually these are controlled via an api of some sort. depending on the distributed system. another popular solution is hola which offers basically the same thing, albeit I think worse prices when I was using it than this one...
- rdbell 7y ago> Or would it cycle through the proxy IPs real fast? Exactly. PacketStream lets you change your visible IP with each request. If you need a persistent IP/geolocation that can be configured as part of the request.
- nickthemagicman 7y agoThat is totally badass. Thanks.
- jackcodes 7y agoThe coolest use of a botnet I’ve seen in a while!
- cyranodb 7y agoSounds a lot like you're resnet: https://krebsonsecurity.com/2019/08/the-rise-of-bulletproof-residential-networks/ https://krebsonsecurity.com/2019/08/the-rise-of-bulletproof-...
- gitgud 7y agoPossibly one more item is "scraping to gather analytics and trends", the revenue comes from using machine learning to predict trends based on the scraped data from social media, news etc... in most cases the raw data isn't as valuable as the patterns or predictions you can see in it... Example -> https://meetglimpse.com/ https://meetglimpse.com/
- pythonbase 7y agoVery interesting. Your project?
- gitgud 7y agoNot my project, was on "Show HN" a few months ago. But I did try to use Web-Scraping a few years ago, when I was trying to set up a crypto-currency [1] arbitrage... but there were a lot of risks and challenges... could be another option for you though. [1] https://en.wikipedia.org/wiki/Arbitrage https://en.wikipedia.org/wiki/Arbitrage