4 ms·

You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questiona

You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective. From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI

andai 2mo ago
So it could just be 1 guy with scrapy and infinite vc money
jancsika 2mo ago
Still not sure about the "who" question, but here's the scale of money changing hands: > Residential proxies are everywhere, so why did proxy DDoS attacks mostly come from the U.S.? The answer is money. If you are committing fraud or circumventing content restrictions, a Russian IP address gets geo-blocked instantly. A fresh U.S. residential IP address (especially one behind carrier-grade NAT and harder to block individually) is “gold.” Customers pay up to $95 to lease a single U.S. residential IP address for 2 weeks (versus $0.30 for an Eastern European IP). Compare that to your own ARPU per subscriber and sit with it for a second. When an individual IP is worth more than the customer relationship behind it, you don’t have a technical problem. You have a market problem.[1] 1: https://www.nokia.com/blog/one-year-later-the-residential-proxy-botnet-problem-got-bigger-not-smaller/ https://www.nokia.com/blog/one-year-later-the-residential-pr...
- sandeepkd 2mo ago
  Here is the fun part, the same companies invests millions of dollars to protect its infrastructure and services from scrapers/bots AND they also invest millions of dollars to circumvent the bots, captchas and rate limits. Unfortunately you cant have it both ways which makes it hard problem to solve for everyone
- br0ceph 2mo ago
  if they pay 190/mo to proxy, i should just sell my ip direct -> profit i pay less than half that for fiber , should get a few more isp accounts, and rake it in.
cozzyd 2mo ago
if they have agents write the scrapers you better believe they're playing every evil trick in the book

andai 2mo ago

So it could just be 1 guy with scrapy and infinite vc money

jancsika 2mo ago

Still not sure about the "who" question, but here's the scale of money changing hands: > Residential proxies are everywhere, so why did proxy DDoS attacks mostly come from the U.S.? The answer is money. If you are committing fraud or circumventing content restrictions, a Russian IP address gets geo-blocked instantly. A fresh U.S. residential IP address (especially one behind carrier-grade NAT and harder to block individually) is “gold.” Customers pay up to $95 to lease a single U.S. residential IP address for 2 weeks (versus $0.30 for an Eastern European IP). Compare that to your own ARPU per subscriber and sit with it for a second. When an individual IP is worth more than the customer relationship behind it, you don’t have a technical problem. You have a market problem.[1] 1: https://www.nokia.com/blog/one-year-later-the-residential-proxy-botnet-problem-got-bigger-not-smaller/ https://www.nokia.com/blog/one-year-later-the-residential-pr...

sandeepkd 2mo ago
Here is the fun part, the same companies invests millions of dollars to protect its infrastructure and services from scrapers/bots AND they also invest millions of dollars to circumvent the bots, captchas and rate limits. Unfortunately you cant have it both ways which makes it hard problem to solve for everyone
br0ceph 2mo ago
if they pay 190/mo to proxy, i should just sell my ip direct -> profit i pay less than half that for fiber , should get a few more isp accounts, and rake it in.

cozzyd 2mo ago

if they have agents write the scrapers you better believe they're playing every evil trick in the book