4 ms·
It has to be behavioral, not IP based for the most part because of the ways these networks operate—primarily on personal computers/routers on US-based residenti
by garciasn 3y ago
It has to be behavioral, not IP based for the most part because of the ways these networks operate—primarily on personal computers/routers on US-based residential connections.
I deal with these constantly for our clients. It’s amazing how smart they function to get around limits (thousands of IPs and valid recent user agents) but how they miss easy things like using the same cookie to submit 1000s of different entries with obvious purchased lists of names/emails, many of whom are deceased.
- TekMol 3y ago"Thousands of IPs" can not hurt Twitters bottom line in any way. With a "read 10 tweets per IP per day" limit, that is 10k tweets. Delivering 10k tweets does not even cost a penny.
- semicolon_storm 3y ago10 per day is a ridiculously low limit. Also, NAT and shared networks (think college campus) mean that hundreds of people often share the same IP, making that limit even more ridiculous. Reality would be that limit would have to be much higher.
- costco 3y agohttps://brightdata.com/proxy-types/residential-proxies https://brightdata.com/proxy-types/residential-proxies gives you access to 70 million IPs. You'd have to be a little insane to scrape millions of tweets like this though because the bandwidth costs would be very high as even with massive volume you're probably not getting any price lower than $5-7/GB. Fortunately Twitter doesn't rate limit by IP (at least as long as you're not posting), they allow guest access and they have an endpoint that lets you get 100 tweets at a time so it's not very difficult to scrape as I described in https://news.ycombinator.com/item?id=36667225 https://news.ycombinator.com/item?id=36667225.