5 ms·

I don't believe that Twitter has to limit requests to avoid bots. They would just have to limit requests by IP and thats it. Say 10 tweets per IP per day. For

by TekMol 3y ago

I don't believe that Twitter has to limit requests to avoid bots. They would just have to limit requests by IP and thats it. Say 10 tweets per IP per day. For requests which do not come from logged in users. I can't see how scrapers would be able to come up with enough IPs to make a dent in Twitters costs.

malfist 3y ago
IPv6 makes it trivial to come up with a mind boggling number of ip addresses. You don't even have to leave your network
- jsnell 3y ago
  Nobody does rate limiting or other abuse work on a full /128. Typically a /64 is treated as the equivalent of an IPv4 address.
  - charcircuit 3y ago
    This will have many false positives. Especially if the address is from a mobile network.
    - zhfliz 3y ago
      people don't care about false positives on CGNAT either, so not much difference to the IPv4 situation if you target /64s
    - costco 3y ago
      Last I read this is how Instagram rate limits IPV6. See https://news.ycombinator.com/item?id=32687694 https://news.ycombinator.com/item?id=32687694 So apparently not enough false positives for them to care.
    - fulafel 3y ago
      Is it common for mobile networks to hand out short (64 or smaller) prefixes to phones? Phones are routers after all (hotspot) so this would seem to break stuff.
      - deleted 3y ago
        [deleted]
- TekMol 3y ago
  First of all, Twitter does not support IPv6. Secondly, if they would, they could simply rate limit by the first x chars of the address.
berkle4455 3y ago
So your solution is to just make it a walled garden. What about the scraping bots who simply log-in?
- TekMol 3y ago
  My solution is to make it less of a walled garden. At the moment, they often refuse even reading a single tweet when not logged in. And for the "simply log-in": It is even more expensive (way more expensive) to get good email adresses as it is to get good IPs. Email providers have very tough checks these days before they give you one.
garciasn 3y ago
It has to be behavioral, not IP based for the most part because of the ways these networks operate—primarily on personal computers/routers on US-based residential connections. I deal with these constantly for our clients. It’s amazing how smart they function to get around limits (thousands of IPs and valid recent user agents) but how they miss easy things like using the same cookie to submit 1000s of different entries with obvious purchased lists of names/emails, many of whom are deceased.
- TekMol 3y ago
  "Thousands of IPs" can not hurt Twitters bottom line in any way. With a "read 10 tweets per IP per day" limit, that is 10k tweets. Delivering 10k tweets does not even cost a penny.
  - semicolon_storm 3y ago
    10 per day is a ridiculously low limit. Also, NAT and shared networks (think college campus) mean that hundreds of people often share the same IP, making that limit even more ridiculous. Reality would be that limit would have to be much higher.
  - costco 3y ago
    https://brightdata.com/proxy-types/residential-proxies https://brightdata.com/proxy-types/residential-proxies gives you access to 70 million IPs. You'd have to be a little insane to scrape millions of tweets like this though because the bandwidth costs would be very high as even with massive volume you're probably not getting any price lower than $5-7/GB. Fortunately Twitter doesn't rate limit by IP (at least as long as you're not posting), they allow guest access and they have an endpoint that lets you get 100 tweets at a time so it's not very difficult to scrape as I described in https://news.ycombinator.com/item?id=36667225 https://news.ycombinator.com/item?id=36667225.
38 3y ago
I dont know what the hell Twitter is doing, but I simply dont use it anymore. I cannot access this page without logging in: https://twitter.com/SarahKSilverman https://twitter.com/SarahKSilverman I check it ONCE per day, thats it. and for weeks every time it just prompts me to login. no thanks, done with Twitter.
- deleted 3y ago
  [deleted]
- pabs3 3y ago
  You can use nitter to access Twitter content anonymously at the moment: https://nitter.net/SarahKSilverman https://nitter.net/SarahKSilverman

by TekMol 3y ago

malfist 3y ago

IPv6 makes it trivial to come up with a mind boggling number of ip addresses. You don't even have to leave your network

jsnell 3y ago
Nobody does rate limiting or other abuse work on a full /128. Typically a /64 is treated as the equivalent of an IPv4 address.
- charcircuit 3y ago
  This will have many false positives. Especially if the address is from a mobile network.
  - zhfliz 3y ago
    people don't care about false positives on CGNAT either, so not much difference to the IPv4 situation if you target /64s
  - costco 3y ago
    Last I read this is how Instagram rate limits IPV6. See https://news.ycombinator.com/item?id=32687694 https://news.ycombinator.com/item?id=32687694 So apparently not enough false positives for them to care.
  - fulafel 3y ago
    Is it common for mobile networks to hand out short (64 or smaller) prefixes to phones? Phones are routers after all (hotspot) so this would seem to break stuff.
    - deleted 3y ago
      [deleted]
TekMol 3y ago
First of all, Twitter does not support IPv6. Secondly, if they would, they could simply rate limit by the first x chars of the address.

berkle4455 3y ago

So your solution is to just make it a walled garden. What about the scraping bots who simply log-in?

TekMol 3y ago
My solution is to make it less of a walled garden. At the moment, they often refuse even reading a single tweet when not logged in. And for the "simply log-in": It is even more expensive (way more expensive) to get good email adresses as it is to get good IPs. Email providers have very tough checks these days before they give you one.

garciasn 3y ago

It has to be behavioral, not IP based for the most part because of the ways these networks operate—primarily on personal computers/routers on US-based residential connections. I deal with these constantly for our clients. It’s amazing how smart they function to get around limits (thousands of IPs and valid recent user agents) but how they miss easy things like using the same cookie to submit 1000s of different entries with obvious purchased lists of names/emails, many of whom are deceased.

TekMol 3y ago
"Thousands of IPs" can not hurt Twitters bottom line in any way. With a "read 10 tweets per IP per day" limit, that is 10k tweets. Delivering 10k tweets does not even cost a penny.
- semicolon_storm 3y ago
  10 per day is a ridiculously low limit. Also, NAT and shared networks (think college campus) mean that hundreds of people often share the same IP, making that limit even more ridiculous. Reality would be that limit would have to be much higher.
- costco 3y ago
  https://brightdata.com/proxy-types/residential-proxies https://brightdata.com/proxy-types/residential-proxies gives you access to 70 million IPs. You'd have to be a little insane to scrape millions of tweets like this though because the bandwidth costs would be very high as even with massive volume you're probably not getting any price lower than $5-7/GB. Fortunately Twitter doesn't rate limit by IP (at least as long as you're not posting), they allow guest access and they have an endpoint that lets you get 100 tweets at a time so it's not very difficult to scrape as I described in https://news.ycombinator.com/item?id=36667225 https://news.ycombinator.com/item?id=36667225.

38 3y ago

I dont know what the hell Twitter is doing, but I simply dont use it anymore. I cannot access this page without logging in: https://twitter.com/SarahKSilverman https://twitter.com/SarahKSilverman I check it ONCE per day, thats it. and for weeks every time it just prompts me to login. no thanks, done with Twitter.

deleted 3y ago
[deleted]
pabs3 3y ago
You can use nitter to access Twitter content anonymously at the moment: https://nitter.net/SarahKSilverman https://nitter.net/SarahKSilverman