30 ms·

I don’t think it’s necessarily saying the self-inflicted DDoS has caused a technical issue that’s forced them to shut down access. I think it’s possible that sh

by evan_ 3y ago

I don’t think it’s necessarily saying the self-inflicted DDoS has caused a technical issue that’s forced them to shut down access. I think it’s possible that shutting down anonymous access caused the DDoS, which led to giant spikes in some metric, which led them (Elon) to conclude that there was an uptick in scraping, so they imposed the 600/tweet/day limit to punish scrapers. Seems like either my quota reset or they changed the policy because I’m able to access the site again.

pschuegr 3y ago
This. I'd bet substantial amounts of money that the evil scraper idea is the result of a) another issue + b) paranoia + c) Musk thinking he understands better than anybody else.
- berkle4455 3y ago
  This is a really ignorant take to dismiss scrapers. LLMs operate by having petabytes of conversational training data. Scraping is how OpenAI trained GPT. It’s how all their copycats are trying to do the same. Elon can be a monumental asshat, and he can be self-DDOS’ing, and can be accurate about scraping at the same time. It’s why every single social media platform is heading toward becoming a walled garden.
  - pschuegr 3y ago
    I'm not denying that scrapers exist, I'm just highly suspicious of this explanation given that: a) he's proven time and time again how willing he is to say shit just to get attention b) he doesn't seem to understand software very well c) if shit was imploding for reasons related to decisions he made, this is precisely the kind of blame externalization I would expect.
    - evan_ 3y ago
      Yeah, scrapers have always existed and while their traffic is undoubtedly higher than it has been in the past, it can't possibly be any significant amount of traffic when compared to the rest of the traffic hitting the site. A real scraper would be stopped by a rate limit set to, like, 100 tweets/minute. 600 tweets/day is a completely pointless, punitive limit.
      - manquer 3y ago
        Depends on how they scrape . 100 tweets / minute is hardly deterrent for a botnet using comprised devices on non data center IPs
      - berkle4455 3y ago
        > A real scraper would be stopped by a rate limit I'm guessing you've never played an offensive or defensive role in scraping because what you've described is in no way a problem for a serious scraping effort. I agree the rate limits are stupid. They fuck over users, they stop amateur scrapers, and do nothing whatsoever to impede professional scraping. If you want to stop most scraping, employ device attestation techniques and TLS fingerprinting.
        costco 3y ago
        But then you have to contend with this: https://github.com/bogdanfinn/tls-client https://github.com/bogdanfinn/tls-client... Just used this to bypass a Cloudflare check! I've never scraped Twitter but Elon said there was a large scraping operation from Oracle IPs. He could substantially raise the cost of scraping by just banning datacenter IPs. Something like p0f would probably help too. I pay for static residential proxies (basically servers running squid that somehow have IPs belonging to consumer ISPs) and with TCP fingerprinting these would be detected as Linux and expose my Windows or iPhone user-agents as inconsistent but I've never encountered a site that checks this. Although maybe sites are doing so silently but I don't notice because I don't otherwise meet the bot threshold.
        berkle4455 3y ago
        for sure, using a custom TLS library like uTLS helps -- need to inject that GREASE cipher selection. I have a suspicion that private residential proxies are out of budget for many outfits, or the IPs are too few and then simple rate limiting kicks in? Who do you use if you're willing to share? I've not been happy with the, uhh, questionable ethics of Luminati/BrightData in the past. There are definitely more and more sites doing TLS/TCP/etc fingerprinting or device attestation for mobile APIs, but it's still pretty rare. I mean Twitter is trying to limit requests by IP, so definitely amateur hour over there.
  - gmerc 3y ago
    It’s quite ignorant to assume petabytes of garbage have any value at this point. See Chinchilla
    - berkle4455 3y ago
      I agree, but there are hundreds if not thousands of AI startups trying to make their own relevant LLM, and they're going to be scraping Twitter. The Onion called it many years ago [1]: "400 billion tweets and not one useful bit of data was ever transmitted". [1] https://www.youtube.com/watch?v=cqggW08BWO0&t=138s https://www.youtube.com/watch?v=cqggW08BWO0&t=138s
      - rightbyte 3y ago
        I can't imagine worse training data than e.g. Twitter and Reddit posts. How about like, dunno, books? Edit: Ah, nvm, if you are trying to do a chat bot it is essentially what you want.
  - dyno12345 3y ago
    How do you feel about search engine indexing?
olalonde 3y ago
According to Elon, shutting down anonymous access was itself an emergency measure to deal with the DDoS[0]. Twitter did increase the quota significantly from earlier today[1]. [0] https://twitter.com/elonmusk/status/1674865731136020505 https://twitter.com/elonmusk/status/1674865731136020505 [1] https://twitter.com/elonmusk/status/1675260424109928449 https://twitter.com/elonmusk/status/1675260424109928449
- idiftl 3y ago
  The links won’t load
  - olalonde 3y ago
    [0] "Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users!" (in reference to Twitter requiring users to be logged in) [1] "Now to 10k, 1k & 0.5k" (in reference to rate limits which were originally 6K 0.6K and 0.3K) And another tweet that confirms disabling anonymous access was an emergency measure: https://twitter.com/elonmusk/status/1674942336583757825 https://twitter.com/elonmusk/status/1674942336583757825 > This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping. > Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data. > It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.

by evan_ 3y ago

pschuegr 3y ago

This. I'd bet substantial amounts of money that the evil scraper idea is the result of a) another issue + b) paranoia + c) Musk thinking he understands better than anybody else.

berkle4455 3y ago
This is a really ignorant take to dismiss scrapers. LLMs operate by having petabytes of conversational training data. Scraping is how OpenAI trained GPT. It’s how all their copycats are trying to do the same. Elon can be a monumental asshat, and he can be self-DDOS’ing, and can be accurate about scraping at the same time. It’s why every single social media platform is heading toward becoming a walled garden.
- pschuegr 3y ago
  I'm not denying that scrapers exist, I'm just highly suspicious of this explanation given that: a) he's proven time and time again how willing he is to say shit just to get attention b) he doesn't seem to understand software very well c) if shit was imploding for reasons related to decisions he made, this is precisely the kind of blame externalization I would expect.
  - evan_ 3y ago
    Yeah, scrapers have always existed and while their traffic is undoubtedly higher than it has been in the past, it can't possibly be any significant amount of traffic when compared to the rest of the traffic hitting the site. A real scraper would be stopped by a rate limit set to, like, 100 tweets/minute. 600 tweets/day is a completely pointless, punitive limit.
    - manquer 3y ago
      Depends on how they scrape . 100 tweets / minute is hardly deterrent for a botnet using comprised devices on non data center IPs
    - berkle4455 3y ago
      > A real scraper would be stopped by a rate limit I'm guessing you've never played an offensive or defensive role in scraping because what you've described is in no way a problem for a serious scraping effort. I agree the rate limits are stupid. They fuck over users, they stop amateur scrapers, and do nothing whatsoever to impede professional scraping. If you want to stop most scraping, employ device attestation techniques and TLS fingerprinting.
      - costco 3y ago
        But then you have to contend with this: https://github.com/bogdanfinn/tls-client https://github.com/bogdanfinn/tls-client... Just used this to bypass a Cloudflare check! I've never scraped Twitter but Elon said there was a large scraping operation from Oracle IPs. He could substantially raise the cost of scraping by just banning datacenter IPs. Something like p0f would probably help too. I pay for static residential proxies (basically servers running squid that somehow have IPs belonging to consumer ISPs) and with TCP fingerprinting these would be detected as Linux and expose my Windows or iPhone user-agents as inconsistent but I've never encountered a site that checks this. Although maybe sites are doing so silently but I don't notice because I don't otherwise meet the bot threshold.
        berkle4455 3y ago
        for sure, using a custom TLS library like uTLS helps -- need to inject that GREASE cipher selection. I have a suspicion that private residential proxies are out of budget for many outfits, or the IPs are too few and then simple rate limiting kicks in? Who do you use if you're willing to share? I've not been happy with the, uhh, questionable ethics of Luminati/BrightData in the past. There are definitely more and more sites doing TLS/TCP/etc fingerprinting or device attestation for mobile APIs, but it's still pretty rare. I mean Twitter is trying to limit requests by IP, so definitely amateur hour over there.
- gmerc 3y ago
  It’s quite ignorant to assume petabytes of garbage have any value at this point. See Chinchilla
  - berkle4455 3y ago
    I agree, but there are hundreds if not thousands of AI startups trying to make their own relevant LLM, and they're going to be scraping Twitter. The Onion called it many years ago [1]: "400 billion tweets and not one useful bit of data was ever transmitted". [1] https://www.youtube.com/watch?v=cqggW08BWO0&t=138s https://www.youtube.com/watch?v=cqggW08BWO0&t=138s
    - rightbyte 3y ago
      I can't imagine worse training data than e.g. Twitter and Reddit posts. How about like, dunno, books? Edit: Ah, nvm, if you are trying to do a chat bot it is essentially what you want.
- dyno12345 3y ago
  How do you feel about search engine indexing?

olalonde 3y ago

According to Elon, shutting down anonymous access was itself an emergency measure to deal with the DDoS[0]. Twitter did increase the quota significantly from earlier today[1]. [0] https://twitter.com/elonmusk/status/1674865731136020505 https://twitter.com/elonmusk/status/1674865731136020505 [1] https://twitter.com/elonmusk/status/1675260424109928449 https://twitter.com/elonmusk/status/1675260424109928449

idiftl 3y ago
The links won’t load
- olalonde 3y ago
  [0] "Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users!" (in reference to Twitter requiring users to be logged in) [1] "Now to 10k, 1k & 0.5k" (in reference to rate limits which were originally 6K 0.6K and 0.3K) And another tweet that confirms disabling anonymous access was an emergency measure: https://twitter.com/elonmusk/status/1674942336583757825 https://twitter.com/elonmusk/status/1674942336583757825 > This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping. > Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data. > It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.