5 ms·
Update from Elon: https://twitter.com/elonmusk/status/1675214274627530754 https://twitter.com/elonmusk/status/1675214274627530754 > Rate limits increasing soon
by minimaxir 3y ago
Update from Elon: https://twitter.com/elonmusk/status/1675214274627530754 https://twitter.com/elonmusk/status/1675214274627530754
> Rate limits increasing soon to 8000 for verified, 800 for unverified & 400 for new unverified
lolwut. Quoting a post from Mastodon: https://mstdn.social/@maxkennerly/110640373859329500 https://mstdn.social/@maxkennerly/110640373859329500
> LOL this is not how you deal with "data scraping," this is how you deal with a catastrophic loss of system capacity.
> You limit data scraping by blocking things a human user couldn't do, like access a thousand posts a minute. This is aimed directly at reducing normal activity across the whole system.
- rideontime 3y agoSo, like, two more feed refreshes per day? Watching the requests in dev tools, it's definitely based on fetches and not impressions.
- cududa 3y agoA cool so I get about 3 more minutes of looking at tweets My guess is he got rate limited himself after a few hours as a verified user so he’s already upping it
- jsheard 3y agoTurning a big dial that says "rate limit" on it and constantly looking back at the audience for approval like a contestant on The Price Is Right
- ssnistfajen 3y agoA garage startup doing this with a thousand users would be cool and fun. An established large platform with hundreds of millions of users doingg this is just undermining its own reputation.
- russdill 3y agoI suspect this is less about scraping and more about artificial manipulation of view counts to both push certain scams/tweets to the top. I don't think they are providing advertising kickbacks yet, but that would only amplify this.
- TillE 3y agoI'm 50/50 on whether they're actually dealing with catastrophic technical issues, or if this is just Musk being mad about something and insisting on an incredibly silly solution. Both are entirely plausible!
- misnome 3y agoI’ve seen speculation correlating “Going down at 9am on the first day of Q3” with the fact that they have apparently been trying to migrate off Google cloud by June 30th because of Melon’s blanket policy of “Not paying bills”. Needless to say, if true, this would be extremely funny.
- joezydeco 3y agoIf Google decides to graveyard GCloud, you can say you were here when it started.
- marginalia_nu 3y ago> You limit data scraping by blocking things a human user couldn't do, like access a thousand posts a minute. This is aimed directly at reducing normal activity across the whole system. This isn't accurate at all. Standard operating procedure for large-scale scraping is to use a botnet, it's a sybil attack where you can have tens or even hundreds of thousands of IPs ticking away at 10 requests per minute or some such. Done correctly, it's nearly impossible to detect which nodes are complicit from their access patterns or user agents. That said, rate limits work relatively poorly against this. A proof of work approach like HashCash might work though.
- CamperBob2 3y agoWouldn't limiting the number of IP addresses that can log in as a single user avoid the botnet problem? That said, someone mentioned that the login page alone involves 11 MB of data transfer, so maybe that is the problem they are confronted with.
- tlogan 3y agoThe solution is not simple but there is one (but it is complex). The question is are you going to spend time and money to implement the right solution or just pick up a cheaper one? Could it be that they just stop paying bills for Akrose and they shut down their service?
- reaperducer 3y agoUpdate from Elon: https://twitter.com/elonmusk/status/1675214274627530754 https://twitter.com/elonmusk/status/1675214274627530754 > Rate limits increasing soon to 8000 for verified, 800 for unverified & 400 for new unverified Well, at least he's engaged over a holiday weekend. If it was my company, this would be a fifteen meetings, a six-month series of sprints, and we still would only decide on a new color for the icon.
- soneca 3y agoI guess a couple of well-run meetings would have prevented this disaster in the first place.
- jakelazaroff 3y agoSix months of bureaucratic paralysis would also be better than actually rolling this out.
- ssnistfajen 3y agoSometimes things move slowly for good reason. Wasted meeting minutes is a reasonable tradeoff over rendering the main product unusable like this.
- evolve2k 3y agoSome are suggesting that the rise in scraping is a direct result from Twitter killing it’s free API, and that there are a range of organisations whos work relies on this information who have now had to resort to other means of attempting to get the same info.
- gmerc 3y agoThere is no actual evidence of any of that though.
- l33tc0de 3y agoAlphabet people talking about Twitter on Mastodon