18 ms·
From Elon's twitter (https://twitter.com/elonmusk/status/1674942336583757825?s=20 https://twitter.com/elonmusk/status/1674942336583757825?s=20) "This will be u
by kakakiki 3y ago
From Elon's twitter (https://twitter.com/elonmusk/status/1674942336583757825?s=20 https://twitter.com/elonmusk/status/1674942336583757825?s=20)
"This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.
Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.
It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation."
- 7e 3y agoThis is a direct consequence of Elon gutting Twitter's infrastructure.
- throwaway290 3y agoI wonder if it's people seeking to move away from Twitter and working around crippled APIs, or if it's ClosedAI in which Musk himself invested before...
- gnicholas 3y ago> Something went wrong. Try reloading.
- meinheld111 3y agoI am trying…
- LeafItAlone 3y agoI get only get > Something went wrong. Try reloading. on that link. Which, frankly, is hilarious.
- jcims 3y agoI used to work for a very large financial institution. Scraping from finance apps was a material source of load even with substantial countermeasures in place. I can’t imagine what it does to sites like Twitter and Reddit (and HN).
- aledalgrande 3y agoHN has an open API look at the footer
- oefrha 3y agoHN volume is absolutely tiny (the ids are sequential so you can easily check how many items there are in a given day) and there’s an API. It’s no comparison.
- zerd 3y agoWell they killed the API, what did they think would happen? It's easier to control access and rate limit with a proper API.
- oefrha 3y agoYou think companies massively scraping right now would respect API rate limit?
- guax 3y agoAPI rate limits are more easily enforceable. If they keep scraping there are methods to detect and thwart behaviour. I don't think twitter has the appropriate talent and work environment to allow proper solutions to be implemented. It's all knee jerk reaction to whatever Elon decides.
- oefrha 3y agoIt's more easily enforced, except when you don't give them enough they just go back to scraping. Or create a million fake developer accounts and pool the free quota if that's possible. These are not hypotheticals, loads of companies have done both against all kinds of APIs over the years, Twitter included.
- UberFly 3y ago"control access and rate limit" Isn't that basically what they did with the API changes?
- spaceman_2020 3y agoBut they were too stingy with the tiers and too greedy with their prices. Even for minor use cases where you need to make, say, 100 API calls a day, you’ll need to pay $100/month. Which just leads people to scrape.
- hcks 3y ago
- sanderjd 3y agoYeah but he's full of crap; why would we believe anything he says about anything?
- kibwen 3y agoThe above comment doesn't deserve to be downvoted. If Musk wants people to believe his statements, he should have refrained from being a serial liar. Now his reputation is in the trash, and he has only himself to blame.
- sanderjd 3y agoIn fairness, the first part of my comment is a personal attack that could have been left out. But the second part is what you're agreeing with, that he doesn't have credibility because of making stuff up a bunch recently. And I think that's right.
- HWR_14 3y agoIsn't Musk a major investor in OpenAI? If he said "my data feeds chatgpt, not its competitors", that would make sense, right?
- zeven7 3y agoHe's been somewhat critical of OpenAI. Specifically the part about it pivoting to a for-profit business. > I’m still confused as to how a non-profit to which I donated ~$100M somehow became a $30B market cap for-profit. If this is legal, why doesn’t everyone do it? https://twitter.com/elonmusk/status/1636047019893481474 https://twitter.com/elonmusk/status/1636047019893481474 > I donated the first $100M to OpenAI when it was a non-profit, but have no ownership or control https://twitter.com/elonmusk/status/1639142924985278466 https://twitter.com/elonmusk/status/1639142924985278466
- wahnfrieden 3y agoYou’re sharing lies. He donated $10 mil and talks it up 10x.
- zeven7 3y agoThere's a lot of lies in tweets. The point was: he's been disparaging OpenAI. It doesn't matter how much he donated or if he donated.
- irthomasthomas 3y agoIsn't the more important point that his consistent lying makes his disparagings unreliable as a signal?
- valval 3y agoCan you point out any of his lies?
- 3y ago
- linuxandrew 3y ago> immediate action was necessary due to EXTREME levels of data scraping It sounds like Elon doesn't "get" the open web
- ipsum2 3y agoThis makes no logical sense. Why would the scraping not restart after its unlocked? More realistically, he got a lot of backlash from users and website owners where embedded tweets suddenly stopped showing up.
- benjaminwootton 3y agoElon has a good point there. Much of the current AI hotness is predicated on stealing peoples content and exploiting the infrastructure that other people have built. I don’t think it’s acceptable. The licenses, compensation models, law, technical solutions, attribution, security and privacy all need time to catch up. Regulation has a role to play as its a bit of a free for all right now. The irony of Elon mentioning “outrageous valuations” though!
- yuuuuuuuu 3y agoThis could easily be solved by making the unauthenticated access hard for machines to consume, like introducing delays or some kind of captcha or even just proof of work (reverse some hash). While the authenticated get all the snappiness they want. I'm strictly anti account, so he just lost me as audience. The next walled garden after Facebook and Instagram that won't ever see me again.
- dragon111 3y agoSounds like outright banning is a stopgap measure, maybe they will implement one of these solutions
- maxlin 3y agoIt already was semi-hard to machine-read, that is the reason I use Nitter for doing my small-scale continuous scraping of twitter which is now temporarily broken. Nitter is tons easier to parse as it's not reliant on JS, etc, and simpler to create screenshots of with headless chrome. However if you mean implementing some even worse obfuscation (kind of like FB putting parts of words in different divs etc) that is not really compatible with the situation that this needed to be done as more of a temporary emergency measure. And PoW doesn't sound reasonable because it sets mobile devices against the scraper's servers. If all of this was just so easy, scraping would be dead. Good that it isn't.
- yuuuuuuuu 3y ago> And PoW doesn't sound reasonable because it sets mobile devices against the scraper's servers. Scraper servers and mobile devices have different access patterns though. I I'm reading tweets then I'm fine waiting 1 second for a tweet to load. Page load times for this kind of bloated stuff are super slow anyway, meanwhile my mobile could spend a second or two on some PoW. But if you want to large-scale scrape, you suddenly have to pay for 1bn CPU seconds. And this PoW could even keep continuously increasing per IP. 0.1% with every tweet. Not noticeablr for the casual surfer sitting on the toilet, neck-breaking for scrapers. > If all of this was just so easy, scraping would be dead. Good that it isn't. Small-scale scraping could still be provided through API access or just a login. The reason they are not doing the "easy" thing is that they don't see a need (yet, perhaps). Just get an account, they'd say, and they are right. It works for Instagram too, except for some weirdos who nobody really cares about.
- syspec 3y agoIs twitter worth scraping though for AI? I mean, Reddit I get, but twitter content has suuuuch a low signal to noise ratio.
- Havoc 3y agoSeems to entertain many so has value in that sense I guess. Plus perhaps post vs replies make some sort of challenge & response pair that can be leveraged?
- lljk_kennedy 3y agoMetadata is probably more interesting than the raw content.
- notfed 3y ago> Reddit I get Reddit is a cesspool of misinformation and phallic references. I guess there may be certain subreddits/tweeters that someone might want to train on, but I dont understand why.
- berkle4455 3y agoAmusing that I called precisely the cause well before Elon tweeted (https://news.ycombinator.com/item?id=36542697 https://news.ycombinator.com/item?id=36542697) and was downvoted for it.
- udev4096 3y agoHN has an audience which is full of hypocrisy
- throw2022110401 3y agoThat's Musk-speak. Translation: "we broke shit again and need to shed load to keep the site up." I wish it would stays that way though so that I wouldn't land on the site by accident.
- choppaface 3y agoLess likely the reason is technical / cost of service (which is very cheap) and more likely he is trying to exercise leverage in pursuit of monetizing engagement (which had already happened [1] ). It’s not that he broke the website’s tech but rather he broke the website’s business. [1] https://www.datacenterdynamics.com/en/news/twitter-pays-google-cloud-bill-fixes-relationship-report https://www.datacenterdynamics.com/en/news/twitter-pays-goog...
- Havoc 3y agoDisables API, gets scraped, needs more servers. Congrats you played yourself.
- 3cats-in-a-coat 3y agoDisables API, gets scraped, needs more servers, disables access without logins, gets millions of fake accounts, has to deal with the fake accounts, in the process deletes tons of real accounts, users pissed, scraping continues, server bills keep rising...
- blipvert 3y agoThere was an old woman who swallowed a fly … Twitter is maybe at the dog stage. Perhaps it’ll die.
- manx 3y agoThey could just provide or even sell well curated datasets instead.
- ricardobayes 3y agoHe's not wrong on this occasion, there are multiple companies out there, some even with a multi-billion dollar valuation that "farm" tweets for many reasons.
- peter_retief 3y agoI was about to say that, I read the same post, I also agree somewhat.
- smcl 3y agoSorry I don’t buy it. Hundreds of millions of people use Twitter, and we are to understand that there are an enough people scraping to the extent that they had to suddenly take drastic action by shuttering unauthenticated access? Any dev would have told him that those supposedly scraping could simply setup Selenium or some other headless browser to login before scraping. This smells of another failed Musk experiment at twiddling with the knobs to increase engagement, to me.
- jaggirs 3y agoYeah no you cant just 'use selenium'. To keep the same scraping volume you might need thousands of accounts and 10x the compute.
- smcl 3y agoIt’s not a little “use selenium” switch you can click, but it absolutely is an option (and there are others) if the barrier is simply to have an authenticated account and be logged in. If these data scraping operations are as sophisticated and determined as he claims this measure is insufficient and actually it really hurts Twitter far more than it helps. Case in point: we stopped sharing Twitter links because when you click them in most iOS apps it opens up an unauthenticated web view and presents you with a login screen. So we just collectively decided “ah ok no sharing Twitter” and moved on. I’m sure there are companies scraping Twitter. I just don’t buy that it’s as big of an issue as he claims it is, and that preventing people from viewing tweets without logging in is a way to mitigate against that (I’d first look at banning problematic IP addresses first, personally). To me it’s either: 1) a very poor and very temporary mitigation against scraping, that could be bypassed with a bit of effort 2) an experiment in optimising metrics - Musk sees lots of unauthenticated users consuming Twitter, tries to steer them into signing up 3) it’s all just a big mistake Option #2 makes the most sense to me, but frankly none of them are good
- sebzim4500 3y agoThey could but signing in, even in selenium, means agreeing to twitter's TOS. See the LinkedIn scraping case.
- SomeBoolshit 3y agoHis tweets are as trustworthy as Trump's.
- kergonath 3y agoAnd as full of random capitals.
- nameless_prole 3y ago[dead]
- devjab 3y agoI think it will be sort of interesting to see what AI scraping does to the open internet. I think that we are already putting too much content into Social Media platforms (HN included). Stuff that we sort of ought to self host because then we would actually own it. But will you even want to run your own sites publicly if they are getting scraped? I guess it’s it really a new issue as such, but I imagine it’ll only get worse as the LLM craze continues to rise.
- gloosx 3y agoBut you can buy verified twitter accounts starting from $0,035 per pc. I really don't understand how can it pose any serious roadblock for scrapers.
- gooseus 3y agoYou haven't been able to look at anything but the /explore endpoint for weeks without an account, and the "content" on there has been total garbage. I was relieved when they started asking for an account this week because now I'll finally be able to break my habit of navigating to Twitter to "see what's happening" only to find a bunch of sports memes, pop music drama, or right-wing trolls pretending like Hunter Biden is the most nefarious person on the planet. Imo, Elon is lying and he locked everything down for PR so he would make a headline and frame it like his site's content is _so valuable_ that he just had to take drastic measures to stop AI from training on it.
- papito 3y agoLeave it to Musk to create a hair-on-fire emergency out of a regular cup of coffee. "EXTREME"! Please... But hey, I am eternally grateful. As a super-compulsive Twitter user, I read accounts and my own lists after I log out. This fixes it. Have your "emergency", Elon.
- marginalia_nu 3y agoMissed opportunity to poison the scrapers' well by showing them AI-generated tweets. Bad data is devastating in a way a HTTP 401 is not.
- belter 3y agoPlanet Earth Inhabitants in 2023: 8 Billion -> Social Media Users -> 4.8 Billion -> Twitter Users -> 368 million active users who engage at least once a month. If those AI Models are being trained on a reduced set of 3% of human beings, they will lose a lot.
- delijati 3y agothat's just a justification to force user to create an account ... he killed the api that already had an rate limit ... its so obvious that it hurts
- danShumway 3y agoI have a bridge to sell anyone who believes this is true. AI companies have been a boon to businesses that want to lock down user data and now have an excuse. It may be true to the extent that Musk is legitimately angry that Twitter isn't getting a piece of that AI VC money (I'm sure he is). But: A) Twitter would probably move in this direction even if AI companies didn't exist, this is an excuse. Nothing about Musk's Twitter has indicated that he cares about Open data access or anonymous access to the site, and this follows a general trend of closing down the platform to non-monetizable users. Musk has abundantly shown in the past that he would prefer everyone browsing Twitter be logged into an account. B) "it's temporary" -- how? You don't have a way to stop this other than forcing login. That situation is not going to change next week. To call this "temporary emergency measures" is so funny; there is no engineering solution for this and you're not going to be able to successfully sue companies for scraping Twitter. Put a captcha in front of it? Sure, let me know how that goes. You going to wait and see if the AI market collapses in the next month? If this does turn out to be temporary, it'll only be because of migrations off of Twitter and because of user criticism, because Musk is impulsive and bends easily under pressure. But nothing about the situation Musk is complaining about is going to change next week.
- m00x 3y agoFWIW: I've been scraping the shit out of social media for my AI training. I also do Amazon, AliExpress, etc. Libs like puppeteer is so good these days that it's impossible to tell real users from fake traffic. Most of the blocks are just IP blocks.
- qingcharles 3y agoRight. And the IP blocks only add a small cost to the scraping because it forces people to use residential IPs which can't sanely be blocked.
- agentgumshoe 3y agoNot to mention Elon is great mates with the VC douchebags that are busy hyping and profiting off the AI hype train.
- benj111 3y agoI just get a 'something went wrong' message from this link. Is that because I'm not logged in, if so why don't they say that?
- geekraver 3y agoA decade ago I worked on building AI systems that made (legitimate, paid) use of the Twitter “firehose”. At that time more than 99% of the data was garbage. It’s worse now. The value then was largely in two areas: historical trends (something like Google trends), and breaking news; and only the latter was really that interesting. I doubt it’s a high value data source scraped in bulk; it could have value in a much more targeted approach. Seems unlikely to require the addition of “large numbers of servers … on an emergency basis”.
- AzzieElbab 3y agoThe whole thing is down now, returning “rate limit exceeded” in the UI. A very hamfisted affair imho
- michelb 3y agoCurious how Elon is going to train his OpenAI competitor. Scraping?
- cyanydeez 3y agoOh how the disruptors hate disruptors
- nurettin 3y agoThen we just add 2fa to the scraper and continue scraping the shit out of Twitter. Checkmate.
- cramjabsyn 3y agoDoesn't really make sense… What prevents a logged in bot from continuing to scrape vast amounts of data?
- zimpenfish 3y agoUpdating from the future (20230703T113430Z) and this has not been unlocked.