8 ms·
Elon has a good point there. Much of the current AI hotness is predicated on stealing peoples content and exploiting the infrastructure that other people have
by benjaminwootton 3y ago
Elon has a good point there. Much of the current AI hotness is predicated on stealing peoples content and exploiting the infrastructure that other people have built. I don’t think it’s acceptable.
The licenses, compensation models, law, technical solutions, attribution, security and privacy all need time to catch up. Regulation has a role to play as its a bit of a free for all right now.
The irony of Elon mentioning “outrageous valuations” though!
- yuuuuuuuu 3y agoThis could easily be solved by making the unauthenticated access hard for machines to consume, like introducing delays or some kind of captcha or even just proof of work (reverse some hash). While the authenticated get all the snappiness they want. I'm strictly anti account, so he just lost me as audience. The next walled garden after Facebook and Instagram that won't ever see me again.
- dragon111 3y agoSounds like outright banning is a stopgap measure, maybe they will implement one of these solutions
- maxlin 3y agoIt already was semi-hard to machine-read, that is the reason I use Nitter for doing my small-scale continuous scraping of twitter which is now temporarily broken. Nitter is tons easier to parse as it's not reliant on JS, etc, and simpler to create screenshots of with headless chrome. However if you mean implementing some even worse obfuscation (kind of like FB putting parts of words in different divs etc) that is not really compatible with the situation that this needed to be done as more of a temporary emergency measure. And PoW doesn't sound reasonable because it sets mobile devices against the scraper's servers. If all of this was just so easy, scraping would be dead. Good that it isn't.
- yuuuuuuuu 3y ago> And PoW doesn't sound reasonable because it sets mobile devices against the scraper's servers. Scraper servers and mobile devices have different access patterns though. I I'm reading tweets then I'm fine waiting 1 second for a tweet to load. Page load times for this kind of bloated stuff are super slow anyway, meanwhile my mobile could spend a second or two on some PoW. But if you want to large-scale scrape, you suddenly have to pay for 1bn CPU seconds. And this PoW could even keep continuously increasing per IP. 0.1% with every tweet. Not noticeablr for the casual surfer sitting on the toilet, neck-breaking for scrapers. > If all of this was just so easy, scraping would be dead. Good that it isn't. Small-scale scraping could still be provided through API access or just a login. The reason they are not doing the "easy" thing is that they don't see a need (yet, perhaps). Just get an account, they'd say, and they are right. It works for Instagram too, except for some weirdos who nobody really cares about.
- maxlin 3y agoOf course the scraper would have to pay too. But it makes for a race between how much they are willing to pay, versus how much worse the experience gets for real users. And for successful mobile apps, reducing average load even during active use is important (example: idle games that don't want to make your phone a drying iron, companies invest in custom engines and make all kinds of compromises to avoid this). And burst-allowing rate limiting is something I'm quite sure was already in place, especially with prejudice towards datacenter/VPN IP's. But similarly to how it is with search engine scraping, professional scrapers already have costly workarounds for these. >The reason they are not doing the "easy" thing is that they don't see a need (yet, perhaps). This argument just doesn't make any sense. Twitter notes that this is hurting them. Previews in chat apps, just clicking links in non-loggedin contexts is are broken. I feel like you just predict that this will turn out to be more accepted in the near future and become more a more permanent decision, which you don't like.
- rvba 3y agoIm not fine waiting 1 second. Most baffling is mobile reddit, where it takes like 6 seconds to load. Do they want us to use their crappy app, or they just dont care?
- nirui 3y agoUnless the captcha is annoying enough to a significant degrees, I doubt that it would work. With all the money in the bucket, scrapers can just hire a captcha farm to get pass the captcha with help from a real human. Also a side note: distributed Web crawler is not unheard of these days, as well as residential IP proxies. Meaning the effectiveness of Proof of Work model maybe also limited.
- yuuuuuuuu 3y agoHow do residential proxies help? Scraping would effectively be bitcoin mining, which costs resources without shortcut.
- nirui 3y agoMany online services (including Twitter) do employ some kind of IP address scoring system as part of their anti-scraping effort. These systems tend to treat residential proxies as normal users, and puts less restrictions on them. On the other hand, if the IP address belongs to some (untrusted) IDCs, then the system will enable more annoying restrictions (say rate limits etc) against it, making scraping less efficient.
- Quarrelsome 3y agoHTTP Status Code 429 exists for this very purpose. While I sympathise with the idea that services need to protect their content from scraping to power AIs, I can't help but feel its a convenient excuse for these companies to re-implement archaic philosophies about online services. i.e. Killing off 3rd party apps and walling their garden higher, both feel very boomer in their retreat from the openness of the internet that seemed to be en vogue prior to smartphones. Perhaps this is just the transition from engineers building services to business, legal and finance trying to force the profit. Correct me if I'm wrong, but surely throttling scrapers (at least ones that are not nefarious in their habits) is a problem that can be mitigated server-side, so I find it somewhat galling that its the excuse.
- yuuuuuuuu 3y ago> is a problem that can be mitigated server-side No matter what you do, this will cost server infra. That's Musk's argument for disabling access altogether. Therefore it would make sense to have a solution which burdens the client disproportionately in relation to the server. A burden so low for the casual user that it's negligible but in aggregate, at scale, would break things. Which is what he wants.
- Quarrelsome 3y ago> Which is what he wants. Looks to me like both reddit and twitter are using the wedge to rather increase the height of the wall of their gardens and kill 3rd party development as opposed to genuinely trying to license bulk-users appropriately. You're gonna need to license api keys so you're already identifying consumers and there's your infra which you need anyway. At which point you can throttle anyone obviously abusing whatever free/open-source tier offering you give out as standard.
- dredmorbius 3y agoThe other option would be to front caches through ISPs and the like. This works far better when the items requested are small in number but large in volume (that is: a large number of requests against a small set of origin resources). When dealing with widespread and deep scraping, other strategies might be necessary, but these aren't impossible to envision. Specifically permitted scraping interfaces or APIs for large-volume data access would be another option. Of course, there's the associated issue that data aggregation itself conveys insights and power, and there might be concerns amongst those who think they're providing incidental and low-volume access to records discovering that there's a wholesale trade occurring in the background (whether that's remunerated or free of charge).
- deleted 3y ago[deleted]
- realusername 3y agoIt's not like Twitter is compensating tweet authors either. For using art the debate is still opened in my opinion even if I'm personally not in favor of it but I don't see how those platforms built on user made content (even more of a clear cut than AI) can have a say on this
- mgiannopoulos 3y agoThey have announced a plan to compensate creators based on ads shown and also have implemented a subscribers feature (people paying users for special access to some tweets)
- realusername 3y agoI had no idea this existed, thanks for the insights.
- benjaminwootton 3y agoPeople are happy to put their content on social networks. Maybe they get some value in return such as sales, exposure, signalling or simple enjoyment. Many people who aren’t that privacy conscious would however object to lots of companies, big and small, sucking their content into their databases for their own uses, then republishing after it’s passed through a few AI models.
- mnky9800n 3y agoYes. AI is becoming the content launderer. I mean what's the difference? You could ask an AI to make not star wars. And what's the difference between that and all the not star wars movies made in the 80s? It's that it was automated this time around? I think this points out that AIs clearly do not work like human brains. Human brains do not need all of the content of humanity to produce a replica of art station mediocre.
- fauigerzigerk 3y ago>People are happy to put their content on social networks. Do they have a choice? A handful of corporations have captured all the network effects. If you need to reach an audience to do your job or find your "friends", what other choice do you have but to give your data to them?
- croes 3y agoBut is Twitter a good source for AIs?
- bob-09 3y agoUsed responsibly, of course it is. A developer is able to ingest current language used in exchanges about current topics, as well as cite prominent sources that are still using the platform.
- larodi 3y agoElon is making a point and a reminder for everyone that what you share on social nets like Twitter is basically not owned by you, but the service. Actually I’m surprised this took so long to do, and in the light of doing so shows that perhaps Twitter was sold for its existing content rather than existing or active user base.
- epidemian 3y ago> [...] perhaps Twitter was sold for its existing content rather than existing or active user base. Not a very significant distinction: if active users stopped posting, scrappers wouldn't have much of a reason to keep scrapping.
- Retric 3y agoMost AI startups don’t particularly care if content is 3 minutes old or 3 years.
- theteapot 3y agoYet.
- valval 3y agoThey absolutely do. What are you even saying?
- Retric 3y agoAI startups training data covers content going back years. DALLE for example was trained on hundreds of year old paintings alongside more modern works. Age may be included as part of the training but they generally want to suck up as much data as possible.
- astrange 3y agoIf you're adding new/marginal data you should want it to be as current as possible so it'll have things like trending slang terms.
- xorcist 3y agoWhy would an AI company start scraping twitter html, instead of using an already existing archive? Something similar to archive.org could earn money from that. If all you want is the content, there's no reason to suck it through a straw. I'd expect those that require real time data, such as stock market bots or sentiment data providers, to scrape twitter (if they don't provide the data by other means, for example the "firehose", which is another great way to earn money). None of this makes much sense. Also, it's much more complicated than it seems. The web works because the data is public. You cannot think of it as "my data". (Especially not twitter, since it is really their users'!) Twitter is not higher quality data than any other web page. If we accept that thinking, every home page would require a login to see that specific company's phone number of opening hours. Those pieces of data are also valuable, in the right circumstances! And then the web would either not exist or the account system required would be so wide spread that accounts would carry no value and the system would become useless.
- kungito 3y agoIf there was more value in requiring login than there is in having this public and easily accessible, it would be behind a login form. The current internet has 99% of the time nothing to do with values someone imagined in the 80s
- fieldcny 3y agoWhat do you think the webs value is today? What do you think the webs value was imagined to be in the “80s”?
- xorcist 3y agoWhat you describe is the Facebook business model. Which seems to be a valid model, but twitter was not built around it and such a pivot would break all business moats around the company. There was no web in the 80s so not sure what values you refer to, or how they are relevant to today's businesses.
- Zak 3y agoValue to whom? Twitter is more valuable to its users, to journalists who embed tweets in stories, and to web users at large who follow links and search results if it does not require a login to view posts. Of course, none of those people own Twitter, and it may well be more valuable to its owners if it does require a login.
- kergonath 3y ago> Elon has a good point there. Much of the current AI hotness is predicated on stealing peoples content and exploiting the infrastructure that other people have built. I don’t think it’s acceptable. But then so is Twitter. They don’t produce any content whatsoever. The data they are having a fit about is not theirs, it’s been volunteered by the users. It’s the same line Reddit is pushing, and it’s bullshit. AI companies scraping the web is no more unethical than Google doing it.
- hashtag-til 3y agoWell, one thing is people going there and putting their data on a platform. That’s their choice. Taking/scrapping/stealing that data out of said platform for the benefit of your over-hyped “disruptive” startup - and implying that others should give you all for free - is the issue.
- kergonath 3y agoThat’s not the point. Twitter has a non-exclusive license to distribute the content; it’s not the owner of the data regardless of the high horse Musk feels like riding today.
- PartiallyTyped 3y agoWe know that quality data is king, with all due respect to people tweeting, that data is most likely garbage.
- matwood 3y ago> stealing peoples content The Twitters and Reddits need to be careful here when complaining because without users generating free content, they also have no business.
- WeylandYutani 3y agoScraping has been a thing since the web started. Happens to every public site. I recall that email at one point was 90% spam.
- johnnyanmac 3y agoI can't really agree. We've already had rulings about data scraping and I don't see the difference here. Just that a lot of people do it now? Also, Twitter is a public platform. Twitter didn't generate comments, and people posting on a public account are indirectly subject to public viewing. Not much different from being indirectly recorded in a public park
- wslh 3y ago> exploiting the infrastructure that other people have built Including you and me. WE built part of the infrastructure.
- quest88 3y agoHow do you define stealing? Is the AI data obtained from accessing private data? Data that users did not make publicly available but kept to themselves on their own devices?
- benj111 3y agoIf I visit Twitter to work out how to sort some JavaScript issue, and that makes my company $X, am I stealing content, or am I just using the platform? There's one major player making money off of other peoples content here, and that's Twitter. Why are they ok doing that, but not anyone else?
- harshreality 3y agoPlease don't throw around the word "stealing" so loosely. Scraping data from a public website is not "stealing". It might be a violation of the terms of service, but then you have the whole issue of click-through (formerly shrink-wrap) licenses and contracts of adhesion. If someone isn't vetting you and potentially signing you to a more meaningful contract before giving you access, for free, to data, then using that data for any purpose whatsoever (except republishing it or derived works, which might, depending on the nature of the data or the derived works, be a violation of someone's copyright) is so far from "stealing" that using that word is wrong, and I suspect intentionally inflammatory. That's why Elon limited access, rather than going to the police to file charges for theft, or suing over copyright violation or breach of contract. Not to say he absolutely couldn't do the latter, but it's hardly a clear win.
- trompetenaccoun 3y agoThere are much bigger issues. 1. Users or one might say content creators don't own their data. Not just do those platform owners make a lot of money with the content (which they have a license to, as per site ToS) but then you have third parties scraping it now for commercial products. Using the data to train models that are then sold back to some of the same social media users who produced the content for free in the first place wasn't a thing until very recently, it used to be a select few doing machine learning research in the past. The laws are lacking behind the tech development and regular internet users are being exploited because of it. 2. It absolutely is stealing in some cases, and even worse. For example when they scrape it for content which they then use to train their bots to impersonate humans. Or on Twitter, there's a very common type of bot that steals content from young attractive female social media users in China, auto-translated to English, to pose as them. If you're in finance and crypto circles they're swarming with these accounts (guess the scammers know their targets). 3. In general this is only going to get worse from here on. LLM are getting better and better. On sites like Twitter you already have no idea if you're interacting with a human or not. But these "AI" can not actually think for themselves, they can only emulate, they can copy other humans. At least so far. So for the sake of making progress and ensuring we can still have intelligent discussions and find novel ideas online, it's imperative to have a way to keep the machines out. Social media must become sybil resistant or it dies in a vicious circle of self-referencing bots ever parroting the same old talking points, or variations thereof. We urgently need human ID!
- isitmadeofglass 3y ago[dead]