11 ms·
With the rise of AI, web crawlers are suddenly controversial
- aaronrobinson 3y agoDrama. Crawlers have always been controversial.
- throwup238 3y ago> For decades, robots.txt governed the behavior of web crawlers. But as unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart. The basic social contract of the web fell apart long ago when almost everyone decided that Google was the only search engine worth serving and started aggressively blocking other crawlers.
- ec109685 3y agoAre there services that have a copy of the Internet for a fee? How did perplexity.ai crawl the internet for their AI?
- zarzavat 3y agoAdd to that the removal of public APIs and things like RSS feeds. I would much rather use an API than scrape you, I will even pay a small fee, but if you don’t provide anything, then you’re getting scraped.
- 1vuio0pswjnm7 3y ago"Add to that the removal of public APIs and things like RSS feeds." You mean like this public API https://hacker-news.firebaseio.com/v0/item/39421253.json https://hacker-news.firebaseio.com/v0/item/39421253.json Or this RSS feed https://www.theverge.com/rss/index.xml https://www.theverge.com/rss/index.xml
- WirelessGigabit 3y agoReddit recently started blocking anonymous GETs to RSS feeds. And if you use your cookies they temp-block your account after x (like x every 30 minutes) times. And I didn't even care that there was no content on the RSS feed, I just wanted the title / notification so I don't have to use ... (I can't put anything in here as there is no alternative that allows me to track everything).
- deleted 3y ago[deleted]
- 1vuio0pswjnm7 3y agoAnother one. This one has a rate limit. curl -si40A "" https://old.reddit.com/new.rss > 1.json sed 10q 1.json HTTP/1.1 200 OK content-length: 40582 content-type: application/atom+xml; charset=UTF-8 x-ua-compatible: IE=edge x-frame-options: SAMEORIGIN x-ratelimit-remaining: 93 x-ratelimit-used: 3 x-ratelimit-reset: 96 x-reddit-pod-ip: 10.104.158.170:80 x-reddit-internal-ratelimit-rls-type: ip-standard In theory, APIs to retrieve public information suck because they are designed to be rate-limited or subject to quotas. Whereeas IME public-facing websites are much less likely to set rate limits and enforce quotas. I prefer to use the public-facing website instead of "Web APIs" and convert HTML to SQLite which I prefer over non-LD JSON. If the public-facing website changes their HTML, I change the code in the filter. This is rare. It is very simple for me to do.
- deleted 3y ago[deleted]
- Plasmoid 3y agoI'm out of the loop on this. What has been happening? Do you get IP blocked? Rate limited?
- realusername 3y agoYou do and since the rise of Cloudflare, you need their approval to create your search engine
- oceanplexian 3y agoUnfortunately for them, AI is a double edged sword. AI can trivially scrape content from a screencap. And there are thousands of residential proxies that can’t be defeated even by companies like CF.
- spencerflem 3y agoIf you have residential proxies, you don't need AI to scrape via video, you have the HTML already.
- oceanplexian 3y agoCompanies spend thousands of engineering hours building elaborate mechanisms to obfuscate JS, protect APIs, apps, trigger captchas, and hide content, all of it completely powerless against the current gen of AI tools. If a human can read it, a bot can.
- Mr_P 3y agoThose same companies often invest in accessibility for vision-impaired users. I'm not sure you need a screen capture to scrape content when the site is designed to be navigable with a screen reader.
- timthelion 3y ago
- elpocko 3y agorobots.txt is relevant and effective, as is my DNT header.
- naiv 3y agoProxy companies are a big winner now
- deleted 3y ago[deleted]
- linkjuice4all 3y ago"With the rise of AI, photos of the exterior of your business are suddenly controversial" Many revenue-based websites tried to have it both ways with web crawlers wherein they wanted to block automated access or repeat viewers while letting first time viewers get a free taste. Others have noted that basically Google gets a free pass for all the traffic it brings in but everyone else has to respect robots declarations. It seems like a no brainer - if your web server is configured to reply to GET requests with a 200 status and some content then they get to do pretty much whatever they want with it. Don't want to give access to everyone? Stop sending your content for free and get them to agree to some contract and authorize/license their access to your stuff.
- spencerflem 3y agoI don't think this is that cut and dry. Its perfectly reasonable to be fine with your content being linked to in an index like google search does, but not fine with the content being read by humans for free or used to train AI. I'm sure the sites would rather that google paid them in licencing too, but without changes to our laws that's not going to happen
- matthewrobertso 3y agoI want to research a topic, so I do a google search about it. If 100% of the search results are readable to google, so that they are indexed, but unreadable to me, due to paywalls, the google search is useless. I'm not sure what the solution is, but paywalled articles in search results are bad. If they want to be indexed they should have to offer that same indexed content to anyone browsing the index.
- carlosjobim 3y agoIf you're doing serious research you pay for the paywall. It's not unreadable to you, just like a coke isn't undrinkable to you because you have to pay for it.
- 3y ago
- amelius 3y agoWhen did robots.txt get a legal status? Or did it ever?
- calibas 3y ago> For decades, robots.txt governed the behavior of web crawlers. It never governed anything, web crawlers were never under any obligation to follow robots.txt. This article seems like they took an existing controversy, rebranded it as something new, then blamed in on AI.
- fallingknife 3y ago> they took an existing controversy, rebranded it as something new, then blamed in on <new thing> Basically describes most of what passes as journalism these days.
- franze 3y ago> web crawlers were never under any obligation to follow robots.txt. other than the fact that you could get successful sued if you do not follow them i.e.: eBay v. Bidder's Edge in 2000
- janalsncm 3y agoNo, I think this is something new. It’s an existential problem for newspapers. Most newspapers have never thrived online but they got a stream of clicks and a few 100x stories every once in a while. But with language models and RAG, we may create a world where no one has to visit their site at all. This really hit home for me in Hard Fork’s interview with the CEO of Perplexity AI [1]. The crux of the problem is, if Perplexity does what it strives to do, provide answers rather than links, no one will be clicking on links and generating revenue for newspapers anymore. I encourage you to watch him struggle with this question because this is really an irreconcilable problem. [1] https://m.youtube.com/watch?v=AJuv_UwxELA https://m.youtube.com/watch?v=AJuv_UwxELA
- andybak 3y ago> But as unscrupulous AI companies seek out more and more data I'm not sure I'm ready to concede the fundemental value judgement being made here. At least I refuse to accept it as a given rather then the core issue to be decided.
- micromacrofoot 3y agomany crawlers have always ignored robots.txt, if you’re monitoring any moderately visited site you’re bound to see random spikes of bots hammering your server no matter what text file or headers you set
- datavirtue 3y agoThis. I stopped reading after the first few sentences. Whoever wrote that verge article is clueless. Data accessible. Data free. Period.
- phone8675309 3y agoMaybe it was the same person that put together the Verge PC build guide (not the person presenting it, the person who wrote the script for it)
- lewhoo 3y agoI don't get it. The crux of it all seems to be that Google isn't competing with owners of data it crawls using the very same data. The crawl part isn't as much of a controversy as usage, isn't it ? The mentioned eBay v. Bidder's Edge (2000) seems to be a dispute over usage.
- pixl97 3y agoI mean Google is making AI models and Google is summarizing sites so they may well be competing. The difference is Google has a near monopoly on search and ads.
- mediumsmart 3y agoThe web comes in 2 versions. One of them has a basic social contract. maybe