18 ms·

Google to explore alternatives to robots.txt

116 points by skilled 3y ago

dbetteridge 3y ago
I notice they don't actually give a good reason that robots.txt isn't suitable. Change for the sake of it?
- strombofulous 3y ago
  AI!
  - asudosandwich 3y ago
    Indeed, AI is called out directly. Google’s laying groundwork for their own version of a regulatory moat.
- Animats 3y ago
  It doesn't require signing up with Google.
  - 0x073 3y ago
    If it would be public the ai could read it and can develop countermeasures ;) .
- vore 3y ago
  To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe you could fudge it today with user agents in robots.txt but who knows what the concrete idea of this is.
  - varenc 3y ago
    robots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing. [0] https://developers.google.com/search/docs/crawling-indexing/robots/intro https://developers.google.com/search/docs/crawling-indexing/...
    - dazc 3y ago
      Indeed, having pages indexed which can't then be crawled is a great way of shooting yourself in the foot.
      - floomk 3y ago
        I think you meant it's a great way for google to punish you for not giving them full access
    - deleted 3y ago
      [deleted]
  - Aerroon 3y ago
    This seems weird to me though, aren't search engines something very similar to AI, if not outright AI?
- helsinkiandrew 3y ago
  > I notice they don't actually give a good reason that robots.txt isn't suitable It's kind of implied: specifying sitemaps/allowance/copyright for different use cases: search, scraping, republishing, training etc. and perhaps adding some of the non standard extensions: Crawl-delay, default host, even sitemap isn't part of the robots.txt standard > We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases.
- h1fra 3y ago
  Came here to say that, seems like nobody as the answer :/ Maybe they want to have finer details on page content, e.g: "you can index those pages but not those nodes" or "those nodes are also AI generated please ignore". Otherwise I don't know, robots.txt is not sexy but definitely does the job.
- JohnFen 3y ago
  I think robots.txt isn't suitable for this for the same reason it's not suitable for keeping other bots from crawling your site: adhering to what robots.txt says is optional, and plenty of bots opt to ignore it.
deleted 3y ago
[deleted]
voytec 3y ago
Seems like it's intended for content stealing from every place that doesn't immediately implement Google's New Web Order as an addition to robots.txt. "Your do not enter sign uses font we don't like, so we'll just ignore it"
- saagarjha 3y ago
  What makes you think this? Why do you think Google actually cares about your sign if all they want to do is steal from you?
  - oneeyedpigeon 3y ago
    IIRC, Google has precedent on this - e.g. scanning full books for search unless the owner explicitly refused.
    - dylan604 3y ago
      They are the ultimate ask for forgiveness rather than permission. Copyright has been a thing for a long long time before googs developed their scanning. They were well aware that it should have been an opt-in, but knew they’d never gain traction for their little project. So they bull in a China shop’d their way to a point of too far to stop them.
      - philipov 3y ago
        They don't even ask for forgiveness. They are "don't admit you've done anything wrong to begin with."
      - remus 3y ago
        Copyright is to do with protecting reproduction of works, no? What google has done here is scanning the book and indexed the content, presumably so it makes it easier for users to search books for relevant material. Assuming they don't reproduce large sections of copyrighted works in their search results I don't feel like they're doing anything wrong here.
stonogo 3y ago
Given the method they decided on for people to opt out of wifi access point scanning -- requiring the rest of the world to change[1], while they continue doing whatever the hell they want -- I expect you'll need to log in to a Google account and write a brief essay about why your content shouldn't belong to them. 1 - https://support.google.com/maps/answer/1725632?hl=en#zippy=%2Chow-do-i-opt-my-access-point-out-of-google-location-services https://support.google.com/maps/answer/1725632?hl=en#zippy=%...
- deleted 3y ago
  [deleted]
- okeuro49 3y ago
  "To opt out, change the SSID (name) of your Wi-Fi access point (your wireless network name) so that it ends with "_nomap."" Wow, that's absurd. It would have been better not to have any mechanism at all.
tannhaeuser 3y ago
Google also introduced XML sitemaps (and noindex) so technically I could see how robots.txt could be consolidated into sitemapindex.xml with additional attributes. But it's not clear why and why now. Are there new requirements (from Google's PoV), such as tagging content as machine-generated? Not sure opening this discussion and changing the legal status of whether something may be crawled is going to end well for Google ie. from a copyright perspective (requiring explicit and individual consent), and in particular by entering legal terra incognita of moral rights vs generative AI, but maybe blocking competitors/AI startups is what they're after?
dazc 3y ago
Since robots.txt is currently ignored by bad actors, why would any alternative be better?
- tjpnz 3y ago
  Because Google can start ripping off content in the open with impunity.
- berkle4455 3y ago
  > bad actors I prefer the term ‘Chad third-party scraper’ [1] https://pbs.twimg.com/media/FxkeJmjakAENFI8?format=jpg&name=900x900 https://pbs.twimg.com/media/FxkeJmjakAENFI8?format=jpg&name=...
masswerk 3y ago
May I suggest a more general "harvest.txt" for all purposes of scraping content? Edit: Alternatively, have a "Harvest" section in "robots.txt", using the same established syntax and semantics. This may come with the advantage of making it clear that agents should default to the general "robots.txt" rules in absence of any such rules. Moreover, existing content management systems will already provide means for maintaining "robots.txt" and there's no need to update those. (We may also introduce an "Index" section for the established purpose of "robots.txt", with any bare, untitled rules defaulting to this, thus providing compatibility.) Example: #file "robots.txt" Index # optional section heading (maybe useful for switching context) User-agent: * Allow: / Disallow: /test/ Disallow: /private/ User-agent: Badbot Disallow: / Harvest # additional rules for scraping User-agent: * Disallow: /blog/ Disallow: /protected-artwork/
blacklight 3y ago
Why are those folks trying to sprinkle AI over everything, even when it's completely inappropriate? There's no AI involved in web crawling. If you come to my site, I'll tell you which pages you can visit/index, and which pages you can't, end of the story Yes, there are security concerns with people putting /very-secret-admin-panel in their robots.txt and letting malicious actors know what URLs they should target. But if /very-secret-admin-panel is never linked by any public page, then the bot won't encounter it, therefore this stuff should never belong to robots.txt. Please keep it as straightforward as this and don't add any AI bullshit to one of the few remaining simple processes in web development and administration.
- iamphilrae 3y ago
  Perhaps they’re intending on a means to say whether your content can be used within an AI training model or not.
  - denton-scratch 3y ago
    Why would any webmaster allow any of their content be used to train an AI? What's in it for them? The deal with searchbots is that you allow indexing because you want to be found. But no such quid-pro-quo occurs when the content is just fed into the maw of an AI trainer.
- simion314 3y ago
  Maybe some websites would like to specify something more then "I allow everything", maybe you could specify a license for the data on the page, like if you are OK for using it in open source research, open source AI training but not allow the data to be used in proprietary AI, or you do not want any kind of AI/research on the data , only search indexing.
Kwpolska 3y ago
Why would AI need a new standard for excluding it? Just add a "Googlebot-AI" user agent to your list [0] and respect these rules when crawling content for use in AIs, and convince OpenAI and Bing to do the same. [0] https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers https://developers.google.com/search/docs/crawling-indexing/...
- bastawhiz 3y ago
  I have no insight, but I suspect it's a question of context: regular old search is about whether a page is indexed or not. Either a URL is part of the index or it isn't. But with AI, there's important questions about what's in those urls. I think Google is probably thinking hard about the problem of training AI: you don't want to train on the output of other AI. That doesn't mean the content shouldn't be processed, just that it shouldn't be used for training. Or maybe it's worth noting that some content is derived from other content that you've manually produced, versus content derived from the content of third parties. Said another way, I expect that Google isn't just implementing a new allowlist/denylist. It's likely about exposing new information about content.
  - 2OEH8eoCRo0 3y ago
    Cool. Sounds like a you problem, you meaning crawlers and AI trainers. Now it will fall on every web developer to tag their data for it to be exploited by megacorps? Now that I think of it- why do we put up with robots.txt at all? > A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests If someone overloads your site with automated requests how is that not criminal? Why aren't they liable?
    - judge2020 3y ago
      You can block the crawler on your entire site. I’m not sure it’s true that it’s primarily used “to avoid overloading your site”.
konschubert 3y ago
If you run a store on Main Street, should you get to decide if people can take pictures of your store?
- mkl95 3y ago
  You will get a lot of replies focused on scraping. Which is exactly what Google do for a living. It's in their best interest to be the only company obtaining that valuable data.
- inglor 3y ago
  If someone goes to your store on Main Street, takes your inventory list and the price list, does the same to all the competitors and then sets up shop in main street selling people the information of where they should buy there is a case for me to be able to decide whether or not I give them access to that information. Especially since they're letting stores pay money to be the first recommended store.
  - Aerroon 3y ago
    Isn't this what gas stations do?
- riffraff 3y ago
  If the shop owner is losing something when people take pictures of the shop, they may start to shoo then away every time. Robots.txt exists because shop photographers want to be allowed to take pictures rather than be blocked tout court.
- manuelmoreale 3y ago
  If you run a store on Main Street, should people be allowed to take pictures of your store, copy its content and put it up for sale on another store? I see this argument made over and over again here on HN and it’s puzzling that people always stop at the first part. Companies won’t stop at the “look at your content” phase. They will use the knowledge gathered by looking at your content to do something else. That’s the problematic part.
foreign_legion 3y ago
WARNING: training data moat building in progress. They want to introduce a line in robots.txt that says "not for training AI". So nobody else can use publci data to train their AI. They already did.
- isodev 3y ago
  An ad company is keen to explore alternative means of controlling what other ad companies can do... After said ad company already scraped what they needed with no regard if it was even OK to do it.
xg15 3y ago
This press release really contains no substantial information (except the signup form), but the amount of "corpospeak words" in there that are usually euphemisms for bad news frankly worries me. In particular those bits: > A principled approach to evolving choice and control for web content > We believe everyone benefits from a vibrant content ecosystem. Key to that is web publishers having choice and control over their content, and opportunities to derive value from participating in the web ecosystem. However, we recognize that existing web publisher controls were developed before new AI and research use cases. > We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases. That's an awful lot of talk about "choice", and even more so "evolving choice". That's particularly odd when the choice of most publishers seems to be rather clear: "don't scrape content for AI training without at least asking before" - and robots.txt is perfectly capable of expressing that choice. So the ones that seem unhappy with the available means of choice seem to be the AI scrapers, not the publishers. So my preliminary translation of this from corpospeak would be: "Look guys, we were fine with robots.txt as long as we were only scraping your sites for search indexing. But now the AI race is on and gathering training data has just become Too Important, so we're planning to ignore robots.txt in the near future and just scrape the entirety of your sites for AI training. Instead, we'll offer you the choice of whether you want to let us scrape in exchange for some yet-to-be-determined compensation or whether you just provide the data for free. If we're particularly nice, we'll also give you an option to opt-out of scraping altogether. However, this option will be separate from robots.txt and you will have to explicitly add it to your site (provided you get to know about it in the first place)" That being said, I find robots.txt a bit strange for a target for this. Robots.txt really is nothing - it's not a license and has no legal significance (afaik) and it never prevented scraping on a technical level either. All it did was give friendly scrapers a hint, so they don't accidentally step on the publisher's toes - but it never prevented anyone from intentionally scraping stuff they weren't supposed to. On the other hand, if some courts did interpret robots.txt as some kind of impromptu licence, that interpretation probably wouldn't change, whether Google likes the standard or not. Also, people who employ real technical measures (ratelimiting, captchas, etc) will probably continue to do too. So if that's what they're planning to do, my only explanation would be that there is a large amount of small and "low-hanging fruit" sites (probably with inexperienced devs) that don't want to be scraped but really only added a robots.txt to block scrapers and didn't do anything else - and Google is planning to use those for AI training when all the large social networks are increasingly blocking them off now.
sys_64738 3y ago
How about they invent a new method that is opt-out by default and lack of presence means move along and leave this server alone.
revskill 3y ago
Singing up to know more updates ? No, sorry.
transformi 3y ago
There are a lot of comment about scraping. But I think that their new standards will try to tag a AI-genererated content. There is not much point to give crawlers a lot of content generation; but rather only the succinct "prompt". In that way, it will be easy to signal to crawlers what to crawl, and user can be read the content after the function of LLM...
tikkun 3y ago
See also: https://content.getsphere.com/ https://content.getsphere.com/
mrkramer 3y ago
In the first sentence they mention AI, that's how I know it's doomed.
- MentallyRetired 3y ago
  Because something annoys you, it's destined to fail? I'm not following the logic.
pentagrama 3y ago
The other day I was trying to search a comment of a YouTube video that I remembered, but wasn't able to find it with the google search "site:youtube.com [phrase of the comment]", later to find out that YouTube disallows search engines to index comments trough robots.txt https://www.youtube.com/robots.txt https://www.youtube.com/robots.txt >Disallow: /comment So I guess that works for them.
- DistractionRect 3y ago
  In fairness, I'd expect YouTube comments to largely be noise and not worth indexing.
thayne 3y ago
Hopefully the result is actually a portable standard, and not filling out forms for google and every other company training AI.
denton-scratch 3y ago
> You can join the web and AI communities’ discussion by signing up on our website and we'll share more information about this process soon. > I accept Google's Terms and Conditions and acknowledge that my information will be used in accordance with Google's Privacy Policy. Why is this being done through a Google mailing list? Why does Google want any public participation anyway? They usually just implement their new gee-whizz scheme, and start strong-arming web publishers into using it. Like, why would I trust a process that is run by Google, to create a new mechanism for controlling search engine behaviour? Fox: meet henhouse.
marginalia_nu 3y ago
There are problems with robots.txt if you actually try to implement it for a crawler. Consider this scenario: Allow: /foo Disallow: /bar Consider the situation where /foo HTTP 301s to /bar, or 200s but with a canonical location header that is /bar. Do you follow the redirect? Do you index /foo? In practice it's also often a directory of the paths the website owners don't want eyes to look at. Pretty common to find a list of uncomfortable content, especially on larger websites... like that time the dean of the college praised the philanthropy of Boko Haram. Real OSINT footgun.
- Animats 3y ago
  > There are problems with robots.txt if you actually try to implement it for a crawler. Yes, although that's not what people are usually worried about. I once tried to deal with that in Sitetruth's crawler. There are redirects at the HTTP level, redirects at the HTML level, and the HTTP->HTTPS thing. Resolving all that honestly is annoying, but possible. Sometimes you do need to look at the beginning of a file blocked by "robots.txt" to find that it is redirecting you elsewhere. It's like a door that says both "Keep Out" and "Please Use Other Door". This is more of a pedantic problem than a real one.
westurner 3y ago
There are a number of opportunities to solve for carbon.txt, security.txt, content licenses, indication of [AI] provenance, and do better than robots.txt; hopefully with JSON-LD Linked Data. > https://news.ycombinator.com/item?id=35888037 https://news.ycombinator.com/item?id=35888037 : security.txt, carbon.txt, SPDX SBOM, OSV, JSON-LD, blockcerts "Google will label fake images created with its A.I" (re: IPTC, Schema org JSON-LD" (2023) https://news.ycombinator.com/item?id=35896000 https://news.ycombinator.com/item?id=35896000 From "Tell HN: We should start to add “ai.txt” as we do for “robots.txt”" (2023) https://news.ycombinator.com/item?id=35888037 https://news.ycombinator.com/item?id=35888037 : > How many parsers should be necessary for https://schema.org/CreativeWork https://schema.org/CreativeWork https://schema.org/license https://schema.org/license metadata for resources with (Linked Data) URIs?
activiation 3y ago
"Google to ignore robots.txt for AI purposes"
lakomen 3y ago
Please stop adding more and more irrelevant text files Google. It's getting annoying. There's nothing wrong with robots.txt. Don't change what works just because you Google developer have to justify your employment
mindcrash 3y ago
Next up: "Google to explore alternatives to HTML". Do you think they developed AMP and are heavily invested in the W3C for "the good of the community"? And they already tried to "Googlify" cookies earlier.
JohnFen 3y ago
I agree that there needs to be better control over such things. Whatever comes from this, it also needs to avoid the fundamental problem with robots.txt: it's advisory only, and bots can freely ignore it. And many do. A real solution has to have an effectiveness greater than just asking nicely and hoping that people are honorable.

116 points by skilled 3y ago

dbetteridge 3y ago

I notice they don't actually give a good reason that robots.txt isn't suitable. Change for the sake of it?

strombofulous 3y ago
AI!
- asudosandwich 3y ago
  Indeed, AI is called out directly. Google’s laying groundwork for their own version of a regulatory moat.
Animats 3y ago
It doesn't require signing up with Google.
- 0x073 3y ago
  If it would be public the ai could read it and can develop countermeasures ;) .
vore 3y ago
To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe you could fudge it today with user agents in robots.txt but who knows what the concrete idea of this is.
- varenc 3y ago
  robots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing. [0] https://developers.google.com/search/docs/crawling-indexing/robots/intro https://developers.google.com/search/docs/crawling-indexing/...
  - dazc 3y ago
    Indeed, having pages indexed which can't then be crawled is a great way of shooting yourself in the foot.
    - floomk 3y ago
      I think you meant it's a great way for google to punish you for not giving them full access
  - deleted 3y ago
    [deleted]
- Aerroon 3y ago
  This seems weird to me though, aren't search engines something very similar to AI, if not outright AI?
helsinkiandrew 3y ago
> I notice they don't actually give a good reason that robots.txt isn't suitable It's kind of implied: specifying sitemaps/allowance/copyright for different use cases: search, scraping, republishing, training etc. and perhaps adding some of the non standard extensions: Crawl-delay, default host, even sitemap isn't part of the robots.txt standard > We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases.
h1fra 3y ago
Came here to say that, seems like nobody as the answer :/ Maybe they want to have finer details on page content, e.g: "you can index those pages but not those nodes" or "those nodes are also AI generated please ignore". Otherwise I don't know, robots.txt is not sexy but definitely does the job.
JohnFen 3y ago
I think robots.txt isn't suitable for this for the same reason it's not suitable for keeping other bots from crawling your site: adhering to what robots.txt says is optional, and plenty of bots opt to ignore it.

deleted 3y ago

[deleted]

voytec 3y ago

Seems like it's intended for content stealing from every place that doesn't immediately implement Google's New Web Order as an addition to robots.txt. "Your do not enter sign uses font we don't like, so we'll just ignore it"

saagarjha 3y ago
What makes you think this? Why do you think Google actually cares about your sign if all they want to do is steal from you?
- oneeyedpigeon 3y ago
  IIRC, Google has precedent on this - e.g. scanning full books for search unless the owner explicitly refused.
  - dylan604 3y ago
    They are the ultimate ask for forgiveness rather than permission. Copyright has been a thing for a long long time before googs developed their scanning. They were well aware that it should have been an opt-in, but knew they’d never gain traction for their little project. So they bull in a China shop’d their way to a point of too far to stop them.
    - philipov 3y ago
      They don't even ask for forgiveness. They are "don't admit you've done anything wrong to begin with."
    - remus 3y ago
      Copyright is to do with protecting reproduction of works, no? What google has done here is scanning the book and indexed the content, presumably so it makes it easier for users to search books for relevant material. Assuming they don't reproduce large sections of copyrighted works in their search results I don't feel like they're doing anything wrong here.

stonogo 3y ago

Given the method they decided on for people to opt out of wifi access point scanning -- requiring the rest of the world to change[1], while they continue doing whatever the hell they want -- I expect you'll need to log in to a Google account and write a brief essay about why your content shouldn't belong to them. 1 - https://support.google.com/maps/answer/1725632?hl=en#zippy=%2Chow-do-i-opt-my-access-point-out-of-google-location-services https://support.google.com/maps/answer/1725632?hl=en#zippy=%...

deleted 3y ago
[deleted]
okeuro49 3y ago
"To opt out, change the SSID (name) of your Wi-Fi access point (your wireless network name) so that it ends with "_nomap."" Wow, that's absurd. It would have been better not to have any mechanism at all.

tannhaeuser 3y ago

Google also introduced XML sitemaps (and noindex) so technically I could see how robots.txt could be consolidated into sitemapindex.xml with additional attributes. But it's not clear why and why now. Are there new requirements (from Google's PoV), such as tagging content as machine-generated? Not sure opening this discussion and changing the legal status of whether something may be crawled is going to end well for Google ie. from a copyright perspective (requiring explicit and individual consent), and in particular by entering legal terra incognita of moral rights vs generative AI, but maybe blocking competitors/AI startups is what they're after?

dazc 3y ago

Since robots.txt is currently ignored by bad actors, why would any alternative be better?

tjpnz 3y ago
Because Google can start ripping off content in the open with impunity.
berkle4455 3y ago
> bad actors I prefer the term ‘Chad third-party scraper’ [1] https://pbs.twimg.com/media/FxkeJmjakAENFI8?format=jpg&name=900x900 https://pbs.twimg.com/media/FxkeJmjakAENFI8?format=jpg&name=...

masswerk 3y ago

May I suggest a more general "harvest.txt" for all purposes of scraping content? Edit: Alternatively, have a "Harvest" section in "robots.txt", using the same established syntax and semantics. This may come with the advantage of making it clear that agents should default to the general "robots.txt" rules in absence of any such rules. Moreover, existing content management systems will already provide means for maintaining "robots.txt" and there's no need to update those. (We may also introduce an "Index" section for the established purpose of "robots.txt", with any bare, untitled rules defaulting to this, thus providing compatibility.) Example: #file "robots.txt" Index # optional section heading (maybe useful for switching context) User-agent: * Allow: / Disallow: /test/ Disallow: /private/ User-agent: Badbot Disallow: / Harvest # additional rules for scraping User-agent: * Disallow: /blog/ Disallow: /protected-artwork/

blacklight 3y ago

Why are those folks trying to sprinkle AI over everything, even when it's completely inappropriate? There's no AI involved in web crawling. If you come to my site, I'll tell you which pages you can visit/index, and which pages you can't, end of the story Yes, there are security concerns with people putting /very-secret-admin-panel in their robots.txt and letting malicious actors know what URLs they should target. But if /very-secret-admin-panel is never linked by any public page, then the bot won't encounter it, therefore this stuff should never belong to robots.txt. Please keep it as straightforward as this and don't add any AI bullshit to one of the few remaining simple processes in web development and administration.

iamphilrae 3y ago
Perhaps they’re intending on a means to say whether your content can be used within an AI training model or not.
- denton-scratch 3y ago
  Why would any webmaster allow any of their content be used to train an AI? What's in it for them? The deal with searchbots is that you allow indexing because you want to be found. But no such quid-pro-quo occurs when the content is just fed into the maw of an AI trainer.
simion314 3y ago
Maybe some websites would like to specify something more then "I allow everything", maybe you could specify a license for the data on the page, like if you are OK for using it in open source research, open source AI training but not allow the data to be used in proprietary AI, or you do not want any kind of AI/research on the data , only search indexing.

Kwpolska 3y ago

Why would AI need a new standard for excluding it? Just add a "Googlebot-AI" user agent to your list [0] and respect these rules when crawling content for use in AIs, and convince OpenAI and Bing to do the same. [0] https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers https://developers.google.com/search/docs/crawling-indexing/...

bastawhiz 3y ago
I have no insight, but I suspect it's a question of context: regular old search is about whether a page is indexed or not. Either a URL is part of the index or it isn't. But with AI, there's important questions about what's in those urls. I think Google is probably thinking hard about the problem of training AI: you don't want to train on the output of other AI. That doesn't mean the content shouldn't be processed, just that it shouldn't be used for training. Or maybe it's worth noting that some content is derived from other content that you've manually produced, versus content derived from the content of third parties. Said another way, I expect that Google isn't just implementing a new allowlist/denylist. It's likely about exposing new information about content.
- 2OEH8eoCRo0 3y ago
  Cool. Sounds like a you problem, you meaning crawlers and AI trainers. Now it will fall on every web developer to tag their data for it to be exploited by megacorps? Now that I think of it- why do we put up with robots.txt at all? > A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests If someone overloads your site with automated requests how is that not criminal? Why aren't they liable?
  - judge2020 3y ago
    You can block the crawler on your entire site. I’m not sure it’s true that it’s primarily used “to avoid overloading your site”.

konschubert 3y ago

If you run a store on Main Street, should you get to decide if people can take pictures of your store?

mkl95 3y ago
You will get a lot of replies focused on scraping. Which is exactly what Google do for a living. It's in their best interest to be the only company obtaining that valuable data.
inglor 3y ago
If someone goes to your store on Main Street, takes your inventory list and the price list, does the same to all the competitors and then sets up shop in main street selling people the information of where they should buy there is a case for me to be able to decide whether or not I give them access to that information. Especially since they're letting stores pay money to be the first recommended store.
- Aerroon 3y ago
  Isn't this what gas stations do?
riffraff 3y ago
If the shop owner is losing something when people take pictures of the shop, they may start to shoo then away every time. Robots.txt exists because shop photographers want to be allowed to take pictures rather than be blocked tout court.
manuelmoreale 3y ago
If you run a store on Main Street, should people be allowed to take pictures of your store, copy its content and put it up for sale on another store? I see this argument made over and over again here on HN and it’s puzzling that people always stop at the first part. Companies won’t stop at the “look at your content” phase. They will use the knowledge gathered by looking at your content to do something else. That’s the problematic part.

foreign_legion 3y ago

WARNING: training data moat building in progress. They want to introduce a line in robots.txt that says "not for training AI". So nobody else can use publci data to train their AI. They already did.

isodev 3y ago
An ad company is keen to explore alternative means of controlling what other ad companies can do... After said ad company already scraped what they needed with no regard if it was even OK to do it.

xg15 3y ago

This press release really contains no substantial information (except the signup form), but the amount of "corpospeak words" in there that are usually euphemisms for bad news frankly worries me. In particular those bits: > A principled approach to evolving choice and control for web content > We believe everyone benefits from a vibrant content ecosystem. Key to that is web publishers having choice and control over their content, and opportunities to derive value from participating in the web ecosystem. However, we recognize that existing web publisher controls were developed before new AI and research use cases. > We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases. That's an awful lot of talk about "choice", and even more so "evolving choice". That's particularly odd when the choice of most publishers seems to be rather clear: "don't scrape content for AI training without at least asking before" - and robots.txt is perfectly capable of expressing that choice. So the ones that seem unhappy with the available means of choice seem to be the AI scrapers, not the publishers. So my preliminary translation of this from corpospeak would be: "Look guys, we were fine with robots.txt as long as we were only scraping your sites for search indexing. But now the AI race is on and gathering training data has just become Too Important, so we're planning to ignore robots.txt in the near future and just scrape the entirety of your sites for AI training. Instead, we'll offer you the choice of whether you want to let us scrape in exchange for some yet-to-be-determined compensation or whether you just provide the data for free. If we're particularly nice, we'll also give you an option to opt-out of scraping altogether. However, this option will be separate from robots.txt and you will have to explicitly add it to your site (provided you get to know about it in the first place)" That being said, I find robots.txt a bit strange for a target for this. Robots.txt really is nothing - it's not a license and has no legal significance (afaik) and it never prevented scraping on a technical level either. All it did was give friendly scrapers a hint, so they don't accidentally step on the publisher's toes - but it never prevented anyone from intentionally scraping stuff they weren't supposed to. On the other hand, if some courts did interpret robots.txt as some kind of impromptu licence, that interpretation probably wouldn't change, whether Google likes the standard or not. Also, people who employ real technical measures (ratelimiting, captchas, etc) will probably continue to do too. So if that's what they're planning to do, my only explanation would be that there is a large amount of small and "low-hanging fruit" sites (probably with inexperienced devs) that don't want to be scraped but really only added a robots.txt to block scrapers and didn't do anything else - and Google is planning to use those for AI training when all the large social networks are increasingly blocking them off now.

sys_64738 3y ago

How about they invent a new method that is opt-out by default and lack of presence means move along and leave this server alone.

revskill 3y ago

Singing up to know more updates ? No, sorry.

transformi 3y ago

There are a lot of comment about scraping. But I think that their new standards will try to tag a AI-genererated content. There is not much point to give crawlers a lot of content generation; but rather only the succinct "prompt". In that way, it will be easy to signal to crawlers what to crawl, and user can be read the content after the function of LLM...

tikkun 3y ago

See also: https://content.getsphere.com/ https://content.getsphere.com/

mrkramer 3y ago

In the first sentence they mention AI, that's how I know it's doomed.

MentallyRetired 3y ago
Because something annoys you, it's destined to fail? I'm not following the logic.

pentagrama 3y ago

The other day I was trying to search a comment of a YouTube video that I remembered, but wasn't able to find it with the google search "site:youtube.com [phrase of the comment]", later to find out that YouTube disallows search engines to index comments trough robots.txt https://www.youtube.com/robots.txt https://www.youtube.com/robots.txt >Disallow: /comment So I guess that works for them.

DistractionRect 3y ago
In fairness, I'd expect YouTube comments to largely be noise and not worth indexing.

thayne 3y ago

Hopefully the result is actually a portable standard, and not filling out forms for google and every other company training AI.

denton-scratch 3y ago

> You can join the web and AI communities’ discussion by signing up on our website and we'll share more information about this process soon. > I accept Google's Terms and Conditions and acknowledge that my information will be used in accordance with Google's Privacy Policy. Why is this being done through a Google mailing list? Why does Google want any public participation anyway? They usually just implement their new gee-whizz scheme, and start strong-arming web publishers into using it. Like, why would I trust a process that is run by Google, to create a new mechanism for controlling search engine behaviour? Fox: meet henhouse.

marginalia_nu 3y ago

There are problems with robots.txt if you actually try to implement it for a crawler. Consider this scenario: Allow: /foo Disallow: /bar Consider the situation where /foo HTTP 301s to /bar, or 200s but with a canonical location header that is /bar. Do you follow the redirect? Do you index /foo? In practice it's also often a directory of the paths the website owners don't want eyes to look at. Pretty common to find a list of uncomfortable content, especially on larger websites... like that time the dean of the college praised the philanthropy of Boko Haram. Real OSINT footgun.

Animats 3y ago
> There are problems with robots.txt if you actually try to implement it for a crawler. Yes, although that's not what people are usually worried about. I once tried to deal with that in Sitetruth's crawler. There are redirects at the HTTP level, redirects at the HTML level, and the HTTP->HTTPS thing. Resolving all that honestly is annoying, but possible. Sometimes you do need to look at the beginning of a file blocked by "robots.txt" to find that it is redirecting you elsewhere. It's like a door that says both "Keep Out" and "Please Use Other Door". This is more of a pedantic problem than a real one.

westurner 3y ago

There are a number of opportunities to solve for carbon.txt, security.txt, content licenses, indication of [AI] provenance, and do better than robots.txt; hopefully with JSON-LD Linked Data. > https://news.ycombinator.com/item?id=35888037 https://news.ycombinator.com/item?id=35888037 : security.txt, carbon.txt, SPDX SBOM, OSV, JSON-LD, blockcerts "Google will label fake images created with its A.I" (re: IPTC, Schema org JSON-LD" (2023) https://news.ycombinator.com/item?id=35896000 https://news.ycombinator.com/item?id=35896000 From "Tell HN: We should start to add “ai.txt” as we do for “robots.txt”" (2023) https://news.ycombinator.com/item?id=35888037 https://news.ycombinator.com/item?id=35888037 : > How many parsers should be necessary for https://schema.org/CreativeWork https://schema.org/CreativeWork https://schema.org/license https://schema.org/license metadata for resources with (Linked Data) URIs?

activiation 3y ago

"Google to ignore robots.txt for AI purposes"

lakomen 3y ago

Please stop adding more and more irrelevant text files Google. It's getting annoying. There's nothing wrong with robots.txt. Don't change what works just because you Google developer have to justify your employment

mindcrash 3y ago

Next up: "Google to explore alternatives to HTML". Do you think they developed AMP and are heavily invested in the W3C for "the good of the community"? And they already tried to "Googlify" cookies earlier.

JohnFen 3y ago

I agree that there needs to be better control over such things. Whatever comes from this, it also needs to avoid the fundamental problem with robots.txt: it's advisory only, and bots can freely ignore it. And many do. A real solution has to have an effectiveness greater than just asking nicely and hoping that people are honorable.