4 ms·
In what circles is it a joke? Google bots seem to respect it on my sites according to logs.
by Demiurge 2y ago
In what circles is it a joke? Google bots seem to respect it on my sites according to logs.
- otikik 2y agoApparently, the regular search crawler does it, but the ai thingie doesn't.
- supriyo-biswas 2y agoHuh? You can add Google-Extended[1] to opt out from Generative AI summaries. [1] https://blog.google/technology/ai/an-update-on-web-publisher-controls/ https://blog.google/technology/ai/an-update-on-web-publisher...
- micromacrofoot 2y agoGoogle will still scrape it for training data either way, this only impacts search results.
- supriyo-biswas 2y ago> Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help *improve Bard and Vertex AI generative APIs*, including future generations of models that power those products.
- micromacrofoot 2y agohttps://www.theverge.com/news/630079/openai-google-copyright-fair-use-exception https://www.theverge.com/news/630079/openai-google-copyright... they're literally asking to break laws to train AI for national security. A sentence in a press release from 2 years ago is worthless... look at what they're actually doing
- lucgagan 2y agoCan confirm. My website is flooded with AI bots despite attempts to block crawlers to certain parts of it.
- mediumsmart 2y agoI know an artist that had noindex turned on by mistake in robots.txt for the last 5 years - google, kagi and duckduckgo find tons of links relevant to the artist and the artwork but not a single one from the website. so not seem to or apparently but matter of fact like. robots.txt works for the intended audience
- joecool1029 2y agoIt's in a small circle of those that do. Blame the internet archive for starting this trend: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- hirako2000 2y agoGiven websites do disappear or worse, get their content adultered. Also given the long history of the internet archive as a non profit - and the commons service it has served so far, the joke would be to see that bot honor it.
- nikisweeting 2y agolol IA did not start that, if anything they were late to the game. only the top handful of US-based search engines ever bothered respecting it in the first place
- wuming2 2y agoSorry to intrude with something unrelated. But YC closed the earlier discussion. Saw your comment about Kannel WAP of few months back and wanted to ask if do you know of any WAP Push full service provider still in operation.
- joecool1029 2y agoNah I just know about that public gateway I linked. I can't use it anymore as 2G was shut down on my local towers back in January.
- micromacrofoot 2y agoA small number of search engines respect it, no one else does. Just about every content scraping bot ignores it, including a number of Google's.