18 ms·
Pulling my site from Google over AI training
- fortyseven 3y agojots down "CRAWL AND SCRAPE A WEB RING" Got it.
- fortyseven 3y agoHonestly, if you're so terrified now of what happens to your information once you post it publicly, just cut the cord. It was ALWAYS like this.
- pixelgeek 3y agoHelpful links. I will be doing the same this evening
- jxramos 3y ago> I’m going to start by pulling my websites out of Google search, then work on adding my sites to directories. Maybe I’ll even join a webring I'm curious, this is the first time I've heard of a webring, I'd like to learn more about these alternate discovery routes. Anyone have any concrete experience or recommendations to share?
- tivert 3y agoDirectory sites and webrings were artifacts of the early web, when search engines were pretty bad. They disappeared once search got better. https://en.wikipedia.org/wiki/Webring https://en.wikipedia.org/wiki/Webring
- stevenicr 3y agoAdd some context, expanding on what wolpoli mentioned, Sure search engines got better, and then they got worse.. Many of the things 'web rings' promoted were 'similar sites'. Yet it was when google become big that it destroyed web rings and blog rolls (which were similar to web rings, but often included a variety of sites not just similars) - It became known that google penalized people for linking to other sites, linking to 'bad neighborhoods' - they would sometimes call out sites publicly for giving 'link juice' to others for profit.. Web rings disappeared and blog rolls as well, because google. Not because search engines got better, but because google threatened to penalize you for linking out.
- jxramos 3y agoI'd love to see a resurgence of them, google is ever increasingly becoming useless as a search tool that when I see the first page is junk I just give up on the search and make note of what I'm looking for and try again at a later time or use alternate means of information retrieval. It feels like at some point in the last 5 years we crossed some threshold where search engines are so optimized to the commercial space and selling junk that obscure searches yield no useful information. Any keyword which is so unfortunate to overlap in the commercial product space will just dominate your results and make them mostly useless. Even my advanced google fu to tack on certain phrases and other bits of language to narrow things down and formerly gave very focused results but now feels like google is atrophying in that domain. It doesn't help that our culture has become so word overload friendly where instead of creating new words and phonetic combinations existing words are used instead and now require disambiguation to say not the product kind of this word.
- JohnFen 3y agoI miss webrings. I don't think that search getting better killed them (they are useful for reasons unrelated to the state of search), but when personal and hobbyist websites started vanishing, webrings went with them.
- tiltowait 3y agoWebrings were a great, essentially curated list of sites the authors of sites you liked thought you might also enjoy or find useful. I miss them, too.
- jxramos 3y agoyah, come to think of it in the curated space, this reminds me of that awesome X family of github pages. Looks like someone compiled a bunch of them here https://github.com/sindresorhus/awesome#databases https://github.com/sindresorhus/awesome#databases. I have found those to be highly valuable treasure troves pregnant with rich and relevant information.
- wolpoli 3y agoIronically, once search got better for the users, there was no further need to build comprehensive sites with great content in the hopes of getting added to webrings and directories. All sites need to worry about after were getting backlinks. After that, sites also stop linking to each other.
- bagpuss 3y agodon't forget Jimmy "Jimbo" Wales's Bomis webring thing, featuring the Bomis Babe Report!
- ModernMech 3y agoStop trying to make the rest of us feel old lol.
- tivert 3y ago> Blocking bots that collect training data for AIs (and more) > In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install (Nope.) > I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl. Instead of blocking, it would be neater to present and alternative version to the crawlers (like many paywalled sites already to for SEO) that's full of dynamically generated LLM-generated garbage. That'll help the LLMs poison themselves.
- JohnFen 3y agoI was thinking about jamming along these lines, but the problem is that it's a game of whack-a-mole -- you have to keep up on what bots are active (robots.txt doesn't really help here, and focusing on Common Crawl is insufficient). My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.
- sebzim4500 3y agoSo if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation? You have a legal right to do this (assuming the LLM bullshit isn't libel) but I don't see how it could be considered a moral act.
- JohnFen 3y ago> So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation? Yes, although I've been blocking googlebot for years, so nobody will get to my sites through google anyway. > I don't see how it could be considered a moral act. I'm curious about this -- why do you think this is in any way an immoral act? If a naïve human comes across the site, they'll quickly realize that it's not useful and move on. No harm done. How does morality enter into it? Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?
- TheCaptain4815 3y agoAn interesting thought are government or foreign actors training a generative Ai. They won't abide by any no-index tax and will scrape any and everything.
- JohnFen 3y agoIt doesn't have to be government or foreign actors. Abiding by the contents of robots.txt isn't required of anybody at all. It's merely a social convention.
- chomp 3y agoAuthor should update their robots.txt for Googlebot as well. It is not clear if noindex means "notrain" too. The entire webpage has to be read in and parsed for Google to extract that meta tag. However, robots.txt should stop the crawler before it proceeds to the rest of your site.
- DanHulton 3y agoThe author mentioned she was going to block Googlebot, only not yet, in order to make sure it can crawl the site again to get the 'noindex' instructions.
- mkl95 3y agoCan you pollute their data with hidden elements, or do they only scrape visible stuff?
- jeffbee 3y agoIf Google thinks a site is serving different content to googlebot vs. real users, it will stop returning that site on the SERP, because that is a malware distribution technique, among other reasons.
- mkl95 3y agoWhat if you serve the same site? Does googlebot know that some text has the same color as the background?
- jeffbee 3y agoI don't know, that's not my game and I never worked on crawl or index, but I imagine they have that handled since invisible keyword spam at the bottom of the page was already a universal spammer technique by 1997.
- barbariangrunge 3y agoAs a substack author, with “permission is not granted to use any portion of this to train an ai” at the bottom of most of my posts, it’s bullshit that you have to do this sort of thing, and that it will almost certainly not work This must be illegal, but how are all the little bloggers going to oppose it?
- arsome 3y agoI don't see why it would be illegal, AI reading it should be no different from anyone else.
- barbariangrunge 3y agoI keep having to say this: An ai is not a person
- anonn77 3y agoYep, there’s a big difference in practice. If an AI could attribute and provide royalties then it may not be so different but that’s never going to happen. A big reason Bard exists is Google trying to ensure they stay profitable and relevant. They don’t care where the knowledge really comes from.
- arsome 3y agoDon't forget to provide royalties for every synapse in your head.
- JohnFen 3y ago> If an AI could attribute and provide royalties then it may not be so different But even that requires the permission of the copyright holder. Nobody is required to accept an infringing use of their work in exchange for royalties.
- JohnFen 3y ago"Illegal" is too strong. But if you specifically disallow the use of your website contents from being used to train AI, then anyone doing so is violating the terms of service. Which doesn't really mean anything. At this point, the only defense I can think of is to not make the content publicly available. Which is what I've done.
- LordOfRiverRun 3y agoWhy the preference not to have Google train their AI using your website content?
- bryzaguy 3y agoOne reason is the same as authors who don’t want their books trained and actors who don’t want their likeness trained. If that content is valuable it allows google to realize that value with no return to the creator.
- vidarh 3y agoThe naivety is assuming a small number of people making access hard will affect the value they're able to realize.
- JohnFen 3y agoI bet the reasons vary from person to person. My reason is because I think that these AI systems pose too great of a risk to society, and I want to make sure that I'm not helping them in any way.
- tasubotadas 3y ago[flagged]
- tasubotadas 3y ago"Ironically, the uniformity of the copies of Gutenberg’s Bible led many superstitious people of the time to equate printing with Satan because it seemed to be magical. Printers’ apprentices became known as the "printer’s devil." In Paris, Fust [a typographer] was charged as a witch. Although he escaped the Inquisition, other printers did not." (The Unsung Heroes, a History of Print by Dr. Jerry Waite 2001)"
- txcvdsfsgh 3y agoThis whole AI scraping argument is so silly to me. If you don't want people downloading and processing your content, then don't post it on the public internet?
- JohnFen 3y agoIt's not about not wanting people to access the content, it's about not wanting AI bots to do so. But yes, I agree, the only realistic defense we have is to remove it from the publicly-accessible web.
- deleted 3y ago[deleted]
- moribvndvs 3y agoThey literally outline their reasoning in a link[0]. There’s a significant gap between offering information for someone to read for free (where I can see the author and choose to respect their terms, if they have any), and a huge tech company aggregating that data where it is assimilated into a model and used in a product they will profit from[1]. They are exploiting gray area regarding digital rights, copyright, etc. [0] https://tracydurnell.com/2023/07/07/the-next-big-theft/ https://tracydurnell.com/2023/07/07/the-next-big-theft/ [1] https://www.tumblr.com/nedroidcomics/41879001445/the-internet https://www.tumblr.com/nedroidcomics/41879001445/the-interne...
- jeffgreco 3y agoMy pitch: A search engine that exclusively indexes noindex sites (you can use other sites while spidering) and builds an LLM model with the results.
- bratao 3y agoI had an (terrible) idea to create an search index that build on DMCA notices.
- JohnFen 3y agoI rather suspect that this is already being done.
- jeffbee 3y agoI seem to recall meta-search sites that only listed sites that had disappeared due to DMCA takedowns. There was also "unsafe search" that ran your google search twice, once with "SafeSearch" enabled, and returned the complement of the union, i.e. just the porn.
- CatWChainsaw 3y agoComments are bound to be spicy on this one. I always love it when techbros say that AI learning and human learning are exactly the same, because reading one thing at a time at a biological pace and remembering takeaway ideas rather than verbatim passages is obviously exactly the same thing as processing millions of inputs at once and still being able to regurgitate sources so perfectly that verbatim copyrighted content can be spit out of an LLM that doesn't 'contain' its training material. It's even better when they get so butthurt at being called out that they have a nice little rage-cry.
- zero_shift 3y agoThey only believe it because they think their own labour isn't under threat. Think.
- CatWChainsaw 3y agoI'm going to savor every last iota of the Shocked Pikachu energy they emit. Especially when they realize, far too late, that building their almighty replacements gets them exactly zero kudos from the people who will inevitably control it. I'm not going to be apologetic about it either, since the same people who think they're invulnerable also tend to espouse sadistic glee over the impending immiseration of millions due to these developments... just practicing what they preach, after all.
- blibble 3y agowhat do you mean, they'll get a whole $20k bonus out of it then be competing with it for the rest of their lives as it slowly reduces their labour potential to zero
- EamonnMR 3y agoThe layout on this linked site: https://vasilis.nl/nerd/how-to-disagree-with-googles-privacy-policy/ https://vasilis.nl/nerd/how-to-disagree-with-googles-privacy... Is incredible.
- kccqzy 3y agoDon't forget about also pulling your site from Bing! It would be naive if you somehow trust that Microsoft won't use your site for AI training.
- talldatethrow 3y agoAnd Yandex! Side note, I have friends that crawled a massive amount of the internet over several months for their own purposes.. at this point it's probably impossible to exclude your site since tons of other people probably link to your site if it's at all of value.
- JohnFen 3y agoYeah, this is why robots.txt is garbage. Too many crawlers completely ignore it, so I stopped bothering with it (I have it set to block all bots, but I don't expect it to be effective.) Instead, I'd just keep an eye on my access logs and block obvious crawlers when I saw them.
- zer0w1re 3y agoAh, yes. More complaining about freely posting content publicly on the internet and then being upset when it's used in a way you don't want. I'm sure foreign companies and even governments are doing something similar, what will U.S. laws do to stop that?
- zo1 3y agoBit of a related rant. Just today I googled (and duck duck go'd?) alternatives for Discord (because reasons). Entire search results page was "X top alternatives to Discord." It was all blog posty kind of stuff with an "author". And like 90% of it was written by indian and african sounding names. These were clearly "content farms" with low paid labour and bad grammar, or just authors with nothing better to do than write Yet Another Blog Post about Top Discord Alternatives. Sure, they weren't generated, but the fact that a human was involved in creating something crappy doesn't make it better or unique. What I was actually looking for was unique content. Either an actual curated list of alternatives (NOT a blog post they update every year). Or an extract from a book where someone posted fiction about a fictional Discord user that meets aliens. Or comments in a forum, or a link to a song-lyrics website for a Weird Al parody song about discord, a website dedicated to expounding the virtues cutting the discord cord, a link to a PDF where someone saved a IRC chat server's logs about a person switching from discord to IRC, or an "IRC-MF do you speak it" crass website, or something. Anything but a damn content blog post by some third-world content creator or hipster-blog-poster from the 1st world. What I got was garbage. Human-level garbage. Garbage that across hundreds of thousands of websites basically took a piece of content and expanded it with every known combination of words, sentences, and mini-stories and pasted it on a stupid blog post with an author. And this garbage is what this AI is training on so we can have content farms make more copies of itself with more variations and in different languages now, all so we can pay Google et al attention-coins to magically sift through all that garbage and present us with something a little less garbage-y for us to consume.
- theonlybutlet 3y agoI can't understand the outrage. In practice absolutely nothing has changed. It is reading and learning. A person would read and learn. This has no bearing on plagiarism or copyright. A work can be considered plagiarized or to breach copyright if the author hasn't even seen or come across the copyrighted/published work. This is no different. I can write some code and use it, subconsciously referencing a work. If I don't check my written work and put it out there, someone might have a claim against me. If I don't check the machine generated work and put it out there, someone might have a claim against me. OpenAI,Meta,et al are providing the model, basically a regression model or tool. I'm adding the variables or secret sauce that makes it output that set of data in that specific order not them. It'd be like suing Parker for making the pen.
- blargey 3y agoA human cannot learn from and re/produce work they view at the speed, volume, and scale that “AI” does, nor can that human be infinitely replicated and farmed out. When describing this gap in capabilities, or the consequences of “learning”, “orders of magnitude” would be a comical understatement. Existing conventions around “learning” are built on assumptions of human scale, and the expected consequences thereof. I can’t understand why one would expect people to go “oh it’s technically ‘learning’ I guess I’ll ignore all the consequences that weren’t present when it was just humans”.
- theonlybutlet 3y agoThat's completely subjective. What you describe is a spectrum. If that were true, thesis' would not have to be run through plagiarism scans (they are), and there would be no copyright lawsuits for familiarity such as Ed Sheeran's Marvin Gaye lawsuit. It's also important to note these laws were always intended to strike a fair balance between the copyright owner and the good of society as a whole. Copyright is not an end in itself.
- JohnFen 3y agoThis is such an important point that I think escapes a lot of people, both on the pro-AI and anti-AI side. Although everyone probably wishes otherwise, there isn't really a hard objective line for determining if something is infringing or not. And that's actually a good thing! But it means that, in the end, it's up to a judge to weigh a bunch of different, rather fuzzy, factors.