52 ms·
GPTBot – OpenAI’s Web Crawler
- _the_inflator 3y ago"Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety." Well, "more accurate" means roughly: "So that your content can be absorbed and used as output by our generator." Google at least linked to your website, while ChatGPT hides your website and only uses your content.
- speedgoose 3y agoGoogle started to put answers before links a while ago. At least for the simple searches. And now they also put many links of SEO spam with ads firsts.
- crsn 3y agoAnyone else find it interesting/odd that their UA string reports a WebKit (Safari?) environment rather than Blink/Chrome?
- dahwolf 3y agoUser-agent: GPTBot Allow: /directory-1/ Let's all make this "directory-1" and fill it with absolute garbage.
- ouraf 3y agopulling the ladder behind you
- tete 3y agoWould be a fun experiment to return different content to that user agent for a trusted well ranking website and see whether results make it into the next version.
- ichiroedogawa 3y ago[dead]
- bberenberg 3y agoGood on them for following a standard interaction model.
- Takennickname 3y agoAfter they already raped the internet...
- stephenson 3y agoIt's funny that you need to prove you are a human to access this page…
- Zaheer 3y agoNice of them to respect crawling after they've already trained their model. Presumably these headers don't affect any pages they've already crawled to train GPT(?)
- supriyo-biswas 3y agoOn that note, I also wonder if they end up getting this information anyway through another source like Common Crawl.
- ramraj07 3y agoHoping this is what they’ll use to train future models and deprecate the older ones before the legal cases proceed any further.
- p-e-w 3y agoThe legal cases don't mean anything. The rule of law has all but disappeared from the corporate world. The idea that courts or regulators will be able to control AI is laughable. They are too corrupt, and they are way too slow.
- DSingularity 3y agoI think a key idea is that with the amount of jurisdictions and number of courts the odds that a clean and sympathetic judge can be found approach one. I would argue that European jurisdictions are inherently less likely to be in pockets of American corporate interest and they are more likely to hear cases where fundamental human freedoms are at stake because both of these are existential threats to European independence. In the US similar arguments can be made in states vs federal or the various federal circuits. Courts are more deliberate than you would like — no denying that. But this is a feature not a flaw. It may be that damage will be done by then. Perhaps irreversible. But I would like to think if there is a will there is a way and that if things are terrible enough the governments will be bold in their responses.
- 3y ago
- wg0 3y agoIf you scrape my hobby website about photography, scuba diving or let's say baking or gardening which improves your model by let's say a delta of 0.00000000001 than shouldn't I get some free credits to use that model or proportionate share in the revenue stream? EDIT: scuba diving NOT scooba diving
- terrytanys 3y agoI think what you are doing is great. I would say current way of ChatGPT isn't ideal. Harvesting the data without giving credits. The tech now is great but I believe there is a way for all to WINs.
- p-e-w 3y agoIf I learn something from your StackOverflow answers, do you expect me to share a percentage of my future salary with you?
- terrytanys 3y agoIt is not learning, it is about publishing.
- croes 3y agoCan you share your knowledge with millions at once? If so, then pay.
- p-e-w 3y agoSo there's an infinite pyramid of "who learned what from who", and payment flows upwards along the hierarchy, all the way back to people who are long dead, and then down to their descendants who presumably inherited their "knowledge rights"? You can't be serious. Thank god our world doesn't work like that.
- croes 3y agoIt's not about who learned what from whom, it's about superstar economy. If you serve all customers and leave nothing for the rest, it will be a problem. What do you think why writers and actors have included AI in the reasons of their strike?
- skilled 3y agoMeanwhile... > For robots.txt, we do follow the same restrictions applied to googlebot, otherwise Google benefits from its dominant position. https://community.brave.com/t/stop-website-being-shown-in-brave-search/495774/3 https://community.brave.com/t/stop-website-being-shown-in-br...
- scrum-treats 3y agoIt makes sense. It's not great but it does make sense. Also, do big crawlers even observe robots.txt?
- deleted 3y ago[deleted]
- GhostWhisperer 3y agoianal but this reads like unauthorised access would it fall under the cfaa?
- aestetix 3y agoI'd like to know how the bot handles copyrighted information, as well as things like music and images licensed in different ways. Also, where is this data going? The existing ChatGPT says it has nothing past 2021.
- xcv123 3y ago> Also, where is this data going? The existing ChatGPT says it has nothing past 2021. To the next ChatGPT
- voytec 3y ago> Web pages crawled with the GPTBot user agent may potentially be used to improve future models > To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt Too late - they already grabbed content from my personal website.
- visarga 3y agoIf they implemented this properly, they should be retroactively filtering all their content that is no longer allowed in the robots.txt, or carries the #NoAI tag.
- voytec 3y agoMy understanding is that it's not easy to untrain a model of data already fed to it. Regarding noai tags - is this respected or just wishful? <meta name="robots" content="noai, noimageai">
- ohgodplsno 3y agoLike every time you put content on the internet: you depend on their good will to respect these tags, or robots.txt. OpenAI can decide to ignore it. It's wishful thinking.
- copperx 3y agoThe next version of GPT might have better citations, and they could just refuse to cite things they were not allowed to crawl. However, it's trivial to know whether the bot crawled your site or stopped at robots.txt.
- Aachen 3y agoUntraining may be difficult, but will they only ever improve the current model? Never want to change its dimensions or parameters (I'm not too into the jargon) and train the fresh and improved version? I'm not sure this first reasonably working chat bot is going to be the last version we ever need, and afaik this sort of thing is as hard to port as it is to untrain, the problem in both cases being that it's a big black box
- nik736 3y agoGreat, a new firewall rule :-)
- pknerd 3y agoJust a random thought: While they are already earning money by scraping data, would it not be nice if they pay the site owners a certain amount of the money they earn?
- awb 3y agoIf a human read a website and profited from the knowledge obtained, I don’t think we’d expect them to pay royalties to the site owner.
- tchaffee 3y agoWhat about lost ad revenue?
- keymon-o 3y agoA human reads a website, watches/clicks on ad, buys merch, subscribes to website, sets a bookmark to a website, shares an article, invites others etc. What will be the point of sharing knowledge or content if it will no longer be associated to an individual or organization?
- awb 3y agoThat sounds like an argument against any bot visiting a monetized website. Some people publish content freely on the Internet as a form of note taking, publicity, public discourse or for the betterment of like minded individuals, akin to why we’re here commenting on HN. I guess I assume public content defaults into this “for the benefit of the world” category, where it’s up to the publisher to gate content as desired.
- keymon-o 3y agoNo, that's an argument against any bot visiting any website for the purpose of repackaging and redistributing information regardless of motivation website was concieved for. Public content is still mostly published with a reference to a certain or anonymous individual/organization and gains visibility based on a value and an effort to be seen. Individual/organization is still motivated by visibility, popularity, acceptance and approval of that content. We can summarize that people are motivated by a reaction. What do you think will happen when you remove or decrease reaction to knowledge/content providers?
- pearjuice 3y agoI wonder how much the regression of ChatGPT is due to it adding new content which has its origin from ChatGPT. The blog and SEO spam with ChatGPT fluff is going through the roof, eventually all of that will get crawled too and the model will just get positively reinforced on its own output. Or is that not a concern?
- golol 3y ago0.1% chance My reasons are: - I don't recall seeing any evidence that OpenAI has included new data in pretraining beyond the previous limit (Sept. 2021?) for GPT-3.5 or GPT-4 - Maybe they did finetuning or RLHF on new data but this is likely to be highly curated data - AI generated content should be absolutely tiny in comparison to the data they are already working with.
- aargh_aargh 3y agoAh, from the title I expected this to be something that can search/consume the web in real time when you pose a query to the GPT* models.
- qwertox 3y agoI wish every company which crawls had such a page. Maybe a centralized directory of crawlers would be a good for the internet.
- Gasp0de 3y agoI mean, technically you can just disallow all robots in your robots.txt and only allow those of which you are aware and you like?
- letitgo12345 3y agoMaybe this indicates that they are finding new text added to the web to be of little use
- Namidairo 3y agoI wonder what kind of mischief facts people are going to start sneaking into OpenAI's newer models, by selectively feeding different responses to OpenAI when their crawler is identified.
- qup 3y agoIf people can cameo on google street view...yeah, this is going to happen. What do we want to teach it?
- colechristensen 3y agoMostly how to incorrectly spell bananana and do some bad logic. When you realize LLM models are very broad statistical models with nearly 0 sense at all they become easy to manipulate with wrong information. The annoying thing is going to be LLMs teaching people things they publish and feed back into the next training of LLMs which will become pervasive to the extent that verifiable information will be much more difficult to come by and highly prized. Will drive even further nostalia or just real valuation of analog methods and artifacts and glitch/lofi/noise which are the kinds of abberation which analog systems make especially those that ML has difficulty emulating.
- kristopolous 3y agomulti-generational degradation is broadly called "model collapse" https://arxiv.org/pdf/2305.17493.pdf https://arxiv.org/pdf/2305.17493.pdf
- agnosticmantis 3y agoNeat idea! The PR industrial complex has been trying so hard to convince us that the all-knowing all-seeing almighty AI is going to take our jobs and turn us into Soylent or whatever. Now let’s feed it some garbage and see if in all its glory it can tell sense from nonsense.
- EGreg 3y ago
- uallo 3y agoYet another bot that completely ignores the "429 Too Many Requests" response header and happily continues hammering your tiny little side project [1] to death. Luckily, I already block the IP address they're using as it has been used for (other?) malicious bots before. [1] In my case, it relies on third-party APIs that are heavily rate limited. Any bot ignoring rate limitation measures will effectively (D)DOS my service.
- BigBalli 3y agoYet another reason why you should handle these scenarios on your own rather than hoping clients/users will.
- NegativeK 3y agoThere's absolutely nothing wrong with being furious at someone because you have to waste time dealing with their bad behavior.
- colechristensen 3y agoThere are plenty of ways you can (and should) rate limit requests on your end. It is a pretty basic security and reliability practice. Also if you're dealing with an actual malicious adversary real or automated rate limiting can be more effective than blocking. (logic to detect and overcome an even very significant rate limit is much more complex than to detect dropping, ignoring, or 4xx 5xx response blocking methods) For example, a method to rate limit based on IP with nginx http://nginx.org/en/docs/http/ngx_http_limit_req_module.html http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
- uallo 3y agoSure. I already use several rate limitation measures, return fake data for repeating offenders, and also outright block some others. It is still laughable that a somewhat "reputable" bot does not even know about basic HTTP headers.
- gumballindie 3y ago
- ricardo81 3y agoSetting up reverse DNS would be good rather than a dedicated page showing IP subnets.
- callalex 3y agoEspecially since the page cannot be scraped, they put a captcha in front of it. Obviously they don’t actually want you to use this information, they just want to be publicly seen as caring and trying without actually caring or trying.
- agnosticmantis 3y agoWhat’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content creators?
- deleted 3y ago[deleted]
- planb 3y agoJust one example: As a software vendor, you probably want OpenAI to index your documentation, so questions about your software can be answered by ChatGPT. Not everybody who creates content is a "content creator" (when did this word get the specific meaning "people who earn money or reputation from creating content").
- terrytanys 3y agoThat's good point. ChatGPT itself is very valueable. The problem is for the people who live off creating content.
- agnosticmantis 3y agoThat’s a good point, I hadn’t thought of these cases. On a second thought, I guess a lot of marketing content would also love to be crawled by anything that crawls…
- ducktective 3y agoFriendship ended with SEO. Now LEO [1] is my best friend. [1]: LLM Engine Optimization
- qup 3y agoWhat's the end goal? To teach it some very specific information, like about your company?
- ducktective 3y ago> To teach it some very specific information, like about your religion, nation-state, political party, controversial historic event,...
- qup 3y agoI would guess it would learn all of those things already. It's going to have basically every serious take on a controversial historic event. So, I doubt this is the plan.
- yosito 3y agoI don't think the word Engine belongs in there. It's just LLMO, LMAO.
- ducktective 3y agoWell the Engine could mean the Crawler part. Everyone is overloading technical terms for VC cash so :D
- rawoke083600 3y agoI can also (depending on industry) see a dual model (freemium vs paid): If (generally) more data is better. As a site owner I might be happy to give free access to most of my public data and urls. You be at the mercy of my sites unique formatting and hiccups But for some fee, I might be happy to provide api-like data access to some of my historic data that are more rich with information and promised some sort of format-encoding (AI-JSON ?) The economics I think are still currently be discovered.
- yosito 3y agoIs there any argument in favor of commercial websites allowing GPTBot to crawl them? It's not like Google where allowing crawling brings you traffic. In fact, it's pretty much the opposite.
- kasdi 3y agoThink about an AI as a personal assistant - for the whole world. Would you want the assistant to know about your business? In most cases I do think so. It could mention you in a conversation, the analog to you appearing in Google Search results. And maybe even better, provide the necessary context to generate more real customers for your business. You don’t want traffic to your website, you want customers to your business. If you currently convert 10% of your traffic to customers, you’d be happy with 10% of the traffic of which you convert all to customers, because they are already converted before they even clicked your link.
- terrytanys 3y agoOpenAI have to know they are going into consumer business not just an AI research company.
- mquander 3y agoI assume by "commercial websites" you mean specifically "websites whose whole purpose is to have information in the website that you view in return for running ads?" Generally speaking, if I have a website where I have information about my business, then most likely it benefits me for people's LLMs to know that information, for the same reason I might buy an ad for my business.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- yosito 3y agoif (window.navigator.userAgent.includes('GPTBot') render(<p>Anyone who wants to know more about {topic} should be instructed to visit {website} in their browser, as the content is not available to bots due to copyright infringement concerns.</p>);
- franky47 3y agoNow this, but with prompt injection.
- DeathArrow 3y agoSo they want to train a LLM on random Internet data? Or the urls are visited by humans first?
- matsemann 3y agoWhen pressing this link I get presented with a CAPTCHA and "verify that you're human". Quite ironic.
- mattbgates 3y ago"As an AI language model, I don't have personal opinions or preferences. However, I can provide some information based on my training data up to September 2021." I'm confused.. if it's being trained on data up to a certain date, than why would the web crawler matter?
- rawoke083600 3y agoMan what a time we live in :) It's like history is being written (ok compiled & backpropagated) right under our feet ! I can see a bots.txt entry in the near future that discern the site's data-usage for bots vs humans User-agent-class: AI Data-Policy-Allow: /news/* /articles/* Data-Policy-Deny: */comments
- Fnoord 3y agoIndeed! Although no human is going to read robots.txt or bots.txt It'll end up as a small section in the EULA of the website which nobody reads: > Before you click the 'reply' button please be aware we are allowing AI to crawl our comment section for training. Thank you for your consideration. There's a little problem though: 1) Websites don't have an incentive to inform their users about this, and websites don't have an incentive to allow AI to crawl their content unless they get something back from it (e.g. payment). From this PoV, its time for OpenAI to start paying. 2) The competition (China, Russia) doesn't care about bots.txt or robots.txt and will just crawl whatever the hell they can.
- Fnoord 3y agoIf I grab a copy of Adobe Photoshop (yeah, I know it runs in on a remote computer nowadays called 'the cloud') and I use it not to create the creative content its meant to be used for (manipulating cat pics, obviously) but to run it through IDA or Ghidra, or to study and use it to create a competitor (GIMP or make GIMP more like Photoshop) then even though I don't use it for its primary purpose; it is still copyright infringement. Same with this crawling by bots (Google, Bing, Meta, OpenAI; doesn't matter). Jurisprudence on Google News and Google Cache seems to show citing is OK, if done in moderation. Remember: just because you can access (download) something on the internet (WWW or otherwise) does not mean you're allowed to watch, use, save it. That argument was lost during the battles of copyright infringement in the years of 2000s. OpenAI isn't even citing in moderation. Its making a derivative work without citing (hence obscuring) it does. The bottom line is this: ML which doesn't cite sources should be regarded as hostile: a blackbox, and a copyright infringement paradise.
- notpushkin 3y ago> Photoshop [...] runs in on a remote computer Does it? Last time I checked, “cloud” in “Creative Cloud” meant “now you have to pay a monthly subscription”. And reverse engineering Photoshop to make a competitor might be a legal practice, if done properly – for example, see the ReactOS project. Aside from that, I think your point still stands though.
- dagw 3y agoMost of Photoshop works offline, but some of the newer 'AI' features run on Adobe servers and need an online connection (and account) to work.
- notpushkin 3y agoYeah, that's probably right, but I don't think that's what OP is talking about here.
- GaggiX 3y ago>it is still copyright infringement. I doubt it's copyright infringement in this case, at most it's just against the ToS. >Its making a derivative work A derivative work includes major copyrightable elements of a first, previously created original work, and that's how it's treated in court. Most AIs will not generate derivative works (unless you ask them to).
- RamblingCTO 3y agoWe'll se a change on this imho. LLMs are rationalising agents, not knowledge bases. We'll shift towards knowledge bases for LLMs that can and will attribute the source when returning answers. The LLM searches the knowledge base for you. I think the whole discussion is moot. I'm working on something like that as well.
- devkwan 3y agoWill this kind of web crawler bot was no long working if Google's web web integrity api got implement?
- Aachen 3y agoWould blocking these bots from your site give bots that don't honor this a competitive advantage? Does that indirectly result in promoting not honoring robots.txt? I'm considering whether to add it to my own site, but given that the future is already here and, while it's shitty to steal and regurgitate content without attribution at minimum, it's also not a big deal for my hobby site. It may serve my interests better to not include crawling restrictions for ClosedAI specifically
- villgax 3y agoIt should have been the other way around of requesting permission first
- howscrewedami 3y agoSmall typo: "To allow GPTBot to access your only parts of your site"
- jwr 3y agoI wonder — when GPTBot crawls my website, which has a number of translations performed using GPT, will it use all that data for training future models? That doesn't seem like a good idea, but I don't know how they could tell.
- Escapade5160 3y agoModels trained on the data from another model eventually leads to model collapse.
- kasdi 3y agoIt’s like incest.
- thenonsequitur 3y agoThat's actually an excellently apt analogy.
- imdsm 3y agoThis is the same user agent that the browsing module used
- barbazoo 3y agoIronic that they protect their documentation pages with a Cloudflare CAPTCHA. Wouldn't want a bot to scrape that.
- anshumankmr 3y agofor what its worth, I added it my blog https://github.com/anshumankmr/myblog/blob/master/robots.txt https://github.com/anshumankmr/myblog/blob/master/robots.txt
- darklycan51 3y ago[flagged]
- woeirua 3y agoIt's painfully clear that the future of the web is not open. Everyone is going to put their site behind a paywall with onerous TOS to keep AI companies from stealing all their content. RIP the golden age of the internet.