8 ms·
OpenAI Finally Allows ChatGPT Complete Internet Access
- rideontime 3y agoWill blocking User-agent: GPTBot in robots.txt work for this too?
- judge2020 3y agoThis time it's not scraping the internet (like a robot) but actually acting as a direct user agent for the human typing their prompt, so I wouldn't be against them ignoring robots.txt.
- JohnFen 3y agoIt's not a crawler, but it seems likely that OpenAI would take the point of view that as long it's going to a website, it may as well keep a cached copy to use for training later.
- stingraycharles 3y agoThat makes no sense. It’s still automated and not an actual human, and it’s still distilling information from the internet.
- rolisz 3y agoYour browser is also automating things, you're not resolving DNS and writing all the http requests by hand.
- worthless-trash 3y agoI have found the next topic for my blog post. Challenge accepted!
- jtbayly 3y agoSo because something is automated somewhere, nobody has to follow robots.TXT anymore?
- Spivak 3y agoYou have it backwards, when a human initiates the request, like clicking on a link, your browser (rightfully) will ignore robots.txt. Unattended requests like scrapers do respect robots.txt. This is a case where ChatGPT is acting more like a browser with a funny way of displaying the final result. Each request is initiated by a human so it would likely be reasonable to ignore it in this case.
- luke-stanley 3y agoI thought the same but OpenAI may train on user chats by default. Maybe if training's off, they could ignore robots.txt, or they could flag the content to be skipped.
- setuid9002 3y agoWait, so a simple web scraper script has to comply with robots.txt. But if I want to completely ignore the robots.txt, I only have to make my script more complicated (ChatGPT)?
- seanthemon 3y agoAnd make it complicated enough (a human) and no .txt can stop them!
- kelseyfrog 3y agoYeah, it's like one weird trick for web scraping.
- pixl97 3y agoI'd like to consider this a difference between script action and user action. For example if you make a web page a user pulls up that calls another webpage, is that a user action, a script action, a mix of both? I personally would consider it a user action.
- JohnFen 3y agoNobody actually has to comply with robots.txt.
- karaterobot 3y agoMy understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product, if not duplicated and recorded wholesale. The important idea seems to be ownership, not the ability to browse. If OpenAI had two agents, one with no memory, and one with a memory, that would be better: you could disallow ChaptGPT-storage and allow ChatGPT-user, for example. Barring that, I'd be afraid allowing ChatGPT access to my website means my website is now part of the ChatGPT corpus.
- spott 3y agohttps://platform.openai.com/docs/plugins/bot/chatgpt-user-and-gptbot https://platform.openai.com/docs/plugins/bot/chatgpt-user-an... There are two agents, but apparently you can't allow one but disallow another.
- JohnFen 3y ago> My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product Not really. That use case is done, of course, but the primary purpose of robots.txt is to help crawlers by indicating what parts of the website are appropriate to be searched and what parts aren't. Robots.txt is not intended primarily as a means to defend a site against crawlers. That's why it relies on the goodwill of crawlers to work.
- tomjen3 3y agoAs I recall it it was also to block access to scripts, so it wouldn't e.g get stuck pulling pointless stuff out of your local search page.
- simonw 3y agoIt uses a different user agent, but yeah it still obeys robots.txt. https://platform.openai.com/docs/plugins/bot https://platform.openai.com/docs/plugins/bot User-agent: ChatGPT-User Disallow: /
- tiahura 3y agoDallE is so nerfed now it balks at ninja battle in the style of bob ross.
- ShamelessC 3y agoThat worked for me just now. I agree that it was deliberately censored though.
- Tao3300 3y agoMakes sense. Signs of human life in Bob Ross paintings come down to one person in one painting, and IIRC one instance of smoke coming out of a cabin chimney. A ninja battle in the style of Bob Ross is going to be some trees and mountains while the ninja battle is assumed to be happening out of view on the other side of it.
- twilightzone 3y agoGoogle search is dead.
- jacobtt21 3y agoI thought Google's given Bard access to the internet?? I guess people still don't think to use Bard over ChatGPT
- generalizations 3y agoEven the Bard UI is worse. Being able to read the ChatGPT output as it's generated, instead of staring at a blank screen, is a massive time saver. And, I prefer ChatGPT trying stuff rather than Bard telling me that 'it can't do that right now' and 'we're trying to get better'. I've tried Bard, it's just not worth it.
- maksimur 3y ago> Being able to read the ChatGPT output as it's generated, instead of staring at a blank screen, is a massive time saver. There's an option in Bard's settings to enable real time replies.
- airstrike 3y agoUnfortunately for Google, Bard is irrelevant by means of being late and inferior to GPT-4
- PunchTornado 3y agoMy experience is different. I think Bard is giving better answers than chatgpt so I really use it daily even more than google search.
- londons_explore 3y agoBetter than the free or the paid chatgpt?
- i_k_k 3y agoSo this means it can actually do things now, at least in principle, right?
- senectus1 3y agoNot according to ChatGPT a few seconds ago Default (GPT-3.5) User do you have access to search the internet yet? ChatGPT I do not have the capability to search the internet or access real-time information. My knowledge is based on the text that I was trained on, and my training only includes information up until September 2021. I can provide information and answer questions to the best of my knowledge up to that date, but I cannot browse the web or access current information.
- schleck8 3y agoIt's a premium feature
- simonw 3y agoAsking ChatGPT about its own capabilities rarely returns useful results. Its training data predates its creation, and the model doesn't get updated every time they ship a new feature for it. See also: https://simonwillison.net/2023/Mar/22/dont-trust-ai-to-talk-about-itself/ https://simonwillison.net/2023/Mar/22/dont-trust-ai-to-talk-...
- lsedgwick 3y agoPlease don't paste ChatGPT or BARD answers as HN comments in general? In this specific case, no, LLMs don't reliably know about themselves. They're trained on a big corpus of internet text, then trained by rough reinforcement learning to say and not say certain things about themselves, then given a little more information about themselves in system prompts.
- jtbayly 3y agoIf you’re right, then this is probably the best proof that I’ve seen that ChatGPT isn’t even remotely conscious.
- simonw 3y agoI've been relatively unimpressed with the ChatGPT browse mode. My problem with it is that it seems to run really obvious, naive searches. Most of the time I ask it something, then see what it's searching for and think "oh no, that's not going to return anything more useful than what I could have found myself". It's also pretty slow. I'm very much looking forward to having a search assistant which can go ahead and wade through eg 20 websites about home battery packs and show me a summary table of my options - ChatGPT Browse isn't quite that yet.
- zachthewf 3y agoHave you tried perplexity.ai? It has become my go-to for LLM + browsing. It is really fast and does a pretty good job with retrieval and summarization.
- simonw 3y agoNot in a few months, I should spend more time with that
- wordpad25 3y agoChatGPT, Claude and prepexity are each $20/mo Chatgpt4 seems most powerful but Claude offers unlimited context which can be used to summarize entire books Prepexity can browse and summarize web Which one should I subscribe to?
- Reviving1514 3y agoKagi ultimate is 25 a month and has Claude, GPT4, and AI web search. Been using it a month or so and it's great.
- deleted 3y ago[deleted]
- spott 3y agoDo you have a source for that? All I can find is here [0], which doesn't mention GPT4 or Claude. [0]https://help.kagi.com/kagi/plans/ultimate-plan.html https://help.kagi.com/kagi/plans/ultimate-plan.html
- singularity2001 3y agoIt's the same feature as before, just without the 'beta' toggle, right?
- ChrisRR 3y agoSo this is when GhatGPT becomes racist?
- Alifatisk 3y agoThat’s a weird take?
- bone_frequency 3y agoAlways has been. Even with all the safeguards, two days ago I was having a conversation with it and it just happily blurted out that one of the reasons Iceland has low crime is because its population is racially homogeneous.
- michelb 3y agoThe amount of sites I regularly use with ChatGPT that are blocking AI agents has increased to the point where this feature is not that useful for me anymore. I can only see that amount increasing.
- shostack 3y agoIf anything, AI work feels like it has accelerated everyone with any dataset of value pulling up their drawbridge, reducing open interconnectivity of the web in hopes of charging for data access. This started with scrapers and aggregation sites and has gotten noticeably worse.
- sebosp 3y agoI wonder if AI generated web pages block AI agents from indexing their content, like, one engine indexing the other's content in a loop until the amount of digital garbage is so gigantic that is the end of the information era, or how are we ever stopping this? What is our failsafe?
- ethbr1 3y agoAI Kessler Syndrome. https://en.m.wikipedia.org/wiki/Kessler_syndrome https://en.m.wikipedia.org/wiki/Kessler_syndrome
- sebosp 3y agoIs there an approximation/ratio in which the amount of digital garbage/hallucinations online generated by AI is so big that it cannot be used to train AI itself? Like are AI companies running against the clock because, say, in 5 years the internet will be flooded by false information to such an extent that it would render the internet as an invalid training ground. In a way requiring a snapshot of the internet pre-AI, because this is click bait problem times infinity it feels like
- ethbr1 3y agoAka "t-minus how many days until OpenAi wants to buy archive.org"
- TechRemarker 3y agoTried Browse with Bing a few times, but unlike the normal conversations with Bing, the results from Browse with Bing other than being more up-to-date presumably, gave seemingly very generic answers based on if I had simply searched rather than using all it's power like normal and simply including the latest up-to-date data and took a while to run. So it's a start and hopefully that feature will improve as it enters this new stage.
- gumballindie 3y agoIs there a complete list of ip addresses openai uses for scraping? I suppose they dont honour robots.txt, despite claiming to do so, and this may be one way to reliably block them.
- londons_explore 3y agoI've seen no suggestion of them not honouring robots.txt... I suspect that any case of them not honoring it is probably due to your site content being available on archive.org or common crawl or some other service.
- kelseyfrog 3y agoIs there an open license or ToS that would disallow OpenAI access?
- gumballindie 3y agoApparently openai and the rest of defendants claim that if it’s on the internet it’s “fair use”. Meaning they’ll do all they can tk steal your work regardless of licensing or robots.txt rules.
- kelseyfrog 3y agoThat's a funny argument. It's a lot like saying, "if it's on Github, I can ignore your LICENSE file" or "if it's on DA I can ignore your CC-NC license." Both of these seem technically enforceable (if not expensive) so I'm not quite understanding what OpenAI is grounding their justification in. It seems much more likely that OpenAI stands to make a lot of money by arguing that it's fair use and then back filling in an argument rather than making one from first principles.
- gumballindie 3y agoA”I” depends on data. The more the better. That’s why a lot of these people push for a disregard of people’s property (digital content, behaviour data, etc). Problem is in stealing all of it they will demotivate people from creating quality content and even alienate them from the dead internet. OpenAI leads the cult.
- charcircuit 3y agoThis is only bing search. It can't yet do things like ask a question on stack overflow, or join a discord server to ask a question yet.
- rsolva 3y agoAre there any projects that enable local AIs (like ollama) to fetch/process information from the web?
- ty29423i434 3y ago[flagged]
- enoch2090 3y agoWhen I use ChatGPT browse it's mainly two cases: 1. I don't want to read through all materials in that topic but want an overview. 2. I can't describe the topic with the precise language for it, hoping ChatGPT can translate my naive description to the vocabulary for that field. ChatGPT does bad in both of these cases. For (1) it just randomly browses 3~4 search results, as for (2) it always try to search in my original vocabulary. Why don't I search myself then, but to wait for the slow response?
- jrflowers 3y agoFinally software to explain ladder theory in timecube language and vice versa
- Alifatisk 3y agoGoogle Bard have had Internet access for a long time.