4 ms·
Or you know you can just ignore robots.txt like everyone else. What do you think robots.txt does exactly?
by JSDevOps 2y ago
Or you know you can just ignore robots.txt like everyone else. What do you think robots.txt does exactly?
- ngkw 2y agoThanks for your comment, JSDevOps! I appreciate your input. I understand that robots.txt can be bypassed by some. The idea behind "Copy from URL" is to address a specific need when interacting with AI chatbots through their websites, like ChatGPT or Claude, where we don't have control over robots.txt settings (that's up to OpenAI or Anthropic). If you use the API, you can just ignore this, but that's not the case I'm targeting. This tool is designed for those times when you want to quickly copy content from a site to use in your AI conversations. It's just a simple way to make that process smoother. Hope that clarifies the intention behind the tool!
- ngkw 2y agoTo be honest, I'm really frustrated that OpenAI and Anthropic don't open certain links. It's not just because of robots.txt restrictions, but also due to copyright issues and their excessive caution regarding media content. Isn't it annoying to deal with that? So, the idea was to create a tool that allows us to easily bypass these limitations ourselves.
- Juliate 2y agoAka "isn't it annoying to respect the rule of law?" Reminded of this post by https://twitter.com/GergelyOrosz/status/1808515221817885017 https://twitter.com/GergelyOrosz/status/1808515221817885017