3 ms·

OpenAI scraped aggressively for years. Why should others put themselves behind an artificial moat? If you want to block access to a site, stop relying on arbit

OpenAI scraped aggressively for years. Why should others put themselves behind an artificial moat? If you want to block access to a site, stop relying on arbitrary opt-in voluntary things like user agent or robots.txt. Make your site authenticated only, that’s literally the only answer here.

diggan 2y ago
> OpenAI scraped aggressively for years. Why should others put themselves behind an artificial moat? Not saying I agree/disagree with the whole "LLMs trained on scraped data is unethical", but this way of thinking seems dangerous. If companies like Theranos can prop up their value by lying, does that make it ok for Theranos competitors to also lie, as another example?
- qup 2y ago
  Theranos was engaged in fraud. There's no way to stretch the situations for a comparison
blackeyeblitzar 2y ago
Agree - the first movers who scraped before changes to websites terms and robots files shouldn’t get an unfair advantage. That’s overall bad for society in terms of choice and competition
- hipadev23 2y ago
  Website terms for unauthenticated users and robots.txt have zero legal standing, so it doesn’t matter how much hand-wringing people like the OP do. It would be irresponsible as a business owner to hamstring themselves.
  - rknightuk 2y ago
    Then they should just say that outright instead of pretending they right thing.
    - lolinder 2y ago
      They're not lying, you just misunderstood their docs [0]. > To provide the best search experience, we need to collect data. We use web crawlers to gather information from the internet and index it for our search engine. > You can identify our web crawler by its user agent To anyone who's familiar with web crawling and indexing, these paragraphs have an obvious meaning: Perplexity has a search engine which needs a crawler which crawls the internet. That crawler can be identified by the User-Agent PerplexityBot and will respect robots.txt. Separately, if you give Perplexity a specific URL then it will go fetch the contents of that URL with a one-off request. That one-off request does not respect robots.txt any more than curl does, and that's 100% normal and ethical. The one-off request handler isn't PerplexityBot, it's a separate part of the application that's probably just a regular Chrome browser that issues the request. [0] https://docs.perplexity.ai/docs/perplexitybot https://docs.perplexity.ai/docs/perplexitybot
Findecanor 2y ago
Most creators still want search engines to index their content to help them get publicity for their work. ... And what would you do to find out if an image or piece of text (that you didn't create) has been plagiarised by an AI model? You search for it on a search engine, of course!

diggan 2y ago

> OpenAI scraped aggressively for years. Why should others put themselves behind an artificial moat? Not saying I agree/disagree with the whole "LLMs trained on scraped data is unethical", but this way of thinking seems dangerous. If companies like Theranos can prop up their value by lying, does that make it ok for Theranos competitors to also lie, as another example?

qup 2y ago
Theranos was engaged in fraud. There's no way to stretch the situations for a comparison

blackeyeblitzar 2y ago

Agree - the first movers who scraped before changes to websites terms and robots files shouldn’t get an unfair advantage. That’s overall bad for society in terms of choice and competition

hipadev23 2y ago
Website terms for unauthenticated users and robots.txt have zero legal standing, so it doesn’t matter how much hand-wringing people like the OP do. It would be irresponsible as a business owner to hamstring themselves.
- rknightuk 2y ago
  Then they should just say that outright instead of pretending they right thing.
  - lolinder 2y ago
    They're not lying, you just misunderstood their docs [0]. > To provide the best search experience, we need to collect data. We use web crawlers to gather information from the internet and index it for our search engine. > You can identify our web crawler by its user agent To anyone who's familiar with web crawling and indexing, these paragraphs have an obvious meaning: Perplexity has a search engine which needs a crawler which crawls the internet. That crawler can be identified by the User-Agent PerplexityBot and will respect robots.txt. Separately, if you give Perplexity a specific URL then it will go fetch the contents of that URL with a one-off request. That one-off request does not respect robots.txt any more than curl does, and that's 100% normal and ethical. The one-off request handler isn't PerplexityBot, it's a separate part of the application that's probably just a regular Chrome browser that issues the request. [0] https://docs.perplexity.ai/docs/perplexitybot https://docs.perplexity.ai/docs/perplexitybot

Findecanor 2y ago

Most creators still want search engines to index their content to help them get publicity for their work. ... And what would you do to find out if an image or piece of text (that you didn't create) has been plagiarised by an AI model? You search for it on a search engine, of course!