3 ms·

Just for context, there is a new post about OpenAI DDoS'ing half the internet every other day on hn https://news.ycombinator.com/item?id=42660377 https://news.

• elp 2y ago
  If you ask OpenAI to stop, using robots.txt, they actually will. What Aaron was trying to achieve was great, how he want about it is what ruined his life.
  • omnimus 2y ago
    Nobody really asked Aaron about anything they collected more evidence and wanted to put him to jail. School should have unplugged his machine bring him for questioning and tell him not to do that.
  • LunaSea 2y ago
    It is a well known fact that OpenAI stole content by scraping sites with illegally uploaded content on it.
  • breck 2y ago
    [dead]
• alphan0n 2y ago
  Just for context, the author of the second link in your comment verifiably lied about blocking crawlers via robots.txt CommonCrawl archives robots.txt For convenience, you can view the extracted data here: https://pastebin.com/VSHMTThJ https://pastebin.com/VSHMTThJ You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here: https://index.commoncrawl.org/ https://index.commoncrawl.org/ The index contains a file name that you can append to the CommonCrawl url to download the archive and view. More detailed information on downloading archives here: https://commoncrawl.org/get-started https://commoncrawl.org/get-started From September to December, the robots.txt at wiki.diasporafoundation.org contained this, and only this: >User-agent: * >Disallow: /w/