5 ms·
Wow, it seems most websites that block it, block it because they data is used for training AI
by emi2k01 3y ago
Wow, it seems most websites that block it, block it because they data is used for training AI
- 1vuio0pswjnm7 3y agoBut CCBot has been a very common robots.txt entry for a long time, long before OpenAI of GPTBot existed, long before site operators were even aware CC was being used to train AI. This claim can be verified. Archive.org has good collection of robots.txt files. CC itself has collections of robots.txt files. If CCBot has been in most robots.txt files since 2008, it would be difficult to argue the reason has been to stop usage of crawls for training AI.