3 ms·

How did OpenAI crawl the web so effectively without getting blocked?

• cratermoon 3y ago
  It didn't. Others did it, OpenAI used it. https://commoncrawl.org/ https://commoncrawl.org/
• arthurcolle 3y ago
  Very carefully
• throwawayadvsec 3y ago
  I'm pretty sure scraping is easier than building a groundbreaking LLM used by 100M people
• JoeyBananas 3y ago
  It's not hard to scrape the internet