>One of the difficulties with these training datasets is the currently understood rules around web scraping. The current legal precedent [0] is that web scrapin

>One of the difficulties with these training datasets is the currently understood rules around web scraping. The current legal precedent [0] is that web scraping is perfectly legal, despite what is in the websites terms of service, "licence" or robots.txt. If a human can navigate to it freely, you can scrape it using automated means. So... the HN crowd seems perfect to ask, since everyone's always clamoring for a Goodreads alternative: where is the line between "data" and "content" (on public pages)? AFAICT, Goodreads has been entrenched for so long because their ToS doesn't allow anyone else to use their book/reviews data, which makes quite a large moat. This comment would make me think it's legal to scrape that moat.