Looking at this court case it seems public data is scrape-able and usable (even sell-able) https://techcrunch.com/2024/02/26/meta-drops-lawsuit-against-web-scra

Looking at this court case it seems public data is scrape-able and usable (even sell-able) https://techcrunch.com/2024/02/26/meta-drops-lawsuit-against-web-scraping-firm-bright-data-that-sold-millions-of-instagram-records/ https://techcrunch.com/2024/02/26/meta-drops-lawsuit-against... HN's /robots.txt doesn't disallow scraping. I wouldn't expect a FAQ. What data is collected and how used is usually part of terms-of-service so I'd expect to be a small part of https://www.ycombinator.com/legal/ https://www.ycombinator.com/legal/ The Reddit deal is probably a short-cut to deliver data in structured format and saves both parties money compared to scraping millions (billions?) of pages. Wikipedia has a program to deliver data for such customers, too. https://enterprise.wikimedia.com/ https://enterprise.wikimedia.com/