I appreciate that attitude and value it myself, but I like to point out that it is not without risk. If the world around HN (including its community) changes,

I appreciate that attitude and value it myself, but I like to point out that it is not without risk. If the world around HN (including its community) changes, stasis can damage or kill it as well. Specifically regarding the issue of the original posting: - HN is already an important data source for large language model training. [1] - To the best of my knowledge there is no freely downloadable and current data dump of HN. [2] - The HN-API does not offer all the data that scraping can get. For example, if a post had ever hit the front page or the highest front page position reached, is an interesting data point that is missing. - The Algolia-HN-API has the same limitations. In my opinion this will lead to increased usage of the API and increased scraping which all costs money. HN might be forced to find a solution for this. [1] For example, the RefinedWeb paper lists HN as one of only 12 websites that were excluded. From what I understand, it was excluded because it went into the final dataset unvetted. RefinedWeb was used for the Falcon model. https://arxiv.org/pdf/2306.01116.pdf https://arxiv.org/pdf/2306.01116.pdf [2] The closest thing is probably the Google BigQuery "bigquery-public-data.hacker_news" dataset. It claims to be updated daily, but really is from late September 2022. Also I could not find the download link which other data sets offered on BigQuery have. Does anyone know if I can download the complete thing anyhow? https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=hacker_news&page=dataset&project=hacker-news-314218&ws=!1m4!1m3!3m2!1sbigquery-public-data!2shacker_news https://console.cloud.google.com/bigquery?p=bigquery-public-...