6 ms·
Show HN: A High-Quality Chinese Internet Language Dataset for AI
Hello HN,
We at BAAI are proud to release the Chinese Corpora Internet (CCI) dataset v1.0.0.
CCI's standout features:
- Extensive Data: With 104GB of rich linguistic data, CCI is one of the most high-quality Chinese language datasets available.
- Broad Temporal Range: Spanning from 2001 to 2023, CCI captures a wide array of linguistic evolution and trends.
- Trusted Data Sources: All data is sourced from reputable Chinese internet platforms, ensuring authenticity and reducing noise.
- Stringent Quality Control: From cleaning to deduplication, CCI has been processed with meticulous attention to detail to maintain data excellence.
- Ethical Standards: CCI adheres to strict ethical guidelines, with sensitive content being carefully filtered out to ensure the dataset's safety and reliability.
- Anti-Cheating Measures: We've taken proactive steps to filter out benchmark datasets, ensuring that AI models trained on CCI are evaluated fairly and accurately.
Get started with CCI:
- BAAI Open Data Repository: https://data.baai.ac.cn/details/BAAI-CCI
- HuggingFace: https://huggingface.co/datasets/BAAI/CCI-Data
We're looking forward to the community's feedback and the advancements in AI that CCI will facilitate. Your engagement is key to the success of this initiative.
- yorwba 3y agoHi BAAI, is your dataset compliant with the draft "Basic security requirements for generative artificial intelligence service" [sic] https://www.tc260.org.cn/upload/2023-10-11/1697008495851003865.pdf https://www.tc260.org.cn/upload/2023-10-11/16970084958510038... especially as far as the data source requirements in section 5 are concerned?
- BAAIBeijing 3y agoThank you for reaching out with your inquiry regarding our dataset's compliance with the draft "Basic security requirements for generative artificial intelligence service." I can confirm that our dataset adheres to the data source security requirements outlined in section 5.1 and the corpus content security requirements detailed in section 5.2 of the document. Additionally, we have implemented safety filters to address the main security risks associated with the corpus and generated content, as listed in the annex of the document, which outlines 31 types of potential risks across five categories. Furthermore, our user agreement explicitly includes terms that mandate the secure use of our services. We take these measures seriously to ensure that our dataset and AI services are not only compliant with current regulations but also maintain the highest standards of data security and integrity. Please let us know if you require any further information or clarification on this matter.