3 ms·
I wonder what code corpus the SalesForce CodeGen model (which this uses) was trained on.
by pabs3 4y ago
I wonder what code corpus the SalesForce CodeGen model (which this uses) was trained on.
- jorams 4y agoFrom the paper[1]: > The family of CODEGEN models is trained sequentially on three datasets: THEPILE, BIGQUERY, and BIGPYTHON. > The natural language dataset THEPILE is an 825.18 GiB English text corpus collected by Gao et al. (2020) for language modeling. The dataset is constructed from 22 diverse high-quality subsets, one of which is programming language data collected from GitHub repositories with >100 stars that constitute 7.6% of the dataset. > The multi-lingual dataset BIGQUERY is a subset of Google’s publicly available BigQuery dataset, which consists of code in multiple programming languages such as C, Python, Ruby. For the multi-lingual training, the following 6 programming languages are chosen: C, C++, Go, Java, JavaScript, and Python. > The mono-lingual dataset BIGPYTHON contains a large amount of data in the programming language, Python. We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code in October 2021. [1]: https://arxiv.org/abs/2203.13474 https://arxiv.org/abs/2203.13474