2 ms·

If you haven't noticed, bigcode has also released "The Stack", a 3TB (!) dataset of code (https://huggingface.co/datasets/bigcode/the-stack https://huggingface.

If you haven't noticed, bigcode has also released "The Stack", a 3TB (!) dataset of code (https://huggingface.co/datasets/bigcode/the-stack https://huggingface.co/datasets/bigcode/the-stack). Also, they have a special policy where "The Stack" only contains permissively-licensed code, and anyone can see if their data is included and opt-out. It's true they haven't actually trained a model on the stack, and this is...not copilot. But I like what they're doing and I think it should be appreciated. Honestly, I may even say they're doing with code what stability.ai is doing with images.

moyix 4y ago
> It's true they haven't actually trained a model on the stack What do you mean? SantaCoder is trained on The Stack: > Dataset > The base training dataset for the experiments in this paper contains 268 GB of Python, Java and JavaScript files from The Stack v1.1 (Kocetkov et al., 2022) after removing data from opt-out requests, near-deduplication, PII-redaction (see Section 4), and filtering based on line-length and percentage of alphanumeric characters. This dataset was also decontaminated by removing files that contained test-samples from the following benchmarks: HumanEval (Chen et al., 2021), APPS (Hendrycks et al., 2021), MBPP (Austin et al., 2021) and MultiPL-E (Cassano et al., 2022). It's definitely not on par with Copilot yet, but SantaCoder is a trial run for a larger & better model that they're planning to train in 2023. Stay tuned! :)