3 ms·

Despite being only 1.1B params, SantaCoder outperforms Facebook's InCoder (6.7B params) and Salesforce's CodeGen-Multi-2.7B. Paper: https://hf.co/datasets/bigc

by moyix 4y ago

Despite being only 1.1B params, SantaCoder outperforms Facebook's InCoder (6.7B params) and Salesforce's CodeGen-Multi-2.7B. Paper: https://hf.co/datasets/bigcode/admin/resolve/main/BigCode_SantaCoder.pdf https://hf.co/datasets/bigcode/admin/resolve/main/BigCode_Sa... Dataset search: https://huggingface.co/spaces/bigcode/santacoder-search https://huggingface.co/spaces/bigcode/santacoder-search Model weights: https://huggingface.co/bigcode/santacoder https://huggingface.co/bigcode/santacoder

isoprophlex 4y ago
Any idea how this, and those other two models, would compare to GitHub Copilot?
- moyix 4y ago
  Based on the reverse engineering done by Parth Thakkar [1], the model used by Copilot is probably about 10x as large (12B parameters), so I would expect Copilot to still win pretty handily (especially since the Codex models are generally a lot better trained than Salesforce CodeGen or InCoder). It's also a little bit hard to compare directly because as Parth documents, there are a lot of extra smarts that go into Copilot on the client side. The SantaCoder paper does have some benchmarks on MultiPL-E though, so you could compare them to the Codex results on that benchmark reported here (but keep in mind that code-davinci-002 is probably even larger than the model used by Copilot): https://arxiv.org/abs/2208.08227 https://arxiv.org/abs/2208.08227 [1] https://thakkarparth007.github.io/copilot-explorer/posts/copilot-internals https://thakkarparth007.github.io/copilot-explorer/posts/cop...
  - obastani 4y ago
    Just out of curiosity, in what sense is Codex is better trained than CodeGen?
    - moyix 4y ago
      OpenAI hasn't said exactly how they trained code-davinci-002 so this is speculative, but I'm reasonably sure it was trained on more data and languages than CodeGen and for longer. It was also trained using fill-in-the middle [1]. [1] https://arxiv.org/abs/2207.14255 https://arxiv.org/abs/2207.14255
- karmasimida 4y ago
  Not even close
Vetch 4y ago
SantaCoder's impressive but that's probably misleading. It's reported that incoder doesn't generate as diverse a set of solutions but does do better at the ones it generates. This means it performs well at a lower number of tries when compared to other similar models, which is what matters in practice. The numbers reported here required many trials. With a fuller context and just a handful of tries, it's unlikely that 6.7B version of incoder will be outperformed by SantaCoder.
- moyix 4y ago
  The amount of context is dictated by the benchmark, but I agree it would be good to see what the pass@1 and pass@10 numbers are – if the raw data is available somewhere that can easily be computed.

by moyix 4y ago

isoprophlex 4y ago

Any idea how this, and those other two models, would compare to GitHub Copilot?

moyix 4y ago
Based on the reverse engineering done by Parth Thakkar [1], the model used by Copilot is probably about 10x as large (12B parameters), so I would expect Copilot to still win pretty handily (especially since the Codex models are generally a lot better trained than Salesforce CodeGen or InCoder). It's also a little bit hard to compare directly because as Parth documents, there are a lot of extra smarts that go into Copilot on the client side. The SantaCoder paper does have some benchmarks on MultiPL-E though, so you could compare them to the Codex results on that benchmark reported here (but keep in mind that code-davinci-002 is probably even larger than the model used by Copilot): https://arxiv.org/abs/2208.08227 https://arxiv.org/abs/2208.08227 [1] https://thakkarparth007.github.io/copilot-explorer/posts/copilot-internals https://thakkarparth007.github.io/copilot-explorer/posts/cop...
- obastani 4y ago
  Just out of curiosity, in what sense is Codex is better trained than CodeGen?
  - moyix 4y ago
    OpenAI hasn't said exactly how they trained code-davinci-002 so this is speculative, but I'm reasonably sure it was trained on more data and languages than CodeGen and for longer. It was also trained using fill-in-the middle [1]. [1] https://arxiv.org/abs/2207.14255 https://arxiv.org/abs/2207.14255
karmasimida 4y ago
Not even close

Vetch 4y ago

SantaCoder's impressive but that's probably misleading. It's reported that incoder doesn't generate as diverse a set of solutions but does do better at the ones it generates. This means it performs well at a lower number of tries when compared to other similar models, which is what matters in practice. The numbers reported here required many trials. With a fuller context and just a handful of tries, it's unlikely that 6.7B version of incoder will be outperformed by SantaCoder.

moyix 4y ago
The amount of context is dictated by the benchmark, but I agree it would be good to see what the pass@1 and pass@10 numbers are – if the raw data is available somewhere that can easily be computed.