3 ms·
Promising. Combine that with llama or BLOOM, perhaps finetuned on code, and perhaps your own codebase/documentation, and you cold have an interesting solution.
by MayeulC 4y ago
Promising. Combine that with llama or BLOOM, perhaps finetuned on code, and perhaps your own codebase/documentation, and you cold have an interesting solution.
- jncraton 4y agoWhat is wrong with the CodeGen model that they are using? It is a reasonably large model (up to 16B params) that has already been trained on both natural language and code. I would expect it to underperform larger models, including GPT-3.5 and GPT-4, but this should still be very useful for autocomplete and boilerplate in simpler cases. It is a bit under trained compared to Chinchilla, T5, or LLaMA, but it still performs well. According to the paper[1], this model is competitive with the largest Codex model that was the basis for Copilot originally. [1] https://arxiv.org/pdf/2203.13474.pdf https://arxiv.org/pdf/2203.13474.pdf
- MayeulC 4y agoI haven't ran any of these models yet, I had just assumed CodeGen was less performant for "understanding" prompts. You are right that it's probably enough, especially if fine-tuning is an option. Now, I wonder: as the code-base grows, how often, and how, should such tuning take place?