2 ms·
"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" - https://arxiv.org/abs/2310.06770 https://arxiv.org/abs/2310.06770 "... Our evaluations sho
by belter 3y ago
"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" - https://arxiv.org/abs/2310.06770 https://arxiv.org/abs/2310.06770
"... Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. Claude 2 and GPT-4 solve a mere 4.8% and 1.7% of instances respectively, even when provided with an oracle retriever..."