5 ms·
On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b. HumanEval is abused but this model is only good for its size, it is no match for Cop
by karmasimida 3y ago
On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b.
HumanEval is abused but this model is only good for its size, it is no match for Copilot … yet
- UncleOxidant 3y ago> On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b. Can you put those numbers into context for those who haven't done HumanEval? Are those percentages so that 40+ means 40+% and 26 is 26%? If so does that imply both would be failing scores?