4 ms·

Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). (Found in model card: pass

by lhl 3y ago

Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). (Found in model card: pass@1) For comparison: * GPT-4 claims 85.4 on HumanEval, in a recent paper https://arxiv.org/pdf/2303.11366.pdf https://arxiv.org/pdf/2303.11366.pdf GPT-4 was tested at 80.1 pass@1 and 91 pass@1 using their Reflexion technique. They also include MBPP and Leetcode Hard benchmark comparisons * WizardCoder, a StarCoder fine-tune is one of the top open models, scoring a 57.3 pass@1, model card here: https://huggingface.co/WizardLM/WizardCoder-15B-V1.0 https://huggingface.co/WizardLM/WizardCoder-15B-V1.0 * The best open model I know of atm is replit-code-instruct-glaive, a replit-code-3b fine tune, which scores a 63.5% pass@1. An independent developer abacaj has reproduced that announcement as part of code-eval, a repo for getting human-eval results: https://github.com/abacaj/code-eval https://github.com/abacaj/code-eval Those interested in this area may also want to take a look at this repo https://github.com/my-other-github-account/llm-humaneval-benchmarks https://github.com/my-other-github-account/llm-humaneval-ben... that also ranks with Eval+, the CanAiCode Leaderboard https://huggingface.co/spaces/mike-ravkine/can-ai-code-results https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... and airate https://github.com/catid/supercharger/tree/main/airate https://github.com/catid/supercharger/tree/main/airate Also, as with all LLM evals, to be taken with a grain of salt... Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” arXiv, June 12, 2023. https://doi.org/10.48550/arXiv.2305.01210 https://doi.org/10.48550/arXiv.2305.01210.

famouswaffles 3y ago
GPT-4's zero shot Human Eval score was 67%
- lhl 3y ago
  While that's what the Technical Report (https://arxiv.org/pdf/2303.08774v3.pdf https://arxiv.org/pdf/2303.08774v3.pdf) says, but GPT-4 out in the wild's (reproducible) performance appears to be much higher now. Testing from 3/15 (presumably on the 0314 model) seems to be at 85.36% (https://twitter.com/amanrsanger/status/1635751764577361921 https://twitter.com/amanrsanger/status/1635751764577361921). And the linked paper from my post(https://doi.org/10.48550/arXiv.2305.01210 https://doi.org/10.48550/arXiv.2305.01210) got a pass@1 of 88.4 from GPT-4 recently (May? June?). Out of curiousity, I was trying out gpt-4-0613 and claude-v2 with https://github.com/getcursor/eval https://github.com/getcursor/eval, but sadly I'm getting hangs at 3% with both of them (maybe hitting rate limits?).
  - gcr 3y ago
    do we have evidence that OpenAI is making new versions of gpt4 available? The training data presumably hasn’t changed since 2021 and the model is absurdly expensive to train; there’s little incentive for them to keep touching it up.
    - jiggawatts 3y ago
      The issue with all of the chat-optimised LLMs is that they can’t be incrementally updated. After the base training there are three seperate sets of additional training to align the model, convince it to do question-response, and to improve the quality via feedback. If you update the original base model, then all the tuning steps need to be repeated. For a model the size of GPT-4 this is expensive and slow, which is why OpenAI hasn’t bothered.
      - lhl 3y ago
        Why would you only update the original base model and not simply continue fine-tuning? (This is a rhetorical question of course, and there are multiple empirical ways to confirm the majority of your answer is "hallucinated," not least of which is the fact that OpenAI keeps rolling out new versions and variants with different capabilities.)
    - lhl 3y ago
      Well there's OpenAI's release notes for one: https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-... Pre-training of a foundational model is what you're thinking of for the "absurdly expensive" part but fine tunes are extremely cheap and undoubtedly are being done constantly. (You can see just how cheap by looking at the papers for Alpaca, Vicuna, Koala, etc). Prices dropped from about $600 to $10 for smaller models. Guanaco, using QLoRA, fine tuned llama-65b in about 1 day on a single GPU. Another way to empirically test btw is to search for all the articles pointing out what ChatGPT gets wrong (3 or 4). I recently tested those when looking for evals and it gets the large majority (maybe 80-90% of those are answered correctly now).

by lhl 3y ago

famouswaffles 3y ago

GPT-4's zero shot Human Eval score was 67%

lhl 3y ago
While that's what the Technical Report (https://arxiv.org/pdf/2303.08774v3.pdf https://arxiv.org/pdf/2303.08774v3.pdf) says, but GPT-4 out in the wild's (reproducible) performance appears to be much higher now. Testing from 3/15 (presumably on the 0314 model) seems to be at 85.36% (https://twitter.com/amanrsanger/status/1635751764577361921 https://twitter.com/amanrsanger/status/1635751764577361921). And the linked paper from my post(https://doi.org/10.48550/arXiv.2305.01210 https://doi.org/10.48550/arXiv.2305.01210) got a pass@1 of 88.4 from GPT-4 recently (May? June?). Out of curiousity, I was trying out gpt-4-0613 and claude-v2 with https://github.com/getcursor/eval https://github.com/getcursor/eval, but sadly I'm getting hangs at 3% with both of them (maybe hitting rate limits?).
- gcr 3y ago
  do we have evidence that OpenAI is making new versions of gpt4 available? The training data presumably hasn’t changed since 2021 and the model is absurdly expensive to train; there’s little incentive for them to keep touching it up.
  - jiggawatts 3y ago
    The issue with all of the chat-optimised LLMs is that they can’t be incrementally updated. After the base training there are three seperate sets of additional training to align the model, convince it to do question-response, and to improve the quality via feedback. If you update the original base model, then all the tuning steps need to be repeated. For a model the size of GPT-4 this is expensive and slow, which is why OpenAI hasn’t bothered.
    - lhl 3y ago
      Why would you only update the original base model and not simply continue fine-tuning? (This is a rhetorical question of course, and there are multiple empirical ways to confirm the majority of your answer is "hallucinated," not least of which is the fact that OpenAI keeps rolling out new versions and variants with different capabilities.)
  - lhl 3y ago
    Well there's OpenAI's release notes for one: https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-... Pre-training of a foundational model is what you're thinking of for the "absurdly expensive" part but fine tunes are extremely cheap and undoubtedly are being done constantly. (You can see just how cheap by looking at the papers for Alpaca, Vicuna, Koala, etc). Prices dropped from about $600 to $10 for smaller models. Guanaco, using QLoRA, fine tuned llama-65b in about 1 day on a single GPU. Another way to empirically test btw is to search for all the articles pointing out what ChatGPT gets wrong (3 or 4). I recently tested those when looking for evals and it gets the large majority (maybe 80-90% of those are answered correctly now).