3 ms·

Looks like they aren't releasing a pretty interesting model too. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model

Looks like they aren't releasing a pretty interesting model too. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane. Meta says later on that they aren't releasing it and give no explanation. I wonder why given how incredible it seems to be.

kapp_in_life 3y ago
Likely trained on internal code.
- mediaman 3y ago
  That model is trained on synthetically AI-generated code, not internal code. It suggests that synthetic training could be the future in increasing capability of smaller models (and perhaps bigger ones too). AI will train AI.
  - sroussey 3y ago
    That is the basis for https://synthesis.ai/ https://synthesis.ai/
    - SubiculumCode 3y ago
      I'm an amateur, but it seems to me that methods to synthesize will have to be distinct from methods of the generative model.
  - haldujai 3y ago
    I thought this specific model was referring to self-instruction using both synthetic prompts (generated from few-shot in-context prompting of presumably some OpenAI model, the original paper used text-davinci-002) as well as synthetic code (presumably Code Llama 7 like for self-instruct) subsequently validated with execution? The differences being it's not just training on unvalidated synthetic data and this specific method (per the unnatural questions paper) results in increased instruction diversity which confers some added advantage and I'm assuming explains the performance gain over the also synthetic self-instruct code? I may be misunderstanding but this seems more nuanced than just training on synthetically AI-generated code and is more validating of synthetic instructions (i.e. low resource setting) rather than synthetic code (i.e. high resource setting).
jonchurch_ 3y ago
The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”). The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo. https://github.com/orhonovich/unnatural-instructions https://github.com/orhonovich/unnatural-instructions
- ilaksh 3y ago
  Maybe they used GPT-4 to train it. OpenAI terms of use don't allow that to be released commercially.
  - nkohari 3y ago
    This is the most likely explanation for both why they wouldn't release it and wouldn't explain why.
  - nabakin 3y ago
    I've seen this argued a lot but is it fact? OpenAI was able to train on data from other platforms and surely, those platforms weren't letting their data go if they could help it. Unless some new laws have been passed, I don't think OpenAI can legally prevent others from using their data to train models. OpenAI can't have their cake and eat it too. After all, any content generated by AI can't be copyrighted.
    - lhl 3y ago
      It is indeed a fact that OpenAI's Terms of Use do state that you can't use their service to develop competing models: Section 2.c.iii - https://openai.com/policies/terms-of-use https://openai.com/policies/terms-of-use Now of course, the terms are not the law (so don't govern the use of the generated data by any third party), they are an agreement between two parties. If you did click "agree" then that's a binding agreement and there could be legal/contractual repercussions (some of which are outlined in the terms).
EvgeniyZh 3y ago
Note that current GPT-4 pass@1 for HumanEval is closer to 90% than to 67% reported in GPT-4 technical report, as reported, e.g., in [1] [1] https://arxiv.org/abs/2305.01210 https://arxiv.org/abs/2305.01210
- nabakin 3y ago
  Good point, I guess Meta should be using that number in their chart
- cosmojg 3y ago
  Unfortunately, we have no idea whether GPT-4 has been further trained or finetuned on contaminated data since then.

kapp_in_life 3y ago

Likely trained on internal code.

mediaman 3y ago
That model is trained on synthetically AI-generated code, not internal code. It suggests that synthetic training could be the future in increasing capability of smaller models (and perhaps bigger ones too). AI will train AI.
- sroussey 3y ago
  That is the basis for https://synthesis.ai/ https://synthesis.ai/
  - SubiculumCode 3y ago
    I'm an amateur, but it seems to me that methods to synthesize will have to be distinct from methods of the generative model.
- haldujai 3y ago
  I thought this specific model was referring to self-instruction using both synthetic prompts (generated from few-shot in-context prompting of presumably some OpenAI model, the original paper used text-davinci-002) as well as synthetic code (presumably Code Llama 7 like for self-instruct) subsequently validated with execution? The differences being it's not just training on unvalidated synthetic data and this specific method (per the unnatural questions paper) results in increased instruction diversity which confers some added advantage and I'm assuming explains the performance gain over the also synthetic self-instruct code? I may be misunderstanding but this seems more nuanced than just training on synthetically AI-generated code and is more validating of synthetic instructions (i.e. low resource setting) rather than synthetic code (i.e. high resource setting).

jonchurch_ 3y ago

The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”). The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo. https://github.com/orhonovich/unnatural-instructions https://github.com/orhonovich/unnatural-instructions

ilaksh 3y ago
Maybe they used GPT-4 to train it. OpenAI terms of use don't allow that to be released commercially.
- nkohari 3y ago
  This is the most likely explanation for both why they wouldn't release it and wouldn't explain why.
- nabakin 3y ago
  I've seen this argued a lot but is it fact? OpenAI was able to train on data from other platforms and surely, those platforms weren't letting their data go if they could help it. Unless some new laws have been passed, I don't think OpenAI can legally prevent others from using their data to train models. OpenAI can't have their cake and eat it too. After all, any content generated by AI can't be copyrighted.
  - lhl 3y ago
    It is indeed a fact that OpenAI's Terms of Use do state that you can't use their service to develop competing models: Section 2.c.iii - https://openai.com/policies/terms-of-use https://openai.com/policies/terms-of-use Now of course, the terms are not the law (so don't govern the use of the generated data by any third party), they are an agreement between two parties. If you did click "agree" then that's a binding agreement and there could be legal/contractual repercussions (some of which are outlined in the terms).

EvgeniyZh 3y ago

Note that current GPT-4 pass@1 for HumanEval is closer to 90% than to 67% reported in GPT-4 technical report, as reported, e.g., in [1] [1] https://arxiv.org/abs/2305.01210 https://arxiv.org/abs/2305.01210

nabakin 3y ago
Good point, I guess Meta should be using that number in their chart
cosmojg 3y ago
Unfortunately, we have no idea whether GPT-4 has been further trained or finetuned on contaminated data since then.