4 ms·

The highlight IMO > The Code Llama models provide stable generations with up to 100,000 tokens of context. All models are trained on sequences of 16,000 tokens

The highlight IMO > The Code Llama models provide stable generations with up to 100,000 tokens of context. All models are trained on sequences of 16,000 tokens and show improvements on inputs with up to 100,000 tokens. Edit: Reading the paper, key retrieval accuracy really deteriorates after 16k tokens, so it remains to be seen how useful the 100k context is.

brucethemoose2 3y ago
Did Meta add scalable rope to the official implementation?
- snippyhollow 3y ago
  We changed RoPE's theta from 10k to 1m and fine-tuned with 16k tokens long sequences.
  - deleted 3y ago
    [deleted]
  - malwrar 3y ago
    Curious, what led you to adjusting the parameters this way? Also, have you guys experimented with ALiBi[1] which claims better extrapolative results than rotary positional encoding? [1]: https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409 (charts on page two if you’re skimming)
    - ttul 3y ago
      Undoubtedly, they have tried ALiBi…
nabakin 3y ago
Looks like they aren't releasing a pretty interesting model too. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane. Meta says later on that they aren't releasing it and give no explanation. I wonder why given how incredible it seems to be.
- kapp_in_life 3y ago
  Likely trained on internal code.
  - mediaman 3y ago
    That model is trained on synthetically AI-generated code, not internal code. It suggests that synthetic training could be the future in increasing capability of smaller models (and perhaps bigger ones too). AI will train AI.
    - sroussey 3y ago
      That is the basis for https://synthesis.ai/ https://synthesis.ai/
      - SubiculumCode 3y ago
        I'm an amateur, but it seems to me that methods to synthesize will have to be distinct from methods of the generative model.
    - haldujai 3y ago
      I thought this specific model was referring to self-instruction using both synthetic prompts (generated from few-shot in-context prompting of presumably some OpenAI model, the original paper used text-davinci-002) as well as synthetic code (presumably Code Llama 7 like for self-instruct) subsequently validated with execution? The differences being it's not just training on unvalidated synthetic data and this specific method (per the unnatural questions paper) results in increased instruction diversity which confers some added advantage and I'm assuming explains the performance gain over the also synthetic self-instruct code? I may be misunderstanding but this seems more nuanced than just training on synthetically AI-generated code and is more validating of synthetic instructions (i.e. low resource setting) rather than synthetic code (i.e. high resource setting).
- jonchurch_ 3y ago
  The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”). The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo. https://github.com/orhonovich/unnatural-instructions https://github.com/orhonovich/unnatural-instructions
riku_iki 3y ago
> The Code Llama models provide stable generations with up to 100,000 tokens of context. what is the trick to achieve 100k context? They can't just use 100k wide transformer layer, it is cost prohibitive, right?..
- littlestymaar 3y ago
  I'm pretty sure they don't do that, but for code the relevant relationship between two tokens is easy to determine with the semantics of the language alone (for instance you can say that tokens related to a local variable have no relationship with tokens outside), so it would lead to a sparse matrix in the transformer, reducing the cost of big contexts by a lot. But it would require language specific preprocessing, and whether you can make it fast is also dubious. I don't think it's been tried so far.

brucethemoose2 3y ago

Did Meta add scalable rope to the official implementation?

snippyhollow 3y ago
We changed RoPE's theta from 10k to 1m and fine-tuned with 16k tokens long sequences.
- deleted 3y ago
  [deleted]
- malwrar 3y ago
  Curious, what led you to adjusting the parameters this way? Also, have you guys experimented with ALiBi[1] which claims better extrapolative results than rotary positional encoding? [1]: https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409 (charts on page two if you’re skimming)
  - ttul 3y ago
    Undoubtedly, they have tried ALiBi…

nabakin 3y ago

Looks like they aren't releasing a pretty interesting model too. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane. Meta says later on that they aren't releasing it and give no explanation. I wonder why given how incredible it seems to be.

kapp_in_life 3y ago
Likely trained on internal code.
- mediaman 3y ago
  That model is trained on synthetically AI-generated code, not internal code. It suggests that synthetic training could be the future in increasing capability of smaller models (and perhaps bigger ones too). AI will train AI.
  - sroussey 3y ago
    That is the basis for https://synthesis.ai/ https://synthesis.ai/
    - SubiculumCode 3y ago
      I'm an amateur, but it seems to me that methods to synthesize will have to be distinct from methods of the generative model.
  - haldujai 3y ago
    I thought this specific model was referring to self-instruction using both synthetic prompts (generated from few-shot in-context prompting of presumably some OpenAI model, the original paper used text-davinci-002) as well as synthetic code (presumably Code Llama 7 like for self-instruct) subsequently validated with execution? The differences being it's not just training on unvalidated synthetic data and this specific method (per the unnatural questions paper) results in increased instruction diversity which confers some added advantage and I'm assuming explains the performance gain over the also synthetic self-instruct code? I may be misunderstanding but this seems more nuanced than just training on synthetically AI-generated code and is more validating of synthetic instructions (i.e. low resource setting) rather than synthetic code (i.e. high resource setting).
jonchurch_ 3y ago
The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”). The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo. https://github.com/orhonovich/unnatural-instructions https://github.com/orhonovich/unnatural-instructions

riku_iki 3y ago

> The Code Llama models provide stable generations with up to 100,000 tokens of context. what is the trick to achieve 100k context? They can't just use 100k wide transformer layer, it is cost prohibitive, right?..

littlestymaar 3y ago
I'm pretty sure they don't do that, but for code the relevant relationship between two tokens is easy to determine with the semantics of the language alone (for instance you can say that tokens related to a local variable have no relationship with tokens outside), so it would lead to a sparse matrix in the transformer, reducing the cost of big contexts by a lot. But it would require language specific preprocessing, and whether you can make it fast is also dubious. I don't think it's been tried so far.