3 ms·

Where is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama.

Where is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama. Almost looks like weights were corrupted or something. --- Update: My results using CodeLlama Instruct 7B, w/ Temperature 0 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba3d78092c1 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...

deleted 3y ago
[deleted]
jasonjmcghee 3y ago
(Note: I posted this before the full benchmark above and uses default temperature, not temperature 0, but point stands) Example (CodeLlama-Instruct 7B) --- Sure, here are two poems, one arguing for the use of Kubernetes and another arguing against it: For: Kubernetes, a mighty tool, Managing clusters with ease, Scaling apps with grace. Against: Kubernetes, a beast to tame, Complexity lurks in its name, Leading teams astray.
YetAnotherNick 3y ago
I think they messed up the templates.
- detente18 3y ago
  Here's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd0d4c1d5a2d4a70b49166/litellm/llms/prompt_templates/factory.py#L5 https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd... Anything I'm doing incorrectly?
  - YetAnotherNick 3y ago
    The format is really weird for chat models. In your code post message is `<</SYS>>\n [/INST]` which is wrong. Instead the overall format is `[INST] <<SYS>>\n{system}\n<</SYS>>\n\n{instruction} [/INST]` So both the system message and first instruction are in same [INST] block.
vincelt 3y ago
I used the API from Together[0]. Thanks for sharing your results, they're indeed pretty different. I looked at the source again and did append a "# " before every prompt made by those 10 `code` models (during testing thought that formatting it as a Python comment might help them). Will re-run the script without that to see if it matches your results. [0] https://docs.together.ai/docs/models-inference#code-models https://docs.together.ai/docs/models-inference#code-models

deleted 3y ago

[deleted]

jasonjmcghee 3y ago

(Note: I posted this before the full benchmark above and uses default temperature, not temperature 0, but point stands) Example (CodeLlama-Instruct 7B) --- Sure, here are two poems, one arguing for the use of Kubernetes and another arguing against it: For: Kubernetes, a mighty tool, Managing clusters with ease, Scaling apps with grace. Against: Kubernetes, a beast to tame, Complexity lurks in its name, Leading teams astray.

YetAnotherNick 3y ago

I think they messed up the templates.

detente18 3y ago
Here's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd0d4c1d5a2d4a70b49166/litellm/llms/prompt_templates/factory.py#L5 https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd... Anything I'm doing incorrectly?
- YetAnotherNick 3y ago
  The format is really weird for chat models. In your code post message is `<</SYS>>\n [/INST]` which is wrong. Instead the overall format is `[INST] <<SYS>>\n{system}\n<</SYS>>\n\n{instruction} [/INST]` So both the system message and first instruction are in same [INST] block.

vincelt 3y ago

I used the API from Together[0]. Thanks for sharing your results, they're indeed pretty different. I looked at the source again and did append a "# " before every prompt made by those 10 `code` models (during testing thought that formatting it as a Python comment might help them). Will re-run the script without that to see if it matches your results. [0] https://docs.together.ai/docs/models-inference#code-models https://docs.together.ai/docs/models-inference#code-models