4 ms·
Tangential question - how well does Llama 2 do on coding tasks on less-mainstream languages like Rust?
by ewuhic 3y ago
Tangential question - how well does Llama 2 do on coding tasks on less-mainstream languages like Rust?
- moffkalast 3y agoWell I'm not too familiar with Rust so I can't gauge correctness, but I do have the Llama 2 13B NewHope fine tune loaded (which is afaik tuned for python coding), so I gave it and 3.5-turbo the same random post request question. 3.5's result: https://chat.openai.com/share/9e1aafd3-631c-4c13-80f6-f99c88daab0a https://chat.openai.com/share/9e1aafd3-631c-4c13-80f6-f99c88... NewHope's result: https://i.imgur.com/dfACQC3.png https://i.imgur.com/dfACQC3.png If you have any ideas for a more comprehensive test let me know and I'll try to run it. Giving it some existing code to fix up or change is usually more of a typical use case for me anyway.
- kristianp 3y agoInterestingly, they have withdrawn their model because they discovered that test data leaked into the training data. Quantised versions are still available on huggingface from others though. https://github.com/SLAM-group/newhope https://github.com/SLAM-group/newhope
- moffkalast 3y agoYeah it seemed suspiciously high for HumanEval and it only ranks 14th for JS and 7th for Python on other benchmarks now: https://huggingface.co/spaces/mike-ravkine/can-ai-code-results https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... WizardCoder is a bit of a problem since it's not llama 1/2 based but is its own 15B model and as such the support for it in anything practical is near nonexistent. WizardLM v1.2 looks like it may be worth testing out. All of the LLama 2 fine tunes I've tried out so far have weird issues though. Saying unrelated things at times, ignoring parts of the conversation and such. Could be fine tuning or prompt template goofs or Llama 1 may actually be a more self consistent base model overall.