5 ms·

I’ve been using mistral and code llama to generate large volumes of code recently, and I have to say… These small models just suck compared to the larger ones.

I’ve been using mistral and code llama to generate large volumes of code recently, and I have to say… These small models just suck compared to the larger ones. It get it, they’re quick and cheap (ha! Relatively) to make and good for research and fine tuning… …but can anyone here speak authoritatively on fine tuning and getting good results out? I’ve been super disappointed by how bad even the q6 13B code llama model is at generating consistent code (Ie. It even compiles, forget doing what you asked) > about 30 lines in length. These smaller model seem good for a line or two, maybe, but gosh… it’s an effort to do anything useful with them out of the box. Carefully crafted prompt. Tests, hand written. Iterate: prompt, compile, run tests, generate code metrics. Accept code that passes the tests and beats the target threshold. You’re looking at like 7-10 iterations per prompt to get anything for simple (< 30 lines) functions, and maybe no candidates after 30 iterations for longer complex requests. Are people just using this for 5 line code snippets and autocomplete? Or is there a way to get better results by fine tuning?

a_bonobo 3y ago
I feel your experience fits with what's described in the OP article in https://news.ycombinator.com/item?id=37830011 https://news.ycombinator.com/item?id=37830011 (the most upvoted discussion goes somewhere else) I've had the same experience as you. Quantitative, these models are decent. Qualitative, in my daily work, they're not good at all! We'll need better tests.
tarruda 3y ago
> Or is there a way to get better results by fine tuning? It seems that the general wisdom around LLMs is that you can get very good performance on small models if you fine tune for a specific task. In the case of code generation, I think you might get a good performance by fine tuning it on a specific programming language + codebase or architectural pattern. The main problem with fine tuning is getting a good dataset, so a cheap alternative would be to put a few examples of what you want to generate in the prompt. Then you would save these prompts as task specific "fine tunes" that you would select when you need to accomplish something. You might find this discussion helpful: https://news.ycombinator.com/item?id=37813806 https://news.ycombinator.com/item?id=37813806
- diggan 3y ago
  > It seems that the general wisdom around LLMs is that you can get very good performance on small models if you fine tune for a specific task It would seem so, but is there any stories/research done that actually proves that someone has done so with good results? FOSS of course, so one could actually inspect there is no fudging and so on.
syntaxing 3y ago
Absolutely not victim blaming, but how do you use it? The biggest issue with local LLM is prompt structure prompting (like how you feed the instructions, not how you say something). If you even deviate a small bit from how it was trained, you’ll get terrible results. I’ve been using codellama 13b + ollama + continue and I’ll be honest, it’s almost on par with GPT-3.5 for my stuff. It’s been amazing as a pair programmer. It’s better to make draft and bounce ideas with it than to ask it to start from scratch. Long story short, try ollama + continue. If you’re using llama cpp by itself, chances are, you’ll get bad results.
- tarruda 3y ago
  Are you using apple silicon? How much RAM do you have, and how many tokens/second with codellama 13b?
  - syntaxing 3y ago
    Yes, 16GB of ram is needed for 13B, 32GB for 34B (both for 4bit). The first time it loads a new models takes some warm up time, I wanna say 30s? After that, the context reading and token generation are usually upward of 8 tk/s. Also, the newer and bigger the die, the faster the token generation. Like a Mac Studio would probably generate 30% or so faster than a MBP
    - tarruda 3y ago
      8tk/s on 34b? I've managed to run Codellama instruct 13b with my laptop's RTX 3070 (8gb VRAM) at 6tk/s by offloading 27 layers into the GPU with llama.cpp I've been considering getting a macbook for running 34b+ LLM inference, but with the speed in which small LLMs are progressing, I think it is better to get a laptop with an RTX 4090 and 16gb vram. Maybe It can run 34b models by offloading layers into the GPU.
      - syntaxing 3y ago
        I only have a 16GB computer so I can’t confirm the 34B performance. I have a 3090 with 24GB of VRAM and 34B just fits and runs above 15 tk/s. If you want a laptop and only plan for inferencing, I think a MBP would be better than a 4090 laptop.
diggan 3y ago
I'm guessing these small models are not meant to be used for writing whole blocks of code but rather to add more intelligent autocomplete for a few characters ahead, then they could probably provide a bit of help at least. I've had the same experience as you when trying anything locally below 30B parameters.
jasfi 3y ago
I suspected this would happen. At the end of the day larger models have more to work with, this makes a big difference. Also, there's a lot of domain knowledge in GPT-4 which isn't code but surely makes a big difference when it comes to understanding what you want, and the context of the problem and the solution.
jokethrowaway 3y ago
They're all pretty bad at replacing a developer, especially in lesser used languages Python is their forte, JS is okay. Rust is a mess, they don't get borrowing. The best so far was early chatgpt4 but it has since been nerfed down considerably. It's good to get ideas and for writing algorithms you are too lazy to google and implement, not great at doing actual work. Funnily enough, I think they'd do great at FANGs interviews

a_bonobo 3y ago

I feel your experience fits with what's described in the OP article in https://news.ycombinator.com/item?id=37830011 https://news.ycombinator.com/item?id=37830011 (the most upvoted discussion goes somewhere else) I've had the same experience as you. Quantitative, these models are decent. Qualitative, in my daily work, they're not good at all! We'll need better tests.

tarruda 3y ago

> Or is there a way to get better results by fine tuning? It seems that the general wisdom around LLMs is that you can get very good performance on small models if you fine tune for a specific task. In the case of code generation, I think you might get a good performance by fine tuning it on a specific programming language + codebase or architectural pattern. The main problem with fine tuning is getting a good dataset, so a cheap alternative would be to put a few examples of what you want to generate in the prompt. Then you would save these prompts as task specific "fine tunes" that you would select when you need to accomplish something. You might find this discussion helpful: https://news.ycombinator.com/item?id=37813806 https://news.ycombinator.com/item?id=37813806

diggan 3y ago
> It seems that the general wisdom around LLMs is that you can get very good performance on small models if you fine tune for a specific task It would seem so, but is there any stories/research done that actually proves that someone has done so with good results? FOSS of course, so one could actually inspect there is no fudging and so on.

syntaxing 3y ago

Absolutely not victim blaming, but how do you use it? The biggest issue with local LLM is prompt structure prompting (like how you feed the instructions, not how you say something). If you even deviate a small bit from how it was trained, you’ll get terrible results. I’ve been using codellama 13b + ollama + continue and I’ll be honest, it’s almost on par with GPT-3.5 for my stuff. It’s been amazing as a pair programmer. It’s better to make draft and bounce ideas with it than to ask it to start from scratch. Long story short, try ollama + continue. If you’re using llama cpp by itself, chances are, you’ll get bad results.

tarruda 3y ago
Are you using apple silicon? How much RAM do you have, and how many tokens/second with codellama 13b?
- syntaxing 3y ago
  Yes, 16GB of ram is needed for 13B, 32GB for 34B (both for 4bit). The first time it loads a new models takes some warm up time, I wanna say 30s? After that, the context reading and token generation are usually upward of 8 tk/s. Also, the newer and bigger the die, the faster the token generation. Like a Mac Studio would probably generate 30% or so faster than a MBP
  - tarruda 3y ago
    8tk/s on 34b? I've managed to run Codellama instruct 13b with my laptop's RTX 3070 (8gb VRAM) at 6tk/s by offloading 27 layers into the GPU with llama.cpp I've been considering getting a macbook for running 34b+ LLM inference, but with the speed in which small LLMs are progressing, I think it is better to get a laptop with an RTX 4090 and 16gb vram. Maybe It can run 34b models by offloading layers into the GPU.
    - syntaxing 3y ago
      I only have a 16GB computer so I can’t confirm the 34B performance. I have a 3090 with 24GB of VRAM and 34B just fits and runs above 15 tk/s. If you want a laptop and only plan for inferencing, I think a MBP would be better than a 4090 laptop.

diggan 3y ago

I'm guessing these small models are not meant to be used for writing whole blocks of code but rather to add more intelligent autocomplete for a few characters ahead, then they could probably provide a bit of help at least. I've had the same experience as you when trying anything locally below 30B parameters.

jasfi 3y ago

I suspected this would happen. At the end of the day larger models have more to work with, this makes a big difference. Also, there's a lot of domain knowledge in GPT-4 which isn't code but surely makes a big difference when it comes to understanding what you want, and the context of the problem and the solution.

jokethrowaway 3y ago

They're all pretty bad at replacing a developer, especially in lesser used languages Python is their forte, JS is okay. Rust is a mess, they don't get borrowing. The best so far was early chatgpt4 but it has since been nerfed down considerably. It's good to get ideas and for writing algorithms you are too lazy to google and implement, not great at doing actual work. Funnily enough, I think they'd do great at FANGs interviews