3 ms·

Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but

Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but still nowhere near ChatGPT for coding questions.

ramesh31 3y ago
>Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? Not only possible but quite easy. Inference for 70B can be done with llama.cpp using CPU only, on any commodity hardware with >64GB of RAM
- beefield 3y ago
  I have 64gb on my 5 year old thinkpad. What kind of performance (tokens per sec) I could expect on that nowadays for a 70B model?
  - jerrygenser 3y ago
    Llama cpp speed is dramatically improved by avx instructions. If your CPU has those it would be much faster than not. And if it doesn't you need to do some workarounds with compiling and it gets a bit harder to run.
Zambyte 3y ago
When you say "coding questions" do you mean questions that should be answered by producing code, or questions about code ("explain this")? Or both?
marcosdumay 3y ago
I imagine that if you take the time to specialize it, you suddenly have a model that is better than anything from the large players on all the cases that you care about. But, well, I am currently not hyped enough about it to actually try.
rvz 3y ago
Possibly. Might need to be further optimized in size and 4-bit quantisation, perhaps and then you have a scaleable and fast self-hosted AI model. Lets just hope that there won’t be any embarrassing vulnerabilities coming out of this when someone could prompt the model to reveal its own environment variables or API keys or the internal prompt that it is using. But it seems the $0 free AI models are eating OpenAI’s lunch and Meta so far is winning the race to zero.
lhl 3y ago
You will want to look at HumanEval (https://github.com/abacaj/code-eval https://github.com/abacaj/code-eval) and Eval+ (https://github.com/my-other-github-account/llm-humaneval-benchmarks https://github.com/my-other-github-account/llm-humaneval-ben...) results for coding. While Llama2 is an improvement over LLaMA v1, it's still nowhere near even the best open models (currently, sans test contamination, WizardCoder-15B, a StarCoder fine tune is at top). It's really not a competition atm though, ChatGPT-4 wipes the floor for coding atm.
- riku_iki 3y ago
  this all numbers can be missleading, and simply indicate that gpt have these tasks in training data, and another model doesn't.
  - lhl 3y ago
    While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive/discussions/3 https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ``` From the WizardCoder paper https://arxiv.org/pdf/2306.08568.pdf https://arxiv.org/pdf/2306.08568.pdf you can see that it hits SOTA (for open models) in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. For those interested in reading more about various considerations for coding models, I highly recommend reading the MSR phi-1 paper: https://arxiv.org/pdf/2306.11644.pdf https://arxiv.org/pdf/2306.11644.pdf Looking forward to if they ever publish code/model/dataset since it has extremely strong performance trained on a very small number of tokens very manageable 1.3B and 350M parameter models.
    - riku_iki 3y ago
      > in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. and how do you know all these benchmarks not leaked? I think they all scrapped from web sites, the same as training data for LLM, so risk of contamination is extremely high. The best way to measure this is through synthetic datasets, which generate new tasks every time and model can't memorize them during training. One example is BigBench has multiple such tasks, but researchers usually(always) not regenerating those datasets.

ramesh31 3y ago

>Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? Not only possible but quite easy. Inference for 70B can be done with llama.cpp using CPU only, on any commodity hardware with >64GB of RAM

beefield 3y ago
I have 64gb on my 5 year old thinkpad. What kind of performance (tokens per sec) I could expect on that nowadays for a 70B model?
- jerrygenser 3y ago
  Llama cpp speed is dramatically improved by avx instructions. If your CPU has those it would be much faster than not. And if it doesn't you need to do some workarounds with compiling and it gets a bit harder to run.

Zambyte 3y ago

When you say "coding questions" do you mean questions that should be answered by producing code, or questions about code ("explain this")? Or both?

marcosdumay 3y ago

I imagine that if you take the time to specialize it, you suddenly have a model that is better than anything from the large players on all the cases that you care about. But, well, I am currently not hyped enough about it to actually try.

rvz 3y ago

Possibly. Might need to be further optimized in size and 4-bit quantisation, perhaps and then you have a scaleable and fast self-hosted AI model. Lets just hope that there won’t be any embarrassing vulnerabilities coming out of this when someone could prompt the model to reveal its own environment variables or API keys or the internal prompt that it is using. But it seems the $0 free AI models are eating OpenAI’s lunch and Meta so far is winning the race to zero.

lhl 3y ago

You will want to look at HumanEval (https://github.com/abacaj/code-eval https://github.com/abacaj/code-eval) and Eval+ (https://github.com/my-other-github-account/llm-humaneval-benchmarks https://github.com/my-other-github-account/llm-humaneval-ben...) results for coding. While Llama2 is an improvement over LLaMA v1, it's still nowhere near even the best open models (currently, sans test contamination, WizardCoder-15B, a StarCoder fine tune is at top). It's really not a competition atm though, ChatGPT-4 wipes the floor for coding atm.

riku_iki 3y ago
this all numbers can be missleading, and simply indicate that gpt have these tasks in training data, and another model doesn't.
- lhl 3y ago
  While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive/discussions/3 https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ``` From the WizardCoder paper https://arxiv.org/pdf/2306.08568.pdf https://arxiv.org/pdf/2306.08568.pdf you can see that it hits SOTA (for open models) in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. For those interested in reading more about various considerations for coding models, I highly recommend reading the MSR phi-1 paper: https://arxiv.org/pdf/2306.11644.pdf https://arxiv.org/pdf/2306.11644.pdf Looking forward to if they ever publish code/model/dataset since it has extremely strong performance trained on a very small number of tokens very manageable 1.3B and 350M parameter models.
  - riku_iki 3y ago
    > in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. and how do you know all these benchmarks not leaked? I think they all scrapped from web sites, the same as training data for LLM, so risk of contamination is extremely high. The best way to measure this is through synthetic datasets, which generate new tasks every time and model can't memorize them during training. One example is BigBench has multiple such tasks, but researchers usually(always) not regenerating those datasets.