3 ms·

I have spent more time than I would like to admit trying to get 3B-34B models working for "serious" usecases like RAG, code comprehension/generation and text su

by ukuina 3y ago

I have spent more time than I would like to admit trying to get 3B-34B models working for "serious" usecases like RAG, code comprehension/generation and text summarization, across dozens of "leading" free LLMs, and none of them (not even Mixtral MoE or its derivatives) come close to gpt-3.5-turbo for consistency and depth, and there is no equal for gpt-4(-turbo). LLM Leaderboards and benchmarks do not show the complete picture for YOUR specific usecase.

dcreater 3y ago
If you can post results that would be extremely extremely helpful. This is what I've been looking for -real world performance - rather than gamed leaderboards
- ukuina 3y ago
  That's just it: It is highly dependent on your inputs, expectations and usecase. My observations below. Mixtral does about average at summarization. Not enough details are presented in the summary. Passable for non-decisionbound workflows. Using quantized models injects occasional spelling errors, so fall back on Mistral unquantized. Skip all the Mistral finetunes. Only choose Instruct models. There are no good local code comprehension/generation models (and that includes CodeLlama, DeepSeek, Starcoder, WizardCoder, Phind). You will spend as much time guiding/rerolling/correcting output as coding it yourself, assuming your task is nontrivial and exceeds 100+ LoC. Code completion via continue.dev using local models did not yield good results, either. No local models are good at agentic tasks (e.g., via AutoGen). You might have better luck rolling your own with function-calling via NexusRaven2 than using Agent frameworks with local LLMs. Claude-Instant edges out gpt-3.5-turbo for longform summarization with its 100k context. It is what I use for https://HackYourNews.com https://HackYourNews.com because chunked/rolling summarization is noisy. For every non-hobby task, I am switching to gpt-4-*, choosing between base, 32k, and turbo depending on speed/correctness/cost/length tradeoffs. They are not perfect, but there is no competition.
  - m_kos 3y ago
    Would you mind sharing how you got access to Claude? I applied a long time ago and have never heard back from them.
  - dcreater 3y ago
    thats really useful insight. Yes it is usecase specific but thats also why its crucial for real world trial results to be shared so that theres a better understanding thats qualitatively accurate. P.S: hackyournews.com is great!!
ramoz 3y ago
And this is my drawn out experience as well

by ukuina 3y ago

dcreater 3y ago

If you can post results that would be extremely extremely helpful. This is what I've been looking for -real world performance - rather than gamed leaderboards

ukuina 3y ago
That's just it: It is highly dependent on your inputs, expectations and usecase. My observations below. Mixtral does about average at summarization. Not enough details are presented in the summary. Passable for non-decisionbound workflows. Using quantized models injects occasional spelling errors, so fall back on Mistral unquantized. Skip all the Mistral finetunes. Only choose Instruct models. There are no good local code comprehension/generation models (and that includes CodeLlama, DeepSeek, Starcoder, WizardCoder, Phind). You will spend as much time guiding/rerolling/correcting output as coding it yourself, assuming your task is nontrivial and exceeds 100+ LoC. Code completion via continue.dev using local models did not yield good results, either. No local models are good at agentic tasks (e.g., via AutoGen). You might have better luck rolling your own with function-calling via NexusRaven2 than using Agent frameworks with local LLMs. Claude-Instant edges out gpt-3.5-turbo for longform summarization with its 100k context. It is what I use for https://HackYourNews.com https://HackYourNews.com because chunked/rolling summarization is noisy. For every non-hobby task, I am switching to gpt-4-*, choosing between base, 32k, and turbo depending on speed/correctness/cost/length tradeoffs. They are not perfect, but there is no competition.
- m_kos 3y ago
  Would you mind sharing how you got access to Claude? I applied a long time ago and have never heard back from them.
- dcreater 3y ago
  thats really useful insight. Yes it is usecase specific but thats also why its crucial for real world trial results to be shared so that theres a better understanding thats qualitatively accurate. P.S: hackyournews.com is great!!

ramoz 3y ago

And this is my drawn out experience as well