5 ms·

Curious if anyone has done a side-by-side analysis of this offering vs just running LLaMA? I'm currently running a side-by-side comparison/evaluation of MSFT G

Curious if anyone has done a side-by-side analysis of this offering vs just running LLaMA? I'm currently running a side-by-side comparison/evaluation of MSFT GPT via Cognitive Services vs LLaMA[7B/13B/70B] and intrigued by the possibility of a truly air-gapped offering not limited by external computer power (nor by metered fees racking up.) Any reads on comparisons would be nice to see. (yes, I realize we'll eventually run into the same scaling issues w/r/t GPUs)

FrenchDevRemote 3y ago
chatgpt is obviously a LOT better, llama doesn't even understand some prompts and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning
- baobabKoodaa 3y ago
  > and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning This is overkill. First of all, ChatGPT isn't even the SOTA, so if you "want SOTA to do anything useful", then this ChatGPT offering would be as useless as LLaMA according to you. Second, there are many individual tasks where even those subpar LLaMA models are useful - even without finetuning.
  - FrenchDevRemote 3y ago
    it's the SOTA for chat(prove me wrong), and you can always use the API directly even for simple tasks they're less reliable and needs more prompt engineering
    - baobabKoodaa 3y ago
      > it's the SOTA for chat(prove me wrong) GPT-4 beats ChatGPT on all benchmarks. You can easily google these.
      - stavros 3y ago
        The distinction between GPT-4 and ChatGPT is blurry, as ChatGPT is a chat frontend for a GPT model, and you can use GPT-4 with ChatGPT. The parent probably means ChatGPT with GPT-4.
        baobabKoodaa 3y ago
        Typically when people say "ChatGPT" without specifying which specific model they refer to, they refer to gpt-3.5-turbo (in case of API - or in case of the web ui, they mean whatever model is its current web ui equivalent). But now OP says they meant GPT-4, so, sure.
        stolsvik 3y ago
        Counterpoint: I don’t refer to 3.5 when I say ChatGPT. I pay for ChatGPT, and always use GPT-4. Which I believe every paying customer do.
      - Kiro 3y ago
        I tried and got nothing useful. What's the difference between GPT-4 and ChatGPT Plus using GPT-4?
      - FrenchDevRemote 3y ago
        that is why i said FOR CHAT. even through the API you can't easily use the regular models for chat, the parsing would be atrocious and there are hundreds of edge cases to handle. ChatGPT4 through the API is the SOTA
- londons_explore 3y ago
  openai offers finetuning too. And it's pretty cheap to do considering.
deleted 3y ago
[deleted]
tikkun 3y ago
I did one. I took a few dozen prompts from my ChatGPT history and ran them through a few LLMs. GPT-4, Bard and Claude 2 came out on top. Llama 2 70b chat scored similarly to GPT-3.5, though GPT-3.5 still seemed to perform a bit better overall. My personal takeaway is I’m going to continue using GPT-4 for everything where the cost and response time are workable. Related: A belief I have is that LLM benchmarks are all too research oriented. That made sense when LLMs were in the lab. It doesn't make sense now that LLMs have tens of millions of DAUs — i.e. ChatGPT. The biggest use cases for LLMs so far are chat assistants and programming assistants. We need benchmarks that are based on the way people use LLMs in chatbots and the type of questions that real users use LLM products, not hypothetical benchmarks and random academic tests.
- TillE 3y ago
  I think tests like "can this LLM pass an English literature exam it's never seen before" are probably useful, but yeah there's a lot of silly stuff like math tests. I suppose the question is where are they most commercially viable. I've found them fantastic for creative brainstorming, but that's sort of hard to test and maybe not a huge market.
  - TuringNYC 3y ago
    >> I suppose the question is where are they most commercially viable. Fair point, though I'm not aiming to start a competing LLM SaaS service, rather i'm evaluating swapping out the TCO of Azure Cognitive Service OpenAI for the TCO of dedicated cloud compute running my own LLM -- to serve my own LLM calls currently being sent to a metered service (Azure Cognitive Service OpenAI) Evaluation points would be: output quality; meter vs fixed breakeven points; latency; cost of human labor to maintain/upgrade in most cases, i'd outsource and not think about it. BUT we're currently in some strange economics where the costs are off the charts for some services
- Q6T46nT668w6i3m 3y ago
  I don’t know what you mean by “too research oriented.” A common complaint in LLM research is the poor quality of evaluation metrics. There’s no consensus. Everyone wants new benchmarks but designing useful metrics is very much an open problem.
robertnishihara 3y ago
We (at Anyscale) have benchmarked GPT-4 versus the Llama-2 suite of models on a few problems: functional representation, SQL generation, grade-school math question answering. GPT-4 wins by a lot out of the box. However, surprisingly, fine-tuning makes a huge difference and allows the 7B Llama-2 model to outperform GPT-4 on some (but not all) problems. This is really great news for open models as many applications will benefit from smaller, faster, and cheaper fine-tuned models rather than a single large, slow, general-purpose model (Llama-2-7B is something like 2% of the size of GPT-4). GPT-4 continues to outperform even the fine-tuned 70B model on grade-school math question answering, likely due to the data Llama-2 was trained on (more data for fine-tuning helps here). https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...

FrenchDevRemote 3y ago

chatgpt is obviously a LOT better, llama doesn't even understand some prompts and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning

baobabKoodaa 3y ago
> and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning This is overkill. First of all, ChatGPT isn't even the SOTA, so if you "want SOTA to do anything useful", then this ChatGPT offering would be as useless as LLaMA according to you. Second, there are many individual tasks where even those subpar LLaMA models are useful - even without finetuning.
- FrenchDevRemote 3y ago
  it's the SOTA for chat(prove me wrong), and you can always use the API directly even for simple tasks they're less reliable and needs more prompt engineering
  - baobabKoodaa 3y ago
    > it's the SOTA for chat(prove me wrong) GPT-4 beats ChatGPT on all benchmarks. You can easily google these.
    - stavros 3y ago
      The distinction between GPT-4 and ChatGPT is blurry, as ChatGPT is a chat frontend for a GPT model, and you can use GPT-4 with ChatGPT. The parent probably means ChatGPT with GPT-4.
      - baobabKoodaa 3y ago
        Typically when people say "ChatGPT" without specifying which specific model they refer to, they refer to gpt-3.5-turbo (in case of API - or in case of the web ui, they mean whatever model is its current web ui equivalent). But now OP says they meant GPT-4, so, sure.
        stolsvik 3y ago
        Counterpoint: I don’t refer to 3.5 when I say ChatGPT. I pay for ChatGPT, and always use GPT-4. Which I believe every paying customer do.
    - Kiro 3y ago
      I tried and got nothing useful. What's the difference between GPT-4 and ChatGPT Plus using GPT-4?
    - FrenchDevRemote 3y ago
      that is why i said FOR CHAT. even through the API you can't easily use the regular models for chat, the parsing would be atrocious and there are hundreds of edge cases to handle. ChatGPT4 through the API is the SOTA
londons_explore 3y ago
openai offers finetuning too. And it's pretty cheap to do considering.

deleted 3y ago

[deleted]

tikkun 3y ago

I did one. I took a few dozen prompts from my ChatGPT history and ran them through a few LLMs. GPT-4, Bard and Claude 2 came out on top. Llama 2 70b chat scored similarly to GPT-3.5, though GPT-3.5 still seemed to perform a bit better overall. My personal takeaway is I’m going to continue using GPT-4 for everything where the cost and response time are workable. Related: A belief I have is that LLM benchmarks are all too research oriented. That made sense when LLMs were in the lab. It doesn't make sense now that LLMs have tens of millions of DAUs — i.e. ChatGPT. The biggest use cases for LLMs so far are chat assistants and programming assistants. We need benchmarks that are based on the way people use LLMs in chatbots and the type of questions that real users use LLM products, not hypothetical benchmarks and random academic tests.

TillE 3y ago
I think tests like "can this LLM pass an English literature exam it's never seen before" are probably useful, but yeah there's a lot of silly stuff like math tests. I suppose the question is where are they most commercially viable. I've found them fantastic for creative brainstorming, but that's sort of hard to test and maybe not a huge market.
- TuringNYC 3y ago
  >> I suppose the question is where are they most commercially viable. Fair point, though I'm not aiming to start a competing LLM SaaS service, rather i'm evaluating swapping out the TCO of Azure Cognitive Service OpenAI for the TCO of dedicated cloud compute running my own LLM -- to serve my own LLM calls currently being sent to a metered service (Azure Cognitive Service OpenAI) Evaluation points would be: output quality; meter vs fixed breakeven points; latency; cost of human labor to maintain/upgrade in most cases, i'd outsource and not think about it. BUT we're currently in some strange economics where the costs are off the charts for some services
Q6T46nT668w6i3m 3y ago
I don’t know what you mean by “too research oriented.” A common complaint in LLM research is the poor quality of evaluation metrics. There’s no consensus. Everyone wants new benchmarks but designing useful metrics is very much an open problem.

robertnishihara 3y ago

We (at Anyscale) have benchmarked GPT-4 versus the Llama-2 suite of models on a few problems: functional representation, SQL generation, grade-school math question answering. GPT-4 wins by a lot out of the box. However, surprisingly, fine-tuning makes a huge difference and allows the 7B Llama-2 model to outperform GPT-4 on some (but not all) problems. This is really great news for open models as many applications will benefit from smaller, faster, and cheaper fine-tuned models rather than a single large, slow, general-purpose model (Llama-2-7B is something like 2% of the size of GPT-4). GPT-4 continues to outperform even the fine-tuned 70B model on grade-school math question answering, likely due to the data Llama-2 was trained on (more data for fine-tuning helps here). https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...