14 ms·

Llama 2 is about as factually accurate as GPT-4 for summaries and is 30X cheaper

143 points by pallas_athena 3y ago

cheema33 3y ago
Almost as good as GPT-4? I hear that claim quite often. And then when I test the claim, it falls far far short. I want real competition in this space. But currently, there is none. Except for maybe some very very corner case.
behindai 3y ago
it costs 0.001$ per 1K which is slightly cheaper than GPT-3.5-turbo. I have just tested it and it shows extremely worse results on the tasks in my pipelines. Not a game change, unfortunately.
- phillipcarter 3y ago
  This seems to be the general pattern so far. A particular benchmark shows better or equal performance for a non-openai model, but then someone else tries a different task and it's just not even close. I think that's really significant. It's arguably a case for fine-tuning models for specific tasks. That's great for teams with ML experience. But for product engineering teams without ML engineers, they can just use a foundational model and get great performance for a low cost.
  - ldjkfkdsjnv 3y ago
    The research is dishonest, they know they arent showing the full picture
    - phillipcarter 3y ago
      I think even if it's dishonest, it's still representative of a bullish case for fine-tuning. I'm excited to see people get involved in that to make it easier, because...whew, it's not easy.
    - waleedk 3y ago
      [Author] Can you be specific about which part of the research is dishonest? We've shared the code openly, you can reproduce it yourself if you want.
      - BoorishBears 3y ago
        I think in a vacuum it's not dishonest, but it is half-baked in a way a lot of these benchmarks are. Time and time again we see benchmarks like this where 0 effort into actually trying to get anything good out of the model. Of course you can make extremely specific changes that essentially game the test, so to me a fair way to test this is to establish a realistic token budget across all the models, maybe even adjusted for cost depending on the goals of your testing. Then for each model, use your budget to get the highest quality result possible. _ You're asking the GPT models for example, in the worst possible way. You've moved detailed instructions into the user prompt, despite their newest updates focusing on steerability via the system. Meanwhile from tinkering LLAMA 2 pretty much doesn't care about the system prompt vs user prompts. You're also not allowing for any form of chain-of-thought. Giving the models a few hundred tokens to form a conclusion instead of spending those tokens trying to force out a mathematically induced order bias (which I wouldn't expect to do much) would have been much better. There's also no way that GPT 4 should have struggled to give you a well defined output: All of the models would have probably benefited from a well defined output format in terms of a schema, rather than asking for a single letter response. The model rarely has to produce a single letter answer and will struggle with that. Something like asking for { "output" : "A_IS_MORE_FACTUAL" | "B_IS_MORE_FACTUAL" } would have been better Overall I think there's no open dishonesty, and obviously there's no objective correct amount of effort to put into the test setup here... but there's a conflict of interest that would have made me want to see more effort put into it. I think there were trivially low hanging fruit ignored here that I'd expect people selling LLMOps to have pick up on.
  - skybrian 3y ago
    They didn’t test it on other tasks and yes, it seems there’s no particular reason to believe that the results generalize. If someone wants to try it, they can, but I think it’s based on some unwarranted assumptions that LLM’s are likely to have balanced performance, just because there are some that do seem to be pretty balanced?
  - Vetch 3y ago
    Something to realize is that different models require different prompting styles. You can't prompt non-gpt4 models with GPT4 tuned stylistic ticks and expect similar results. I've gotten great performance from llama2 derivatives. Out of the box performance is not near GPT4 but it is still very strong in its own right. And, if you are able to break down your problem so precise logit control coupled with guidance from forward or backwards chaining on knowledge graphs is applicable, you can easily exceed gpt4's reasoning ability for your domain. No fine-tuning necessary either. I've been getting useful things out of LLMs since the days of roberta and raw T5, when Large stood for hundreds of millions of parameters. I am flabbergasted when people say a 7B parameter model is no good for them.
    - sthatipamala 3y ago
      > precise logit control coupled with guidance from forward or backwards chaining on knowledge graphs What do you mean by this?
      - thewataccount 3y ago
        I'm not 100% sure they're talking about this specifically but logit control/manipulation is often used to conform to a specific schema. https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance I'm going to butcher this explanation - after you've generated your selection of logits but before you sample from them, you check which ones conform to your schema. If you want the only two options to be "true" or "false", then you take any of the logits that would provide invalid answers and lower their probabilities manually. Another example is structures like JSON can be validated so when your sample is "{'name':'Carl'" you lower the probability of "{" since that would invalidate the json. In fact the only valid ones you'd likely have left would be ",", " ", and "}"
  - waleedk 3y ago
    [Author] Can you cite some examples of this so we're not talking in the void? I've actually tested Llama 2 for summarization but haven't blogged about that yet, and across multiple domains, Llama 2 is pretty good. I do see some differences -- Llama 2 doesn't follow instructions as cleanly, and is more verbose. It's not a plugin replacement -- in the article itself I point it out. GPT 3.5 largely followed instructions to return A or B. Llama 2 didn't so I had to use another LLM to post-process. I'm not saying you should always use Llama 2 or ChatGPT. It's that for some use cases, you can save a lot of money by using an open source alternative.
- waleedk 3y ago
  [Author] Can you share more details? How does it fail? What type of domain? Perhaps there is some tweaking to prompts that's required? It would help to understand what you're seeing.
- 19h 3y ago
  What are those “tasks in your pipeline”? We dedicated an entire team of 6 for two months on evaluating LLMs and while Claude 2 was the final choice, we found Llama 2 70b to be absolutely great (for summaries and structured data generation from free text). We chose Claude because running Llama on an A100 or H100 comes with a baseline cost that doesn’t go to zero when you don’t need it (you could spawn a new instance but right now GPUs are so rare everywhere except for expensive cloud providers that it’s possible you don’t get one). That said, we found the smaller Llama models to be so hilariously bad we have an internal slack channel where Llama 7b writes jokes and the funny part isn’t the jokes but how utterly stupid and random they are.
yieldcrv 3y ago
also means that OpenAI can just swap in Llama 2 and increase their capacity by orders of magnitude this is the age (or year) of token price arbitrage
- croddin 3y ago
  This is ignoring OpenAI's margins. We don't know how much GPT-3 and GPT-4 actually cost for them to run but it isn't what they are charging us. For Llama 2 the quoted cost is just compute, but for OpenAI you are also paying for using their software.
  - waleedk 3y ago
    [Author] As an end user, you pay the published price of $1 per million tokens for alternatives (such as Anyscale Endpoints -- https://app.endpoints.anyscale.com/landing https://app.endpoints.anyscale.com/landing, but there are others -- Replikate, Fireworks etc). This isn't just the compute price -- there's some margin in there as well.
- the-dude 3y ago
  IIRC Facebook explicitly licensed against that use case.
  - yieldcrv 3y ago
    Did they? For llama 2? I remember the first llama license
    - samspenc 3y ago
      Per the Llama 2 license, companies with "greater than 700 million monthly active users in the preceding calendar month" must get approval and permission from Meta to use it. For OpenAI - that is very interesting though. The 700M cap means the big FAANG+ companies that Meta competes with - Google, Apple, Amazon, Netflix, even Microsoft - can't use Llama 2. But OpenAI has less than 700 million users? So theoretically, if Llama was better than GPT models, OpenAI could replace their GPT engine with Llama :)
deleted 3y ago
[deleted]
Michelangelo11 3y ago
Nice idea and all, but I think their methodology was totally unsuited to the task. What they actually assessed was whether an LLM can recognize an accurate summary of a given article, not whether the LLM can produce an accurate summary itself. Here's their description of it: > We used a 3-way verified hand-labeled set of 373 news report statements and presented one correct and one incorrect summary of each. Each LLM had to decide which statement was the factually correct summary. The problem with this approach is its assumption that if an LLM can recognize an accurate summary, it'll be able to reliably produce accurate summaries. We know very little about the inner workings of LLMs right now, and what we do know suggests that they work highly counterintuitively, so I think there's no basis to make this assumption.
- waleedk 3y ago
  [Author] I acknowledge this and the article does as well. What I would say is that -- at the very least -- you could apply factual verification as a post process (so stage 1 is create the summary, stage 2 is validate the summary is factually accurate).
ldjkfkdsjnv 3y ago
Anyscale's business model was completely disrupted by OpenAI. They are trying to shift to provide hosting/fine tuning for open source LLMs, but the model performance will get crushed by Gpt4/newer open AI models. In theory the alternative to openAI models is nice, in reality anyscale is now competing with Azure/AWS/etc to provide model hosting. Their original compute platform for running arbitrary ml workloads will become obsolete as the industry consolidates around LLMs.
- deleted 3y ago
  [deleted]
- nightski 3y ago
  I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way there are a lot of problems that LLMs just aren't suited for.
  - ldjkfkdsjnv 3y ago
    Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed: 1. a model for sentiment analysis 2. a model for summarization 3. ...other NLP tasks other tasks: 4. a model for object detection in images 5. a model for face recognition in images Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis
    - losteric 3y ago
      LLMs come with a higher cost per transaction compared to dedicated models. imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models. Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)
m3kw9 3y ago
OpenAI is probably laughing their asses off internally read all the “it’s equivalent to GPT4” results
- adventured 3y ago
  Almost equivalent to, which is usually the magic stretch word. With GPT4 being OpenAI's year ago tech.
born-jre 3y ago
if someone reputable could maintain a blind benchmark that is not public, that would be great.
- waleedk 3y ago
  [Author] Can you elaborate on what you mean here? I'd like to understand what you mean by "blind benchmark" and "not public"?
  - lossolo 3y ago
    What OP probably means is not releasing benchmark data to the public so you can't train/fine tune your model based on that data.
- daniel-cussen 3y ago
  [dead]
sorokod 3y ago
"It is not too much of a stretch to conclude that a system that is better at telling factual from non-factual sentences is better at not making them up in the first place – or alternatively could decide through a two stage process if it was being inconsistent." Stretching aside, how does one follow from the other?
- alfalfasprout 3y ago
  They don't.
  - waleedk 3y ago
    [Author] The problem is: how could you objectively evaluate the factuality in any other way? Summary ranking is an established approach in the literature.
- mullingitover 3y ago
  If the system can evaluate the truthfulness of a statement, and it can generate statements, then it can create arbitrary numbers of candidate statements and filter for truthfulness.
  - sorokod 3y ago
    Looks like you take the "alternatively" fork in the original statemet - is it not a tautology?
  - waleedk 3y ago
    [Author] This is what I meant. That even if it doesn't work at a single layer, you could build a "verifier" stage whose approach is similar to this verification process and it can make the decision.

143 points by pallas_athena 3y ago

Almost as good as GPT-4? I hear that claim quite often. And then when I test the claim, it falls far far short. I want real competition in this space. But currently, there is none. Except for maybe some very very corner case.

behindai 3y ago

it costs 0.001$ per 1K which is slightly cheaper than GPT-3.5-turbo. I have just tested it and it shows extremely worse results on the tasks in my pipelines. Not a game change, unfortunately.

phillipcarter 3y ago
This seems to be the general pattern so far. A particular benchmark shows better or equal performance for a non-openai model, but then someone else tries a different task and it's just not even close. I think that's really significant. It's arguably a case for fine-tuning models for specific tasks. That's great for teams with ML experience. But for product engineering teams without ML engineers, they can just use a foundational model and get great performance for a low cost.
- ldjkfkdsjnv 3y ago
  The research is dishonest, they know they arent showing the full picture
  - phillipcarter 3y ago
    I think even if it's dishonest, it's still representative of a bullish case for fine-tuning. I'm excited to see people get involved in that to make it easier, because...whew, it's not easy.
  - waleedk 3y ago
    [Author] Can you be specific about which part of the research is dishonest? We've shared the code openly, you can reproduce it yourself if you want.
    - BoorishBears 3y ago
      I think in a vacuum it's not dishonest, but it is half-baked in a way a lot of these benchmarks are. Time and time again we see benchmarks like this where 0 effort into actually trying to get anything good out of the model. Of course you can make extremely specific changes that essentially game the test, so to me a fair way to test this is to establish a realistic token budget across all the models, maybe even adjusted for cost depending on the goals of your testing. Then for each model, use your budget to get the highest quality result possible. _ You're asking the GPT models for example, in the worst possible way. You've moved detailed instructions into the user prompt, despite their newest updates focusing on steerability via the system. Meanwhile from tinkering LLAMA 2 pretty much doesn't care about the system prompt vs user prompts. You're also not allowing for any form of chain-of-thought. Giving the models a few hundred tokens to form a conclusion instead of spending those tokens trying to force out a mathematically induced order bias (which I wouldn't expect to do much) would have been much better. There's also no way that GPT 4 should have struggled to give you a well defined output: All of the models would have probably benefited from a well defined output format in terms of a schema, rather than asking for a single letter response. The model rarely has to produce a single letter answer and will struggle with that. Something like asking for { "output" : "A_IS_MORE_FACTUAL" | "B_IS_MORE_FACTUAL" } would have been better Overall I think there's no open dishonesty, and obviously there's no objective correct amount of effort to put into the test setup here... but there's a conflict of interest that would have made me want to see more effort put into it. I think there were trivially low hanging fruit ignored here that I'd expect people selling LLMOps to have pick up on.
- skybrian 3y ago
  They didn’t test it on other tasks and yes, it seems there’s no particular reason to believe that the results generalize. If someone wants to try it, they can, but I think it’s based on some unwarranted assumptions that LLM’s are likely to have balanced performance, just because there are some that do seem to be pretty balanced?
- Vetch 3y ago
  Something to realize is that different models require different prompting styles. You can't prompt non-gpt4 models with GPT4 tuned stylistic ticks and expect similar results. I've gotten great performance from llama2 derivatives. Out of the box performance is not near GPT4 but it is still very strong in its own right. And, if you are able to break down your problem so precise logit control coupled with guidance from forward or backwards chaining on knowledge graphs is applicable, you can easily exceed gpt4's reasoning ability for your domain. No fine-tuning necessary either. I've been getting useful things out of LLMs since the days of roberta and raw T5, when Large stood for hundreds of millions of parameters. I am flabbergasted when people say a 7B parameter model is no good for them.
  - sthatipamala 3y ago
    > precise logit control coupled with guidance from forward or backwards chaining on knowledge graphs What do you mean by this?
    - thewataccount 3y ago
      I'm not 100% sure they're talking about this specifically but logit control/manipulation is often used to conform to a specific schema. https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance I'm going to butcher this explanation - after you've generated your selection of logits but before you sample from them, you check which ones conform to your schema. If you want the only two options to be "true" or "false", then you take any of the logits that would provide invalid answers and lower their probabilities manually. Another example is structures like JSON can be validated so when your sample is "{'name':'Carl'" you lower the probability of "{" since that would invalidate the json. In fact the only valid ones you'd likely have left would be ",", " ", and "}"
- waleedk 3y ago
  [Author] Can you cite some examples of this so we're not talking in the void? I've actually tested Llama 2 for summarization but haven't blogged about that yet, and across multiple domains, Llama 2 is pretty good. I do see some differences -- Llama 2 doesn't follow instructions as cleanly, and is more verbose. It's not a plugin replacement -- in the article itself I point it out. GPT 3.5 largely followed instructions to return A or B. Llama 2 didn't so I had to use another LLM to post-process. I'm not saying you should always use Llama 2 or ChatGPT. It's that for some use cases, you can save a lot of money by using an open source alternative.
waleedk 3y ago
[Author] Can you share more details? How does it fail? What type of domain? Perhaps there is some tweaking to prompts that's required? It would help to understand what you're seeing.
19h 3y ago
What are those “tasks in your pipeline”? We dedicated an entire team of 6 for two months on evaluating LLMs and while Claude 2 was the final choice, we found Llama 2 70b to be absolutely great (for summaries and structured data generation from free text). We chose Claude because running Llama on an A100 or H100 comes with a baseline cost that doesn’t go to zero when you don’t need it (you could spawn a new instance but right now GPUs are so rare everywhere except for expensive cloud providers that it’s possible you don’t get one). That said, we found the smaller Llama models to be so hilariously bad we have an internal slack channel where Llama 7b writes jokes and the funny part isn’t the jokes but how utterly stupid and random they are.

yieldcrv 3y ago

also means that OpenAI can just swap in Llama 2 and increase their capacity by orders of magnitude this is the age (or year) of token price arbitrage

croddin 3y ago
This is ignoring OpenAI's margins. We don't know how much GPT-3 and GPT-4 actually cost for them to run but it isn't what they are charging us. For Llama 2 the quoted cost is just compute, but for OpenAI you are also paying for using their software.
- waleedk 3y ago
  [Author] As an end user, you pay the published price of $1 per million tokens for alternatives (such as Anyscale Endpoints -- https://app.endpoints.anyscale.com/landing https://app.endpoints.anyscale.com/landing, but there are others -- Replikate, Fireworks etc). This isn't just the compute price -- there's some margin in there as well.
the-dude 3y ago
IIRC Facebook explicitly licensed against that use case.
- yieldcrv 3y ago
  Did they? For llama 2? I remember the first llama license
  - samspenc 3y ago
    Per the Llama 2 license, companies with "greater than 700 million monthly active users in the preceding calendar month" must get approval and permission from Meta to use it. For OpenAI - that is very interesting though. The 700M cap means the big FAANG+ companies that Meta competes with - Google, Apple, Amazon, Netflix, even Microsoft - can't use Llama 2. But OpenAI has less than 700 million users? So theoretically, if Llama was better than GPT models, OpenAI could replace their GPT engine with Llama :)

deleted 3y ago

[deleted]

Michelangelo11 3y ago

Nice idea and all, but I think their methodology was totally unsuited to the task. What they actually assessed was whether an LLM can recognize an accurate summary of a given article, not whether the LLM can produce an accurate summary itself. Here's their description of it: > We used a 3-way verified hand-labeled set of 373 news report statements and presented one correct and one incorrect summary of each. Each LLM had to decide which statement was the factually correct summary. The problem with this approach is its assumption that if an LLM can recognize an accurate summary, it'll be able to reliably produce accurate summaries. We know very little about the inner workings of LLMs right now, and what we do know suggests that they work highly counterintuitively, so I think there's no basis to make this assumption.

waleedk 3y ago
[Author] I acknowledge this and the article does as well. What I would say is that -- at the very least -- you could apply factual verification as a post process (so stage 1 is create the summary, stage 2 is validate the summary is factually accurate).

ldjkfkdsjnv 3y ago

Anyscale's business model was completely disrupted by OpenAI. They are trying to shift to provide hosting/fine tuning for open source LLMs, but the model performance will get crushed by Gpt4/newer open AI models. In theory the alternative to openAI models is nice, in reality anyscale is now competing with Azure/AWS/etc to provide model hosting. Their original compute platform for running arbitrary ml workloads will become obsolete as the industry consolidates around LLMs.

deleted 3y ago
[deleted]
nightski 3y ago
I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way there are a lot of problems that LLMs just aren't suited for.
- ldjkfkdsjnv 3y ago
  Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed: 1. a model for sentiment analysis 2. a model for summarization 3. ...other NLP tasks other tasks: 4. a model for object detection in images 5. a model for face recognition in images Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis
  - losteric 3y ago
    LLMs come with a higher cost per transaction compared to dedicated models. imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models. Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)

m3kw9 3y ago

OpenAI is probably laughing their asses off internally read all the “it’s equivalent to GPT4” results

adventured 3y ago
Almost equivalent to, which is usually the magic stretch word. With GPT4 being OpenAI's year ago tech.

born-jre 3y ago

if someone reputable could maintain a blind benchmark that is not public, that would be great.

waleedk 3y ago
[Author] Can you elaborate on what you mean here? I'd like to understand what you mean by "blind benchmark" and "not public"?
- lossolo 3y ago
  What OP probably means is not releasing benchmark data to the public so you can't train/fine tune your model based on that data.
daniel-cussen 3y ago
[dead]

sorokod 3y ago

"It is not too much of a stretch to conclude that a system that is better at telling factual from non-factual sentences is better at not making them up in the first place – or alternatively could decide through a two stage process if it was being inconsistent." Stretching aside, how does one follow from the other?

alfalfasprout 3y ago
They don't.
- waleedk 3y ago
  [Author] The problem is: how could you objectively evaluate the factuality in any other way? Summary ranking is an established approach in the literature.
mullingitover 3y ago
If the system can evaluate the truthfulness of a statement, and it can generate statements, then it can create arbitrary numbers of candidate statements and filter for truthfulness.
- sorokod 3y ago
  Looks like you take the "alternatively" fork in the original statemet - is it not a tautology?
- waleedk 3y ago
  [Author] This is what I meant. That even if it doesn't work at a single layer, you could build a "verifier" stage whose approach is similar to this verification process and it can make the decision.