4 ms·

Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/

by alecco 3y ago

Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closing the gap with 4. Source for GPT-3.5 and GPT-4.0 values (but mind it might not be the same # of shots) https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/30#64c08a5550f0d6578dc67069 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

lhl 3y ago
It depends on the eval, but I think it's fair to say that it's close. Here is the AGI Eval results organized into a table w/ averages (also I put in the new Hermes LLama2 13B model as well: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=711776706 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It beats out ChatGPT in every category except SAT-Math. We definitely need harder benchmarks. So far, there's BIG-Bench Hard https://github.com/suzgunmirac/BIG-Bench-Hard https://github.com/suzgunmirac/BIG-Bench-Hard and just published, Advanced Reasoning Benchmark https://arb.duckai.org/ https://arb.duckai.org/
Tostino 3y ago
That seems more in-line with my experience. I have been using GPT-3.5 and GPT-4 for data cleaning pipelines, and have tried to swap out LLaMA2 70B in a few of the "easier" tasks, and it hasn't performed well enough yet for any of my tasks done by GPT-3.5.
- dr_kiszonka 3y ago
  Hi! Could you please share a few words on what type of data you are cleaning using GPT? It is an intriguing idea and I would love to learn more to see if I could use a similar approach.
  - Tostino 3y ago
    Youtube transcripts. It only works with single-person channels at the moment, as I haven't worked on disambiguating multiple speakers. They are very messy if they are just an auto transcription from Google. Practically unusable in most cases. So first step in the pipeline is cleaning up the transcripts for incorrectly transcribed words or sentences. Using the context of the rest of the transcript, it is able to fix the vast majority of them. Then we add punctuation and format it with paragraphs. Then I have another check over the whole transcript for any remaining issues. After all of that, I have a relatively clean transcript that represents the original audio very closely. From there, I am doing things like: 1. creating question/answer pairs from the transcript 2. creating a document of additional context that fills in details about what the speaker is talking about but may not have explicitly said 3. creating a summary of the transcript identifying the main purpose 4. creating a knowledge graph from the transcript with nodes and edges 5. creating an annotated version of the transcript using that knowledge graph I plan on putting some of this data into a vector database, and some of it will be used for fine tuning LLaMA2 models on specific tasks (like knowledge graph creation, annotation using a knowledge graph, and writing using a knowledge graph to keep track of events)
    - dr_kiszonka 3y ago
      That's very informative - thanks so much for the explanation! My only experience with transcripts is in the context of transcribing short interviews. I used Whisper and it was pretty good. I mostly work with quantitative data, though. In terms of the disambiguation of speakers, I haven't done it, but I remember blind signal separation discussed in a signal processing seminar I attended. There is also this paper, in case you haven't seen it already: https://enk100.github.io/speaker_separation/ https://enk100.github.io/speaker_separation/ Thanks again!
      - Tostino 3y ago
        I have not read that yet, thank you! Also, I haven't tried using Whisper for getting a transcription from the audio. I went the route of downloading the automatically generated transcripts from Youtube for a set of videos. An audio processing pipeline is definitely something I could add later though as an additional input channel for the overall pipeline.
- popinman322 3y ago
  Was this with or without fine-tuning?
  - Tostino 3y ago
    That is with fine-tuning: https://stability.ai/blog/freewilly-large-instruction-fine-tuned-models https://stability.ai/blog/freewilly-large-instruction-fine-t...
    - Tostino 3y ago
      I should be clear though, I didn't fine-tune on my specific tasks myself (as that is the end result of what I am doing with the pipelines outputs). I just tried the LLaMA2-chat model (which is fine-tuned), and the Freewilly 2 finetune.
sytelus 3y ago
LLaMA2 is far and away from GPT 3.5. Just look at HumanEval and other code generation metrics. All these GPT-4 based "chat evals" are extremely misleading and people should take it with a bag of salt.
- alecco 3y ago
  That's what I say in my comment, OP's ranking is quite misleading. The ranking I linked and quoted in my comment's is much better. See the About tab. It has 4 evaluations and it doesn't use GPT-4 to evaluate. Also the top one is a tuned Llama 2. Also clarified in my original comment.
- swyx 3y ago
  many usecases that arent code, + people can finetune llama2 with code. its a great start for a free model.

by alecco 3y ago

lhl 3y ago

It depends on the eval, but I think it's fair to say that it's close. Here is the AGI Eval results organized into a table w/ averages (also I put in the new Hermes LLama2 13B model as well: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=711776706 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It beats out ChatGPT in every category except SAT-Math. We definitely need harder benchmarks. So far, there's BIG-Bench Hard https://github.com/suzgunmirac/BIG-Bench-Hard https://github.com/suzgunmirac/BIG-Bench-Hard and just published, Advanced Reasoning Benchmark https://arb.duckai.org/ https://arb.duckai.org/

Tostino 3y ago

That seems more in-line with my experience. I have been using GPT-3.5 and GPT-4 for data cleaning pipelines, and have tried to swap out LLaMA2 70B in a few of the "easier" tasks, and it hasn't performed well enough yet for any of my tasks done by GPT-3.5.

dr_kiszonka 3y ago
Hi! Could you please share a few words on what type of data you are cleaning using GPT? It is an intriguing idea and I would love to learn more to see if I could use a similar approach.
- Tostino 3y ago
  Youtube transcripts. It only works with single-person channels at the moment, as I haven't worked on disambiguating multiple speakers. They are very messy if they are just an auto transcription from Google. Practically unusable in most cases. So first step in the pipeline is cleaning up the transcripts for incorrectly transcribed words or sentences. Using the context of the rest of the transcript, it is able to fix the vast majority of them. Then we add punctuation and format it with paragraphs. Then I have another check over the whole transcript for any remaining issues. After all of that, I have a relatively clean transcript that represents the original audio very closely. From there, I am doing things like: 1. creating question/answer pairs from the transcript 2. creating a document of additional context that fills in details about what the speaker is talking about but may not have explicitly said 3. creating a summary of the transcript identifying the main purpose 4. creating a knowledge graph from the transcript with nodes and edges 5. creating an annotated version of the transcript using that knowledge graph I plan on putting some of this data into a vector database, and some of it will be used for fine tuning LLaMA2 models on specific tasks (like knowledge graph creation, annotation using a knowledge graph, and writing using a knowledge graph to keep track of events)
  - dr_kiszonka 3y ago
    That's very informative - thanks so much for the explanation! My only experience with transcripts is in the context of transcribing short interviews. I used Whisper and it was pretty good. I mostly work with quantitative data, though. In terms of the disambiguation of speakers, I haven't done it, but I remember blind signal separation discussed in a signal processing seminar I attended. There is also this paper, in case you haven't seen it already: https://enk100.github.io/speaker_separation/ https://enk100.github.io/speaker_separation/ Thanks again!
    - Tostino 3y ago
      I have not read that yet, thank you! Also, I haven't tried using Whisper for getting a transcription from the audio. I went the route of downloading the automatically generated transcripts from Youtube for a set of videos. An audio processing pipeline is definitely something I could add later though as an additional input channel for the overall pipeline.
popinman322 3y ago
Was this with or without fine-tuning?
- Tostino 3y ago
  That is with fine-tuning: https://stability.ai/blog/freewilly-large-instruction-fine-tuned-models https://stability.ai/blog/freewilly-large-instruction-fine-t...
  - Tostino 3y ago
    I should be clear though, I didn't fine-tune on my specific tasks myself (as that is the end result of what I am doing with the pipelines outputs). I just tried the LLaMA2-chat model (which is fine-tuned), and the Freewilly 2 finetune.

sytelus 3y ago

LLaMA2 is far and away from GPT 3.5. Just look at HumanEval and other code generation metrics. All these GPT-4 based "chat evals" are extremely misleading and people should take it with a bag of salt.

alecco 3y ago
That's what I say in my comment, OP's ranking is quite misleading. The ranking I linked and quoted in my comment's is much better. See the About tab. It has 4 evaluations and it doesn't use GPT-4 to evaluate. Also the top one is a tuned Llama 2. Also clarified in my original comment.
swyx 3y ago
many usecases that arent code, + people can finetune llama2 with code. its a great start for a free model.