7 ms·

Is there a place that compares/scores all of these local LLMs? Would be nice to know how close they are to the paid ones. Edit: Found one here: https://huggi

Is there a place that compares/scores all of these local LLMs? Would be nice to know how close they are to the paid ones. Edit: Found one here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

pocketarc 3y ago
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... It doesn't include a comparison to the paid ones, but it's a great place to look at different ones and see which ones you should experiment with and sink time into.
- tmikaeld 3y ago
  Found it a minute after i posted ;-) Have you tried any of the high-scored ones?
  - garciasn 3y ago
    They change somewhat frequently. We were previously working with some of the highest ranked that we could handle and we were getting acceptable results for the parameters of our tests. That said, if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close.
    - tmikaeld 3y ago
      > if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close. I figured as much, but it's incredibly hard to find any comparison that's kept up to date. What I was after was a local javascript LLM coder that i could train on a local codebase, but even that was fruitless except for some unmaintained "promptr" project. I guess OpenAI embeddings is the only option.
  - politelemon 3y ago
    I have tried Stable Beluga which was released recently. Even at 7B, the smallest, it does a decent job.
  - pocketarc 3y ago
    I've been playing a lot with Llama 2 13B recently, and it's really not bad at all. With oobabooga[1] you get a proper UI for it and even get an OpenAI-compatible API, so you just change the endpoint in your OpenAI library and it all works. I've been using that to test changes to my bots. As another poster mentioned though, it's nowhere near the level of GPT-4. It's close enough to GPT 3.5 though, you should try it out! [1]: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
    - tmikaeld 3y ago
      Thanks, I've had oogabooga bookmarked for a while, it's time!
- optimalsolver 3y ago
  What happens when the average approaches 100%? i.e., do these benchmarks taken together capture anything resembling general intelligence?
  - CuriouslyC 3y ago
    When the benchmarks approach 100% it'll be time to create harder benchmarks. We'll know we're getting close to general intelligence when it's a major challenge for humans to create harder benchmarks.
  - gpderetta 3y ago
    you trigger the on-site atomics to prevent a containment breach.
smallerfish 3y ago
It would be good to know what the hosting/scaling costs of each of these are too on cloud providers. Do all of the llama based ones basically share the same performance characteristics? If cost is the primary concern, is OpenAI (3.5) the only game in town right now or should we look elsewhere?
hexomancer 3y ago
Off topic: Jesus Christ that table is bad. How badly have they messed it up that reversing sort order takes multiple seconds on my pc? It was so slow that I thought maybe it is sending a new request to the server every time I change the sort order but no, it is just that slow.
- archerx 3y ago
  The power of modern Javascript. So what if my pentium 133mhz could invert an entire table with hundreds of cells in lotus 123 back in the early 90s in the blink of an eye. The fact that 20 years later with computers 100x faster inverting a table takes longer is progress! Also don't forget to mock people who care about performance by screaming "premature optimization" without fully understanding what it means.
  - capableweb 3y ago
    Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language. Bad programming is just bad programming, no need to pull what language is being used into it. Just happens to be JavaScript because that's what browsers natively support. But if Python (or whatever) was used instead, the very same programmer would have done the very same mistake.
    - zarathustreal 3y ago
      Careful. I just got a slap on the wrist for calling out lazy commentary, apparently it’s “taking a swipe” at someone.
    - ParetoOptimal 3y ago
      > Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language. Counterpoint: It's not what programming languages do, it's what they shepherd you to https://nibblestew.blogspot.com/2020/03/its-not-what-programming-languages-do.html?m=1 https://nibblestew.blogspot.com/2020/03/its-not-what-program...
lhl 3y ago
There are more evals than you can shake a stick at, but none of them are comprehensive or without their own issues. If you're looking for a lists of lists, I keep one here: https://llm-tracker.info/books/evals/page/list-of-evals https://llm-tracker.info/books/evals/page/list-of-evals
fastball 3y ago
https://tatsu-lab.github.io/alpaca_eval/ https://tatsu-lab.github.io/alpaca_eval/

pocketarc 3y ago

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... It doesn't include a comparison to the paid ones, but it's a great place to look at different ones and see which ones you should experiment with and sink time into.

tmikaeld 3y ago
Found it a minute after i posted ;-) Have you tried any of the high-scored ones?
- garciasn 3y ago
  They change somewhat frequently. We were previously working with some of the highest ranked that we could handle and we were getting acceptable results for the parameters of our tests. That said, if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close.
  - tmikaeld 3y ago
    > if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close. I figured as much, but it's incredibly hard to find any comparison that's kept up to date. What I was after was a local javascript LLM coder that i could train on a local codebase, but even that was fruitless except for some unmaintained "promptr" project. I guess OpenAI embeddings is the only option.
- politelemon 3y ago
  I have tried Stable Beluga which was released recently. Even at 7B, the smallest, it does a decent job.
- pocketarc 3y ago
  I've been playing a lot with Llama 2 13B recently, and it's really not bad at all. With oobabooga[1] you get a proper UI for it and even get an OpenAI-compatible API, so you just change the endpoint in your OpenAI library and it all works. I've been using that to test changes to my bots. As another poster mentioned though, it's nowhere near the level of GPT-4. It's close enough to GPT 3.5 though, you should try it out! [1]: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
  - tmikaeld 3y ago
    Thanks, I've had oogabooga bookmarked for a while, it's time!
optimalsolver 3y ago
What happens when the average approaches 100%? i.e., do these benchmarks taken together capture anything resembling general intelligence?
- CuriouslyC 3y ago
  When the benchmarks approach 100% it'll be time to create harder benchmarks. We'll know we're getting close to general intelligence when it's a major challenge for humans to create harder benchmarks.
- gpderetta 3y ago
  you trigger the on-site atomics to prevent a containment breach.

smallerfish 3y ago

It would be good to know what the hosting/scaling costs of each of these are too on cloud providers. Do all of the llama based ones basically share the same performance characteristics? If cost is the primary concern, is OpenAI (3.5) the only game in town right now or should we look elsewhere?

hexomancer 3y ago

Off topic: Jesus Christ that table is bad. How badly have they messed it up that reversing sort order takes multiple seconds on my pc? It was so slow that I thought maybe it is sending a new request to the server every time I change the sort order but no, it is just that slow.

archerx 3y ago
The power of modern Javascript. So what if my pentium 133mhz could invert an entire table with hundreds of cells in lotus 123 back in the early 90s in the blink of an eye. The fact that 20 years later with computers 100x faster inverting a table takes longer is progress! Also don't forget to mock people who care about performance by screaming "premature optimization" without fully understanding what it means.
- capableweb 3y ago
  Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language. Bad programming is just bad programming, no need to pull what language is being used into it. Just happens to be JavaScript because that's what browsers natively support. But if Python (or whatever) was used instead, the very same programmer would have done the very same mistake.
  - zarathustreal 3y ago
    Careful. I just got a slap on the wrist for calling out lazy commentary, apparently it’s “taking a swipe” at someone.
  - ParetoOptimal 3y ago
    > Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language. Counterpoint: It's not what programming languages do, it's what they shepherd you to https://nibblestew.blogspot.com/2020/03/its-not-what-programming-languages-do.html?m=1 https://nibblestew.blogspot.com/2020/03/its-not-what-program...

lhl 3y ago

There are more evals than you can shake a stick at, but none of them are comprehensive or without their own issues. If you're looking for a lists of lists, I keep one here: https://llm-tracker.info/books/evals/page/list-of-evals https://llm-tracker.info/books/evals/page/list-of-evals

fastball 3y ago

https://tatsu-lab.github.io/alpaca_eval/ https://tatsu-lab.github.io/alpaca_eval/