5 ms·
Best 7B LLM on leaderboards made by an amateur following a medium tutorial
- brcmthrowaway 3y agoVery interesting this was managed with a 6 months out of date course.
- sour-taste 3y agoOn a free collab instance...
- brcmthrowaway 3y agoHow much more low hanging fruit is there?
- SushiHippie 3y agoThe GPU he used is not free, you need to buy compute units for these "premium GPUs". https://colab.research.google.com/signup https://colab.research.google.com/signup
- deleted 3y ago[deleted]
- cjbprime 3y agoIs it possible that they fine-tuned on the leaderboard test set?
- idibidiart 3y ago[dead]
- nabakin 3y agoThe only leaderboard this model is good on is the HuggingFace LLM Leaderboard which is known to be manipulated and victim to gross overfit. The Lmsys Arena Leaderboard is a better representation of the best models.
- refulgentis 3y agoThank you!!! _And_ it has the proprietary models...insanely more useful. It's a bug, not a feature, that the stock leaderboard ends up with endless fine tunes, and as you point out and is demonstrated by the article, its more about something else than about quality
- Der_Einzige 3y agoEven that chatbot arena shows that many models freely available and open source are better than some versions of GPT-3.5 and are within a small stones throw of the latest GPT 3.5
- int_19h 3y agoNote that it only includes gpt-3.5-turbo (the current iteration), not the original gpt-3.5. It's not exactly a secret that "turbo" models are noticeably dumber than the originals, whatever OpenAI says. There's no free lunch - that's why it's so much cheaper and faster... That said, we do have public 120b models now that genuinely feel better than the original gpt-3.5. The holy grail remains beating gpt-4 (or even gpt-4-turbo). This seems to be out of reach on consumer hardware at least...
- modeless 3y agoUm, the lmsys elo ranking clearly shows that GPT-4 Turbo is better than GPT-4.
- anoncareer0212 3y agoSame for the ChatGPTs post launch (let's not talk about 11_02 :) ) -- and as long as we're asserting andecdotes freely, I work in the field and have a couple years in before ChatGPT -- it most certainly is not a well-kept secret or a secret or true or anything else other than standard post-millenial self-peasantization. "Outright lie" is kinder toward the average reader via being more succinct, but usually causes explosive reactions because people take a while to come to terms with their ad-hoc knowledge via consuming commentary is fundamentally flawed, if ever.
- barnabee 3y agoIf only all people and companies were as honest as this guy about how much of their success they owe to luck and others!
- wavemode 3y ago"Leaderboards", meh. This tweet is still very true: https://twitter.com/karpathy/status/1737544497016578453 https://twitter.com/karpathy/status/1737544497016578453
- not2b 3y agoGoodhart's law would seem to apply: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Nevertheless, scoring so well on this benchmark is an accomplishment, though I'm not in a position to evaluate how significant it is.
- make3 3y agothat's why the huggingface llm arena exists
- int_19h 3y agoNothing beats an actual human spending a couple hours with the model when it comes to meaningful evaluation.
- jasonjmcghee 3y agoWhy does huggingface list this as a 9B model?
- make3 3y agoit's trained with a lora adaptor, so it's either an error or they also count the adaptor. they use a 16 param inner lora dimension however, so it's unlikely that that's the reason (too small) an important point to keep in mind is that at inference, the lora adaptors are made to be merged into the base model so they don't affect inference speed. (you need to explicitly do it though, if you train your own adaptor)
- latexr 3y agoAre we sure the tutorial was medium? It might have been quite good, or at least above average. Ba dum tss. “medium” should be capitalised in the title, as it refers to the blogging platform. https://medium.com https://medium.com
- politelemon 3y agoGood thread I saw on Reddit about this a few days ago. https://www.reddit.com/r/LocalLLaMA/comments/18xbevs/open_llm_leaderboard_is_disgusting/ https://www.reddit.com/r/LocalLLaMA/comments/18xbevs/open_ll... Many top models are overfitting to top leaderboards rather than be actually useful.
- SubiculumCode 3y agoIt seems so easy for just one poisoned model (trained with test data) to infect a ton of finetune model mixtures...it could happen without intention. Under this scenario, the ones that achieve the top performance are the closest relation to the poison model?
- mbb70 3y agoI'm as quick to jump on the Medium roastwagon as anyone else, but I will say Towards Data Science has a surprising number of quality tutorials running the full spectrum of data science tasks. That and they have great SEO, you basically can't avoid them.
- behnamoh 3y agoI avoid them easily using Kagi :))
- rck 3y agoEveryone knows the hf leaderboard is actively being gamed (Goodhart's law strikes again), but the guy who wrote the Medium post is active in doing stuff with models, and the tutorial is (clearly) pretty good.
- josu 3y agoWell, "Amateur following tutorial games HF leaderboard to become the #1 model" seems just as impressive to me.
- SubiculumCode 3y agoIt seems that a lot of the leaders are the results of mixing finetunes, which really makes me think that there was a leak of test sets into the training data.
- SubiculumCode 3y agoTo reply to myself. I am not saying that this model did, or that even if it did, that it was done intentionally. ML is hard, and there are so many ways for data to leak. What I AM surprised about is that it is not clear what CultriX did that was better than what a ton of others have done. Any clues?
- ramoz 3y agoWho is using 7Bs in a serious manner, instead of OpenAI, in a cost efficient way?
- justinl33 3y ago1. fine tune to specific tasks 2. Not subject to OpenAI’s censorshi 3. Can run on local instead of cloud compute (offline) 4. experimentation
- ramoz 3y agoNot sure what you’re responding to
- dingdingdang 3y agoUsing Mistral ft optimized 1218 7b q5km - very useful for basic queries/creative input. Often just as useful as ChatGPT and feels far more "real" to have it fully local, don't want to depend fully on one proprietary service for something as fundamentally useful as this!
- ukuina 3y agoI have spent more time than I would like to admit trying to get 3B-34B models working for "serious" usecases like RAG, code comprehension/generation and text summarization, across dozens of "leading" free LLMs, and none of them (not even Mixtral MoE or its derivatives) come close to gpt-3.5-turbo for consistency and depth, and there is no equal for gpt-4(-turbo). LLM Leaderboards and benchmarks do not show the complete picture for YOUR specific usecase.
- Havoc 3y agoNobody really trusts the leaderboards anymore. Overfitted and in some cases just outright cheating by training against common test sets https://twitter.com/karpathy/status/1737544497016578453 https://twitter.com/karpathy/status/1737544497016578453