5 ms·
Vicuna v1.5 series, featuring 4K and 16K context, based on Llama 2
- davidkunz 3y agoThey should've called it Vicuna v2.
- codetrotter 3y agoVicuna, Vicdos, Victres
- anotherhue 3y agoViicuna perhaps.
- deleted 3y ago[deleted]
- tmikaeld 3y agoIs there a place that compares/scores all of these local LLMs? Would be nice to know how close they are to the paid ones. Edit: Found one here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- pocketarc 3y agohttps://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... It doesn't include a comparison to the paid ones, but it's a great place to look at different ones and see which ones you should experiment with and sink time into.
- tmikaeld 3y agoFound it a minute after i posted ;-) Have you tried any of the high-scored ones?
- garciasn 3y agoThey change somewhat frequently. We were previously working with some of the highest ranked that we could handle and we were getting acceptable results for the parameters of our tests. That said, if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close.
- tmikaeld 3y ago> if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close. I figured as much, but it's incredibly hard to find any comparison that's kept up to date. What I was after was a local javascript LLM coder that i could train on a local codebase, but even that was fruitless except for some unmaintained "promptr" project. I guess OpenAI embeddings is the only option.
- politelemon 3y agoI have tried Stable Beluga which was released recently. Even at 7B, the smallest, it does a decent job.
- pocketarc 3y agoI've been playing a lot with Llama 2 13B recently, and it's really not bad at all. With oobabooga[1] you get a proper UI for it and even get an OpenAI-compatible API, so you just change the endpoint in your OpenAI library and it all works. I've been using that to test changes to my bots. As another poster mentioned though, it's nowhere near the level of GPT-4. It's close enough to GPT 3.5 though, you should try it out! [1]: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- tmikaeld 3y ago
- smallerfish 3y agoIt would be good to know what the hosting/scaling costs of each of these are too on cloud providers. Do all of the llama based ones basically share the same performance characteristics? If cost is the primary concern, is OpenAI (3.5) the only game in town right now or should we look elsewhere?
- hexomancer 3y agoOff topic: Jesus Christ that table is bad. How badly have they messed it up that reversing sort order takes multiple seconds on my pc? It was so slow that I thought maybe it is sending a new request to the server every time I change the sort order but no, it is just that slow.
- archerx 3y agoThe power of modern Javascript. So what if my pentium 133mhz could invert an entire table with hundreds of cells in lotus 123 back in the early 90s in the blink of an eye. The fact that 20 years later with computers 100x faster inverting a table takes longer is progress! Also don't forget to mock people who care about performance by screaming "premature optimization" without fully understanding what it means.
- capableweb 3y agoBlaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language. Bad programming is just bad programming, no need to pull what language is being used into it. Just happens to be JavaScript because that's what browsers natively support. But if Python (or whatever) was used instead, the very same programmer would have done the very same mistake.
- zarathustreal 3y agoCareful. I just got a slap on the wrist for calling out lazy commentary, apparently it’s “taking a swipe” at someone.
- ParetoOptimal 3y ago> Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language. Counterpoint: It's not what programming languages do, it's what they shepherd you to https://nibblestew.blogspot.com/2020/03/its-not-what-programming-languages-do.html?m=1 https://nibblestew.blogspot.com/2020/03/its-not-what-program...
- lhl 3y agoThere are more evals than you can shake a stick at, but none of them are comprehensive or without their own issues. If you're looking for a lists of lists, I keep one here: https://llm-tracker.info/books/evals/page/list-of-evals https://llm-tracker.info/books/evals/page/list-of-evals
- fastball 3y agohttps://tatsu-lab.github.io/alpaca_eval/ https://tatsu-lab.github.io/alpaca_eval/
- bhouston 3y agoVicuña in my experience is by far the best training for llama models [1]. Way way better than alpaca or base llama. Recently I tried the wizard+vicuña uncensored on llama and it was excellent. Can not wait to try that combination on llama2. my favourite leaderboard is this one as it compares open source and closed source: https://tatsu-lab.github.io/alpaca_eval/ https://tatsu-lab.github.io/alpaca_eval/ [1] https://news.ycombinator.com/item?id=36975048 https://news.ycombinator.com/item?id=36975048
- azeirah 3y agoWhat do you mean by wizard+vicuna? I use wizardlm myself right now and have used vicuna in the past. Do you mean there's a model trained on the datasets of both wizardlm and vicuna?
- bhouston 3y agoI do not know. I just know that this model I found and used is very excellent: Vicuna + Wizard - Censoring: https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GGML https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... Seems to be based upon this: https://github.com/melodysdreamj/WizardVicunaLM https://github.com/melodysdreamj/WizardVicunaLM
- Dig1t 3y ago"Wizard's dataset + ChatGPT's conversation extension + Vicuna's tuning method"
- capableweb 3y agoHow accurate do you think that leaderboard is? It puts LLaMA2 Chat 70B at 92.66% and GPT-4 at 95.28%, only a ~3% difference.
- bhouston 3y agoI don't know exactly, and all benchmarks are subject to problems. But in my experience the ordering in this list is roughly right for the models I've tried.
- covi 3y agoPart of the Vicuna team wrote a guide on finetuning Llama2: https://blog.skypilot.co/finetuning-llama2-operational-guide/ https://blog.skypilot.co/finetuning-llama2-operational-guide...
- VadimPR 3y agoHow good is recall at 4k and 16k prompt sizes?
- 2bitencryption 3y agoHow well does it adhere to the system prompt? The base Llama-Chat models use something called "ghost attention" (they describe it in their paper). No clue how it works, but the result is, the model sticks to the system prompt extremely well. If you tell it that it's Marvin the Paranoid Android in the system prompt, it will stick 100% to that. In llama-derived models like Vicuna, if you tell it to act as Marvin the Paranoid Android, eventually the regular "assistant" voice starts to bleed through, after only a few chat turns. Doesn't sound like a big deal, but in cases where you have strict rules you want the model to follow, then the base llama-2-chat models are far better than any derived ones that do not implement ghost attention.
- loudmax 3y agoMy understanding of ghost attention is that the interface will inject periodic reminders into the conversation. These are seen by the model but hidden from the user. They do use up some of the tokens available in the context window.
- htrp 3y agoCouldn't you replicate that by doing the same thing and prepending system prompts in the Vicuna models?
- 2bitencryption 3y agoBut from the paper, it sounds like this happens only during training. Some trick about constantly re-injecting the system prompt during chat conversations. But during inference, there's no trick. The system message remains once at the top.
- kromem 3y agoGhost attention is only used in the 70B model in llama 2, FWIW. So would need to make sure comparing apples to apples.
- IronWolve 3y agoBeen playing with llama2_7b_chat_uncensored, nice speed, no censoring and gives me full details on censored questions. Still get the warnings about unsafe topic before it answers, but it answers. Its interesting to see how fast these models evolve.