7 ms·
In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderbo
by brucethemoose2 3y ago
In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good:
https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-xaberius-34b-v1beta
https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16
I mention this because it could theoretically be applied to Mistral Moe. If the uplift is the same as regular Mistral 7B, and Mistral Moe is good, the end result is a scary good model.
This might be an inflection point where desktop-runnable OSS is really breathing down GPT-4's neck.
- deleted 3y ago[deleted]
- _boffin_ 3y agoInteresting. One thing i noticed is that Mistral has a `max_position_embeddings` of ~32k while these have it at 4096. Any thoughts on that?
- brucethemoose2 3y agoIs complicated. The 7B model (cybertron) is trained on Mistral. Mistral is technically a 32K model, but it uses a sliding window beyond 32K, and for all practical purposes in current implementations it behaves like an 8K model. The 34B model is based on Yi 34B, which is inexplicably marked as a 4K model in the config but actually works out to 32K if you literally just edit that line. Yi also has a 200K base model... and I have no idea why they didn't just train on that. You don't need to finetune at long context to preserve its long context ability.
- ComputerGuru 3y agoDid you mean "but it uses a sliding window beyond" *8K*? Because I don't understand how the sentence would work otherwise.
- brucethemoose2 3y agoYeah exactly, sorry.
- stavros 3y agoAren't LLM benchmarks at best irrelevant, at worst lying, at this point?
- typon 3y agoYes. The only thing that is relevant is a hidden benchmark that's never released and run by a trusted third party.
- nabakin 3y agoMore or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possible. You can rank models yourself with a private automated benchmark which models don't have a chance to overfit to or with a good human evaluation study. Edit: also, I guess OP is talking about Mistral finetunes (ones overfitting to the benchmarks) beating out 70b models on the leaderboard because Mistral 7b is lower than Llama 2 70b chat.
- brucethemoose2 3y agoI'm not saying its better than 70B, just that its very strong from what others are saying. Actually I am testing the 34B myself (not the 7B), and it seems good.
- fblgit 3y agoUNA: Uniform Neural Alignment. Haven't u noticed yet? Each model that I uniform, behaves like a pre-trained.. and you likely can fine-tune it again without damaging it. If you chatted with them, you know .. that strange sensation, you know what is it.. Intelligence. Xaberius-34B is the highest performer of the board, and is NOT contaminated.
- 3y ago
- fblgit 3y agoCorrect. UNA can align the MoE at multiple layers, experts, nearly any part of the neural network I would say. Xaberius 34B v1 "BETA".. is the king, and its just that.. the beta. I'll be focusing on the Mixtral, its a christmas gift.. modular in that way, thanks for the lab @mistral!
- brucethemoose2 3y agoDo a Yi 200K version as well! That would make my Christmas, as Mistral Moe is only maybe 32K.
- inciampati 3y agoDo you have any docs describing the method?
- swyx 3y agowhat is neural alignment? who came up with it?
- brucethemoose2 3y ago@fblgit apparently, from earlier in this thread.
- 3abiton 3y agoHF leaderboards are rarely reflective of real world performance especially in small variations, but nonetheless, this is promising. What are the HW requirements for this latest Mistral7B?
- brucethemoose2 3y ago> What are the HW requirements for this latest Mistral7B Pretty much anything with ~6-8GB of memory that's not super old. It will run on my 6GB laptop RTX 2060 extremely quickly. It will run on my IGP or Phone with MLC-LLM. It will run fast on a laptop with a small GPU, with the rest offloaded to CPU. Small, CPU only servers are kinda the only questionable thing. It runs, just not very fast, especially with long prompts (which are particularly hard for CPUs). There's also not a lot of support for AI ASICs.
- eyegor 3y agoAny 7b can run well (~50 tok/s) on an 8gb gpu if you tune the context size. 13b can sometimes run well but typically you'll end up with a tiny context window or slow inference. For cpu, I wouldn't recommend going above 1.3b unless you don't mind waiting around.
- aarushsah 3y agoHow so? I'm only getting 12 t/s using Mistral in LM Studio.
- eyegor 3y agoThe lazy way is to use text-generation-webui, use an exllamav2 conversion of your model, and turn down context length until it fits (and tick the 8 bit cache option). If you go over your vram it will cut your speed substantially. Like 60/s down to 15/s for an extra 500 context length over what fits. Similar idea applies to any other backends, but you need to shove all the layers into vram if you want decent tok/s. To give you a starting point, typically for 7b models I have to use 4k-6k context length and I use 4-6 bit quantizations for an 8gb gpu. So start at 4 bit, 4k context and adjust up as you can. You can find most popular models converted for you on huggingface.co if you add exl2 to your search and start with the 4 bit quantized version. Don't bother going above 6 bits even if you have spare vram, practically it doesn't offer much benefit. For reference I max out around 60 tok/s at 4bit, 50 tok/s at 5bit, 40 at 6bit for some random 7b parameter model on a rtx 2070.
- eurekin 3y agoI just played with 7b version. It really feels different than anything I tried before. It could explain a docker compose file. It generated a simple vue application component. I asked around a bit about the example and it was strangely coherent and focused across the whole conversation. It was really well detecting, where I'm starting a new thread (without clearing a context) or referring to things before. It caught me off guard as well with this: > me: What does following mean [content of the docker compose] > cybertron-7b: In the provided YAML configuration, "following" refers to specifying dependencies I've never seen any model using my exact wording in quotes in conversation like that.
- mark_l_watson 3y agoHow did you run it? Are there model files in Ollama format? Are you running on NVidia or Apple Silicon? EDIT: just saw this “ Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA.”
- brucethemoose2 3y agoMy recommendation is: - Exui with exl2 files on good GPUs. - Koboldcpp with gguf files for small GPUs and Apple silicon. There are many reasons, but in a nutshell they are the fastest and most VRAM efficient. I can fit 34Bs with about 75K context on a single 24GB 3090 before the quality drop from quantization really starts to get dramatic.
- mark_l_watson 3y agoThanks! I will check out Koboldcpp.
- eurekin 3y agoIn the textgeneration web ui on NVidia gpu
- whimsicalism 3y agoyour edit is entirely unrelated to this topic
- screye 3y agoYeah, and Mistral doesn't particularly care about lobotomizing the model with 'safety-training'. So it can achieve much better performance per-parameter than anthropic/google/OpenAI while being more steerable as well.
- behnamoh 3y agountil Mistral gets too big for lawyers to ignore.
- nikvdp 3y agoThis piqued my interest so I made an ollama modelfile of it for the smallest variant (from TheBloke's GGUF [1] version). It does indeed seem impressively gpt4-ish for such a small model! Feels more coherent than openhermes2.5-mistral which was my previous goto local llm. If you have ollama installed you can try it out with `ollama run nollama/una-cybertron-7b-v2`. [1]: https://huggingface.co/TheBloke/una-cybertron-7B-v2-GGUF https://huggingface.co/TheBloke/una-cybertron-7B-v2-GGUF
- whimsicalism 3y agoDPO is pretty good as well. I think that the '7b beating 70b' is mostly due to the fact that Mistral is likely trained on considerably more tokens than Chinchilla optimal. So is llama-70b, but not to the same degree.