3 ms·
Qwen1.5-Moe: Matching 7B Model Performance with 1/3 Activated Parameters
- transformi 3y agoHOW is it compare to phi-2?
- sp332 3y agoHigher on MMLU (62.5 vs 56.7 for phi-2) and GSM8k (61.5 vs 61.1). https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ https://www.microsoft.com/en-us/research/blog/phi-2-the-surp... The phi-2 numbers are for 5-shot MMLU and 8-shot GSM8k. The blog post doesn't get that specific for Qwen, but it's very likely they tested the same way.
- hidelooktropic 3y agoThe demo link 404s.
- Havoc 3y agoDoes anyone know the correct template & EOS tokens for this?
- SushiHippie 3y agoFor the chat model https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B-Chat/blob/8a2dca2866423cb6f8d4d0c7bda4c4a0f0417308/tokenizer_config.json#L31 https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B-Chat/blob/8a2d...
- radq 3y ago1/3rd "activated parameters", while also requiring 2x the VRAM.
- deleted 3y ago[deleted]
- YetAnotherNick 3y agoThat's the point of MoE. Sacrificing VRAM for compute/RAM bandwidth which makes it harder sell for consumer devices but easier for server devices where things are more likely to be compute or RAM bandwidth bound.
- cchance 3y agoSilly question, but is it possible for something similar to MoE for diffusion models like Stable Diffusion / DallE
- kaliqt 3y agoSomething tells me that image models are sufficiently small, it's easier to just have your differently tuned models sitting side by side so you can easily swap and inference on them rather than compiling it into one model.
- kristianp 3y agoIt's actually a 14.3B parameter model. It's irritating that they don't follow the convention of naming the model according to size. Qwen1.5-MoE-A2.7B is named for the 2.7B activated parameters. I guess it helps to obfuscate the size, given that it performs about as well as Mistral 7B.