4 ms·
Looks to be Mixture of Experts, here is the params.json: { "dim": 4096, "n_layers": 32, "head_dim": 128, "hidden_dim": 1433
by nulld3v 3y ago
Looks to be Mixture of Experts, here is the params.json:
{
"dim": 4096,
"n_layers": 32,
"head_dim": 128,
"hidden_dim": 14336,
"n_heads": 32,
"n_kv_heads": 8,
"norm_eps": 1e-05,
"vocab_size": 32000,
"moe": {
"num_experts_per_tok": 2,
"num_experts": 8
}
}
- sp332 3y agoI don't see any code in there. What runtime could load these weights?
- brucethemoose2 3y agoIts presumably llama just like Mistral. Everything open source is llama now. Facebook all but standardized the architecture. I dunno about the moe. Is there existing transformers code for that part? It kinda looks like there is based on the config.
- jasonjmcghee 3y agoMistral is not llama architecture. https://github.com/mistralai/mistral-src https://github.com/mistralai/mistral-src
- brucethemoose2 3y agoIts basically llama architecture, all but drop in compatible with llama runtimes.
- refulgentis 3y agoBecause it's JSON? :)
- sockaddr 3y agoWhat does expert mean in this context?
- moffkalast 3y agoIt means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.
- deleted 3y ago[deleted]
- dragonwriter 3y ago> GPT 4 is based on the same architecture, but at 8*222B. Do we actually either no that it is MoE or that size? IIRC both if those started as outsidr guesses that somehow just became accepted knowledge without any actual confirmation.
- moffkalast 3y agoIirc some of the other things the same source stated were later confirmed, so this is likely to be true as well, but I might be misremembering.
- sockaddr 3y agoFascinating. Thanks
- tavavex 3y agoDoes anyone here know roughly how an expert gets chosen? It seems like a very open-ended problem, and I'm not sure on how it can be implemented easily.
- rishabhjain1198 3y ago[Relevant paper](https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538). TL;DR you can think of it as the initial part of the model is essentially dedicated to learning which experts to choose.