6 ms·
I notice a few divergences to common models: - The feedforward hidden size is 16x the d_model, unlike most models which are typically 4x; - The vocabulary siz
by espadrine 3y ago
I notice a few divergences to common models:
- The feedforward hidden size is 16x the d_model, unlike most models which are typically 4x;
- The vocabulary size is 10x (256K vs. Mistral’s 32K);
- The training token count is tripled (6T vs. Llama2's 2T)
Apart from that, it uses the classic transformer variations: MQA, RoPE, RMSNorm.
How big was the batch size that it could be trained so fast?
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/blob/main/config.json https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/bl...
- GaggiX 3y agoLooking at the config.json of Gemma 7B the feedfoarward hidden size is 8x, not 16x
- espadrine 3y agoHuh, indeed, that's what the config.json[0] says; the report[1] indicates “Feedforward hidden dims: 49152”. [0]:https://huggingface.co/google/gemma-7b-it/blob/main/config.json https://huggingface.co/google/gemma-7b-it/blob/main/config.j... [1]: https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf https://storage.googleapis.com/deepmind-media/gemma/gemma-re...
- GaggiX 3y agoI don't see the number 49152 reported in the config.json, what line are you referring to? I just see the intermediate_size of 24576 (so 8x). EDIT: I didn't read the comment correctly, you have noticed the same thing.
- SahAssar 3y agoRead the parent comment again. It says the paper says 49152, not the config.json.
- deleted 3y ago[deleted]
- voxgen 3y agoThe *GLU-based activations functions like GEGLU and SwiGLU use 2 input values to produce 1 output value, which makes these numbers weird. In each value pair, one goes through the GELU/SiLU activation function and is then multiplied by the other "gate" value. In the report, "hidden dim" matches the number of GEGLU inputs. In the config, "intermediate_size" matches the number of GEGLU outputs. Most *GLU models so far have used intermediate_size=8/3*d_model as this makes have the same number of matmul FLOPS & parameters as a 4x-expanded non-GLU model, and PaLM vaguely showed that 4x is better than a smaller expansion factor. If one considers Llama-2-7B's FFN expansion factor to be ~5.33x, Gemma's expansion factor is 16x.
- GaggiX 3y agoMakes perfect sense thx
- lalaithion 3y agoWhat does tokenization look like in 256k vs 32k?
- espadrine 3y agoIt mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilingually): “説明書” (instruction manual) has its own token, and so does “Nixon”, “آباد” (a city suffix, I believe), and the HTML sequence "\"><!--".
- lalaithion 3y agoI understand the theory, I was looking for an example of the same text tokenized with the two different vocabularies.
- espadrine 3y agoDo you have an example text in mind? You can use this playground to test it out: https://huggingface.co/spaces/Xenova/the-tokenizer-playground https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...
- moffkalast 3y agoInteresting, it's actually worse than GPT-4s 100k tokenizer by quite a bit despite being over twice the size and only marginally better than LLama's 30k. At least for some random articles in English latin script that I tried anyway, but Llama and Gemma are English-only models so no point in testing anything else. Doesn't seem like a well made tokenizer at first glance or it's heavily biased towards languages the model can't even generate coherently, lol. If they really wanted it to be SOTA at something they could've at least made it the first open source truly multilingual model, but that's apparently more effort than the lame skin colour oriented virtue signalling Google wants to do.
- visarga 3y ago
- andy_xor_andrew 3y ago> The training token count is tripled (6T vs. Llama2's 2T) Damn, 6T? That's a lot! Given that this model seems to roughly match Mistral (according to the numbers from Google), this makes me think we have saturated the 7B parameter space, and couldn't possibly make it much better unless new techniques are discovered.
- espadrine 3y agoHard to say definitively. Mistral’s token embeddings only account for <2% of the 7B parameters, while Gemma’s larger token vocabulary vampirized over 10%, leaving less space for the more important parts of the network. It is a somewhat surprising tradeoff given that it was pretrained towards an English bias.