Thank you very much for releasing these models! It's great to see Google enter the battle with a strong hand. I'm wondering if you're able to provide any insig

Thank you very much for releasing these models! It's great to see Google enter the battle with a strong hand. I'm wondering if you're able to provide any insight into the below hyperparameter decisions in Gemma's architecture, as they differ significantly from what we've seen with other recent models? * On the 7B model, the `d_model` (3072) is smaller than `num_heads * d_head` (16*256=4096). I don't know of any other model where these numbers don't match. * The FFN expansion factor of 16x is MUCH higher than the Llama-2-7B's 5.4x, which itself was chosen to be equi-FLOPS with PaLM's 4x. * The vocab is much larger - 256k, where most small models use 32k-64k. * GQA is only used on the 2B model, where we've seen other models prefer to save it for larger models. These observations are in no way meant to be criticism - I understand that Llama's hyperparameters are also somewhat arbitrarily inherited from its predecessors like PaLM and GPT-2, and that it's non-trivial to run hyperopt on such large models. I'm just really curious about what findings motivated these choices.