Someone [1] inspected dimensions of the embedding component of model and it seems GP was on the right track. Assuming I understood correctly in [2], it does see

Someone [1] inspected dimensions of the embedding component of model and it seems GP was on the right track. Assuming I understood correctly in [2], it does seem to be the embedding of the input tokens which is passed directly into each layer. I have not looked at the model but since the embedding dimension of 256 seems quite small (for reference according to [3] the old Gemma 1B had 1152 dimension input embedding), I'm guessing that this is not done _in lieu_ of the main input embedding to first layer, but in addition to it. [1] https://twitter.com/cccntu/status/1925043973170856393 https://twitter.com/cccntu/status/1925043973170856393 [2] https://news.ycombinator.com/edit?id=44048662 https://news.ycombinator.com/edit?id=44048662 [3] https://developers.googleblog.com/en/gemma-explained-whats-new-in-gemma-3/ https://developers.googleblog.com/en/gemma-explained-whats-n...