3 ms·

Which well explains why they aren't competitive with OpenAI yet.

• anankaie 2y ago
  You are probably right that there is massive disparity between the two re: compute. With that said, it is conceivably possible to use a larger number of weaker chips rather than fewer bigger ones, and come out ahead per unit-time. Also, given their strategy seems to be trying to do as much on-device as possible, they are targeting smaller models, so they likely have less of a packing problem with smaller chips. So in the goal of producing a strong model, it could go either way; especially for smaller models, data seems to be much more important than compute, as per the “Textbooks are all you need” paper (https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644), especially if you are not looking for a fully generalist model.
• boroboro4 2y ago
  There is no connection whatsoever between hardware used to train models and performance of the models if hardware permits truly large models. Google's hardware does permit it.