3 ms·

How would you respond to the argument that the size of the database should be accounted for when computing the size of the total model? How does the latency of

• mountainriver 5y ago
  I’ve been thinking on this too, the real driver is just that it’s hard to scale models to that size, I think with the MoE work happening like GLAM it will hopefully be easier to scale models in a distributed fashion.
• jayalammar 5y ago
  The size of the database, the training set, the details of the architecture, as well as results on benchmark tasks should all be considered in the comparison. I'm also a fan of Behavioral Testing [1]. Parameter count is not very accurate measure of model performance. Mixture of Expert models like the Switch Transformer [2] can be 1 trillion parameters in size, but are not 5X the performance, for example. They clock the retrieval at 10 ms, unclear if that includes the BERT inference, however. My assumption is that it does not. [1] https://arxiv.org/abs/2005.04118 https://arxiv.org/abs/2005.04118 [2] https://arxiv.org/abs/2101.03961 https://arxiv.org/abs/2101.03961