3 ms·
Model inference is actually comparatively very cheap. If you have the resources to train a model, you most definitely have the resources to run it.
by dwohnitmok 5y ago
Model inference is actually comparatively very cheap. If you have the resources to train a model, you most definitely have the resources to run it.
- shock-value 5y agoDoes that hold as the workload scales up? E.g. could this or similar models be used as part of a general-purpose search engine whereby (at least) one inference is completed per unique search? Aside from computation, I know these models consume an intense amount of memory -- would that scale horizontally easily / economically? Would it need to?
- niklasd 5y agoGoogle is using BERT for most/all search queries [1]. BERT is far smaller than Megatron (340M < 530B), but still "big" in a traditional sense (in the blog they say they are using TPUs for inference). [1] https://blog.google/products/search/search-language-understanding-bert/ https://blog.google/products/search/search-language-understa...
- shock-value 5y agoInteresting, though as you noted it's less than 1/1000th the size of the 530B model and according to the article is only used in about one in ten U.S.-based English-language searches, at least when that was written.
- gwern 5y agoDon't forget that Google is not stopping at BERT. I dunno if they used any of the T5s or Switch Transformers in production, but they've said that MUM (O(100b)?) is going to run for production search queries, and no one knows what 'Pathways' is (multimodal O(1000b) MoE?) that Jeff Dean has been enthusing about.
- manquer 5y agoNot necessarily, you train once , you run inferences billions of times maybe. The compute required could be beyond your resources.