Some details that might interest you from SemiAnalysis [1] just published yesterday. There's quite a bit that goes into optimizing inference with lots of dials

Some details that might interest you from SemiAnalysis [1] just published yesterday. There's quite a bit that goes into optimizing inference with lots of dials to turn. One thing that does seem to have a large impact is batch size which is a benefit of scale. 1. https://www.semianalysis.com/p/inference-race-to-the-bottom-make https://www.semianalysis.com/p/inference-race-to-the-bottom-...