5 ms·

That is a crazy speedup!!

• GordonS 3y ago
  Is it really? Going from CPU to GPU, I would have expected a much better improvement.
  • qwertox 3y ago
    I feel the same. For example some stats from Whisper [0] (audio transcoding, 30 seconds) show the following for the medium model (see other models in the link): --- GPU medium fp32 Linear 1.7s CPU medium fp32 nn.Linear 60.7s CPU medium qint8 (quant) nn.Linear 23.1s --- So the same model runs 35.7 times faster on GPU, and compared to an "optimized" model still 13.6. I was expecting around an order or magnitude of improvement. Then again, I do not know if in the case of this article the entire model was in the GPU, or just a fraction of it (22 layers) and the remainder on CPU, which might explain the result. Apparently that's the case, but I don't know much about this stuff. [0] https://github.com/MiscellaneousStuff/openai-whisper-cpu https://github.com/MiscellaneousStuff/openai-whisper-cpu
    • rahimnathwani 3y ago
      You last paragraph is correct. Only about half the model was running on the GPU.
    • pavelstoev 3y ago
      Training and inference on GPUs significantly underutilize …the GPUs. So tuning and various tricks need to be applied to achieve dramatic performance gains. If I am not good at cooking, giving me a larger kitchen will not make me faster or better.
  • rahimnathwani 3y ago
    You can think of it this way: if half the model is running on the GPU, and the GPU is infinitely fast, then the total calculation time would go down by 50%, compared with everything running on the CPU.
    • ethbr0 3y ago
      Ref Amdahl's Law: https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law
      • quickthrower2 3y ago
        Wow seems like common sense turned into a law. Maybe I can get a law :-). Anyone who has commuted on public transport probably knows this intuitively. (Using a kick scooter instead of walking cut my travel time by a good 5% which was excellent, as I still needed to be on a bus where that made no difference.)
        • ethbr0 3y ago
          It's a little more formalized and generalized observation: https://en.m.wikipedia.org/wiki/Amdahl%27s_law#Derivation https://en.m.wikipedia.org/wiki/Amdahl%27s_law#Derivation But yeah! There's a lot of common sense that, with some mathematical formalization, yields useful and extensible laws. Amdahl's has been robust because it opined on the 60s supercomputer coprocessor architectures, and then in more recent decades with consumer multicore chips. Laws aren't famous because they're insightful -- they're famous because they're useful.
        • mejutoco 3y ago
          These laws make more sense if one reads them as "heuristics", which is what they are.