4 ms·

Their tech report says one inference deployment is around 400 GPUs...

• fspeech 2y ago
  You need that to optimize load balancing. Unfortunately that gain is not available to small or individual deployment.