3 ms·

Isn't equating active parameters with cost a little unfair since you still need full memory for all the inactive parameters?

• tartrate 3y ago
  Well, since it affects inference speed it means you can handle more in less time, needing less concurrency.
• sa-code 3y ago
  Fewer parameters at inference time makes a massive difference in cost for batch jobs, assuming vram usage is the same