3 ms·
How does this work in terms of utilization? The isolation presumably means buying gpu capacity and only using a %?
by Havoc 3y ago
How does this work in terms of utilization? The isolation presumably means buying gpu capacity and only using a %?
- asabla 3y agoBasically you get N tokens/second (or if it was minute, can check tomorrow if you're really interested) per deployment. So if you would outgrow on deployment, just deploy another one (with the associated costs of course). One deployment = a deployed model which you can query On top of that, depending on the model you're using, you also see a cost increment for each 1000 request you make.
- Havoc 3y agoAh right so it is somewhat shared. Not like your own gpu type situation
- weird-eye-issue 3y agoWhy did you make that assumption?