2 ms·
70 somethings per second, is slow. So that means it does take a very significant amount of resources, considering it's running on the same or better hardware. T
by vrighter 2y ago
70 somethings per second, is slow. So that means it does take a very significant amount of resources, considering it's running on the same or better hardware. To sustain 70 things per second for thousands of users, it gets expensive really quickly.
- lukev 2y agoMy point is that at current API pricing the users are paying enough to cover inference costs.