3 ms·
Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and de
by janalsncm 3mo ago
Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.
But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
- pooploop64 3mo agoTokens in a casino is pretty accurate if you think about it. You never really know what you'll get so it's tempting to "roll" over and over, thinking every roll puts you closer to a bellringer. It can even get addicting for some people.
- HarHarVeryFunny 3mo agoTokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
- janalsncm 3mo agoProviders amortize the compute across a batch. If yours is the only request in the batch it will cost them one full pass through the model. If yours is one of 1024 inputs in the batch the per token cost is 1024x less.
- HarHarVeryFunny 3mo agoApparently hardware depreciation is the dominant cost rather that operational cost (electricity etc), and this is occuring at a fixed rate per the planned replacement lifetime. So, the cost to provide the service is essentially fixed regardless of load, but the revenue they are generating is variable. In practice most GPU's are going to be capacity-maxxed since the providers sell cheap batch APIs that they queue to keep the machine loaded. They'd be losing money over a given time interval if revenue generated during that interval wasn't greater than depreciation (etc), but it seems that will rarely happen.
- janilowski 3mo agoI am not sure if pricing actually reflects inference costs