3 ms·
Why is the 32K context only twice as expensive as the 8K context? Are they using sparse attention or something? I don't think flash attention on it's own can e
by sebzim4500 4y ago
Why is the 32K context only twice as expensive as the 8K context?
Are they using sparse attention or something? I don't think flash attention on it's own can explain it.
EDIT: Oh right, if the cost is per token so if you actually fill the context then it is 8x more, which makes much more sense.
- cscheid 4y agoAsk it the other way around, "Why is the 8K context only twice as cheap as the 32K context?" and your original answer is clearer: because they think the demand curve supports it. Price is not determined by cost, but by how much people are willing to pay.
- sebzim4500 4y agoNo I'm just an idiot. Since the price is per token the actual ratio is more like 8x more (assuming you fill both contexts) which is plausibly the ratio between the costs.
- Maxion 4y agoIn the end the cost for you is how many tokens consumed, if you can get more value out of one request with 32k context that multiple with lower, then it’s cheaper (and faster). Some problems also require a larger context to be able to solve.