3 ms·
this is not the point - we were previously billed by the number of reasoning tokens produced. now we aren't. so this means, even if Astra uses a lot of reasonin
by simianwords 1mo ago
this is not the point - we were previously billed by the number of reasoning tokens produced. now we aren't. so this means, even if Astra uses a lot of reasoning we may not be billed for it because they aren't real tokens.
what they could be doing is billing us by virtual tokens meaning number of loops?
but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.