This is the right way of thinking end-to-end. Tokenizer changes are one piece to understand for sure, but as you say, you need to evaluate $/task not $/token o

This is the right way of thinking end-to-end. Tokenizer changes are one piece to understand for sure, but as you say, you need to evaluate $/task not $/token or #tokens/task alone.