This is a great piece of data, but only a piece of the actual question that we need to answer, which is: For a given input, how many tokens will be used for an

This is a great piece of data, but only a piece of the actual question that we need to answer, which is: For a given input, how many tokens will be used for an answer, and how high quality will that answer be? Measuring the tokenizer is just one input into the cost-benefit tradeoff.