It looks like you can use the gemma tokenizer to count tokens up to at least the 1.5 models. The docs claim that there's a local compute_tokens function in goog

It looks like you can use the gemma tokenizer to count tokens up to at least the 1.5 models. The docs claim that there's a local compute_tokens function in google-genai, but it looks like it just does an API call. Example for 1.5: https://github.com/googleapis/python-aiplatform/blob/main/vertexai/tokenization/_tokenizer_loading.py https://github.com/googleapis/python-aiplatform/blob/main/ve...