3 ms·

Calculate the number of language model tokens for a string

• sva_ 2y ago
  import tiktoken enc = tiktoken.encoding_for_model('o1-mini') len(enc.encode('some string'))
  • NitpickLawyer 2y ago
    Does that work on anything other than openai?
    • topicseed 2y ago
      No but most models have their own libs you can use