Hmm I have no background but how can they determine what a 'term' is for the purposes of frequency without some form of tokenization unless they are using an ar

Hmm I have no background but how can they determine what a 'term' is for the purposes of frequency without some form of tokenization unless they are using an arbitrary maximum length on 'term' sizes and are eliminating small terms.