2 ms·
Most heavy text-processing applications I know actually tokenize text to words (well, technically terms), keep a lexicon mapping from the term ID to textual rep
by nostrademons 13y ago
Most heavy text-processing applications I know actually tokenize text to words (well, technically terms), keep a lexicon mapping from the term ID to textual representation, and then work in term space. Individual letters are usually not semantically meaningful in most languages (both human and machine), and so your analysis becomes much easier if you operate in a space that is semantically meaningful.