3 ms·I'd think it's really good at tokenization to word. That's exactly what it's trained to do.by lamroger 3y agoI'd think it's really good at tokenization to word. That's exactly what it's trained to do.