4 ms·
They are almost certainly tokenized in most LLM multi-modal models. https://en.wikipedia.org/wiki/Large_language_model#Multimodality https://en.wikipedia.org/wi
by pseudosavant 2y ago
They are almost certainly tokenized in most LLM multi-modal models. https://en.wikipedia.org/wiki/Large_language_model#Multimodality https://en.wikipedia.org/wiki/Large_language_model#Multimoda...
- viraptor 2y agoAh, an overloaded "tokenizer" meaning. "split into tokens" vs "turned into a single embedding matching a token" I've never heard it used that way before, but it makes sense kinda.