4 ms·
We're clearly heading towards a "Generalist Multimodal Large Language Model" that autonomously selects the appropriate specialized LLM for any given task, rathe
by mdrzn 3y ago
We're clearly heading towards a "Generalist Multimodal Large Language Model" that autonomously selects the appropriate specialized LLM for any given task, rather than requiring us to switch between multiple LLMs. The combination of a mixture of experts approach and multimodality appears to be the way forward. Very excited for the future.
- ilaksh 3y agoI don't think it's accurate to call them language models anymore actually. Large Multimodal Model?
- phh 3y agoYou're right, that calling them language model is no longer accurate. But "Language Model" has a precise signification (which is that it tries to create a perfect representation of the language). Saying "multimodal model", you're not really saying what does it try to represent. "World Model" or "Human Model" would be more accurate at this point
- t_mann 3y agoWorld model sounds too much of physics than it deserves imho. If you want to name it after what's being modeled (which it doesn't need to be, it could also be named after the methods used eg), why not just call it something like general token model?
- throwanem 3y agoWhat they model is meaning. I'd call them "semantic models".
- Smith42 3y ago"Large Observation Model" has a nice ring to it
- ShamelessC 3y agoI’m fond of Large Sequence Model, which I’ve seen used a bit recently. If your data can be defined as a sequence of unique symbols, actions, letters, tokens, patches, whatever - a transformer is a great way to model it.
- cs702 3y agoI'm partial to Large World Model (LWM). Large Sequence Model (LSM) is OK too.
- kristjansson 3y agoAs long as the underlying representation is a sequence of (embeddings of) tokens , and the training depends on next-token prediction, they should still be called language models. The 'language' part of 'language models' is about the architecture, not the modalit(y/ies).
- CuriouslyC 3y agoIs it a "language" model though, if you have a multimodal autoencoder that converts input into concept tokens, and you do sequence extension in concept space?
- kristjansson 3y agoUnambiguously, yes. To elaborate a bit, that's effectively what happens in a natural-language-only LLM. Text tokens are immediately mapped to wide embeddings, which are processed by a transformer, producing another embedding, which is decoded back to a discrete token. To a first approximation, unified multimodal models substitute f: text -> [embedding] with g: image -> [embedding,] and act on a sequence like [*f(t), *g(i), ...]. So... the multimodal model maps text/image parts to wide embeddings, which are processed by a transformer producing another embedding, which is decoded back to a discrete text/image part. Which sounds somewhat similar to the former :). Of course the function g is a bit more complicated, and decoding takes more work, but the underlying machinery is unchanged. The ability to handle multi-modal inputs/outputs uniformly is exactly what's so exciting about work like this!
- jasfi 3y agoApparently used in GPT-4 and coming in Gemini 1.5.
- kristjansson 3y agoTFA is the exact opposite of that though? It's a single model that acts on a single representation, which is derived from (and mapped back to) different input (output) modalities.
- 3abiton 3y agoAre there any MoE multimodal approaches?