5 ms·
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
- mdrzn 3y agoWe're clearly heading towards a "Generalist Multimodal Large Language Model" that autonomously selects the appropriate specialized LLM for any given task, rather than requiring us to switch between multiple LLMs. The combination of a mixture of experts approach and multimodality appears to be the way forward. Very excited for the future.
- ilaksh 3y agoI don't think it's accurate to call them language models anymore actually. Large Multimodal Model?
- phh 3y agoYou're right, that calling them language model is no longer accurate. But "Language Model" has a precise signification (which is that it tries to create a perfect representation of the language). Saying "multimodal model", you're not really saying what does it try to represent. "World Model" or "Human Model" would be more accurate at this point
- t_mann 3y agoWorld model sounds too much of physics than it deserves imho. If you want to name it after what's being modeled (which it doesn't need to be, it could also be named after the methods used eg), why not just call it something like general token model?
- throwanem 3y agoWhat they model is meaning. I'd call them "semantic models".
- Smith42 3y ago"Large Observation Model" has a nice ring to it
- ShamelessC 3y agoI’m fond of Large Sequence Model, which I’ve seen used a bit recently. If your data can be defined as a sequence of unique symbols, actions, letters, tokens, patches, whatever - a transformer is a great way to model it.
- cs702 3y agoI'm partial to Large World Model (LWM). Large Sequence Model (LSM) is OK too.
- kristjansson 3y agoAs long as the underlying representation is a sequence of (embeddings of) tokens , and the training depends on next-token prediction, they should still be called language models. The 'language' part of 'language models' is about the architecture, not the modalit(y/ies).
- CuriouslyC 3y agoIs it a "language" model though, if you have a multimodal autoencoder that converts input into concept tokens, and you do sequence extension in concept space?
- kristjansson 3y agoUnambiguously, yes. To elaborate a bit, that's effectively what happens in a natural-language-only LLM. Text tokens are immediately mapped to wide embeddings, which are processed by a transformer, producing another embedding, which is decoded back to a discrete token. To a first approximation, unified multimodal models substitute f: text -> [embedding] with g: image -> [embedding,] and act on a sequence like [*f(t), *g(i), ...]. So... the multimodal model maps text/image parts to wide embeddings, which are processed by a transformer producing another embedding, which is decoded back to a discrete text/image part. Which sounds somewhat similar to the former :). Of course the function g is a bit more complicated, and decoding takes more work, but the underlying machinery is unchanged. The ability to handle multi-modal inputs/outputs uniformly is exactly what's so exciting about work like this!
- jasfi 3y agoApparently used in GPT-4 and coming in Gemini 1.5.
- kristjansson 3y agoTFA is the exact opposite of that though? It's a single model that acts on a single representation, which is derived from (and mapped back to) different input (output) modalities.
- 3abiton 3y agoAre there any MoE multimodal approaches?
- ilaksh 3y agoI feel like this type of capability or architecture might be the future of interactive agents. The quality of the voices leaves a little to be desired, but otherwise it seems very powerful.
- thfuran 3y agoGiven how quickly we've been churning through sota architectures in the last decade or so, I'd not bet on anything we currently have being "the future"
- cs702 3y agoThe key takeaway for me: Whether data is continuous or discrete, no matter its modality (text, video, music, etc.), we now have an array of proven methods for representing it with discrete tokens, enabling us to use existing sequence modeling architectures (Transformers, linear RNNs). We live in interesting times!
- aaron_wang 3y agoTotally agree. Another takeaway is that data from different modality has some shared intrinisity, so one can be mapped to another.
- justanotherjoe 3y agoWhat's with the dummy github page? Anyway. This feels like the right step forward. Just like OpenAI, i have near religious faith in the transformers architecture. The question is how these modalities can work together better.
- rustcleaner 3y agoSay I am between sophomore and junior, what's the best way to bootstrap from calc+linalg+stat to being able to competently configure and piece these components together into something like this? Any good lecturers or courses?