5 ms·
ARIA: An Open Multimodal Native Mixture-of-Experts Model
- vessenes 2y agoThis looks worth a try. Great test results, very good example output. No way to know if it’s cherry picked / overtuned without giving it a spin, but it will go on my list. Should fit on an M2 Max at full precision.
- Onavo 2y agoWhat's the size of your M2 Max memory?
- treefry 2y agoLooks like 64GB or more
- SubiculumCode 2y agoHow do you figure out the required memory? The MoE aspect complicates it.
- ProofHouse 2y agoEach model added, no?
- vessenes 2y agoIt does; in this case, though, a 25b f16 model will fit. The paper mentions an A100 80G is sufficient but a 40 is not; M2 Max has up to 192G. That said, MoEs are popular in lower memory devices because you can swap out the experts layers -- their expert layers are like 3-4b parameters, so if you are willing to have a sort of pause on generation where you load up the desired expert, you could do it in a lot less RAM. They pitch the main benefit here as faster generation, it's a lot less matmul to do per token generated.
- SomewhatLikely 2y ago"Here, we provide a quantifiable definition: A multimodal native model refers to a single model with strong understanding capabilities across multiple input modalities (e.g. text, code, image, video), that matches or exceeds the modality specialized models of similar capacities."
- niutech 2y agoI’m curious how it compares with recently announced Molmo: https://molmo.org/ https://molmo.org/
- bsenftner 2y agoKnow of where Molmo is being discussed? Looks interesting.
- deleted 2y ago[deleted]
- espadrine 2y agoThe Pixtral report[0] compares positively to Molmo. (Also, beware, molmo.org is an AI-generated website to absorb through SEO Allen AI’s efforts; the real website is molmo.allenai.org. Note for instance that all tweets listed here are from fake accounts since suspended: https://molmo.org/#how-to-use https://molmo.org/#how-to-use) [0]: https://arxiv.org/pdf/2410.07073 https://arxiv.org/pdf/2410.07073
- petemir 2y agoModel should be available for testing here [0], although I tried to upload a video and got an error in Chinese, and whenever I write something it says that the API key is invalid or missing. [0] https://rhymes.ai/ https://rhymes.ai/
- theanonymousone 2y agoIn an MoE model such as this, are all "parts" loaded in Memory at the same time, or at any given time only one part is loaded? For example, does Mixtral-8x7B have the memory requirement of a 7B model, or a 56B model?
- dartos 2y agoCloser to 56. All part are loaded in as any could be called upon to generate the next token.
- theanonymousone 2y agoAha, so it's decided per token, not per input. I thought at first the LLM chooses a "submodel" based on the input and then follows it to generate the whole output. Thanks a lot.
- dartos 2y agoYeah, this image helped solidify that for me. https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ea80480-0809-4626-86e5-4775e44132ab_1403x805.png https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr... Each different color highlight is a generated by a different expert. You can see that the "experts" are more experts of syntax than concepts. Notice how the light blue one almost always generates puncuation and operators. (until later layers when the red one does so) I'm honestly not too sure the mechanism behind which experts gets chosen. I'm sure it's encoded in the weights somehow, but I haven't gone too deep into MoE models.
- MacsHeadroom 2y agoI see a whitespace expert, a punctuation expert, and a first word expert. It's interesting to see how the experts specialize.
- 2y ago
- cantSpellSober 2y ago> outperforms Pixtral-12B and Llama3.2-11B Cool, maybe needs of a better name for SEO though. ARIA has meaning in web apps.
- panarchy 2y agoThey could call it Maria (MoE Aria) won't help with standing out in searches however. Maybe MarAIa so it would be more unique. I'm here all night if anyone else needs some other lazy name suggestions.