3 ms·
Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit
- LaurensBER 3mo agoIt's really cool and interesting to see the kind of engineering that goes into Xiaomi (and Deepseeks) inference optimizations. Z.ai has also published some interesting papers although I haven't had a chance to go through them yet. It does inspire hope that the Chinese labs seem to be so open although the sceptic in me does wonder what their end game is. Surely, from a purely economic perspective it would be wiser to keep this proprietary and benefit from the increased API traffic?
- ignoramous 3mo ago> the kind of engineering that goes into Xiaomi (and Deepseeks) inference optimizations At Xiaomi, MiMo is now led by Luo Fuli. She is a former Alibaba & DeepSeek employee: https://newsen.pku.edu.cn/news_events/news/people/15385.html https://newsen.pku.edu.cn/news_events/news/people/15385.html (https://archive.vn/I8Pmu https://archive.vn/I8Pmu) / https://e.vnexpress.net/news/tech/personalities/who-is-luo-fuli-the-30-year-old-ai-genius-girl-behind-deepseek-s-success-4953445.html https://e.vnexpress.net/news/tech/personalities/who-is-luo-f... (https://archive.vn/sb3B6 https://archive.vn/sb3B6) Don't know if it is due to Luo, but it is striking how similar performance & pricing of the models, DeepSeek v4 Pro & MiMo v2.5 Pro, is.
- xyzsparetimexyz 3mo agoChrist, she had such a nicer vibe than Altman or Amodel.
- bee_rider 3mo agoThe bet could be that they’ll ultimately be able to sell hardware capable enough of running local models comfortably.
- notnullorvoid 3mo agoIf China is able to undercut Nvidia on high performance local AI hardware, they will pop the AI bubble in a matter of days. They wouldn't even need to make something equivalent to the latest hardware. A Chinese RTX 3090 equivalent would be enough.
- onlyrealcuzzo 3mo agoStandard commoditize your complement.
- trollbridge 3mo agoTheir game? Sell me tokens instead of me buying them from an American lab for a higher price. Publishing open weights gives me more confidence in the model, and ironically makes me less anxious about making sure I can replace the cloud usage with a local alternative. Whereas I’m very nervous right now with relying on 5.6-Sol - what if they triple the price, nerf it, etc.?
- TurdF3rguson 3mo ago> Publishing open weights gives me more confidence in the model Why? It's not like you can audit weights like you can with code. > what if they triple the price, nerf it, etc.? What if an open weights infra provider does that? What's the difference?
- alightsoul 3mo agoyou keep the model. it's never deprecated. with closed ai, you are forced into a new more expensive model every few months. if an open model infra provider does that, you simply switch to another one. it's not in their interest to do that.
- TurdF3rguson 3mo agoThat's not really true though, providers are deprecating models and I have at least 10 emails to prove it.
- alightsoul 3mo agoNothing stops you from downloading the model and hosting it on a cloud virtual machine
- TurdF3rguson 3mo agoCommon sense does, but other than that I suppose you're right.
- 3mo ago
- yogthos 3mo agoWhat Chinese firms are doing makes perfect sense from the commercial perspective actually because they understand how a classic commoditization spiral works. The reality is that models themselves are general commodities and there's just not enough difference between them. A company can get ahead of others by a few months, but then the rest quickly close the gap. It's a really low margin business because there's no way to differentiate yourself. Chinese companies know that there's no profit in general purpose models in the long run, and they're treating models as shared infrastructure akin to Linux. They're amortizing the cost of research by keeping models open, and rapidly closing the gap and driving prices towards the marginal cost of inference. The money is going to be in customization niches. Companies will charge to tune models for specific use cases and charge support for that. There's also going to be money at the bottom for hardware vendors making chips and memory. But the middle tier of generic LLMs is seeing involution where there's relentless competition driving profits towards the bottom.
- CuriouslyC 3mo agoThe Chinese labs incentives is to run inference for the world, because inference can run on the homegrown Chinese chips (giving them a guaranteed market for their hardware) and they have cheap plentiful power. The US frontier labs have an incentive to do deals with large firms to act like a contract research organization, taking royalties on creations/discoveries. Alex Karp called this out in his rant ("Why charge for tokens, take a %") and he's basically right about this.
- jingpostmedia 3mo ago[flagged]
- jononor 3mo agoYeah China has a huge (and growing) advantage in power generation. And they have been looking to break into high-end chip manufacturing. The latter has high cost of entry and needs large scale to become viable. AI inference on own hardware would allow them to bootstrap the chip demand. Both for memory and accelerators.
- 3mo ago
- alightsoul 3mo agoits a governement mandate that states that AI research must be open source ,that's one benefit of communism
- yorwba 3mo agoThere is no such mandate. ByteDance keeps their models closed. So does iFlyTek. Qwen Max is closed as well.
- alightsoul 3mo agoBut they are not groundbreaking. They are simply copies of other similar Chinese models The mandate is worded differently from what I said
- yorwba 3mo agoWhat is the literal wording of this "mandate" then?
- ricardobeat 3mo agoSuch a well written article, refreshing to read in between all the slop. I've used Mimo extensively in the past few months, can't wait to see what v3 will bring.
- trollbridge 3mo agoAre you using UltraSpeed? It's my favourite thing about MiMo.
- ricardobeat 3mo agoI did, it’s amazing. Shame it’s going away and not available permanently.
- zozbot234 3mo agoIt's a very nice summary of KV cache storage requirements for the leading open weights models, but ultimately it still shows the DeepSeek V4 series winning by a huge margin on that front. And DeepSeek actually does quite well on needle-in-haystack context recall benchmark so it seems like the outstanding context compression it has isn't even impacting smarts all that much. Keep in mind that KV cache memory requirements usually has a direct impact on how much you can parallelize inference on both large-scale platforms and lower-end edge compute, so this is a very important metric.
- killingtime74 3mo agoWill they lower the price or is this documenting past work
- limbicsystem 3mo agoTheir pricing is incredible on the token plan - something like 50b tokens for $60!
- polski-g 3mo agoNo that's not right. It's 50b credits, not tokens. What is a credit? Nobody knows. Someone did the math a few months ago and paying API prices was the same as the monthly subscription.
- trollbridge 3mo agoYep, roughly the same. If you max out the sub each month it’s roughly 20% cheaper if you also carefully use all their promotions. I’ve thrown $50 at it, use UltraSpeed liberally and have yet to exhaust it.
- LaurensBER 3mo agoI've found these kind of models (Deepseek + Xiaomi) to be absolutely excellent when it comes to writing documentation for code. We have a bunch of internal tasks that need to be documented for a non-technical team. I added 20 USD in credits for the Xiaomi models a while ago and they've been happily writing and updating hundreds if not thousand of pages and I still have 7 USD left!
- crazylogger 3mo agoTo be fair while almost no company publishes what a "credit" or what a 5h window in their subscription plan really is in terms of input/cached input/output tokens, Xiaomi does: https://mimo.mi.com/docs/en-US/tokenplan/Token%20Plan/subscription https://mimo.mi.com/docs/en-US/tokenplan/Token%20Plan/subscr...
- amanharshx 3mo agoThis is really neat. They've done some really impressive engineering here ngl with the ~95% KV cache hit rates. MiMo and Deepseek both do seem to get the job done for me. Hope they can keep this pace while staying open source.
- SwellJoe 3mo agoI've been saying efficiency is the next "frontier" in AI, at least for LLMs that people use daily. Companies have started to really balk at token costs from the major providers, and there's some evidence that the cheaper Chinese models are chipping away at Anthropic/OpenAI dominance from below (as cheaper Chinese products have done in other industries for many years). I continue to think that the vendor that figures out how to efficiently serve a Good Enough model at a much lower price will be the one to win in the end. DeepSeek is remarkably efficient at caching and their cached token rates are crazy cheap; using it with Reasonix is free real estate, like 97% cached tokens, ends up costing like 30 cents an hour to use DeepSeek V4 Pro. I hadn't dug into MiMo's caching behavior as I haven't used it as heavily as DeepSeek, but this indicates it's close to DeepSeek. At this point I don't see a reason to use Sonnet, Haiku, or the smaller GPT models, because their API rates are much higher than the best models from MiMo and DeepSeek. We're still figuring out the upper bounds of capability and I am still finding next generation models are unlocking things I couldn't readily accomplish before and I'm willing to pay more for them (at least, I'll pay the $100 or $200 subscription rates for them, I couldn't justify the token expense for most of my dev work), but we're already at a point where someone building standard CRUD web apps doesn't need the top models and probably doesn't benefit much from using them.
- mapontosevenths 3mo agoI agree, but maybe for different reasons. I think Karpathy is right. We need models that reason, not models that memorize. Karpathy calls it a "Cognitive Core", and it's essentially a small model that learns to reason and look up the data it needs as opposed to a giant model that memorizes all the data in the world and tries to process large chunks of it all at once with every thought. I think it will be based on the thing that grokking, the lottery ticket hypothesis, and the universal weight subpspace hypothesis all point to. Eventually someone will figure out how to build it and the entire economy that we've now built on top of the wacky idea that nothing can possibly ever get more efficient will collapse overnight. Sometimes I wonder how much Nvidia would pay someone not to release a thing like that, and then I wonder if that's already happened.
- 3mo ago
- geenkeuse 3mo ago[dead]
- nmfisher 3mo agoRelated, I was given access to mimo-v2.5-ultraspeed, which is amazing. This is now my expectation for speed, it’s fast enough for me to stay mentally engaged rather than getting distracted waiting for the agent to churn.
- embedding-shape 3mo agoThe -spark variant of GPT was a ton of fun indeed, such a shame it's so dumb though so really hard to rely on. If you were to compare the quality of mimo-v2.5-ultraspeed with anything from OpenAI/Anthropic, where would it be placed ~more or less in your view?
- perching_aix 3mo agoIs it the same quality as base mimo v2.5, or different? I've been enjoying regular mimo v2.5 quite a bit via opencode, if ultraspeed provides the same quality, that's crazy.
- nmfisher 3mo agoI'm not a regular v2.5 user, so I don't really know. But given the TileRT team write-up says the entire network gets quantized to FP8 (and experts get quantized to FP4)[0], I'm assuming there's at least a modest drop in quality. [0] https://www.tilert.ai/blog/breaking-1000-tps.html https://www.tilert.ai/blog/breaking-1000-tps.html
- deleted 3mo ago[deleted]