8 ms·
Qwen3-Omni: Native Omni AI model for text, image and video
- sinpor1 1y ago[flagged]
- simonw 1y agoYou can try it out on https://chat.qwen.ai/ https://chat.qwen.ai/ - sign in with Google or GitHub (signed out users can't use the voice mode) and then click on the voice icon. It has an entertaining selection of different voices, including: *Dylan* - A teenager who grew up in Beijing's hutongs *Peter* - Tianjin crosstalk, professionally supporting others *Cherry* - A sunny, positive, friendly, and natural young lady *Ethan* - A sunny, warm, energetic, and vigorous boy *Eric* - A Sichuan Chengdu man who stands out from the crowd *Jada* - The fiery older sister from Shanghai
- indigodaddy 1y agoI only see Omni Flash, is that the one?
- furiousteabag 1y agosame, did you figure it out?
- simonw 1y agoI think so, you need to click the big jagged audio icon to start a voice session.
- karimf 1y agoIs the Qwen3-Omni-Flash the same as Qwen3-Omni-30B-A3B, or is the Omni-Flash a different closed-source model?
- fivestones 1y agoMy question too
- ltdemey 1y agoIn Section 5 of their [technical report](https://arxiv.org/pdf/2509.17765v1 https://arxiv.org/pdf/2509.17765v1) they mention them "... A comprehensive evaluation was performed on a suite of models, including Qwen3-Omni-30B-A3B- Instruct, Qwen3-Omni-30B-A3B-Thinking, and two in-house developed variants, designated Qwen3- Omni-Flash-Instruct and Qwen3-Omni-Flash-Thinking. These “Flash” models were designed to improve both computational efficiency and performance efficacy, integrating new functionalities, notably the support for various dialects. ..."
- flockonus 1y agoThe voices are really fun, thanks for the laughs :)
- vladgur 1y agoMany of these voices are especially hillarious when you switch the language. In Russian, Ryan sounds like a westerner who started reading Russian words a month ago. Dylan sounds somewhat authentic, while everyone else is a different degree of heavy-asian-accented Russian.
- hadlock 1y agoSpeech input + speech output is a big deal. In theory you can talk to it using voice, and it can respond in your language, or translate for someone else, without intermediary technologies. Right now you need wakeword, speech to text, and then text to speech, in addition to your core LLM. A couple can input speech, or output speech, but not both. It looks like they have at least 3 variants in the ~32b range. Depending on the architecture this is something you could feasibly have in your house in a couple of years or in an expensive "ai toaster"
- CamperBob2 1y agoSeems like a big win for language learning, if nothing else. Also seems possible to run locally, especially once the unsloth guys get their hands on it.
- data-ottawa 1y agoThe opportunities of plugging this into your home automation through tool calls is huge. Ever since ChatGPT added this feature I've been waiting for anyone else to catch up. They're are tons of hands free situations like cooking where this would be amazing ("read the next step please, my hands are covered in raw pork", "how much flour for the roux", "crap, I don't have any lemons, what can I substitute")
- chisleu 1y agoHere is the demo video on it. The video w/ sound input -> sound output while doing translation from the video to another language was the most impressive display I've seen yet. https://www.youtube.com/watch?v=_zdOrPju4_g https://www.youtube.com/watch?v=_zdOrPju4_g
- state_less 1y agoInteresting, the pacing seemed very slow when conversing in english, but when I spoke to it in spanish, it sounded much faster. It's really impressive that these models are going to be able to do real time translation and much more. The Chinese are going to end up owning the AI market if the American labs don't start competing on open weights. Americans may end up in a situation where they have some $1000-2000 device at home with an open Chinese model running on it, if they care about privacy or owning their data. What a turn of events!
- tedivm 1y agoThis is exactly what I do. I have two 3090s at home, with Qwen3 on it. This is tied into my Home Assistant install, and I use esp32 devices as voice satellites. It works shockingly well.
- tomatoman 1y agoSeems interesting setup, do you have it documented anywhere, thinking of building one!
- joshgachnang 1y agoOoo interesting, I'd love to hear more about the esp32's as voice satellites!
- sho_hn 1y agoI assume it's very similar to what Home Assistant's backing commercial entity Nabu Casa sells with the "Home Assistant Voice PE" device, which is also esp32-based. The code is open and uses the esphome framework so it's fairly easy to recreate on custom HW you have laying around.
- tedivm 1y agoFor the physical hardware I use the esp32-s3-box[1]. The esphome[2] suite has firmware you can flash to make the device work with HomeAssistant automatically. I have an esphome profile[3] I use, but I'm considering switching to this[4] profile instead. For the actual AI, I basically set up three docker containers: one for speech to text[5], one for text to speech[6], and then ollama[7] for the actual AI. After that it's just a matter of pointing HomeAssistant at the various services, as it has built in support for all of these things. 1. https://www.adafruit.com/product/5835 https://www.adafruit.com/product/5835 2. https://esphome.io/ https://esphome.io/ 3. https://gist.github.com/tedivm/2217cead94cb41edb2b50792a8bea8e6 https://gist.github.com/tedivm/2217cead94cb41edb2b50792a8bea... 4. https://github.com/BigBobbas/ESP32-S3-Box3-Custom-ESPHome/ https://github.com/BigBobbas/ESP32-S3-Box3-Custom-ESPHome/ 5. https://github.com/rhasspy/wyoming-faster-whisper https://github.com/rhasspy/wyoming-faster-whisper 6. https://github.com/rhasspy/wyoming-piper https://github.com/rhasspy/wyoming-piper 7. https://ollama.com/ https://ollama.com/
- edude03 1y agoThe qwen thinker/speaker architecture is really fascinating and is more in line with how I imagine human multi modality works - IE, a picture of an apple, the text a p p l e and the sound all map to the same concept without going to text first.
- adastra22 1y agoIsn’t that how all LLMs work?
- simonw 1y agoThe existing vision LLMs all work like this, which is most of the major models these days. Multi-modal audio models are a lot less common. GPT-4o was meant to be able to do this natively from the start but they ended up shipping separate custom models based on it for their audio features. As far as I can tell GPT-5 doesn't have audio input/output at all - the OpenAI features for that still use GPT-4o-audio. I don't know if Gemini 2.5 (which is multi-modal for vision and audio) shares the same embedding space for all three, but I expect it probably does.
- adastra22 1y agoWhat I mean is that all processing in an LLM occurs in state space. The next-token prediction is the very last step.
- uniqueuid 1y agoThere are many more weird and complex architectures in models for video understanding. For example, beyond video->text->llm and video->embedding in llm, you can also have an llm controlling/guiding a separate video extractor. See this paper for a pretty thorough overview. Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., Vosoughi, A., Huang, C., Zhang, Z., Liu, P., Feng, M., Zheng, F., Zhang, J., Luo, P., Luo, J., & Xu, C. (2025). Video Understanding with Large Language Models: A Survey (No. arXiv:2312.17432). arXiv. https://doi.org/10.48550/arXiv.2312.17432 https://doi.org/10.48550/arXiv.2312.17432
- pzo 1y agowill be interesting how it will compare with pricing with audio modality comparing to gemini 2.0 flash once many providers offer it. Even though gemini 2.0 flash is quite old I still like it. Very cheap (each second of audio is just 32 tokens), support even more languages, non-reasoning so very fast, big rate limits.
- richardlblair 1y agoI recently needed to scan hundreds of low quality invoices and run them through OCR for invoice numbers and dates. I really took for granted how seamless this is in some applications, and was shocked how much work went into producing decent results. I was obviously really naive. Either way, it gets me excited any time I see progress with OCR. I should give this a try against my (small) dataset.
- tomrod 1y agoI don't think I understand your comment. What were your results for Qwen? Or is that what you meant for how much work was needed?
- mrbonner 1y agoWhat is your point about Qwen? Or is it just a general statement regarding LLM?
- richardlblair 1y agoI did it the old school way with openai as the fallback. All I'm saying is I'm excited to try Qwen to see if it out performs my gnarly algorithm.
- richardlblair 1y agoI just ran Qwen against some of invoices that my gnarly algorithm really struggled with (with openai fallback) and Qwen was able to extract all relevant data without any issues. I'm pretty damn impressed to be honest.
- neilmovva 1y agoThe multilingual example in the launch graphic has Qwen3 producing the text: > "Bonjour, pourriez-vous me dire comment se rendreà la place Tian'anmen?" translation: "Hello, could you tell me how to get to Tiananmen Square?" a bold choice!
- ripped_britches 1y agoWesterners only know it from the massacre but it’s actually just like Times Square for them
- whimsicalism 1y agoonly really a reference with the date or at least 89
- OJFord 1y agoNot really, it's a significant place which is why the protest (and hence massacre) was there, so especially for Chinese people (I expect) merely referencing it doesn't so immediately refer to the massacre, they have plenty of other connotations for it. e.g. if something similar happened in Trafalgar Square, I expect it would still be primarily a major square in London to me, not oh my god they must be referring to that awful event. (In fact I think it was targeted in the 7/7 bombings for example.) Or a better example to go with your translation - you can refer to the Bastille without 'boldly' invoking the histoire of its storming in the French Revolution. No doubt the US media has referred to the Capitol without boldness many times since 6 Jan '21.
- em500 1y agoNot to mention, Tiananmen Square is one of the major tourist destinations in Beijing (similar to National Mall in Washington DC), for both domestic and foreign visitors.
- hedgehog 1y agoThis is true. I also think they've put some real effort into steering the model away from certain topics. If you ask too closely you'll get a response like: "As an AI assistant, I must remind you that your statements may involve false and potentially illegal information. Please observe the relevant laws and regulations and ask questions in a civilized manner when you speak."
- simonw 1y agoThe model weights are 70GB (Hugging Face recently added a file size indicator - see https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct/tree/main https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct/tree... ) so this one is reasonably accessible to run locally. I wonder if we'll see a macOS port soon - currently it very much needs an NVIDIA GPU as far as I can tell.
- dcreater 1y agois there an inference engine for this on macos?
- simonw 1y agoNot yet as far as I can tell - might take a while for someone to pull that together given the complexity involved in handling audio and image and text and video at once.
- growthwtf 1y agoA fun project for somebody who has more time than myself would be to see if they can get it working with the new Mojo stuff from yesterday for Apple. I don't know if the functionality would be fully baked out enough yet to actually do the port successfully, but it would be an interesting try.
- wsintra2022 1y agoNew Mojo stuff from Apple?
- wsintra2022 1y agoNvm found it https://news.ycombinator.com/item?id=45326388 https://news.ycombinator.com/item?id=45326388
- a_e_k 1y agoThat's at BF16, so it should fit fairly well on 24GB GPUs after quantization to Q4, I'd think. (Much like the other 30B-A3B models in the family.) I'm pretty happy about that - I was worried it'd be another 200B+.
- vunderba 1y agoNeat. I threw a couple simple audio clips at it and it was able to at least recognize the instrumentation (piano, drums, etc). I haven't seen a lot of multimodal LLM focus around recognizing audio outside of speech, so I'd love to see a deep dive of what the SOTA is.
- wills_forward 1y agohttps://x.com/whowillrickwill/status/1920723985311903767 https://x.com/whowillrickwill/status/1920723985311903767
- yencabulator 1y agoThe Break Fast Club.
- ndr_ 1y agoAny insights into what "native video support" actually means? Is it just good at interpreting consecutive full frame images taken at intervals (thus missing out on fast events) or is there something more elaborate to it?
- nmitchko 1y agoNext steps for AI in general: - additional modalities - Faster FPS (inferences per second) - Reaction time tuning (latency vs quality tradeoff) for visual and audio inputs/outputs - built-in planning modules in the architecture (think premotor frontal lobe) - time awareness during inference (towards an always inferring / always learning architecture)
- __mharrison__ 1y agoI'm running Q3-Next on my MBP and seeing ~ GPT4.1 performance from it. Impressive what these local models are now capable of.
- rapatel0 1y agoThe real point of leverage for here is performance/size. Getting traction in the open weights space kinda forces that the models need to innovate on efficiency. This means the open weight models may get leverage that the closed weight ones don't think about. If we had some aggregated cluster reasoning mechanisms, When would 8x 30B models running on an h100 server out perform in terms of accuracy 1 240B model on the same server.
- syndacks 1y agoDoes this support realtime speech to speech via API? If so, where is this hosted/documented? I wasn’t able to see any info. I’d love to use this in lieu of OAIs (expensive) real time speech to speech offering.
- syndacks 1y agohttps://www.alibabacloud.com/help/en/model-studio/realtime?spm=a2c63.p38356.help-menu-2400256.d_0_2_5.7bc833b7Wgu4et https://www.alibabacloud.com/help/en/model-studio/realtime?s...
- karimf 1y agoThank you. I'm looking for this as well. The realtime model is a closed-source model and it's different than the open Qwen3-Omni-30B-A3B, right? I wonder how hard is it to turn the open-source model to be a realtime model.
- nmfisher 1y agoWhy do you say they are different models? I've been looking at this today and haven't seen anything explicitly state that.
- karimf 1y agoThis is just my assumption given that they listed a lot of different models here: https://modelstudio.console.alibabacloud.com/?spm=a3c0i.28768018.1579141730.2.5cd35c10kPPWff&tab=doc#/doc/?type=model&url=2840914&modelId=qwen-plus https://modelstudio.console.alibabacloud.com/?spm=a3c0i.2876... This is an older link, but they listed two different sections here, commercial and open source models: https://modelstudio.console.alibabacloud.com/?tab=doc#/doc/?type=model&url=2867839 https://modelstudio.console.alibabacloud.com/?tab=doc#/doc/?... For the realtime multimodal, I'm not seeing the open source models tab: https://modelstudio.console.alibabacloud.com/?tab=doc#/doc/?type=model&url=2880812 https://modelstudio.console.alibabacloud.com/?tab=doc#/doc/?...
- j-bos 1y agoI usually ask these models to tell me a short story, and most times the prose is stiff and the story reads like a mass market straight to KDP kids book. But wow, first shot generated something, light, mildly funny, and chill. Quite a surprise. Pasted here for your own judgement: *Title: The Last Lightbulb* The power had been out for three days. Rain drummed against the windows of the old cabin, and the only light came from a flickering candle on the kitchen table. Maggie, wrapped in a wool blanket, squinted at the last working flashlight. “We’ve got one bulb left, Jack. One.” Jack, hunched over a board game he’d dug out of the closet, didn’t look up. “Then don’t turn it on unless you’re reading Shakespeare or delivering a baby.” She rolled her eyes. “I need it to find the can opener. I’m not eating cold beans with my fingers again.” Jack finally glanced up, grinning. “You did that yesterday and called it ‘rustic dining.’” “Desperate times,” she muttered, clicking the flashlight on. The beam cut through the gloom—and immediately began to dim. “No—!” Jack lunged, but too late. The light sputtered… then died. Silence. Then Maggie sighed. “Well. There goes civilization.” Jack leaned back, chuckling. “Guess we’re officially cavemen now.” “Cavewoman,” she corrected, fumbling in the dark. “And I’m going to bed. Wake me when the grid remembers we exist.” As she shuffled off, Jack called after her, “Hey—if you find the can opener in the dark, you’re officially magic.” A pause. Then, from down the hall: “I found socks that match. That’s basically witchcraft.” Jack smiled into the dark. “Goodnight, witch.” “Goodnight, caveman.” Outside, the rain kept falling. Inside, the dark didn’t feel so heavy anymore. — The End —
- adastra22 1y agoThat's not bad actually.
- djohnston 1y agoDoes anyone have good resources for learning about multimodal models? I'm not sure where to begin.
- nothrowaways 1y agoSo much detailed documentation.
- wahnfrieden 1y agoHow’s the Japanese performance?
- ttul 1y agoとてもおいしです！
- indigodaddy 1y agoAll, what's the best model right now to bring a photo to life (create a short video from a photo etc) ?
- lossolo 1y agoOpen source? Wan 2.2 i2v
- elng 1y ago[flagged]
- hanghhahhh 1y ago[flagged]
- hanghhahhh 1y ago[flagged]
- Jgoauh 1y agohas anyone figured out how to ask a question with text and have it speak the answer in the app ? i can generate text or talk but not jump between i was lead to believe that it was possibly by the first image here: https://qwen.ai/blog?id=1f04779964b26eacd0025e68698258faacc784ba&from=research.latest-advancements-list https://qwen.ai/blog?id=1f04779964b26eacd0025e68698258faacc7... that shows a voice output (top left) next to the written out detail of the thinking mode
- wccrawford 1y agoI tried using it for a planning a trip that we're taking soon. We've actually already planned the trip, but thought it'd be fun. It asked me follow up questions, and then took forever. I finally clicked the button to see what was happening and it had started outputting Chinese in the middle. I gave up on it.
- gercius 1y agoI kept asking if it knows what "sussy baka" means, but it constantly ignored "sussy" part, even when I specifically asked to not ignore it, interesting.