4 ms·

Does Qwen3-Omni support real-time conversation like GPT-4o? Looking at their documentation it doesn't seem like it does. Are there any open weight models that

• dsrtslnd23 10mo ago
  it seems to be able to do native speech-speech
  • sosodev 10mo ago
    It does for sure. I did some more digging and it does real-time too. That's fascinating.
• red2awn 10mo ago
  None of inference frameworks (vLLM/SGLang) supports the full model, let alone non-nvidia.
  • sosodev 10mo ago
    That's unfortunate but not too surprising. This type of model is very new to the local hosting space.
  • AndreSlavescu 10mo ago
    We actually deployed working speech to speech inference that builds on top of vLLM as the backbone. The main thing was to support the "Talker" module, which is currently not supported on the qwen3-omni branch for vLLM. Check it out here: https://models.hathora.dev/model/qwen3-omni https://models.hathora.dev/model/qwen3-omni
    • red2awn 10mo ago
      Nice work. Are you working on streaming input/output?
      • AndreSlavescu 10mo ago
        Yeah, that's something we currently support. Feel free to try the platform out! No cost to you for now, you just need a valid email to sign up on the platform.
        • valleyer 10mo ago
          I tried this out, and it's not passing the record (n.) vs. record (v.) test mentioned elsewhere in this thread. (I can ask it to repeat one, and it often repeats the other.) Am I not enabling the speech-to-speech-ness somehow?
          • AndreSlavescu 10mo ago
            From my understanding of the above problem, this would be something to do with the model weights. Have you tested this with the transformers inference baseline that is shown on huggingface? In our deployment, we do not actually tune the model in any way, this is all just using the base instruct model provided on huggingface: https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct And with the potential concern around conversation turns, our platform is designed for one-off record -> response flows. But via the API, you can build your own conversation agent to use the model.
    • sosodev 10mo ago
      Is your work open source?
      • AndreSlavescu 10mo ago
        At the moment, no unfortunately. However, to my recent knowledge of open source alternatives, the vLLM team published a separate repository for omni models now: https://github.com/vllm-project/vllm-omni https://github.com/vllm-project/vllm-omni I have not yet tested out if this does full speech to speech, but this seems like a promising workspace for omni-modal models.
  • whimsicalism 10mo ago
    Makes sense, I think streaming audio->audio inference is a relatively big lift.
    • red2awn 10mo ago
      Correct, it's breaks the single prompt, single completion assumption baked into the frameworks. Conceptually it's still prompt/completion but for low latency response you have to do streaming KV cache prefill with a websocket server.
      • whimsicalism 10mo ago
        I imagine you have to start decoding many speculative completions in parallel to have true low latency.
• ivape 10mo ago
  That's exciting. I doubt there are any polished voice chat local apps yet that you can easily plug this into (I doubt the user experience is "there" yet). Even stuff like Silly Tavern is near unusable, lots of work to be done on the local front. Local voice models are what's going to enable that whole Minority Report workflow soon enough (especially if commands and intent are determined at the local level, and the meat of the prompt is handled by a larger remote model). This is part of programming that I think is the new field. There will be tons of work for those that can build the new workflows which will need to be primarily natural language driven.
  • sosodev 10mo ago
    I did find this app: https://github.com/gabber-dev/gabber https://github.com/gabber-dev/gabber The creator posted a little demo of it working with Qwen3 Omni that is quite impressive: https://www.youtube.com/watch?v=5DBFVe3cLto https://www.youtube.com/watch?v=5DBFVe3cLto He didn't include any details regarding how the model was running though
• potatoman22 10mo ago
  From what I can tell, their official chat site doesn't have a native audio -> audio model yet. I like to test this through homophones (e.g. record and record) and asking it to change its pitch or produce sounds.
  • sosodev 10mo ago
    Huh, you're right. I tried your test and it clearly can't understand the difference between homophones. That seems to imply they're using some sort of TTS mechanism. Which is really weird because Qwen3-Omni claims to support direct audio input into the model. Maybe it's a cost saving measure?
    • sosodev 10mo ago
      Weirdly, I just tried it again and it seems to understand the difference between record and record just fine. Perhaps if there's heavy demand for voice chat, like after a new release, they load shed by using TTS to a smaller model. However, It still doesn't seem capable of producing any of the sounds, like laughter, that I would expect from a native voice model.
    • potatoman22 10mo ago
      To be fair, discerning heteronyms might just be a gap in its training.
  • djtango 10mo ago
    Is record a homophone? At least in the UK we use different pronunciations for the meanings. Re-cord for the verb, rec-ord for the noun.
    • potatoman22 10mo ago
      I was mistaken about what homophone means!
  • dragonwriter 10mo ago
    “record and record”, if you mean the verb for persisting something and the noun for the thing persisted, are heteronyms (homographs which are not homophones), which incidentally is also what you would probably want to test what you are talking about here (distinguishing homophones would test use of context to understand meaning, but wouldn’t test anything about whether or not logic was working directly on audio or only working on text processed from audio, failing to distinguish heteronyms is suggestive of processing occurring on text, not audio directly.)