4 ms·

We actually deployed working speech to speech inference that builds on top of vLLM as the backbone. The main thing was to support the "Talker" module, which is

We actually deployed working speech to speech inference that builds on top of vLLM as the backbone. The main thing was to support the "Talker" module, which is currently not supported on the qwen3-omni branch for vLLM. Check it out here: https://models.hathora.dev/model/qwen3-omni https://models.hathora.dev/model/qwen3-omni

red2awn 10mo ago
Nice work. Are you working on streaming input/output?
- AndreSlavescu 10mo ago
  Yeah, that's something we currently support. Feel free to try the platform out! No cost to you for now, you just need a valid email to sign up on the platform.
  - valleyer 10mo ago
    I tried this out, and it's not passing the record (n.) vs. record (v.) test mentioned elsewhere in this thread. (I can ask it to repeat one, and it often repeats the other.) Am I not enabling the speech-to-speech-ness somehow?
    - AndreSlavescu 10mo ago
      From my understanding of the above problem, this would be something to do with the model weights. Have you tested this with the transformers inference baseline that is shown on huggingface? In our deployment, we do not actually tune the model in any way, this is all just using the base instruct model provided on huggingface: https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct And with the potential concern around conversation turns, our platform is designed for one-off record -> response flows. But via the API, you can build your own conversation agent to use the model.
sosodev 10mo ago
Is your work open source?
- AndreSlavescu 10mo ago
  At the moment, no unfortunately. However, to my recent knowledge of open source alternatives, the vLLM team published a separate repository for omni models now: https://github.com/vllm-project/vllm-omni https://github.com/vllm-project/vllm-omni I have not yet tested out if this does full speech to speech, but this seems like a promising workspace for omni-modal models.

red2awn 10mo ago

Nice work. Are you working on streaming input/output?

AndreSlavescu 10mo ago
Yeah, that's something we currently support. Feel free to try the platform out! No cost to you for now, you just need a valid email to sign up on the platform.
- valleyer 10mo ago
  I tried this out, and it's not passing the record (n.) vs. record (v.) test mentioned elsewhere in this thread. (I can ask it to repeat one, and it often repeats the other.) Am I not enabling the speech-to-speech-ness somehow?
  - AndreSlavescu 10mo ago
    From my understanding of the above problem, this would be something to do with the model weights. Have you tested this with the transformers inference baseline that is shown on huggingface? In our deployment, we do not actually tune the model in any way, this is all just using the base instruct model provided on huggingface: https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct And with the potential concern around conversation turns, our platform is designed for one-off record -> response flows. But via the API, you can build your own conversation agent to use the model.

sosodev 10mo ago

Is your work open source?

AndreSlavescu 10mo ago
At the moment, no unfortunately. However, to my recent knowledge of open source alternatives, the vLLM team published a separate repository for omni models now: https://github.com/vllm-project/vllm-omni https://github.com/vllm-project/vllm-omni I have not yet tested out if this does full speech to speech, but this seems like a promising workspace for omni-modal models.