9 ms·

New developments in the Opus audio codec

108 points by nullc 13y ago

ChuckMcM 13y ago
This is an awesome page. The discussion about tonality really resonated with me (no pun intended) because it irritates me when switching audio sources at how "squashed" they can sound. For a while I had some macros on my G15 keyboard to change settings in the audio mixer based on music, games, or general computer sounds. If Opus can do a reasonable job on a wide variety of inputs automatically it will be a wonderful contribution.
zanny 13y ago
> And we haven't even started using NEON yet. That is almost concerning, because even though it seems like overkill to gpu accelerate audio decoding (that the copy and latency overhead isn't worth it) I imagine it will require quite a bit of refactoring to make a SIMD decoder (or encoder). But that will be absolutely necessary to see Opus adoption as the web audio standard where mobile devices need every watt of energy savings possible.
- nullc 13y ago
  NEON isn't GPU, it's just the ARM equivalent to SSE/MMX. (Though a fair bit nicer to work with). It doesn't require extensive refactoring. There are patches to use SIMD that aren't merged yet— though pure C improvements have tended to be more important since NEON isn't available on all ARM parts.
revelation 13y ago
I wish support for Opus was better in popular players. I tried building a RTP + RTSP streaming server using libopus, but even VLC couldn't play it.
- nullc 13y ago
  There is Opus support in VLC, but I don't believe that RTP is working for it there yet. (Obviously simple HTTP streaming works)
  - revelation 13y ago
    I think it's because the Opus decoder in VLC expects to find a Ogg container, which is a no-go for streaming.
    - nullc 13y ago
      No go for RTP— as thats not the standard and an additional container doesn't provide anything useful, but its fantastic for HTTP streaming.
      - revelation 13y ago
        I guess I should clarify streaming as in low-latency live streaming, where RTP is necessary - and Opus would be a perfect fit with its very low algorithmic delay.
        tveita 13y ago
        Chrome and Firefox should both support Opus over RTP now as part of WebRTC.
gwu78 13y ago
Given how contentious the environment surrounding audio codecs have been over the years, Opus (and Silk, too) just seems too good to be true. Am I missing a devil in the details? Otherwise, I'm excited at the possibilities this provides when used in open source projects that utilize open protocols to deliver audio end-to-end (peer-to-peer) over the network.
moneyrich2 13y ago
how does the cpu usage compare to vorbis? (and then to mp3?) i assume the variable bitrate uses less cpu, or? (b/c i just have to have 40 tabs open, steam, and a mp3 player at all times)
- jmvalin 13y ago
  CPU usage in the decoder is currently higher than Vorbis, though most of the difference is due to having less optimizations. As for the encoder, it's already faster than Vorbis and should become even faster in the future. I haven't checked, but I suspect the same is true when compared to MP3. That being said, the complexity is pretty much negligible on a desktop machine.
  - graue 13y ago
    Interesting. I tested a while back and found opusenc consistently slower than oggenc. I don't remember the specifics, but I'm surprised a 27% improvement was enough to make up the difference. It's not negligible to me, as I'm running a music server on a Raspberry Pi that needs to transcode audio, and the Pi can only transcode FLAC to Vorbis at 1.8x realtime. This makes it terribly sensitive to background tasks — anything else going on tends to prevent it from keeping up. So I'm quite glad you all are optimizing for ARM :)
    - keeperofdakeys 13y ago
      Now I'm not sure how much floating point arithmetic the ogg vorbis encoder would use, but ARM can be sensitive to this. If you have the choice between "hardfp" and "softfp" for you distro, you want hardfp, as this will use the floating point hardware (which is present in the pi). Some ARM systems require softfp, since they lack fp hardware. Also, what optimisations have you got turned on (on the compiler). For a realtime system, it might make sense to turn them down, trading stream size for processing time.
    - nullc 13y ago
      Uh. On a Raspberry Pi you should be using the Opus encoder compiled as fixed point. It will be a _ton_ faster than Vorbis, as the floating point on the rpi is very slow and there is no fixed point vorbis encoder. (Unfortunately, if your input requires resampling, there isn't currently an option to compile the opus-tools front end to use the fixed point resampler, so that will add some slowdown— I guess that should go up in priority) Although ... rpi ugh. I often feel that device was an evil scheme to turn people off of arm. It is _remarkably_ slow for its cost and power consumption. For $100 you can have an arm device which is easily 32x faster for most DSP-ish stuff, and which draws similar power.
hnha 13y ago
the following is not true, see below: warning: don't visit the linked site on opera mobile. It seems to download something bigger and also froze the browser for me. edit: sorry, false alarm! that was something else. the site loads perfectly fine (and I am honestly happy about every site that does not differentiate between desktop and mobile clients but serves a readable, clearly structured, "static" page like that!). edit 2: even the audio samples worked flawlessly. the accompanying displays not though.
TempleOS 13y ago
VMware removed support for PC Speaker. God says... experts act oh_no it_was_nothing I_was_just_thinking a_flag_on_that_play impossible flat China frown stoked I'm_bored look_buddy I_had_a_crazy_dream why_is_it music the I_planned_that nasty wanna_bet ba_ha honesty You_can_count_on_that maybe_I_didn't_make_it_clear homo grumble I'll_get_right_on_it charged listen_buddy game_over rockstar scum spunky I'm_gonna_smack_someone sess_me nut_job geek special_case That's_gonna_leave_a_mark you_owe_me chump_change big_fish good skills beam_me_up what_would_Jesus_do but_of_course maybe_I_didn't_make_it_clear Bam endeared virtue stoked do_it joker duck_the_shoe well_golly umm you_think_you_could_do_better why_didn'_you_tell_me who_are_you_to_judge sex I_pitty_the_fool I_didn't_see_that whiner how_high but_of_course let_me_count_the_ways to_infinity_and_beyond I'll_let_you_know you_talkin'_to_me you'll_see well_obviously I'm_beginning_to_wonder if_anything_can_go_wrong flat I'm_tired_of_this silly_human rose_colored_glasses fabulous bad hype crash_and_burn ohh_thank_you Icarus freak astrophysics look_out shist experts once_upon_a_time daunting Ramsay I'll_get_right_on_it silly_human boss Tomorrow basically choose_one astronomical don't_you_love_me rockstar do_you_have_a_problem wanna_bet genius is_it_just_me_or liberal doh God_smack air_head big_fish that's_much_better I_love_this what_a_nightmare play What_are_you_doing_dave cheerful watch_it_buddy listen_buddy it_was_nothing vice Varoom Burp jobs insane I_love_you bastard
kragen 13y ago
This is awesome. Who would have guessed that ultra-low-latency low-bandwidth perceptual coding of audio would involve neural networks and hidden Markov models? It seems obvious when you put it that way, but I certainly wouldn't have predicted it.

108 points by nullc 13y ago

This is an awesome page. The discussion about tonality really resonated with me (no pun intended) because it irritates me when switching audio sources at how "squashed" they can sound. For a while I had some macros on my G15 keyboard to change settings in the audio mixer based on music, games, or general computer sounds. If Opus can do a reasonable job on a wide variety of inputs automatically it will be a wonderful contribution.

zanny 13y ago

> And we haven't even started using NEON yet. That is almost concerning, because even though it seems like overkill to gpu accelerate audio decoding (that the copy and latency overhead isn't worth it) I imagine it will require quite a bit of refactoring to make a SIMD decoder (or encoder). But that will be absolutely necessary to see Opus adoption as the web audio standard where mobile devices need every watt of energy savings possible.

nullc 13y ago
NEON isn't GPU, it's just the ARM equivalent to SSE/MMX. (Though a fair bit nicer to work with). It doesn't require extensive refactoring. There are patches to use SIMD that aren't merged yet— though pure C improvements have tended to be more important since NEON isn't available on all ARM parts.

revelation 13y ago

I wish support for Opus was better in popular players. I tried building a RTP + RTSP streaming server using libopus, but even VLC couldn't play it.

nullc 13y ago
There is Opus support in VLC, but I don't believe that RTP is working for it there yet. (Obviously simple HTTP streaming works)
- revelation 13y ago
  I think it's because the Opus decoder in VLC expects to find a Ogg container, which is a no-go for streaming.
  - nullc 13y ago
    No go for RTP— as thats not the standard and an additional container doesn't provide anything useful, but its fantastic for HTTP streaming.
    - revelation 13y ago
      I guess I should clarify streaming as in low-latency live streaming, where RTP is necessary - and Opus would be a perfect fit with its very low algorithmic delay.
      - tveita 13y ago
        Chrome and Firefox should both support Opus over RTP now as part of WebRTC.

gwu78 13y ago

Given how contentious the environment surrounding audio codecs have been over the years, Opus (and Silk, too) just seems too good to be true. Am I missing a devil in the details? Otherwise, I'm excited at the possibilities this provides when used in open source projects that utilize open protocols to deliver audio end-to-end (peer-to-peer) over the network.

moneyrich2 13y ago

how does the cpu usage compare to vorbis? (and then to mp3?) i assume the variable bitrate uses less cpu, or? (b/c i just have to have 40 tabs open, steam, and a mp3 player at all times)

jmvalin 13y ago
CPU usage in the decoder is currently higher than Vorbis, though most of the difference is due to having less optimizations. As for the encoder, it's already faster than Vorbis and should become even faster in the future. I haven't checked, but I suspect the same is true when compared to MP3. That being said, the complexity is pretty much negligible on a desktop machine.
- graue 13y ago
  Interesting. I tested a while back and found opusenc consistently slower than oggenc. I don't remember the specifics, but I'm surprised a 27% improvement was enough to make up the difference. It's not negligible to me, as I'm running a music server on a Raspberry Pi that needs to transcode audio, and the Pi can only transcode FLAC to Vorbis at 1.8x realtime. This makes it terribly sensitive to background tasks — anything else going on tends to prevent it from keeping up. So I'm quite glad you all are optimizing for ARM :)
  - keeperofdakeys 13y ago
    Now I'm not sure how much floating point arithmetic the ogg vorbis encoder would use, but ARM can be sensitive to this. If you have the choice between "hardfp" and "softfp" for you distro, you want hardfp, as this will use the floating point hardware (which is present in the pi). Some ARM systems require softfp, since they lack fp hardware. Also, what optimisations have you got turned on (on the compiler). For a realtime system, it might make sense to turn them down, trading stream size for processing time.
  - nullc 13y ago
    Uh. On a Raspberry Pi you should be using the Opus encoder compiled as fixed point. It will be a _ton_ faster than Vorbis, as the floating point on the rpi is very slow and there is no fixed point vorbis encoder. (Unfortunately, if your input requires resampling, there isn't currently an option to compile the opus-tools front end to use the fixed point resampler, so that will add some slowdown— I guess that should go up in priority) Although ... rpi ugh. I often feel that device was an evil scheme to turn people off of arm. It is _remarkably_ slow for its cost and power consumption. For $100 you can have an arm device which is easily 32x faster for most DSP-ish stuff, and which draws similar power.

hnha 13y ago

the following is not true, see below: warning: don't visit the linked site on opera mobile. It seems to download something bigger and also froze the browser for me. edit: sorry, false alarm! that was something else. the site loads perfectly fine (and I am honestly happy about every site that does not differentiate between desktop and mobile clients but serves a readable, clearly structured, "static" page like that!). edit 2: even the audio samples worked flawlessly. the accompanying displays not though.

TempleOS 13y ago

VMware removed support for PC Speaker. God says... experts act oh_no it_was_nothing I_was_just_thinking a_flag_on_that_play impossible flat China frown stoked I'm_bored look_buddy I_had_a_crazy_dream why_is_it music the I_planned_that nasty wanna_bet ba_ha honesty You_can_count_on_that maybe_I_didn't_make_it_clear homo grumble I'll_get_right_on_it charged listen_buddy game_over rockstar scum spunky I'm_gonna_smack_someone sess_me nut_job geek special_case That's_gonna_leave_a_mark you_owe_me chump_change big_fish good skills beam_me_up what_would_Jesus_do but_of_course maybe_I_didn't_make_it_clear Bam endeared virtue stoked do_it joker duck_the_shoe well_golly umm you_think_you_could_do_better why_didn'_you_tell_me who_are_you_to_judge sex I_pitty_the_fool I_didn't_see_that whiner how_high but_of_course let_me_count_the_ways to_infinity_and_beyond I'll_let_you_know you_talkin'_to_me you'll_see well_obviously I'm_beginning_to_wonder if_anything_can_go_wrong flat I'm_tired_of_this silly_human rose_colored_glasses fabulous bad hype crash_and_burn ohh_thank_you Icarus freak astrophysics look_out shist experts once_upon_a_time daunting Ramsay I'll_get_right_on_it silly_human boss Tomorrow basically choose_one astronomical don't_you_love_me rockstar do_you_have_a_problem wanna_bet genius is_it_just_me_or liberal doh God_smack air_head big_fish that's_much_better I_love_this what_a_nightmare play What_are_you_doing_dave cheerful watch_it_buddy listen_buddy it_was_nothing vice Varoom Burp jobs insane I_love_you bastard

kragen 13y ago

This is awesome. Who would have guessed that ultra-low-latency low-bandwidth perceptual coding of audio would involve neural networks and hidden Markov models? It seems obvious when you put it that way, but I certainly wouldn't have predicted it.