8 ms·

Spatial Audio in VR

122 points by bharatkhatri14 9y ago

cromwellian 9y ago
I've never really been impressed by HRTFs compared to true Binaural recordings (e.g. virtual barbershop) There's something happening in the binaural recordings for positioning that just isn't captured in HRTFs it seems.
Theodores 9y ago
Back in the day 'spatial audio' was excellent for what it was in VRML, as played through the SGI viewer. I was sad VRML never happened as I quite enjoyed worlds where you could play different songs in different virtual rooms able to mix albeit not to the beat between different tracks by moving around. A fantastic music experience could be had. This new take on 3D audio just makes me want to go back to 1998 and what was achievable then in 3D on the web if you were willing to do a 20Mb download over a modem to get the plugin required to 'see' VRML. The actual hurdle was use case, instead of a 3D web we ended up with a 2D flat table web. If you got good at VRML and used all of the features and built significant sized worlds then you soon realised that you were on your own, few others in the world had got that far, their models were more like Duplo Lego rather than Lego Technical. The alleged 'bar' back then was the download for the plugin, that allegedly prevented VRML from catching on. The authorship tools were awesome - Maya like - and yet there were few takers. CosmoWorlds was awesome, has to be said. The lack of content creators was the problem, 2D Flash was the way people went, nobody was thinking in 3D. So when can we start doing cool 3D just on the web in a web browser that you don't need specialist hardware for? With 3D spatial audio working too?
mdf 9y ago
Nice to see audio finally getting the attention it deserves! And seems like VR is the driving factor. I wonder how this compares to NVIDIA VRWorks Audio[1]? When you think about it, 3D audio has many similarities to 3D video: rays of sound(/light) traveling through open space and attenuating in the process, bouncing off surfaces whose material filters different frequencies. From this point of view, in hindsight, the accelerated sound cards from ’00s seem a bit of a misstep – isn’t this something that’s better computed on a GPU? [1] http://www.roadtovr.com/nvidias-vrworks-audio-brings-physically-based-3d-gpu-accelerated-sound/ http://www.roadtovr.com/nvidias-vrworks-audio-brings-physica...
pfista 9y ago
This is actually quite amazing and one of the fields I'm most interested in. Spatial audio / 3d audio is much more immersive and less gimmicky than 3D video, but I feel like people don't really care too much because they were so underwhelmed with 3D video. It's one of those things with high fidelity audio you can't tell a difference between what is real and what is fake. Edgar Choueiri, a Princeton professor, was the first I know of that tried to commercialize his algorithm for spatial audio. I believe he got a deal with Jawbone to put in Jambox as a feature called "live audio mode". He did try to sell some hardware for audiophiles to build out home systems, though I'm pretty sure they were insanely expensive (10-100k range). https://www.princeton.edu/3D3A/index.html https://www.princeton.edu/3D3A/index.html There's also a great podcast that tells stories using only spatial audio. They're really immersive and sometimes I can't tell if the audio is happening in the podcast or outside my room. It's called The Truth: http://www.thetruthpodcast.com/ http://www.thetruthpodcast.com/ Makes me happy to see tools like this are open to developers now. I expect to see a lot more content featuring this technology now.
JetSpiegel 9y ago
What's the difference between this and OpenAL? It has 3D audio for ages, but not many games really use it.
- munchbunny 9y ago
  I don't think OpenAL simulates some of the more complex phenomena having to do with how sound interacts with your body and ears? Valve also released an audio toolkit that does a similar thing to this one called Steam Audio.
  - gens 9y ago
    > I don't think OpenAL simulates some of the more complex phenomena having to do with how sound interacts with your body and ears? It does. This blog post [0] seems to be a nice reference. What you want is called "head-related transfer function". (There are tons of papers on HRTF) One can even turn it on for any game that use openal, even old ones. You can go download The Dark Mod now and play it with HRTF [1]. Note: am not an expert What openal doesn't do, while this google thing does, is reverb (reflection of waves, echo). One way to do reverb is.. well simply raytrace it. Just send out a bunch of rays from the source, probably do some filtering of the results, and apply to the audio. Another way is to take a real room that you are "porting" to digital, ping somewhere in it (play the impulse response of an ideal 20kHz low pass filter, aka the sync function), then measure somewhere else. Then repeat in a.. grid or something. What you get is a bunch of filters (num_points if there is a single static "listener", or num_points^2 if there are more and or moving; as it is source->destination dependent). This can also be done by shooting rays in an arbitrary digital space, but i wanted people to imagine a room and waves going around in it. AFAIK this is how the valve bought software works. Another thing is diffraction[2]... that idk if either gugl or volvo do. Calculate it with voxel grids then add to refraction ? Any renderers do proper Rayleigh scattering ? [0]http://www.bitoutsidethebox.com/shabda/hrtf-info/ http://www.bitoutsidethebox.com/shabda/hrtf-info/ [1]https://www.youtube.com/watch?v=MVqFbu4gsqs https://www.youtube.com/watch?v=MVqFbu4gsqs [2]http://hyperphysics.phy-astr.gsu.edu/hbase/Sound/diffrac.html http://hyperphysics.phy-astr.gsu.edu/hbase/Sound/diffrac.htm...
- stevenwoo 9y ago
gasbag 9y ago
If you want to hear a great example of spatial audio, check out The Quiet American [0] and be sure to use earbuds. He records his environment using two in-ear microphones to get all of the effects discussed in the article: sound shaped by the external parts of the ear, timing differences between the left and right ear, etc. I remember listening to one recording where he is at the edge of a lake or ocean, and listening while standing with my eyes closed I almost expected to feel the waves around my ankles. It's incredibly immersive, and will work wonders for VR. [0] https://www.quietamerican.org/introduction.html https://www.quietamerican.org/introduction.html
AceJohnny2 9y ago
So, are we going to revive A3D for mobile/VR? Have any of the patents expired?
- Terr_ 9y ago
  If Creative ever used A3D to improve their own product -- after lawsuiting Aureal into oblivion and buying the company -- I never noticed. For readers who aren't familiar with the story: http://www.roadtovr.com/sightline-chair-goes-binaural-developer-shares-thoughts-3dception-integration/2/ http://www.roadtovr.com/sightline-chair-goes-binaural-develo... https://toni.org/a3d/ https://toni.org/a3d/
- deleted 9y ago
  [deleted]

I've never really been impressed by HRTFs compared to true Binaural recordings (e.g. virtual barbershop) There's something happening in the binaural recordings for positioning that just isn't captured in HRTFs it seems.

Theodores 9y ago

Back in the day 'spatial audio' was excellent for what it was in VRML, as played through the SGI viewer. I was sad VRML never happened as I quite enjoyed worlds where you could play different songs in different virtual rooms able to mix albeit not to the beat between different tracks by moving around. A fantastic music experience could be had. This new take on 3D audio just makes me want to go back to 1998 and what was achievable then in 3D on the web if you were willing to do a 20Mb download over a modem to get the plugin required to 'see' VRML. The actual hurdle was use case, instead of a 3D web we ended up with a 2D flat table web. If you got good at VRML and used all of the features and built significant sized worlds then you soon realised that you were on your own, few others in the world had got that far, their models were more like Duplo Lego rather than Lego Technical. The alleged 'bar' back then was the download for the plugin, that allegedly prevented VRML from catching on. The authorship tools were awesome - Maya like - and yet there were few takers. CosmoWorlds was awesome, has to be said. The lack of content creators was the problem, 2D Flash was the way people went, nobody was thinking in 3D. So when can we start doing cool 3D just on the web in a web browser that you don't need specialist hardware for? With 3D spatial audio working too?

mdf 9y ago

Nice to see audio finally getting the attention it deserves! And seems like VR is the driving factor. I wonder how this compares to NVIDIA VRWorks Audio[1]? When you think about it, 3D audio has many similarities to 3D video: rays of sound(/light) traveling through open space and attenuating in the process, bouncing off surfaces whose material filters different frequencies. From this point of view, in hindsight, the accelerated sound cards from ’00s seem a bit of a misstep – isn’t this something that’s better computed on a GPU? [1] http://www.roadtovr.com/nvidias-vrworks-audio-brings-physically-based-3d-gpu-accelerated-sound/ http://www.roadtovr.com/nvidias-vrworks-audio-brings-physica...

pfista 9y ago

This is actually quite amazing and one of the fields I'm most interested in. Spatial audio / 3d audio is much more immersive and less gimmicky than 3D video, but I feel like people don't really care too much because they were so underwhelmed with 3D video. It's one of those things with high fidelity audio you can't tell a difference between what is real and what is fake. Edgar Choueiri, a Princeton professor, was the first I know of that tried to commercialize his algorithm for spatial audio. I believe he got a deal with Jawbone to put in Jambox as a feature called "live audio mode". He did try to sell some hardware for audiophiles to build out home systems, though I'm pretty sure they were insanely expensive (10-100k range). https://www.princeton.edu/3D3A/index.html https://www.princeton.edu/3D3A/index.html There's also a great podcast that tells stories using only spatial audio. They're really immersive and sometimes I can't tell if the audio is happening in the podcast or outside my room. It's called The Truth: http://www.thetruthpodcast.com/ http://www.thetruthpodcast.com/ Makes me happy to see tools like this are open to developers now. I expect to see a lot more content featuring this technology now.

JetSpiegel 9y ago

What's the difference between this and OpenAL? It has 3D audio for ages, but not many games really use it.

munchbunny 9y ago
I don't think OpenAL simulates some of the more complex phenomena having to do with how sound interacts with your body and ears? Valve also released an audio toolkit that does a similar thing to this one called Steam Audio.
- gens 9y ago
  > I don't think OpenAL simulates some of the more complex phenomena having to do with how sound interacts with your body and ears? It does. This blog post [0] seems to be a nice reference. What you want is called "head-related transfer function". (There are tons of papers on HRTF) One can even turn it on for any game that use openal, even old ones. You can go download The Dark Mod now and play it with HRTF [1]. Note: am not an expert What openal doesn't do, while this google thing does, is reverb (reflection of waves, echo). One way to do reverb is.. well simply raytrace it. Just send out a bunch of rays from the source, probably do some filtering of the results, and apply to the audio. Another way is to take a real room that you are "porting" to digital, ping somewhere in it (play the impulse response of an ideal 20kHz low pass filter, aka the sync function), then measure somewhere else. Then repeat in a.. grid or something. What you get is a bunch of filters (num_points if there is a single static "listener", or num_points^2 if there are more and or moving; as it is source->destination dependent). This can also be done by shooting rays in an arbitrary digital space, but i wanted people to imagine a room and waves going around in it. AFAIK this is how the valve bought software works. Another thing is diffraction[2]... that idk if either gugl or volvo do. Calculate it with voxel grids then add to refraction ? Any renderers do proper Rayleigh scattering ? [0]http://www.bitoutsidethebox.com/shabda/hrtf-info/ http://www.bitoutsidethebox.com/shabda/hrtf-info/ [1]https://www.youtube.com/watch?v=MVqFbu4gsqs https://www.youtube.com/watch?v=MVqFbu4gsqs [2]http://hyperphysics.phy-astr.gsu.edu/hbase/Sound/diffrac.html http://hyperphysics.phy-astr.gsu.edu/hbase/Sound/diffrac.htm...
stevenwoo 9y ago

gasbag 9y ago

If you want to hear a great example of spatial audio, check out The Quiet American [0] and be sure to use earbuds. He records his environment using two in-ear microphones to get all of the effects discussed in the article: sound shaped by the external parts of the ear, timing differences between the left and right ear, etc. I remember listening to one recording where he is at the edge of a lake or ocean, and listening while standing with my eyes closed I almost expected to feel the waves around my ankles. It's incredibly immersive, and will work wonders for VR. [0] https://www.quietamerican.org/introduction.html https://www.quietamerican.org/introduction.html

AceJohnny2 9y ago

So, are we going to revive A3D for mobile/VR? Have any of the patents expired?

Terr_ 9y ago
If Creative ever used A3D to improve their own product -- after lawsuiting Aureal into oblivion and buying the company -- I never noticed. For readers who aren't familiar with the story: http://www.roadtovr.com/sightline-chair-goes-binaural-developer-shares-thoughts-3dception-integration/2/ http://www.roadtovr.com/sightline-chair-goes-binaural-develo... https://toni.org/a3d/ https://toni.org/a3d/
deleted 9y ago
[deleted]