5 ms·

I hope the current giants in DAWs are doing much experimentation with VR tools, with the addition of multi-modal interfaces (like the mimu gloves and others), w

by cmehdy 6y ago

I hope the current giants in DAWs are doing much experimentation with VR tools, with the addition of multi-modal interfaces (like the mimu gloves and others), which could open up some marvelous ways of re-thinking music creation as a whole-body experience. The tech for VR might still be a bit limited but the potential for the intersection of it and multi-modal interfaces is, in my opinion, very interesting.

qppo 6y ago
Almost every DAW fails to handle spatial audio rendering properly, so I doubt this is the case. Spatial computing that relies on spatial audio rendering is basically impossible to predict in current DAW offerings - you have to rely on game engine technology for the most part. The most accessible DAW that can handle spatial audio is Reaper, which doesn't even allow an odd number of channels.
- paublyrne 6y ago
  Could you expand on what you mean by not handling properly? I don't work with non stereo audio and mostly use Logic but I assumed something like Nuendo would have excellent multi channel support as it's targeted at audio for film.
  - qppo 6y ago
    The problem is deeper than multichannel support, which is poorly supported everywhere except Pro Tools. Even that is fairly deep, Nuendo relies on VST3's model of channel set configuration which is incapable of representing the kinds of multichannel configurations that are in vogue for spatial audio rendering, like you'd find in VR. But even without leaving stereo, DAWs are already really bad. Think about how a pan or balance control works, instead of representing the perceptual localization of a sound they ask you to control the gain feeding two channels, left/right. The semantics of those two channels change after rendering, which makes it difficult to convey the same intent of a mixer/producer to the audience using that interface. Compare a stereo mix on near field monitors positioned perfectly, vs reference headphones, vs a car, vs a club system... the pan/balance control you set in your DAW changes the localization of the sound depending on ultimate render target, whereas what you need for spatial audio is for the localization to be independent from those targets. The solution is object based audio, which is not perfect and a misnomer 90% of the time, but even that is because of how terribly DAWs handle spatialization. And it's not something you can really control with a plugin without hacking the mixbus configurations inside a DAW, and you can't propagate position down to other plugins.
TheOtherHobbes 6y ago
This is harder than it sounds. Waving your arms around explicitly conflicts with the very fine motor control required to play an instrument accurately and expressively. You need between two and 12 or so (including feet) channels of very precise movement data to make a useful musical instrument. VR hardware isn't designed to have that level of precision. Of course you can have virtual pads/buttons triggering samples, with a few faders for control of effects. But that's more like sample-based DJing than instrumental performance. It might well have commercial value, but it's a very limited subset of music production as a whole.
- cmehdy 6y ago
  No doubt it's hard and technically limited too, with the state of customer-grade sensors (like the mimu, or eye-tracking devices) and VR. There are still many hurdles to simply running VR devices, so we're far from the kind of adoption that would lead to meaningful inventive approaches to something as complex as what DAWs and their physical extensions offer. However it's more about thinking of musical creation very laterally. It could mean leveraging tilts of any place where one can put an accelerometer (heck, the OP1 does it in a gimmicky way to change any specific setting you want for a given track for example), your finger movements position in 3d space (mimu does some of the finger movement stuff), heck even smoothed-out eye "gestures" could be explored (google-glass-y devices would need to succeed since you're not going anywhere with current tech without invasive things like the EyeLink II). Pivoting around 360 degrees in a "musical cockpit" could also be a thing. Voice too, and NLP could be leveraged in creative ways (to bring forward different aspects of the tool or to give instructions at a high-level - "take the first four measures of the first guitar track and apply melodic inversion, shift down one octave and paste at the 2min mark on the bass track"). Different parts of the music-making process could take on different shapes in spaces that add multiple dimensions. As the mimu shows, a lot of what those dimensions do could be entirely configurable and fitting the creator. Conductors are a good example of someone leveraging rotations and depth (over half-sphere surfaces) to create expressions and rhythm and so on. Plus nobody has to throw away the current tools either, once could think of the advantages of AR (Augmented Reality) which could leave perfectly clear the areas of your synths (or under a certain line of vision or even physically marked delimitations) during a performance, and all of the glory of ye olde knobs and faders can remain intact. Current VR is out of reach for too many people for much experimentation (basically forces you to be on Windows, requires either heavy-duty hardware or has very little room for fine details), which means all the exploratory stuff to lead to even a subset of what I mentioned is only accessible to a handful of people so far. Same for multi-modal interfaces (the mimu is still a toy for rich artsy-techy people, the Tobiis are either too basic or too expensive, etc). But we're still in the infancy of much of these kinds of tools. And while the usual engineer will talk about why everything's impossible, a designer or an artist will always be curious to see what just could be eventually.

by cmehdy 6y ago

qppo 6y ago

Almost every DAW fails to handle spatial audio rendering properly, so I doubt this is the case. Spatial computing that relies on spatial audio rendering is basically impossible to predict in current DAW offerings - you have to rely on game engine technology for the most part. The most accessible DAW that can handle spatial audio is Reaper, which doesn't even allow an odd number of channels.

paublyrne 6y ago
Could you expand on what you mean by not handling properly? I don't work with non stereo audio and mostly use Logic but I assumed something like Nuendo would have excellent multi channel support as it's targeted at audio for film.
- qppo 6y ago
  The problem is deeper than multichannel support, which is poorly supported everywhere except Pro Tools. Even that is fairly deep, Nuendo relies on VST3's model of channel set configuration which is incapable of representing the kinds of multichannel configurations that are in vogue for spatial audio rendering, like you'd find in VR. But even without leaving stereo, DAWs are already really bad. Think about how a pan or balance control works, instead of representing the perceptual localization of a sound they ask you to control the gain feeding two channels, left/right. The semantics of those two channels change after rendering, which makes it difficult to convey the same intent of a mixer/producer to the audience using that interface. Compare a stereo mix on near field monitors positioned perfectly, vs reference headphones, vs a car, vs a club system... the pan/balance control you set in your DAW changes the localization of the sound depending on ultimate render target, whereas what you need for spatial audio is for the localization to be independent from those targets. The solution is object based audio, which is not perfect and a misnomer 90% of the time, but even that is because of how terribly DAWs handle spatialization. And it's not something you can really control with a plugin without hacking the mixbus configurations inside a DAW, and you can't propagate position down to other plugins.

TheOtherHobbes 6y ago

This is harder than it sounds. Waving your arms around explicitly conflicts with the very fine motor control required to play an instrument accurately and expressively. You need between two and 12 or so (including feet) channels of very precise movement data to make a useful musical instrument. VR hardware isn't designed to have that level of precision. Of course you can have virtual pads/buttons triggering samples, with a few faders for control of effects. But that's more like sample-based DJing than instrumental performance. It might well have commercial value, but it's a very limited subset of music production as a whole.

cmehdy 6y ago
No doubt it's hard and technically limited too, with the state of customer-grade sensors (like the mimu, or eye-tracking devices) and VR. There are still many hurdles to simply running VR devices, so we're far from the kind of adoption that would lead to meaningful inventive approaches to something as complex as what DAWs and their physical extensions offer. However it's more about thinking of musical creation very laterally. It could mean leveraging tilts of any place where one can put an accelerometer (heck, the OP1 does it in a gimmicky way to change any specific setting you want for a given track for example), your finger movements position in 3d space (mimu does some of the finger movement stuff), heck even smoothed-out eye "gestures" could be explored (google-glass-y devices would need to succeed since you're not going anywhere with current tech without invasive things like the EyeLink II). Pivoting around 360 degrees in a "musical cockpit" could also be a thing. Voice too, and NLP could be leveraged in creative ways (to bring forward different aspects of the tool or to give instructions at a high-level - "take the first four measures of the first guitar track and apply melodic inversion, shift down one octave and paste at the 2min mark on the bass track"). Different parts of the music-making process could take on different shapes in spaces that add multiple dimensions. As the mimu shows, a lot of what those dimensions do could be entirely configurable and fitting the creator. Conductors are a good example of someone leveraging rotations and depth (over half-sphere surfaces) to create expressions and rhythm and so on. Plus nobody has to throw away the current tools either, once could think of the advantages of AR (Augmented Reality) which could leave perfectly clear the areas of your synths (or under a certain line of vision or even physically marked delimitations) during a performance, and all of the glory of ye olde knobs and faders can remain intact. Current VR is out of reach for too many people for much experimentation (basically forces you to be on Windows, requires either heavy-duty hardware or has very little room for fine details), which means all the exploratory stuff to lead to even a subset of what I mentioned is only accessible to a handful of people so far. Same for multi-modal interfaces (the mimu is still a toy for rich artsy-techy people, the Tobiis are either too basic or too expensive, etc). But we're still in the infancy of much of these kinds of tools. And while the usual engineer will talk about why everything's impossible, a designer or an artist will always be curious to see what just could be eventually.