4 ms·

On the other hand, the Mozilla counter proposal, which was trying to do low-latency audio by using Javascript as a DSP was pretty awful. It's like saying you'll

On the other hand, the Mozilla counter proposal, which was trying to do low-latency audio by using Javascript as a DSP was pretty awful. It's like saying you'll do 3D by handing a framebuffer to JS and doing rasterization in software. "Forcing it through the standards process" sounds like the kind of rhetoric Republicans use when bills get passed they don't like.

kevingadd 13y ago
Web Audio API has a similar mechanism for generating samples with JS and it's just as bad. In fact, Web Audio is worse for playback of JS-synthesized audio, not better. And lots of use cases demand playback of JS-synthesized audio: emulators, custom audio codecs, synths, etc. The Mozilla API did one or two things and did them adequately; it did them by extending an existing API (the <audio> element). Use cases it didn't cover could have been handled with the introduction of new APIs or extensions to existing APIs. The Web Audio API introduced an interconnected web of dozens of poorly tested, poorly specified components that covered some arbitrary subset of cases users care about. Most of the oversights still haven't been fixed and the spec is still full of unaddressed issues and deficiencies.
- cromwellian 13y ago
  >Web Audio API has a similar mechanism for generating samples with JS and it's just as bad. In fact, Web Audio is worse for playback of JS-synthesized audio, not better. And lots of use cases demand playback of JS-synthesized audio: emulators, custom audio codecs, synths, etc. Generating audio samples via JS is an escape hatch, the same way rasterizing to a raw framebuffer is an escape hatch. If your system had audio acceleration hardware, and many systems do (e.g. DirectSound/OpenAL), you want to leverage that. If you are deploying a game to a mobile device, the last thing you want to do is waste CPU cycles burning JS to do DSP effects in software. This is especially awful for low end phones like the ones that Firefox OS runs on. Latency on audio is already terrible, you don't want the browser UI event loop involved IMHO in processing it to feed it. Maybe if you had speced it being available to Web Workers without needing the UI thread involved it would make more sense. >The Web Audio API introduced an interconnected web of dozens of poorly tested, poorly specified components that covered some arbitrary subset of cases users care about The arbitrary subset being, those that have been on OSX for years? AFAIK, Chris Rogers developed Web Audio based on years of experience working on Core Audio at Apple, and therefore, at minimum, it represents at least some feedback from the use cases of the professional audio market, at the very least, Apple's own products like Garage Band and Logic Express which sit atop Core Audio. You assert that the other use cases could have been handled by just extending existing APIs, but to me this argument sounds like the following: "You've just dumped this massively complex WebGL API on us, it has hundreds of untested functions. It would be better to just have <canvas> or <img> with a raw array that you manipulate with JS. Any other functions could be done [hand-wave] by just bolting on additional higher level apis. Like, if you want to draw a polygon, we'd add that." APIs like Core Audio, Direct Sound, OpenGL have evolved because of low level optimization to match the API to what the hardware can accelerate efficiently. In many cases, bidirectional, so that HW influences the API and vice-versa. Putting the burden on the WG to reinvent what has already been done for years is the wrong way to go about it. Audio is a solved problem outside JS, all that was needed was good idiomatic bindings for either Core Audio, OpenAL, or DirectSound. Whenever I see these threads on HN, I always get a sense of a big dose of NIH from Mozilla. Whenever anyone else proposes a spec, there's always a complaint about complexity, like Republicans complaining about laws because they are too long, when in reality, they don't like them either for ideological reasons, or political ones. Mozilla is trying to build a web-based platform for competing with native apps. You can see it in asm.js and Firefox OS. And they are not going to get there if they shy away from doing the right things because they are complex. Mobile devices need all the hardware acceleration they can get, and specing out a solution that requires JS to do DSP sound processing is just an egregious waste of battery life and cycles IMHO for a low end web-OS based mobile HW.
  - _delirium 13y ago
    > Generating audio samples via JS is an escape hatch, the same way rasterizing to a raw framebuffer is an escape hatch. But that escape hatch is where all the interesting innovation happens! It's great that canvas exists, and it's much more widely used than WebGL is, because it's more flexible and depends on less legacy cruft. You don't have to use it, but I do, and I'd like a "canvas for audio" too. To make matters worse, the Web Audio stuff is much less flexible than OpenGL. You can at least write more or less arbitrary graphics code in OpenGL: it's not just an API for playing movie clips filtered through a set of predefined MovieFilter nodes. You can generate textures procedurally, program shaders, render arbitrary meshes with arbitrary lighting, do all kinds of stuff. If this were still the era of the fixed-function OpenGL 1.0 pipeline, it'd be another story, but today's OpenGL at least is a plausible candidate for a fully general programmable graphics pipeline. Web Audio seems targeted more at just being an audio player with a fixed chain of filter/effect nodes, not a fully programmable audio pipeline. How are you going to do real procedural music on the web, something more like what you can do in Puredata or SuperCollider or even Processing, without being able to write to something resembling an audio sink? Apple cares about stuff like Logic Express, yes, but that isn't a programmable synth or capable of procedural music; while I care about is the web becoming a usable procedural-music platform. One alternative is to do DSP in JS; another is to require you to write DSP code in a domain-specific language, like how you hand off shaders to WebGL. But Web Audio does the first badly and the 2nd not at all! > Audio is a solved problem outside JS Yeah, and the way it's solved is that outside JS, you can just write a synth that outputs to the soundcard...
    - cromwellian 13y ago
      WebGL is less used than Canvas for the most part, because 3D and linear algebra are much more difficult to work with for most people than 2D. Also, people work with raw canvas image arrays much more rarely than they do the high level functions (stroke/fill/etc) OpenGL was still a better API than a raw framebuffer even when it was just a fixed function pipeline. Minecraft for example is implemented purely with fixed-function stuff, no shaders. It isn't going to work if done via JS rasterization. Yes, there are people on the edge cases doing procedural music, but that is a rare use case compared to the more general case of people writing games and needing audio with attenuation, 3D positional HRTF, doppler effects, etc. That's the sweet spot that the majority of developers need. Today's 3D hardware includes features like geometry shaders/tessellation, but most games don't use them. OpenSL/AL would work a lot better if it had "audio shaders". Yes. But if your argument is that you want to write a custom DSP, then you don't want Data Audio API, what you want is some form of OpenAL++ that exposes an architecture neutral shader language for audio DSPs, that actually compiles your shader and uploads it to the DSP. Or, you want OpenCL plus a pathway to schedule running the shaders and copying the data to the HW that does not involve the browser event loop. That said, if there was a compelling need for the stuff you're asking for, it would have been done years ago. None of the professional apps, nor game developers, have been begging for Microsoft Direct Sound, Apple, or Khronos to make audio shaders. There was a company not to long ago, Aureal 3D, which tried to be the "3dfx of audio", but failed, but it turns out, most people just need a set of basic sound primitives they can change together. I have real sympathy for your use case. For years, I dreamed of sounds being generated in games ala PhysX, really simulating sound waves in the environment, and calculating true binaural audio, the way Oculus Rift wants to deliver video to your senses, taking into account head position. To literally duplicate the quality of binaural audio recordings programmatically. But we're not there, the industry seems to have let us down, there is no SGI, nor 3dfx, nor Nvidia/AMD "of audio" to lead the way, and we certainly aren't going to get there by dumping a frame buffer from JS. Right now, the target for all this stuff, Web GL, Web Audio, et al, it exposing APIs to bring high performance, low latency games to the web. I just don't see doing attenuation or HRTF in JS as compatible with that.

kevingadd 13y ago

Web Audio API has a similar mechanism for generating samples with JS and it's just as bad. In fact, Web Audio is worse for playback of JS-synthesized audio, not better. And lots of use cases demand playback of JS-synthesized audio: emulators, custom audio codecs, synths, etc. The Mozilla API did one or two things and did them adequately; it did them by extending an existing API (the <audio> element). Use cases it didn't cover could have been handled with the introduction of new APIs or extensions to existing APIs. The Web Audio API introduced an interconnected web of dozens of poorly tested, poorly specified components that covered some arbitrary subset of cases users care about. Most of the oversights still haven't been fixed and the spec is still full of unaddressed issues and deficiencies.

cromwellian 13y ago
>Web Audio API has a similar mechanism for generating samples with JS and it's just as bad. In fact, Web Audio is worse for playback of JS-synthesized audio, not better. And lots of use cases demand playback of JS-synthesized audio: emulators, custom audio codecs, synths, etc. Generating audio samples via JS is an escape hatch, the same way rasterizing to a raw framebuffer is an escape hatch. If your system had audio acceleration hardware, and many systems do (e.g. DirectSound/OpenAL), you want to leverage that. If you are deploying a game to a mobile device, the last thing you want to do is waste CPU cycles burning JS to do DSP effects in software. This is especially awful for low end phones like the ones that Firefox OS runs on. Latency on audio is already terrible, you don't want the browser UI event loop involved IMHO in processing it to feed it. Maybe if you had speced it being available to Web Workers without needing the UI thread involved it would make more sense. >The Web Audio API introduced an interconnected web of dozens of poorly tested, poorly specified components that covered some arbitrary subset of cases users care about The arbitrary subset being, those that have been on OSX for years? AFAIK, Chris Rogers developed Web Audio based on years of experience working on Core Audio at Apple, and therefore, at minimum, it represents at least some feedback from the use cases of the professional audio market, at the very least, Apple's own products like Garage Band and Logic Express which sit atop Core Audio. You assert that the other use cases could have been handled by just extending existing APIs, but to me this argument sounds like the following: "You've just dumped this massively complex WebGL API on us, it has hundreds of untested functions. It would be better to just have <canvas> or <img> with a raw array that you manipulate with JS. Any other functions could be done [hand-wave] by just bolting on additional higher level apis. Like, if you want to draw a polygon, we'd add that." APIs like Core Audio, Direct Sound, OpenGL have evolved because of low level optimization to match the API to what the hardware can accelerate efficiently. In many cases, bidirectional, so that HW influences the API and vice-versa. Putting the burden on the WG to reinvent what has already been done for years is the wrong way to go about it. Audio is a solved problem outside JS, all that was needed was good idiomatic bindings for either Core Audio, OpenAL, or DirectSound. Whenever I see these threads on HN, I always get a sense of a big dose of NIH from Mozilla. Whenever anyone else proposes a spec, there's always a complaint about complexity, like Republicans complaining about laws because they are too long, when in reality, they don't like them either for ideological reasons, or political ones. Mozilla is trying to build a web-based platform for competing with native apps. You can see it in asm.js and Firefox OS. And they are not going to get there if they shy away from doing the right things because they are complex. Mobile devices need all the hardware acceleration they can get, and specing out a solution that requires JS to do DSP sound processing is just an egregious waste of battery life and cycles IMHO for a low end web-OS based mobile HW.
- _delirium 13y ago
  > Generating audio samples via JS is an escape hatch, the same way rasterizing to a raw framebuffer is an escape hatch. But that escape hatch is where all the interesting innovation happens! It's great that canvas exists, and it's much more widely used than WebGL is, because it's more flexible and depends on less legacy cruft. You don't have to use it, but I do, and I'd like a "canvas for audio" too. To make matters worse, the Web Audio stuff is much less flexible than OpenGL. You can at least write more or less arbitrary graphics code in OpenGL: it's not just an API for playing movie clips filtered through a set of predefined MovieFilter nodes. You can generate textures procedurally, program shaders, render arbitrary meshes with arbitrary lighting, do all kinds of stuff. If this were still the era of the fixed-function OpenGL 1.0 pipeline, it'd be another story, but today's OpenGL at least is a plausible candidate for a fully general programmable graphics pipeline. Web Audio seems targeted more at just being an audio player with a fixed chain of filter/effect nodes, not a fully programmable audio pipeline. How are you going to do real procedural music on the web, something more like what you can do in Puredata or SuperCollider or even Processing, without being able to write to something resembling an audio sink? Apple cares about stuff like Logic Express, yes, but that isn't a programmable synth or capable of procedural music; while I care about is the web becoming a usable procedural-music platform. One alternative is to do DSP in JS; another is to require you to write DSP code in a domain-specific language, like how you hand off shaders to WebGL. But Web Audio does the first badly and the 2nd not at all! > Audio is a solved problem outside JS Yeah, and the way it's solved is that outside JS, you can just write a synth that outputs to the soundcard...
  - cromwellian 13y ago
    WebGL is less used than Canvas for the most part, because 3D and linear algebra are much more difficult to work with for most people than 2D. Also, people work with raw canvas image arrays much more rarely than they do the high level functions (stroke/fill/etc) OpenGL was still a better API than a raw framebuffer even when it was just a fixed function pipeline. Minecraft for example is implemented purely with fixed-function stuff, no shaders. It isn't going to work if done via JS rasterization. Yes, there are people on the edge cases doing procedural music, but that is a rare use case compared to the more general case of people writing games and needing audio with attenuation, 3D positional HRTF, doppler effects, etc. That's the sweet spot that the majority of developers need. Today's 3D hardware includes features like geometry shaders/tessellation, but most games don't use them. OpenSL/AL would work a lot better if it had "audio shaders". Yes. But if your argument is that you want to write a custom DSP, then you don't want Data Audio API, what you want is some form of OpenAL++ that exposes an architecture neutral shader language for audio DSPs, that actually compiles your shader and uploads it to the DSP. Or, you want OpenCL plus a pathway to schedule running the shaders and copying the data to the HW that does not involve the browser event loop. That said, if there was a compelling need for the stuff you're asking for, it would have been done years ago. None of the professional apps, nor game developers, have been begging for Microsoft Direct Sound, Apple, or Khronos to make audio shaders. There was a company not to long ago, Aureal 3D, which tried to be the "3dfx of audio", but failed, but it turns out, most people just need a set of basic sound primitives they can change together. I have real sympathy for your use case. For years, I dreamed of sounds being generated in games ala PhysX, really simulating sound waves in the environment, and calculating true binaural audio, the way Oculus Rift wants to deliver video to your senses, taking into account head position. To literally duplicate the quality of binaural audio recordings programmatically. But we're not there, the industry seems to have let us down, there is no SGI, nor 3dfx, nor Nvidia/AMD "of audio" to lead the way, and we certainly aren't going to get there by dumping a frame buffer from JS. Right now, the target for all this stuff, Web GL, Web Audio, et al, it exposing APIs to bring high performance, low latency games to the web. I just don't see doing attenuation or HRTF in JS as compatible with that.