4 ms·

imho MIDI 2.0 hits the low hanging fruit but doesn't go far enough at fixing the biggest problem in professional audio. Deterministic rendering. Basically same

imho MIDI 2.0 hits the low hanging fruit but doesn't go far enough at fixing the biggest problem in professional audio. Deterministic rendering. Basically same input makes the same output, which doesn't happen even in totally digital systems - doubly so with live (or recreated) midi events. I'll give an example - you have your PC with $DAW_OF_CHOICE running and plug in $MIDI_2_CONTROLLER to USB and enable a track and hit record, play your stuff, and stop. When you play it back, unless the cosmic forces are exceptionally on your side, it will not sound the same as when you played it. It's subtle and sometimes ignored or even desirable, but it's there. There are a lot of reasons why this problem hasn't been solved, some technical and others artistic. But imho, a certain grade of equipment (namely, recording/reproduction, of which the MIDI protocol is a key component) should behave identically under the same conditions and be able to reproduce a performance exactly. It's a goal that borders on absurd, but I think we could do it! Namely - enough of this "transport agnostic" horseshit. Give me a professional event protocol that is sent synchronously with the audio, on the wire and on the board. I want my codec chips interleaving midi messages (don't care if they're 16/24/32 bit) as a separate audio channel, even if it is undersampled (eg send LRLRLRMLRLRLRM over I2C at the appropriate clock to have audio in time and MIDI undersampled by 3x per word), I want MIDI events in my interrupt and in the callback, synchronized exactly to whatever audio is coming in at the same time. And give me unadulterated total, absolute dictatorial control over the audio drivers using a ping/pong buffer that gets mapped to memory in the audio callback for my professional application and mine alone. Minimize the kernel time spent mapping physical memory to virtual memory, like ASIO drivers but without the bullshit. I want as little overhead between the frame buffer coming in off DMA to the chip as I want without the possibility of pwning the system. Hell, give me a dedicated core!

analognoise 7y ago
Use an FPGA? I mean that sounds pretty simple, if you think there's a market let's talk.
- unlinked_dll 7y ago
  Silicon isn't the bottleneck here, you could probably rig it up on existing chips. I can think of a way to do it with a stereo codec and surround codec with a mux in there, it wouldn't be super cheap but for pro gear, who cares. The big blocker is the drivers and compatibility with existing software. Audio people do not like tooling changes, and what I'm talking about is a rather fundamental change to very low level components in MacOS and Windows, the former of which is more important and already does things very well and would take a massive engineering effort with little ROI (CoreAudio is a marvel). On Linux you could do some impressive shit, and how I'd like to do it is via a hypervisor that hogs a core for audio processing and provides an API back to the system for communication. I know there has been some work to do that already, but incorporating hardware changes to support it would be fairly high cost with fairly low ROI. This kind of thing could be done, sure, but the money in pro audio and speed of adoption are non-ideal. You're talking 3-5 years dev cycle to get a prototype shipped and in stores, ask users to give up a lot of hardware, and for a subtle change in what they hear. This would be a project for my free time after an extremely lucrative exit event from my current venture, and I'd have to add it to the list of pro audio paradigm shifts I'd want to work on.
  - PaulDavisThe1st 7y ago
    Given your later explanations of what you actually mean, none of what you're talking about can possibly help. MIDI is a serial protocol without timestamps. It is not possible for it to have "N notes with the same timestamp" because MIDI messages/events do not have timestamps. There is no notion of any time other than "now" in the MIDI protocol. In the early 2000's there were MIDI hardware interfaces that did accept a timestamped event stream, and claimed to provide much better timing than those that just "send it out ASAP". These gained zero traction in the industry because nobody could actually tell the difference, and it required h/w-specific code, which nobody likes. Your "how i'd like to do it" on Linux is impressively wrong in the sense that you do not need a hypervisor and you do not need hardware changes. We already do this on Linux, when desired (e.g. embedded Linux in the mixer consoles on several pro-audio companies). I've spent 20 years writing pro-audio+MIDI software after "an extremely lucrative exit" from a previous venture. I don't think you understand what the actual pro-audio paradigms are, nor how they could be changed.
jerrysievert 7y ago
> There are a lot of reasons why this problem hasn't been solved, some technical and others artistic. But imho, a certain grade of equipment (namely, recording/reproduction, of which the MIDI protocol is a key component) should behave identically under the same conditions and be able to reproduce a performance exactly. It's a goal that borders on absurd, but I think we could do it! why should this always be the case? or better yet, why should this often be the case? quite a lot of digital effects attempt to behave like their analog counterparts, not their digital counterparts - we should expect a non-deterministic result. in digital: I send 1, then I send 1, that makes 2. in analog: I sent 1, then I sent 1, that makes 1 + some other stuff + 1. it shouldn't necessarily be a goal, bordering on "absurd" or not, it's just a different expression in a different medium. having more bits (25 of them) shouldn't change the sound profoundly, when MIDI was introduced, analog was king, and a note pressed was typically different than the same note pressed a second later. this is the same environment that has been pushed forward with hardware (again) via eurorack, and emulated quite effectively in software. all that we've really done is smooth the digital by adding more steps, which is fantastic, but to try to "solve" a "problem" with this, other than some smoothness, is just silly. this said as someone on their nth career writing audio software (https://svmodular.com https://svmodular.com if you're interested).
- unlinked_dll 7y ago
  It's not about quantization error (which is quantifiable as noise) or the kind of nonlinearities you're talking about, but timing concerns. It's basically the difference between naive automation in a DAW and sample accurate automation, it's not about the granularity of your changes but the fact that sample accuracy allows your system to reproduce the same thing every time. Not so many years ago, online renders in certain DAWs were perceptually and quantifiably different than offline renders because of things like this - you want to be able to tell a user what they hear while they work is the same when they go back and render. With MIDI 1 and 2.0 that's rather difficult when factoring in live input due to the fact that your production system has wack drivers on top of a non-realtime OS and can't provide guarantees. MIDI 2.0 goes a good step in the direction with synchronization, but I have doubts that it will be utilized to where we can guarantee received events are replicated accordingly, due to the accuracy of reception and synchronization of clocks. Maybe we'll get it, idk.
rewgs 7y ago
> I'll give an example - you have your PC with $DAW_OF_CHOICE running and plug in $MIDI_2_CONTROLLER to USB and enable a track and hit record, play your stuff, and stop. When you play it back, unless the cosmic forces are exceptionally on your side, it will not sound the same as when you played it. It's subtle and sometimes ignored or even desirable, but it's there. I'm sorry, but...what? I'm the rare non-professional-programmer on this site. I'm a professional composer -- I spend my days writing/producing/mixing music on a computer. Can you go into more detail of what you're talking about here? Because, no offense, but if you were right, I think I would have noticed by now.
- blattimwind 7y ago
  I wouldn't be at all surprised if the process is not bit perfect; the question is: can you hear a difference on your own? Can you hear the difference if it is pointed out ahead of time (through waveform comparison)?
  - rewgs 7y ago
    I would be absolutely surprised if it weren't bit perfect. We're dealing with 7 bits of MIDI data controlling, typically at most, 48,000 samples per second. These numbers are chump change for modern computers. The bit depths can add some more complexity to that in terms of dynamic range (I'm usually working at 32 bit float), but that doesn't apply as much to the situation here. Yes, real-time performance is a tough task (as is always stated, "the audio thread waits for nothing"), but OP here is talking about recording MIDI data, which -- once recorded -- acts as a static input controlling either a synthesizer or sampler. So let's break that down. A digital synthesizer, unless designed with some amount of randomness (typically for "analog-like" behavior purposes), by definition is the same every time. The MIDI data in this case is going to be something like CC7, controlling output volume; CC1, controlling some pre-defined parameter (i.e. opening/closing a filter); etc. A solid representation of OP's example in this case would be "CC7 controlling output volume of a synth over a 4 second period, linearly from totally silent upwards to 0 dB." I fail to see how that could possibly change from one playback to the next, unless, again, some amount of randomness-with-same-MIDI-data is a feature of the synth's programming. A sampler is just, at its most basic form, playing back audio. Audio is a static file; MIDI is controlling which audio is playing back. Round Robin sampling, where, say, C4 is played N number of times and X number of similar samples are called randomly so as to avoid the "machine gun effect," where literally the same sample is called, could account for "it not sounding the same," but like the "analog-like" programming of the synth above, that's on purpose, not a flaw. I routinely deal with situations where phase cancellation null tests would reveal the kind of behavior that OP is talking about, and I simply have never come across them. And that's not even going into sensitivity in listening, which while subjective and impossible to prove, is something I put a lot of faith in. Sorry, unless OP can point me to a solid source laying out a further explanation, I call horse shit.
djaychela 7y ago
>I'll give an example - you have your PC with $DAW_OF_CHOICE running and plug in $MIDI_2_CONTROLLER to USB and enable a track and hit record, play your stuff, and stop. When you play it back, unless the cosmic forces are exceptionally on your side, it will not sound the same as when you played it. It's subtle and sometimes ignored or even desirable, but it's there. I've never encountered this with any modern system; I've been sequencing from the early 90s on Atari, still doing it now, and it makes up most of my (varied) day job - plus recording, etc. I've done tests of my own and not found anything measurable (I was an instrument technician in a nuclear facility before turning to music, FTR). There were definitely issues with early DAWs (Cubase Audio springs to mind), but I've not found anything in the modern era. Care to share some measurable results? I'm interested in what the problem is.
vortico 7y ago
Why would you ever want perfectly deterministic rendering? I've never met an audio engineer that would care about this, and I've never personally run into the problem of my audio projects being too undeterministic. We're not doing LHC experiments here, we're just making music. Who cares if a MIDI note arrives 100 microseconds late? A musician surely wouldn't care. If it's truly a problem for you because of phasing or something, why not bake the MIDI track into an audio track before mixing?
boomlinde 7y ago
MIDI addresses a broader category of problems where there might not necessarily even be an audio stream. Not unusual now, I have an analog desktop synthesizer module with a MIDI-to-CV control interface, and I simply want to connect a MIDI keyboard to it to be able to play it. My application is unconcerned with digital audio streams. Actually, I have a bunch of synthesizers, some analog and some digital. I use a hardware sequencer to sequence them. Some use their own internal sequencers, so they simply use the start/stop/clock real-time messages to stay on beat. The point at which any of this becomes an audio stream is if I start my PC and record from the mixer output. MIDI 2.0 at least opens up to some improvements to synchronization with fine grained event timestamps. Using that together with timestamped audio buffer delivery at least allows for jitter mitigation.

analognoise 7y ago

Use an FPGA? I mean that sounds pretty simple, if you think there's a market let's talk.

unlinked_dll 7y ago
Silicon isn't the bottleneck here, you could probably rig it up on existing chips. I can think of a way to do it with a stereo codec and surround codec with a mux in there, it wouldn't be super cheap but for pro gear, who cares. The big blocker is the drivers and compatibility with existing software. Audio people do not like tooling changes, and what I'm talking about is a rather fundamental change to very low level components in MacOS and Windows, the former of which is more important and already does things very well and would take a massive engineering effort with little ROI (CoreAudio is a marvel). On Linux you could do some impressive shit, and how I'd like to do it is via a hypervisor that hogs a core for audio processing and provides an API back to the system for communication. I know there has been some work to do that already, but incorporating hardware changes to support it would be fairly high cost with fairly low ROI. This kind of thing could be done, sure, but the money in pro audio and speed of adoption are non-ideal. You're talking 3-5 years dev cycle to get a prototype shipped and in stores, ask users to give up a lot of hardware, and for a subtle change in what they hear. This would be a project for my free time after an extremely lucrative exit event from my current venture, and I'd have to add it to the list of pro audio paradigm shifts I'd want to work on.
- PaulDavisThe1st 7y ago
  Given your later explanations of what you actually mean, none of what you're talking about can possibly help. MIDI is a serial protocol without timestamps. It is not possible for it to have "N notes with the same timestamp" because MIDI messages/events do not have timestamps. There is no notion of any time other than "now" in the MIDI protocol. In the early 2000's there were MIDI hardware interfaces that did accept a timestamped event stream, and claimed to provide much better timing than those that just "send it out ASAP". These gained zero traction in the industry because nobody could actually tell the difference, and it required h/w-specific code, which nobody likes. Your "how i'd like to do it" on Linux is impressively wrong in the sense that you do not need a hypervisor and you do not need hardware changes. We already do this on Linux, when desired (e.g. embedded Linux in the mixer consoles on several pro-audio companies). I've spent 20 years writing pro-audio+MIDI software after "an extremely lucrative exit" from a previous venture. I don't think you understand what the actual pro-audio paradigms are, nor how they could be changed.

jerrysievert 7y ago

> There are a lot of reasons why this problem hasn't been solved, some technical and others artistic. But imho, a certain grade of equipment (namely, recording/reproduction, of which the MIDI protocol is a key component) should behave identically under the same conditions and be able to reproduce a performance exactly. It's a goal that borders on absurd, but I think we could do it! why should this always be the case? or better yet, why should this often be the case? quite a lot of digital effects attempt to behave like their analog counterparts, not their digital counterparts - we should expect a non-deterministic result. in digital: I send 1, then I send 1, that makes 2. in analog: I sent 1, then I sent 1, that makes 1 + some other stuff + 1. it shouldn't necessarily be a goal, bordering on "absurd" or not, it's just a different expression in a different medium. having more bits (25 of them) shouldn't change the sound profoundly, when MIDI was introduced, analog was king, and a note pressed was typically different than the same note pressed a second later. this is the same environment that has been pushed forward with hardware (again) via eurorack, and emulated quite effectively in software. all that we've really done is smooth the digital by adding more steps, which is fantastic, but to try to "solve" a "problem" with this, other than some smoothness, is just silly. this said as someone on their nth career writing audio software (https://svmodular.com https://svmodular.com if you're interested).

unlinked_dll 7y ago
It's not about quantization error (which is quantifiable as noise) or the kind of nonlinearities you're talking about, but timing concerns. It's basically the difference between naive automation in a DAW and sample accurate automation, it's not about the granularity of your changes but the fact that sample accuracy allows your system to reproduce the same thing every time. Not so many years ago, online renders in certain DAWs were perceptually and quantifiably different than offline renders because of things like this - you want to be able to tell a user what they hear while they work is the same when they go back and render. With MIDI 1 and 2.0 that's rather difficult when factoring in live input due to the fact that your production system has wack drivers on top of a non-realtime OS and can't provide guarantees. MIDI 2.0 goes a good step in the direction with synchronization, but I have doubts that it will be utilized to where we can guarantee received events are replicated accordingly, due to the accuracy of reception and synchronization of clocks. Maybe we'll get it, idk.

rewgs 7y ago

> I'll give an example - you have your PC with $DAW_OF_CHOICE running and plug in $MIDI_2_CONTROLLER to USB and enable a track and hit record, play your stuff, and stop. When you play it back, unless the cosmic forces are exceptionally on your side, it will not sound the same as when you played it. It's subtle and sometimes ignored or even desirable, but it's there. I'm sorry, but...what? I'm the rare non-professional-programmer on this site. I'm a professional composer -- I spend my days writing/producing/mixing music on a computer. Can you go into more detail of what you're talking about here? Because, no offense, but if you were right, I think I would have noticed by now.

blattimwind 7y ago
I wouldn't be at all surprised if the process is not bit perfect; the question is: can you hear a difference on your own? Can you hear the difference if it is pointed out ahead of time (through waveform comparison)?
- rewgs 7y ago
  I would be absolutely surprised if it weren't bit perfect. We're dealing with 7 bits of MIDI data controlling, typically at most, 48,000 samples per second. These numbers are chump change for modern computers. The bit depths can add some more complexity to that in terms of dynamic range (I'm usually working at 32 bit float), but that doesn't apply as much to the situation here. Yes, real-time performance is a tough task (as is always stated, "the audio thread waits for nothing"), but OP here is talking about recording MIDI data, which -- once recorded -- acts as a static input controlling either a synthesizer or sampler. So let's break that down. A digital synthesizer, unless designed with some amount of randomness (typically for "analog-like" behavior purposes), by definition is the same every time. The MIDI data in this case is going to be something like CC7, controlling output volume; CC1, controlling some pre-defined parameter (i.e. opening/closing a filter); etc. A solid representation of OP's example in this case would be "CC7 controlling output volume of a synth over a 4 second period, linearly from totally silent upwards to 0 dB." I fail to see how that could possibly change from one playback to the next, unless, again, some amount of randomness-with-same-MIDI-data is a feature of the synth's programming. A sampler is just, at its most basic form, playing back audio. Audio is a static file; MIDI is controlling which audio is playing back. Round Robin sampling, where, say, C4 is played N number of times and X number of similar samples are called randomly so as to avoid the "machine gun effect," where literally the same sample is called, could account for "it not sounding the same," but like the "analog-like" programming of the synth above, that's on purpose, not a flaw. I routinely deal with situations where phase cancellation null tests would reveal the kind of behavior that OP is talking about, and I simply have never come across them. And that's not even going into sensitivity in listening, which while subjective and impossible to prove, is something I put a lot of faith in. Sorry, unless OP can point me to a solid source laying out a further explanation, I call horse shit.

djaychela 7y ago

>I'll give an example - you have your PC with $DAW_OF_CHOICE running and plug in $MIDI_2_CONTROLLER to USB and enable a track and hit record, play your stuff, and stop. When you play it back, unless the cosmic forces are exceptionally on your side, it will not sound the same as when you played it. It's subtle and sometimes ignored or even desirable, but it's there. I've never encountered this with any modern system; I've been sequencing from the early 90s on Atari, still doing it now, and it makes up most of my (varied) day job - plus recording, etc. I've done tests of my own and not found anything measurable (I was an instrument technician in a nuclear facility before turning to music, FTR). There were definitely issues with early DAWs (Cubase Audio springs to mind), but I've not found anything in the modern era. Care to share some measurable results? I'm interested in what the problem is.

vortico 7y ago

Why would you ever want perfectly deterministic rendering? I've never met an audio engineer that would care about this, and I've never personally run into the problem of my audio projects being too undeterministic. We're not doing LHC experiments here, we're just making music. Who cares if a MIDI note arrives 100 microseconds late? A musician surely wouldn't care. If it's truly a problem for you because of phasing or something, why not bake the MIDI track into an audio track before mixing?

boomlinde 7y ago

MIDI addresses a broader category of problems where there might not necessarily even be an audio stream. Not unusual now, I have an analog desktop synthesizer module with a MIDI-to-CV control interface, and I simply want to connect a MIDI keyboard to it to be able to play it. My application is unconcerned with digital audio streams. Actually, I have a bunch of synthesizers, some analog and some digital. I use a hardware sequencer to sequence them. Some use their own internal sequencers, so they simply use the start/stop/clock real-time messages to stay on beat. The point at which any of this becomes an audio stream is if I start my PC and record from the mixer output. MIDI 2.0 at least opens up to some improvements to synchronization with fine grained event timestamps. Using that together with timestamped audio buffer delivery at least allows for jitter mitigation.