14 ms·

Pretty cool, though I wonder what the latency of this would be if used as a plugin? The author says it works in real-time, but to non music/audio folks this co

by fab1an 6y ago

Pretty cool, though I wonder what the latency of this would be if used as a plugin? The author says it works in real-time, but to non music/audio folks this could mean '100 ms latency is real-time enough, right?' Generally, I think the audio VST business is a really fun space to be in for a lifestyle business, as it is way too small to be attractive for VCs. It seems like a space that provides many niches for lots of small players to thrive in. As an aside, it's really quite interesting that a lot of cutting edge tech is now used to emulate the hardware-based tech of yesteryear. Think film filters for photoshop, and about 90% of all audio plugins that emulate high end hardware, compressors, pedals, etc etc.

eru 6y ago
Real-time has a few slightly different meanings. So it's hard to say what the author means. One meaning is just that you can guarantee specific deadlines. So if your programme can react within an hour guaranteed, that would be real-time. (Though usually we are talking about tighter deadlines, like what's needed to make ABS brakes work.) For 'real time' music usage you wouldn't need strict guarantees, but something that's usually fast enough.
- aea12 6y ago
  Implementing a VST plugin is literally the exact definition of requiring strict latency guarantees. Your comment winds through a lot of unrelated comparisons to ultimately not make any sense. “Usually fast enough” are three words that guarantee failure in a live show/MIDI environment, which is a large use case of VST and its peers beyond production. By extension, “usually fast enough” further guarantees nobody will ever use your software. That’s noticeable right away. The question isn’t about compsci real-time theorycrafting, it’s “here’s a buffer of samples, if you don’t give it back in a dozen milliseconds the entire show collapses.” That’s pretty clearly meant by “real time“ contextually.
  - microcolonel 6y ago
    Not to mention if the inference is done on the CPU, it shouldn't be that hard to control it. The matrices are of a set size by the time you're running a VST; this is the actual simple answer. The medium answer is "this is a wavenet model, so inference is probably really expensive unless the continuous output is a huge improvement to performance".
    - mochomocha 6y ago
      Indeed. Having myself spent some time in the "VST lifestyle business" when I was in grad school (was selling a guitar emulation based on physical modelling synthesis), and now working in ML, I think there's no chance for such an approach to hit "mainstream" anytime soon. Even if you do your inference on CPU, most deep learning libraries are designed for throughput, not latency. In a VST plugin environment, you're also only one of the many components requiring computation, so your computational requirements better be low.
      - jerf 6y ago
        You might be able to combine it with the recent work on minimizing models to obtain something that is small enough to run reliably in real time. Although the unusual structure of the net here may mean you're doing original and possibly publication-level work to adapt that stuff to this net structure. If you were really interested in this, there could also be some profit in minimizing the model and then figuring out how to replicate it in a non-neural net way. Direct study of the resulting net may be profitable. (I'm not in the ML field. I haven't seen anyone report this but I may just not be seeing it. But I'd be intrigued to see the result of running the size reduction on the net, running training on that network, then seeing if maybe you can reduce the resulting network again, then training that, and iterating until you either stop getting reduced sizes or the quality degrades too far. I've also wondered if there is something you could do to a net to encourage it not to have redundancies in it... although in this case the structure itself may do that job.)
      - microcolonel 6y ago
        I wonder if teddykoker has looked at applying FFTNet or similar methods as a replacement for Wavenet. I'm not sure but it seems to me like FFTNet is a lot more tractable than Wavenet, and not necessarily that much worse for equivalent training data.
  - eru 6y ago
    The show won't collapse, if you have one glitch an hour.
    - aea12 6y ago
      Yes, it absolutely 100% will, depending on what you mean by handwaving “glitch”. VST is built into chains, and a flaky plugin will derail an entire performance, often making downstream plugins crash. I’m speaking from extensive experience writing plugins and performing with them in multiple hosts and trigger setups. It’s not a robust protocol, but it gets the job done. Are you speaking from some experience with which I’m unfamiliar where it’s okay for DSP code to fail hourly? Trying to understand your viewpoint.
      - nseggs 6y ago
        Agreed. If anyone wants to see some of the more successful DSP work being done today for pro or prosumer audio, I recommend checking out Strymon and Universal Audio products. Both make use of SHARC SoCs and achieve great results.
      - boomlinde 6y ago
        A VST that doesn't fill its buffer on time shouldn't crash another plugin. It's your other plugins that are flaky.
        deleted 6y ago
        [deleted]
        boomlinde 6y ago
        To expand on this, each plugin will receive host managed buffers that they're requested to fill and the input they're expected to process. If they don't do that in time for the host to mix the buffers and deliver the mixed buffer to the audio driver, it simply won't. Nowhere do the plugins directly interact through this process. If your plugins are crashing because of an underrun you have a much more serious problem than underruns. Then you have plugins writing to or reading from memory that wasn't either handed to them by the host or allocated by themselves. That bad code running in your process can cause it to crash is an orthogonal problem to buffer underruns causing skips or stuttering in audio.
  - swebs 6y ago
    No, the other guy is right. Technically the definition of real-time can have a lot of leeway. Here's the paper linked in the article. Note how the authors never define what they really mean by real-time. They even make statements like "runs 1.9 times faster than real-time". They certainly imply your definition, but there's plenty of wiggle room to say "Well technically, I wasn't lying" https://www.mdpi.com/2076-3417/10/3/766/pdf https://www.mdpi.com/2076-3417/10/3/766/pdf
    - deleted 6y ago
      [deleted]
    - zodiac 6y ago
      In the context of e.g. offline video encoders "1.9x realtime" is a statement about throughput, not latency
  - boomlinde 6y ago
    "Usually fast enough" is unfortunately the only guarantee a preemptive multitasking OS can give you. Unless your system is guaranteeing your program x cycles of uninterrupted processing per frame of audio and you can consistently process the frame in that amount of cycles, the only mitigation is to deliver frames in large enough chunks that you never run out of time in practice under agreeable circumstances. That said, I agree that the question of what "real-time" might mean is irrelevant given the context.
    - aea12 6y ago
      It is completely irrelevant, given the context. The only, only, only thing real-time means here is “can be run on a live signal passing through it” rather than “is a slow, offline effect for a DAW”. No hard real-time, no soft real-time, no QNX, no pulling out the college compsci textbook. There IS real-time in that sense in DSP, it just isn’t in a VST plugin. I’ll repeat again that any compsci theorycrafting is not the concern here, and real-time has a very specific meaning in DSP. Computer science does not own the concept of real-time, and the only people tripping over the terminology are those with more compsci experience than DSP. I appreciate everyone trying to explain this to me, but (a) I understand both, and (b) this is like saying “no, Captain, a vector could mean anything like a mathematical collection, air traffic control should learn a thing or two from mathematics.”
      - boomlinde 6y ago
        Just to be perfectly clear here because I'm not sure you're just using my post as a soapbox or if you have misunderstood my argument: I agree that it's clear what real-time means in this context. I disagree that "usually fast enough" guarantees failure for a VST, because in the case of VST, "usually fast enough" is the only guarantee the host operating system will offer your software. It's not "theorycrafting" to say that real-time music software running in a preemptive multitasking operating system without deterministic process time allocation will have to suffer the possibility of occasional drops. It happens in practice and audio drivers have to be implemented to account for the bulk of it, and the VST API is designed in such a way that failure to fill a buffer on time needn't be fatal.
alexlarsson 6y ago
I assume by real-time he meant "able to produce samples at a rate equal to or higher than the audio output sample rate".
- amelius 6y ago
  Latency is equally important.
- TheRealPomax 6y ago
  That's not what real time, means though. Real time processing means taking signals as they come in, and outputting the transformed result such that there is as close to no signal lag as possible. The output can in fact be wildly lower or higher resolution, real-time does not particularly say anything about that. It's all about whether the output plays (for practical purposes) at the perceived "same time" as the input signal. There will always be some delay, but that delay can't get perceivable, and for obvious reasons there can't be any (significant) buffering.
  - jeffbee 6y ago
    Is that your private definition of "real-time"? I think it is common to define real-time processing by a specified, finite time between input and output. Many real-time processes are concerned more with the consistency of the latency than with its absolute value.
    - avisser 6y ago
      For guitar pedals, there is an implied sub-perceptibility. The output needs to happen as I play - if the delay is too long, it's now a delay pedal. So realtime might match your definition, but it is consistent in audio production. For humans, you can start to notice the lag @ 50ms. (A selection of experimental results summarized here https://gamedev.stackexchange.com/a/74975 https://gamedev.stackexchange.com/a/74975)
      - NobodyNada 6y ago
whiddershins 6y ago
Do solo or small shop vst plugin developers make any money? I’m curious if anyone has any direct knowledge about that. There are so many professional activities similar to that where no one makes any money and people really just do it for the love, and then there are seemingly similar things like that where people make surprisingly large amounts of money.
- ff7f00 6y ago
  There are definitely big players making a lot of money from plugins they develop. Here are a few to check out: ($1200) https://www.native-instruments.com/en/products/komplete/bundles/komplete-12-ultimate/ https://www.native-instruments.com/en/products/komplete/bund... ($300) https://www.soundtoys.com/product/soundtoys-5/ https://www.soundtoys.com/product/soundtoys-5/ ($500) https://www.arturia.com/products/analog-classics/v-collection/overview https://www.arturia.com/products/analog-classics/v-collectio... However, piracy is also pretty big when it comes to plugins.
  - moralestapia 6y ago
    Sure, but, >Do solo or small shop vst plugin developers make any money?
    - kleer001 6y ago
      The implication seems to be "no".
      - TheRealPomax 6y ago
        That implication would be wildly incorrect.
        kleer001 6y ago
        With the operative word being "developers" I disagree.
deleted 6y ago
[deleted]
samplenoise 6y ago
There's latency and there's the somewhat separate question of how much time is needed to make a prediction. Wavenet is causal (no look-ahead) and operates on the sample level so there are no buffers and thus no latency in the strict sense, beyond encoding/decoding into the sample rate and format required by the ML model, which should take <1ms. Whether a model manages to make a prediction in that amount of time depends on things like the receptive field and number of layers. The linked paper says their custom implementation runs at 1.1x real-time. I guess this isn't impossible; their receptive field is ~40ms, vs. 300 for the original (notoriously slow) wavenet, and the model is likely to have less layers and channels.
- cjlars 6y ago
  "Round trip," or guitar to processing to speakers needs to be sub 10ms to be transparent to the musician. Source: spent years playing guitar through my guitar -> DAC -> PC -> DAC -> speaker signal chain
  - samplenoise 6y ago
    The receptive field size is how much 'history' the algorithm requires, it doesn't affect the round trip time, which can still be sub ms
TheRealPomax 6y ago
while training? terrible. As finalised model running in an AU/VST3 wrapper? probably extremely low.
qppo 6y ago
I know of a few shops that took VC money. The big problem isn't the market size so much as how slow the market moves. The product lifetime of a plugin is around a decade. And users hate subscriptions. And it's really hard to determine the value you add to your customers. And no one wants to pay you. It's basically a terrible place to be a developer in it for the money. Really fun work otherwise. The cool gigs are the ones where you build custom plugins for someone's crazy idea. In consumer applications, plugins are used all the time for prototyping before you go to hardware. MATLAB is way too slow for anything useful.
- nil-sec 6y ago
  The success of splice would disagree with your notion that “users hate subscriptions”. Given the horrendous price point of many of these plugins it seems to be perfect for a subscription based model. To me it always seemed there is more of a pushback from the industry producing vsts than from the consumers.
  - qppo 6y ago
    Splice's numbers aren't public so I can't comment on their success. Avid's are, and they had a terrible quarter - and they're the poster child (alongside Adobe) for subscription licensing in creative software. But I'd be interested to see what the breakdown in revenue is for plugin licenses versus preset/sample packs (bit of a blade & razor model there). The price points really aren't horrendous if you consider how expensive the engineering is, how little demand there is, and how long you need to maintain a product. You aren't being ripped off by spending a couple hundred bucks on a plugin. I think we'll end up at a place where everything is a subscription, but I can tell you from experience that it creates friction for the users.
    - nil-sec 6y ago
      Agreed. The business model seems to be to give access to the rent to own deals via the sample subscription fee. Don’t think they make any money of their plugin deals. I’m also not arguing it’s too expensive or a rip off. But it’s still a large amount of money for software, in the private space at least. The rent to own thing seems like a smart tool to get rid of the barrier of entry.

by fab1an 6y ago

eru 6y ago

Real-time has a few slightly different meanings. So it's hard to say what the author means. One meaning is just that you can guarantee specific deadlines. So if your programme can react within an hour guaranteed, that would be real-time. (Though usually we are talking about tighter deadlines, like what's needed to make ABS brakes work.) For 'real time' music usage you wouldn't need strict guarantees, but something that's usually fast enough.

aea12 6y ago
Implementing a VST plugin is literally the exact definition of requiring strict latency guarantees. Your comment winds through a lot of unrelated comparisons to ultimately not make any sense. “Usually fast enough” are three words that guarantee failure in a live show/MIDI environment, which is a large use case of VST and its peers beyond production. By extension, “usually fast enough” further guarantees nobody will ever use your software. That’s noticeable right away. The question isn’t about compsci real-time theorycrafting, it’s “here’s a buffer of samples, if you don’t give it back in a dozen milliseconds the entire show collapses.” That’s pretty clearly meant by “real time“ contextually.
- microcolonel 6y ago
  Not to mention if the inference is done on the CPU, it shouldn't be that hard to control it. The matrices are of a set size by the time you're running a VST; this is the actual simple answer. The medium answer is "this is a wavenet model, so inference is probably really expensive unless the continuous output is a huge improvement to performance".
  - mochomocha 6y ago
    Indeed. Having myself spent some time in the "VST lifestyle business" when I was in grad school (was selling a guitar emulation based on physical modelling synthesis), and now working in ML, I think there's no chance for such an approach to hit "mainstream" anytime soon. Even if you do your inference on CPU, most deep learning libraries are designed for throughput, not latency. In a VST plugin environment, you're also only one of the many components requiring computation, so your computational requirements better be low.
    - jerf 6y ago
      You might be able to combine it with the recent work on minimizing models to obtain something that is small enough to run reliably in real time. Although the unusual structure of the net here may mean you're doing original and possibly publication-level work to adapt that stuff to this net structure. If you were really interested in this, there could also be some profit in minimizing the model and then figuring out how to replicate it in a non-neural net way. Direct study of the resulting net may be profitable. (I'm not in the ML field. I haven't seen anyone report this but I may just not be seeing it. But I'd be intrigued to see the result of running the size reduction on the net, running training on that network, then seeing if maybe you can reduce the resulting network again, then training that, and iterating until you either stop getting reduced sizes or the quality degrades too far. I've also wondered if there is something you could do to a net to encourage it not to have redundancies in it... although in this case the structure itself may do that job.)
    - microcolonel 6y ago
      I wonder if teddykoker has looked at applying FFTNet or similar methods as a replacement for Wavenet. I'm not sure but it seems to me like FFTNet is a lot more tractable than Wavenet, and not necessarily that much worse for equivalent training data.
- eru 6y ago
  The show won't collapse, if you have one glitch an hour.
  - aea12 6y ago
    Yes, it absolutely 100% will, depending on what you mean by handwaving “glitch”. VST is built into chains, and a flaky plugin will derail an entire performance, often making downstream plugins crash. I’m speaking from extensive experience writing plugins and performing with them in multiple hosts and trigger setups. It’s not a robust protocol, but it gets the job done. Are you speaking from some experience with which I’m unfamiliar where it’s okay for DSP code to fail hourly? Trying to understand your viewpoint.
    - nseggs 6y ago
      Agreed. If anyone wants to see some of the more successful DSP work being done today for pro or prosumer audio, I recommend checking out Strymon and Universal Audio products. Both make use of SHARC SoCs and achieve great results.
    - boomlinde 6y ago
      A VST that doesn't fill its buffer on time shouldn't crash another plugin. It's your other plugins that are flaky.
      - deleted 6y ago
        [deleted]
      - boomlinde 6y ago
        To expand on this, each plugin will receive host managed buffers that they're requested to fill and the input they're expected to process. If they don't do that in time for the host to mix the buffers and deliver the mixed buffer to the audio driver, it simply won't. Nowhere do the plugins directly interact through this process. If your plugins are crashing because of an underrun you have a much more serious problem than underruns. Then you have plugins writing to or reading from memory that wasn't either handed to them by the host or allocated by themselves. That bad code running in your process can cause it to crash is an orthogonal problem to buffer underruns causing skips or stuttering in audio.
- swebs 6y ago
  No, the other guy is right. Technically the definition of real-time can have a lot of leeway. Here's the paper linked in the article. Note how the authors never define what they really mean by real-time. They even make statements like "runs 1.9 times faster than real-time". They certainly imply your definition, but there's plenty of wiggle room to say "Well technically, I wasn't lying" https://www.mdpi.com/2076-3417/10/3/766/pdf https://www.mdpi.com/2076-3417/10/3/766/pdf
  - deleted 6y ago
    [deleted]
  - zodiac 6y ago
    In the context of e.g. offline video encoders "1.9x realtime" is a statement about throughput, not latency
- boomlinde 6y ago
  "Usually fast enough" is unfortunately the only guarantee a preemptive multitasking OS can give you. Unless your system is guaranteeing your program x cycles of uninterrupted processing per frame of audio and you can consistently process the frame in that amount of cycles, the only mitigation is to deliver frames in large enough chunks that you never run out of time in practice under agreeable circumstances. That said, I agree that the question of what "real-time" might mean is irrelevant given the context.
  - aea12 6y ago
    It is completely irrelevant, given the context. The only, only, only thing real-time means here is “can be run on a live signal passing through it” rather than “is a slow, offline effect for a DAW”. No hard real-time, no soft real-time, no QNX, no pulling out the college compsci textbook. There IS real-time in that sense in DSP, it just isn’t in a VST plugin. I’ll repeat again that any compsci theorycrafting is not the concern here, and real-time has a very specific meaning in DSP. Computer science does not own the concept of real-time, and the only people tripping over the terminology are those with more compsci experience than DSP. I appreciate everyone trying to explain this to me, but (a) I understand both, and (b) this is like saying “no, Captain, a vector could mean anything like a mathematical collection, air traffic control should learn a thing or two from mathematics.”
    - boomlinde 6y ago
      Just to be perfectly clear here because I'm not sure you're just using my post as a soapbox or if you have misunderstood my argument: I agree that it's clear what real-time means in this context. I disagree that "usually fast enough" guarantees failure for a VST, because in the case of VST, "usually fast enough" is the only guarantee the host operating system will offer your software. It's not "theorycrafting" to say that real-time music software running in a preemptive multitasking operating system without deterministic process time allocation will have to suffer the possibility of occasional drops. It happens in practice and audio drivers have to be implemented to account for the bulk of it, and the VST API is designed in such a way that failure to fill a buffer on time needn't be fatal.

alexlarsson 6y ago

I assume by real-time he meant "able to produce samples at a rate equal to or higher than the audio output sample rate".

amelius 6y ago
Latency is equally important.
TheRealPomax 6y ago
That's not what real time, means though. Real time processing means taking signals as they come in, and outputting the transformed result such that there is as close to no signal lag as possible. The output can in fact be wildly lower or higher resolution, real-time does not particularly say anything about that. It's all about whether the output plays (for practical purposes) at the perceived "same time" as the input signal. There will always be some delay, but that delay can't get perceivable, and for obvious reasons there can't be any (significant) buffering.
- jeffbee 6y ago
  Is that your private definition of "real-time"? I think it is common to define real-time processing by a specified, finite time between input and output. Many real-time processes are concerned more with the consistency of the latency than with its absolute value.
  - avisser 6y ago
    For guitar pedals, there is an implied sub-perceptibility. The output needs to happen as I play - if the delay is too long, it's now a delay pedal. So realtime might match your definition, but it is consistent in audio production. For humans, you can start to notice the lag @ 50ms. (A selection of experimental results summarized here https://gamedev.stackexchange.com/a/74975 https://gamedev.stackexchange.com/a/74975)
    - NobodyNada 6y ago

whiddershins 6y ago

Do solo or small shop vst plugin developers make any money? I’m curious if anyone has any direct knowledge about that. There are so many professional activities similar to that where no one makes any money and people really just do it for the love, and then there are seemingly similar things like that where people make surprisingly large amounts of money.

ff7f00 6y ago
There are definitely big players making a lot of money from plugins they develop. Here are a few to check out: ($1200) https://www.native-instruments.com/en/products/komplete/bundles/komplete-12-ultimate/ https://www.native-instruments.com/en/products/komplete/bund... ($300) https://www.soundtoys.com/product/soundtoys-5/ https://www.soundtoys.com/product/soundtoys-5/ ($500) https://www.arturia.com/products/analog-classics/v-collection/overview https://www.arturia.com/products/analog-classics/v-collectio... However, piracy is also pretty big when it comes to plugins.
- moralestapia 6y ago
  Sure, but, >Do solo or small shop vst plugin developers make any money?
  - kleer001 6y ago
    The implication seems to be "no".
    - TheRealPomax 6y ago
      That implication would be wildly incorrect.
      - kleer001 6y ago
        With the operative word being "developers" I disagree.

deleted 6y ago

[deleted]

samplenoise 6y ago

There's latency and there's the somewhat separate question of how much time is needed to make a prediction. Wavenet is causal (no look-ahead) and operates on the sample level so there are no buffers and thus no latency in the strict sense, beyond encoding/decoding into the sample rate and format required by the ML model, which should take <1ms. Whether a model manages to make a prediction in that amount of time depends on things like the receptive field and number of layers. The linked paper says their custom implementation runs at 1.1x real-time. I guess this isn't impossible; their receptive field is ~40ms, vs. 300 for the original (notoriously slow) wavenet, and the model is likely to have less layers and channels.

cjlars 6y ago
"Round trip," or guitar to processing to speakers needs to be sub 10ms to be transparent to the musician. Source: spent years playing guitar through my guitar -> DAC -> PC -> DAC -> speaker signal chain
- samplenoise 6y ago
  The receptive field size is how much 'history' the algorithm requires, it doesn't affect the round trip time, which can still be sub ms

TheRealPomax 6y ago

while training? terrible. As finalised model running in an AU/VST3 wrapper? probably extremely low.

qppo 6y ago

I know of a few shops that took VC money. The big problem isn't the market size so much as how slow the market moves. The product lifetime of a plugin is around a decade. And users hate subscriptions. And it's really hard to determine the value you add to your customers. And no one wants to pay you. It's basically a terrible place to be a developer in it for the money. Really fun work otherwise. The cool gigs are the ones where you build custom plugins for someone's crazy idea. In consumer applications, plugins are used all the time for prototyping before you go to hardware. MATLAB is way too slow for anything useful.

nil-sec 6y ago
The success of splice would disagree with your notion that “users hate subscriptions”. Given the horrendous price point of many of these plugins it seems to be perfect for a subscription based model. To me it always seemed there is more of a pushback from the industry producing vsts than from the consumers.
- qppo 6y ago
  Splice's numbers aren't public so I can't comment on their success. Avid's are, and they had a terrible quarter - and they're the poster child (alongside Adobe) for subscription licensing in creative software. But I'd be interested to see what the breakdown in revenue is for plugin licenses versus preset/sample packs (bit of a blade & razor model there). The price points really aren't horrendous if you consider how expensive the engineering is, how little demand there is, and how long you need to maintain a product. You aren't being ripped off by spending a couple hundred bucks on a plugin. I think we'll end up at a place where everything is a subscription, but I can tell you from experience that it creates friction for the users.
  - nil-sec 6y ago
    Agreed. The business model seems to be to give access to the rent to own deals via the sample subscription fee. Don’t think they make any money of their plugin deals. I’m also not arguing it’s too expensive or a rip off. But it’s still a large amount of money for software, in the private space at least. The rent to own thing seems like a smart tool to get rid of the barrier of entry.