6 ms·

Declarative, non-intrusive, compile-time C++ reflection for audio plug-ins

69 points by jcelerier 5y ago

pininja 5y ago
What are some things this can be used for? From a short skim this seems like a way to make sharable, tweakable effects and patches for any OS. Stuff that historically soundflower could be use for?
- jcelerier 5y ago
  If you wrote a software that worked with soundflower it means that at some point you used to call either the CoreAudio API directly or any abstraction on top of it (RtAudio, PortAudio, ...). Thus harder to port to another OS :-) Here the idea is to write the algorithms in a way that is more future-proof, by not having them to depend on any run-time API, just a generic specification (given as a set of C++20 concepts). This way the algorithms will still be useful in 10 years when everyone has moved to API N+1, unlike a metric ton of existing audio software which depends on a specific audio / media-object API for no good reason (today ! When they were written C++ wasn't advanced enough to allow this at all) - all the objects in https://github.com/pcastine-lp/LitterPower https://github.com/pcastine-lp/LitterPower for instance - all the algorithms in https://github.com/MTG/essentia/tree/master/src/algorithms https://github.com/MTG/essentia/tree/master/src/algorithms - ditto for https://github.com/cycfi/q/tree/master/q_lib/include/q https://github.com/cycfi/q/tree/master/q_lib/include/q - ditto for all the VCV algorithms: https://github.com/VCVRack/Fundamental/tree/v1/src https://github.com/VCVRack/Fundamental/tree/v1/src - ditto for all the Bespoke objects: https://github.com/BespokeSynth/BespokeSynth/tree/main/Source https://github.com/BespokeSynth/BespokeSynth/tree/main/Sourc... - ditto for all the LMMS plugins: https://github.com/LMMS/lmms/tree/master/plugins https://github.com/LMMS/lmms/tree/master/plugins etc etc, there's a couple hundred of those, which always depend on some API and are thus not easily portable across environments: if tomorrow you want to make an audio software and want to use one of the, say, VCVRack plug-ins you're going to have to bring the whole VCV run-time API along. Related projects are Faust (https://faust.grame.fr/ https://faust.grame.fr/) and SOUL (https://juce.com/discover/stories/soul-first-public-beta-release https://juce.com/discover/stories/soul-first-public-beta-rel...) but they both are domain-specific languages with their own compilers. I wanted a pure-C++ thing instead, which allows to call directly native code, and enables more than just generic audio processing: unlike Faust and SOUL (last time I checked) it's possible to make a message-based object for Pd or Max, not just an audio filter or synthesizer. My end goal for this is that when I make an object for the main software I'm developing, ossia score (https://ossia.io https://ossia.io) then the whole media arts community can benefit :-)
  - einpoklum 5y ago
    > Here the idea is to write the algorithms in a way that is more future-proof More baroque, for sure. More future-proof? Only if this is the one true form of reflected code that gets adopted. > This way the algorithms will still be useful in 10 years when everyone has moved to API N+1 Actually, this way algorithms are not useful today since nobody writes them this way, and in 10 years, whoever writes this way right now will probably have moved on. Now, if some small community (like C++-reflection-and-audio-buffs) commits to this, then maybe it makes sense for them. But not generally. General reflection (not to mention runtime reflection) is probably the way to go.
    - jcelerier 5y ago
      > More baroque, for sure. More future-proof? Only if this is the one true form of reflected code that gets adopted. ah, actually not :) the library tries to do things in two steps in many places: 1/ map the user's code to a proxy depending on which concepts it conforms to: for instance whether your audio processing function is written per-sample (that's still in progress tho): float operator()(float input) { return input * 0.5; } or per-buffer: void operator()(float** inputs, float** outputs, int frames) { for(each channel, each sample) { ... } } 2/ the back-end accesses the class through these proxies to do whatever it needs to do. this means than if tomorrow we get reflection and metaclasses (I hint to this in the readme), and that we can write the "ideal" struct which (to me) would look like struct my_effect { [[name: "Main inputs"]] audio_input main_input; [[name: "Main outputs"]] audio_output main_input; [[range: {20, 20000}]] [[unit: hertz]] slider frequency; // will create an appropriate message in max / pd with the name some_function void some_function(int, std::ranges::view<std::variant<float, int>> list_of_stuff); auto operator()(std::floating_point auto in) { return in * gain; } }; the only thing that one will have to do is map the new "shape" in these proxies (which are as far as possible doing their stuff at compile-time) and all the existing backends will keep working with the algos defined in the better way. of course the existing algorithms will stay uglier, but they'll also keep working with new environments (and, as far as possible, without a runtime perf. hit) > General reflection (not to mention runtime reflection) is probably the way to go. yes, we all want that !
deleted 5y ago
[deleted]
ognarb 5y ago
This is very cool and a nice use of C++20 new features.
gavinray 5y ago
Going to ask some really stupid questions, apologies in advance: This is mindblowing work. I'm having some trouble wrapping my head around all of it. Will lead with the context that: A) I am not very familiar with audio programming specifics, or DSP at all. I have done some things with VST2 + VST3, and looked into LV2. B) I have very little background with C++ I am curious: 1. Is this targeted more towards the DSP and algorithm side of audio plugin development, or is there interest/use for the plugin side too (IE, generating VST/LV2 compatible interfaces)? 2. Would a byproduct of this be that you would be able to implement functionality + interact with C++ classes from other languages? Or generate C ABI's? I see the note: "Automatically generate Python bindings for a C++ class" I ask about the second one because I have been trying to work towards getting a C ABI for VST3 so that language bindings can be codegen'ed. I know VST3 works through COM so you can use that, but the alternative is to emulate the VTable layouts in C structs. This can make it easier than implementing COM in a language. ----- Edit: I also wanted to say that when reading your blogpost, the way you write C++ is very easy to understand. Not sure if it's a stylistic thing or a purposeful choice. I usually have to spend a second reading C++ code to comprehend it, but in your snippets, I was able to (mostly) grok it first pass. Same for the example code in the repo.
- jcelerier 5y ago
  Thanks ! I don't think those questions are stupid at all, if anything those are things I should put in the README ! > 1. Is this targeted more towards the DSP and algorithm side of audio plugin development, or is there interest/use for the plugin side too (IE, generating VST/LV2 compatible interfaces)? it does generate something compatible with VST2.x and I've been working on VST3 this week-end (and pulling a few hairs, that API is horrendous, the smallest example is a few thousand lines of codes). For LV2 I don't think it will be directly possible as LV2 requires separate .ttl files which describe the plug-in. Technically one could make a small program which from the C++ reflection, outputs the .ttl on stdout and kindly ask CMake to call it and generate the .ttl file from it. > 2. Would a byproduct of this be that you would be able to implement functionality + interact with C++ classes from other languages? yes, that's a goal, make your DSP processor in C++ and then write a small UI for it in python or QML without having to write binding code. I've tried to find ways to auto-generate a C ABI but sadly: - asm("...") only wants string literals, strings generated through constexpr don't cut it (otherwise it'd be possible and even relatively easy to generate e.g. asm(R"( .global foo_set_x .text foo_set_x: jmp _ZN3foo5set_xEi )"); to wrap for instance a foo::set_x(int) method. - I tried writing directly the binary code and appending it to the ELF sections but it seems that there's no easy way to do append to .symtab / .stringtab where the name of the symbols are without using a linker script.
jdc 5y ago
Looks promising. I wonder if this could be used to load arbitrary VST and LV2 plugins into EasyEffects.
Negitivefrags 5y ago
There are examples here of making objects that will be reflected, but no examples of what it's like to consume those objects. It would be nice if there was an example of that too.
- jcelerier 5y ago
  > but no examples of what it's like to consume those objects. I'm not sure I understand: the whole idea is that since the objects are just C++ structs, to "consume" them from some C++ program would just mean including the file and using it. Otherwise, if you want to know how you can do similar reflection on it (for instance to generate an UI from any of those processor) the simplest example is the python binding, the meat of it fits in ~100 lines of code: https://github.com/celtera/avendish/blob/main/src/python/processor.hpp https://github.com/celtera/avendish/blob/main/src/python/pro... if that is what you mean, then yeah what I plan to do is to write some documentation to explain the techniques and reflection API
loxias 5y ago
I just need to say: I've spent some (the happiest part!) of my career (and ~2/3rd of my life, hobby wise) involved with sound/music technology, designing and writing audio (and some video) plugins (VST, AU, PD, and others), audio middle-ware, plugin host frameworks, low level DSP algorithms optimized for performance, authoring music/sound protocols (MIDI-over-BLE, and tons of unfinished network sync things...), translation layers for seamless porting of effects between APIs (ie: auto-generation of an iOS app, or embedded microcontroller code, from VST source code), algorithms/processes for "JITish" C/ASM code generation from a concise higher level mathematical definitions (think: something you'd type in a CAS), so you can edit the math at runtime, and see the result within seconds. 8 years ago I made a PoC of something similar to this, using dark C++ magic (policy based design, lots of TMP) to enable compile-time optimization of filter chains, where each filter is encapsulated at a source level then connected in a dataflow graph, to produce an optimized binary of the whole chain, suitable for running as an application or on bare metal. Apparently, much of what I was going for eons ago (with c++11) is possible now with c++20! I've just begun to skim the code, with an arrogant "This is in my domain of expertise!" hat on. So far, I really, really like what I see, an uncommon experience when I look at Other People's Audio Code. All sorts of little clues that point towards "good code smell". ("Oh good, they used <such and such> library..", "Hah! Correct usage of memory fences!", &c) I tip my hat to you, author! I am embarrassed I hadn't heard of this project before! As soon as I can afford to focus on anything non-paying, I look forward to contributing, competing, or both, with this project and author. :)
- udbhavs 5y ago
  How would you recommend someone wanting to get into creating audio plugins (with very basic c++ and close to no experience in DSP) start? Also another thing I was wondering, does the math used in audio processing also come in useful for other broader applications like video?
  - fit2rule 5y ago
    Learn JUCE, watch some tutorials... http://juce.org http://juce.org
  - 5y ago

69 points by jcelerier 5y ago

What are some things this can be used for? From a short skim this seems like a way to make sharable, tweakable effects and patches for any OS. Stuff that historically soundflower could be use for?

jcelerier 5y ago
If you wrote a software that worked with soundflower it means that at some point you used to call either the CoreAudio API directly or any abstraction on top of it (RtAudio, PortAudio, ...). Thus harder to port to another OS :-) Here the idea is to write the algorithms in a way that is more future-proof, by not having them to depend on any run-time API, just a generic specification (given as a set of C++20 concepts). This way the algorithms will still be useful in 10 years when everyone has moved to API N+1, unlike a metric ton of existing audio software which depends on a specific audio / media-object API for no good reason (today ! When they were written C++ wasn't advanced enough to allow this at all) - all the objects in https://github.com/pcastine-lp/LitterPower https://github.com/pcastine-lp/LitterPower for instance - all the algorithms in https://github.com/MTG/essentia/tree/master/src/algorithms https://github.com/MTG/essentia/tree/master/src/algorithms - ditto for https://github.com/cycfi/q/tree/master/q_lib/include/q https://github.com/cycfi/q/tree/master/q_lib/include/q - ditto for all the VCV algorithms: https://github.com/VCVRack/Fundamental/tree/v1/src https://github.com/VCVRack/Fundamental/tree/v1/src - ditto for all the Bespoke objects: https://github.com/BespokeSynth/BespokeSynth/tree/main/Source https://github.com/BespokeSynth/BespokeSynth/tree/main/Sourc... - ditto for all the LMMS plugins: https://github.com/LMMS/lmms/tree/master/plugins https://github.com/LMMS/lmms/tree/master/plugins etc etc, there's a couple hundred of those, which always depend on some API and are thus not easily portable across environments: if tomorrow you want to make an audio software and want to use one of the, say, VCVRack plug-ins you're going to have to bring the whole VCV run-time API along. Related projects are Faust (https://faust.grame.fr/ https://faust.grame.fr/) and SOUL (https://juce.com/discover/stories/soul-first-public-beta-release https://juce.com/discover/stories/soul-first-public-beta-rel...) but they both are domain-specific languages with their own compilers. I wanted a pure-C++ thing instead, which allows to call directly native code, and enables more than just generic audio processing: unlike Faust and SOUL (last time I checked) it's possible to make a message-based object for Pd or Max, not just an audio filter or synthesizer. My end goal for this is that when I make an object for the main software I'm developing, ossia score (https://ossia.io https://ossia.io) then the whole media arts community can benefit :-)
- einpoklum 5y ago
  > Here the idea is to write the algorithms in a way that is more future-proof More baroque, for sure. More future-proof? Only if this is the one true form of reflected code that gets adopted. > This way the algorithms will still be useful in 10 years when everyone has moved to API N+1 Actually, this way algorithms are not useful today since nobody writes them this way, and in 10 years, whoever writes this way right now will probably have moved on. Now, if some small community (like C++-reflection-and-audio-buffs) commits to this, then maybe it makes sense for them. But not generally. General reflection (not to mention runtime reflection) is probably the way to go.
  - jcelerier 5y ago
    > More baroque, for sure. More future-proof? Only if this is the one true form of reflected code that gets adopted. ah, actually not :) the library tries to do things in two steps in many places: 1/ map the user's code to a proxy depending on which concepts it conforms to: for instance whether your audio processing function is written per-sample (that's still in progress tho): float operator()(float input) { return input * 0.5; } or per-buffer: void operator()(float** inputs, float** outputs, int frames) { for(each channel, each sample) { ... } } 2/ the back-end accesses the class through these proxies to do whatever it needs to do. this means than if tomorrow we get reflection and metaclasses (I hint to this in the readme), and that we can write the "ideal" struct which (to me) would look like struct my_effect { [[name: "Main inputs"]] audio_input main_input; [[name: "Main outputs"]] audio_output main_input; [[range: {20, 20000}]] [[unit: hertz]] slider frequency; // will create an appropriate message in max / pd with the name some_function void some_function(int, std::ranges::view<std::variant<float, int>> list_of_stuff); auto operator()(std::floating_point auto in) { return in * gain; } }; the only thing that one will have to do is map the new "shape" in these proxies (which are as far as possible doing their stuff at compile-time) and all the existing backends will keep working with the algos defined in the better way. of course the existing algorithms will stay uglier, but they'll also keep working with new environments (and, as far as possible, without a runtime perf. hit) > General reflection (not to mention runtime reflection) is probably the way to go. yes, we all want that !

deleted 5y ago

[deleted]

ognarb 5y ago

This is very cool and a nice use of C++20 new features.

gavinray 5y ago

Going to ask some really stupid questions, apologies in advance: This is mindblowing work. I'm having some trouble wrapping my head around all of it. Will lead with the context that: A) I am not very familiar with audio programming specifics, or DSP at all. I have done some things with VST2 + VST3, and looked into LV2. B) I have very little background with C++ I am curious: 1. Is this targeted more towards the DSP and algorithm side of audio plugin development, or is there interest/use for the plugin side too (IE, generating VST/LV2 compatible interfaces)? 2. Would a byproduct of this be that you would be able to implement functionality + interact with C++ classes from other languages? Or generate C ABI's? I see the note: "Automatically generate Python bindings for a C++ class" I ask about the second one because I have been trying to work towards getting a C ABI for VST3 so that language bindings can be codegen'ed. I know VST3 works through COM so you can use that, but the alternative is to emulate the VTable layouts in C structs. This can make it easier than implementing COM in a language. ----- Edit: I also wanted to say that when reading your blogpost, the way you write C++ is very easy to understand. Not sure if it's a stylistic thing or a purposeful choice. I usually have to spend a second reading C++ code to comprehend it, but in your snippets, I was able to (mostly) grok it first pass. Same for the example code in the repo.

jcelerier 5y ago
Thanks ! I don't think those questions are stupid at all, if anything those are things I should put in the README ! > 1. Is this targeted more towards the DSP and algorithm side of audio plugin development, or is there interest/use for the plugin side too (IE, generating VST/LV2 compatible interfaces)? it does generate something compatible with VST2.x and I've been working on VST3 this week-end (and pulling a few hairs, that API is horrendous, the smallest example is a few thousand lines of codes). For LV2 I don't think it will be directly possible as LV2 requires separate .ttl files which describe the plug-in. Technically one could make a small program which from the C++ reflection, outputs the .ttl on stdout and kindly ask CMake to call it and generate the .ttl file from it. > 2. Would a byproduct of this be that you would be able to implement functionality + interact with C++ classes from other languages? yes, that's a goal, make your DSP processor in C++ and then write a small UI for it in python or QML without having to write binding code. I've tried to find ways to auto-generate a C ABI but sadly: - asm("...") only wants string literals, strings generated through constexpr don't cut it (otherwise it'd be possible and even relatively easy to generate e.g. asm(R"( .global foo_set_x .text foo_set_x: jmp _ZN3foo5set_xEi )"); to wrap for instance a foo::set_x(int) method. - I tried writing directly the binary code and appending it to the ELF sections but it seems that there's no easy way to do append to .symtab / .stringtab where the name of the symbols are without using a linker script.

jdc 5y ago

Looks promising. I wonder if this could be used to load arbitrary VST and LV2 plugins into EasyEffects.

Negitivefrags 5y ago

There are examples here of making objects that will be reflected, but no examples of what it's like to consume those objects. It would be nice if there was an example of that too.

jcelerier 5y ago
> but no examples of what it's like to consume those objects. I'm not sure I understand: the whole idea is that since the objects are just C++ structs, to "consume" them from some C++ program would just mean including the file and using it. Otherwise, if you want to know how you can do similar reflection on it (for instance to generate an UI from any of those processor) the simplest example is the python binding, the meat of it fits in ~100 lines of code: https://github.com/celtera/avendish/blob/main/src/python/processor.hpp https://github.com/celtera/avendish/blob/main/src/python/pro... if that is what you mean, then yeah what I plan to do is to write some documentation to explain the techniques and reflection API

loxias 5y ago

I just need to say: I've spent some (the happiest part!) of my career (and ~2/3rd of my life, hobby wise) involved with sound/music technology, designing and writing audio (and some video) plugins (VST, AU, PD, and others), audio middle-ware, plugin host frameworks, low level DSP algorithms optimized for performance, authoring music/sound protocols (MIDI-over-BLE, and tons of unfinished network sync things...), translation layers for seamless porting of effects between APIs (ie: auto-generation of an iOS app, or embedded microcontroller code, from VST source code), algorithms/processes for "JITish" C/ASM code generation from a concise higher level mathematical definitions (think: something you'd type in a CAS), so you can edit the math at runtime, and see the result within seconds. 8 years ago I made a PoC of something similar to this, using dark C++ magic (policy based design, lots of TMP) to enable compile-time optimization of filter chains, where each filter is encapsulated at a source level then connected in a dataflow graph, to produce an optimized binary of the whole chain, suitable for running as an application or on bare metal. Apparently, much of what I was going for eons ago (with c++11) is possible now with c++20! I've just begun to skim the code, with an arrogant "This is in my domain of expertise!" hat on. So far, I really, really like what I see, an uncommon experience when I look at Other People's Audio Code. All sorts of little clues that point towards "good code smell". ("Oh good, they used <such and such> library..", "Hah! Correct usage of memory fences!", &c) I tip my hat to you, author! I am embarrassed I hadn't heard of this project before! As soon as I can afford to focus on anything non-paying, I look forward to contributing, competing, or both, with this project and author. :)

udbhavs 5y ago
How would you recommend someone wanting to get into creating audio plugins (with very basic c++ and close to no experience in DSP) start? Also another thing I was wondering, does the math used in audio processing also come in useful for other broader applications like video?
- fit2rule 5y ago
  Learn JUCE, watch some tutorials... http://juce.org http://juce.org
- 5y ago