To convey freeform "modularity" sends you in one of two directions: visual diagram notation of this sort, and source code "audio programming languages", eg Supe

To convey freeform "modularity" sends you in one of two directions: visual diagram notation of this sort, and source code "audio programming languages", eg Supercollider, which let you define things more symbolically/declaratively. To simplify this interface you have to relax the assumption that you will create a graph with arbitrary feedback loops: then your signal path can be acyclic or linear. Most things you will want to do in sound design use feedback in a very limited sense(e.g. a delay line effect) and can be turned into black boxes for a linear chain. Linear by itself is slightly too little power since you do want to mix and split signals at various points, and most commercial synthesizers use a "semi-modular" design that assumes a linear default and then adds particular methods of choosing modulation input at various points. Regardless, modular's power to do anything is tempting to dive into. It's just a case of "not worth the effort" most of the time since having any input anywhere creates multitudes of obscure ways to get no output.