6 ms·
AudioFlux: A C/C++ library for audio and music analysis
- deleted 2y ago[deleted]
- gosub100 2y agoCan this be used for audio fingerprinting?
- bravura 2y agoIf this is supposed to be used for deep-learning, shouldn't all the transforms be GPU-accelerated torch functions?
- herogary 2y agoMaybe for the convenience of mobile usage?
- tgv 2y agoBy the looks of it, those functions extract features (like frequency peaks). You do that once for a sound. The output could function as input for an NN, in which case it would be a tokenizer for sound.
- bravura 2y agoGiven what I've seen in audio ML research: 1) Tuning hyperparameters of your audio preprocessing is a pain if it's a preprocessed CPU step. You have to redo preprocessing every time you want to tune your audio feature hyperparams 2) It's quite common to use torchaudio spectrograms, etc. purely because they are faster (I can link to a handful of recent high-impact audio ML github repos if you like) 3) If you use nnAudio, you can actually backprop the STFT or mel filters and tune them if you like. With that said, this is not so commonplace. 4) Sometimes the audio is GENERATED by a GPU. For example, in a neural vocoder, you decode the audio from a mel to a waveform. Then, you compute the loss over the true versus predict audio mel spectrograms. You can't do this with these C++ features. (Again, I can link a handful of recent high-impact audio ML github repos if you like.) Again, I just don't get it.
- codetrotter 2y ago> I can link to a handful of recent high-impact audio ML github repos if you like Yes please :D
- bravura 2y agoFor instance: https://github.com/descriptinc/descript-audio-codec/blob/main/dac/nn/loss.py#L142 https://github.com/descriptinc/descript-audio-codec/blob/mai... https://github.com/NVIDIA/BigVGAN/blob/main/loss.py#L23 https://github.com/NVIDIA/BigVGAN/blob/main/loss.py#L23 https://arxiv.org/pdf/2210.13438 https://arxiv.org/pdf/2210.13438 (the github repo doesn't include training, just inference) It is INCREDIBLY common to use multi-scale spectral loss as the audio distance / objective measure in audio generation. They have some issues (i.e. they aren't always well correlated with human perception) but they are the known-current-best.
- tgv 2y agoBackpropping filter coefficients sounds clever, but can't you just do that on any layer that takes a spectrum as input?
- bravura 2y agoBackpropping filter coefficients is clever, but it hasn't really caught on much. Google also tried with LEAF (https://github.com/google-research/leaf-audio https://github.com/google-research/leaf-audio) to have a learnable audio filterbank. Anyway, in audio ML what is very common is: a) Futzing with the way you do feature extraction on the input. (Oh, maybe I want CQT for this task or a different scale Mel etc) b) Doing feature extraction on generated audio output, and constructing loss functions from generated audio features. So, as I said, I don't exactly see the utility of this library for deep learning. With that said, it is definitely nice to have really high speed low latency audio algorithms in C++. I just wouldn't market it as "useful for deep learning" because a) during training, you need more flexibility than non-GPU methods without backprop b) if you are doing "deep learning" then your inferred model will presumably be quite large, and there will be a million other things you'll need to optimize to get real-time inference or inference on CPUs to work well. Is just my gut reaction. It seems like a solid project, I just question the one selling point of "useful for deep learning" that's all.
- aa-jv 2y agoA GPU is useful, but DSP's are also still useful - for example there is a compelling case to have frameworks around such as AudioFlux, JUCE and others, in order to support portability and also realtime analysis competitively, which is important in this domain, where such things as Qualcomms' ADK, and others, is quite literally being put inside peoples ears... Not to say that big-AI shouldn't have audio analysis as a compelling sphere of application, but more that, until the chips arrive, in-ear AI is less of a specification/requirement, than in-ear DSP. We don't need AI to isolate discrete audio components and do things with them, in-Ear. Offline/big-AI, however, is still compelling. But we don't yet have GPU neckbands ..
- nesarkvechnep 2y agoWhat's this C/C++ language?
- n4r9 2y agoSome preliminary analysis suggests that if C is an integer greater than 1, C/C++ will always evaluate to 1 [0]. [0] https://www.programiz.com/online-compiler/9fkHTct0Mybpu https://www.programiz.com/online-compiler/9fkHTct0Mybpu
- gpderetta 2y agoActually it could be 2 or 1. IIRC the order of evaluation of operator parameters is unspecified (this used to be UB, now is merely unspecified). /extremelypedantic
- troymc 2y ago1/++
- rossant 2y agoor 1++ if you got your C operator precedence wrong
- iExploder 2y agomean you are using C++ but only the features that dont suck, so probably 10% of the language, rest is plain C
- deleted 2y ago[deleted]
- pjmlp 2y agoIt is the use of English grammar rules to mean C and C++, naturally not everyone was that great on English classes, and specially those that never attended WG21 and WG14 meetings, or work for said companies, and enjoy being pedantic online about it. To make it easier for those skipping English classes "A forward dash can be used to state alternatives. A sentence that uses a forward slash in this way can be read to mean that any or all of the stated words could apply." https://www.thesaurus.com/e/grammar/slash/ https://www.thesaurus.com/e/grammar/slash/ And from the world of reference C and C++. "The C/C++ Users Journal" https://en.wikipedia.org/wiki/C/C%2B%2B_Users_Journal https://en.wikipedia.org/wiki/C/C%2B%2B_Users_Journal "Visual Studio C/C++ IDE and Compiler for Windows" https://visualstudio.microsoft.com/vs/features/cplusplus/ https://visualstudio.microsoft.com/vs/features/cplusplus/ Random job post from Microsoft, "Perform software development in C/C++, Python, and other languages." https://jobs.careers.microsoft.com/global/en/job/1752991/Principal-Software-Engineer https://jobs.careers.microsoft.com/global/en/job/1752991/Pri... Random job post from Apple, "Develop/maintain bit-accurate function C/C++ model for hardware verification Develop/maintain cycle-approximate perf C/C++ model for performance analysis - Analyze model Excellent C/C++ programming skills" https://jobs.apple.com/en-us/details/200448639/graphics-modeling-engineer?team=HRDWR https://jobs.apple.com/en-us/details/200448639/graphics-mode... Random job post from Google, "4 years of experience coding with one or more programming languages (e.g., Java, C/C++, Python)" https://www.google.com/about/careers/applications/jobs/results/110995728691536582-product-solutions-engineer-payments-platform?q=%22C%2FC%2B%2B%22#!t=jo&jid=127025001& https://www.google.com/about/careers/applications/jobs/resul... Random job post from NVidia, "Strong C/C++ programming skills" https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/details/Display-Driver-Software-Engineer--RDSS-Intern-_JR1974479?q=%22C/C++%22 https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCar... More examples from WG14, WG21 members, C and C++ compiler vendors can be provided.
- dsego 2y agoIt's also for Python, I just discovered it a few days ago. This is the website https://audioflux.top/ https://audioflux.top/
- jcelerier 2y agoIt would be nice to have a comparison with any of the many C++ MIR (music information retrieval) libraries in the wild: - https://essentia.upf.edu/ https://essentia.upf.edu/ - https://github.com/marsyas/marsyas https://github.com/marsyas/marsyas - https://github.com/ircam-ismm/pipo https://github.com/ircam-ismm/pipo - https://github.com/flucoma/flucoma-core/tree/main/include/algorithms/public https://github.com/flucoma/flucoma-core/tree/main/include/al...
- BrannonKing 2y agoIf a person wanted to transcribe sheet music from recorded audio, do you know which library and features would be the best starting point?
- bckr 2y agoStart with source separation using demucs
- bravura 2y agoI have had mixed luck with this model, which is supposedly state-of-the-art: https://github.com/magenta/mt3 https://github.com/magenta/mt3 What kind of music are you trying to transcribe? Feel free to email me.
- cpdomina 2y agoFor MIDI: https://github.com/Music-and-Culture-Technology-Lab/omnizart https://github.com/Music-and-Culture-Technology-Lab/omnizart and https://basicpitch.spotify.com/ https://basicpitch.spotify.com/ They work better if you apply some source separation before (e.g, https://github.com/sigsep/open-unmix-pytorch https://github.com/sigsep/open-unmix-pytorch, https://github.com/facebookresearch/demucs https://github.com/facebookresearch/demucs, or https://mvsep.com https://mvsep.com) Still, I think the best results are from proprietary models (specifically https://www.ableton.com/en/manual/converting-audio-to-midi/ https://www.ableton.com/en/manual/converting-audio-to-midi/ and https://www.celemony.com/en/melodyne/what-is-melodyne https://www.celemony.com/en/melodyne/what-is-melodyne)
- 2y ago
- dekken_ 2y agoIt's C and Python, not C++
- dsego 2y agoC can be used in C++ code, no?
- Galanwe 2y agoSomething very close, but that's not what you would expect for something that markets itself as a C++ library IMHO. Especially in 2024, most people would hope (or assume) that "C++" means "C++ 11" at least. Definitely doesn't count as _lying_, but still underwhelming.
- dekken_ 2y agoDepends, not all C is C++, eg, there is no (yet) `restrict` keyword in C++ (even if lots of C++ compilers support __restrict__, it's not in the spec)
- bregma 2y agoYes. And C can also be used with Python and Rust. That does not make this a Rust library.
- dsego 2y agoRight, but C++ started as an extension of C and is mostly compatible and historically you could compile C with the C++ compiler. I don't think it's a good comparison.
- codetrotter 2y agoZig can compile C. That makes this C/C++/Zig library. Right? :^)
- jcelerier 2y ago> historically you could compile C with the C++ compiler. not any C, only the C++-compatible subset. int* foo = malloc(sizeof(int)); has never worked in C++ for instance while it's valid C. Code that worked is code that people actually did effort to express in a way compatible with a C++ compiler.
- leonardohn 2y ago[flagged]
- morning-coffee 2y agoDo you have one in safe Rust? See, we've only just met, and I don't know how you handle your ptr/len arguments in C just yet. ;)
- BrannonKing 2y agoSo are they going for feature parity with librosa? I think that would be great.
- zombot 2y agoHow can a Python library support iOS?
- adamnemecek 2y agoIt’s a cpp library with python bindings.