3 ms·
Can you provide some more details on your use case? In general a powerful approach is to use a neural network to extract vector embeddings and then do simple mo
by jononor 3y ago
Can you provide some more details on your use case?
In general a powerful approach is to use a neural network to extract vector embeddings and then do simple models on top. The neural network could for example be a pretrained audio classifier (like YAMNet mentioned in article), or a model trained with self-supervised learning on representative (unlabeled) data.
A small model for unsupervised event detection / segmentation could be agglomerative clustering.
A data-efficient supervised approach could be K Nearest Neighbors.
- mnky9800n 3y agoAs an example, I have 9 months of hydrophone cable data that has a large number of sounds that occur across a large variety of time lengths and frequencies. It would be nice to have a method that highlights certain signals given specific meta data. Tbh, what I think I want is a pipe dream.
- jononor 3y agoIf you are willing to label 100 instances of each event type, then the approaches suggested above should work. And half of the data should be used to quantify how well it works.
- akomtu 3y agoWhat kind of metadata?
- gopher_space 3y agoI think some of the folks working with cetaceans might be able to point you towards interesting areas. I'm aware of work on processing petabytes of highest-res hydrophone data and I'm pretty sure they're tagging things that look like new clicktrains in some manner. Does this sound appropriate? As an aside, I've been kicking around the thought of what a software resonator might look like and where the analogy would work. I feel like there's a really efficient filter you could pull out of the concept.
- gnulinux 3y agoOne problem I wasn't able to solve very satisfyingly is music -> midi. Even with solo instruments, the output data is a hot jumbled mess with all kinds of errors from pitch to rhythm to dynamics. Once anything more than one timbre is involved, or e.g. an orchestra, it seems completely impossible to solve it with current tools since the model needs to be smart enough to tell apart fundamental frequencies with multiple timbres.
- jononor 3y agoPolyphonic / multi-track music transcription is a very challenging tasks and still an open area of research. Plenty of progress made the last 5 years, but still not at a very practical level. Source separation and single/lead instrument transcription has gotten pretty good though. So if the music fits with the 5-stem split of say Spleeter it can do an OK job