5 ms·

TuneNN: A transformer-based network model for pitch detection

112 points by CMLab 3y ago

squidsoup 3y ago
It might be worth pointing out that the banjo model is for a four string banjo, given a five string banjo is the more common instrument.
CMLab 3y ago
A transformer-based network model, pitch tracking for musical instruments. The timbre of musical notes is the result of various combinations and transformations of harmonic relationships, harmonic strengths and weaknesses, instrument resonant peaks, and structural resonant peaks over time. It utilizes the transformer-based tuneNN network model for abstract timbre modeling, supporting tuning for 12+ instrument types.
- azinman2 3y ago
  This smells like an automated summary.
  - advisedwang 3y ago
    It is from the submitter. I think it is intended as a submission statement.
    - azinman2 3y ago
      That’s the first time I’ve seen/noticed that. I think it makes me feel better?
      - willy_k 3y ago
        Not sure what you were feeling bad about, but it’s also directly from the readme, so OP just saw something cool on GitHub and posted it along with a comment with the project’s own description, which is pretty standard on HN (I think, that’s how I’ve done it at least).
vessenes 3y ago
This is cool! The very best software-based tuning tech out there is probably in piano tuning apps; they cost hundreds of dollars+ and are specifically made to report on harmonics and other piano nuances. Do you have any comparisons against other pitch detection tech? Accuracy? Delay/Responsiveness? I assume it's much more compute work than a handcoded FFT type pitch detector. I think it's possible this would find utilization in the piano world if the output offers something new / something that can analyze what a piano tuning maestro can hear and make it accessible to a mid-tier tuner.
- jansommer 3y ago
  Sounds like you know a thing or two about pitch detection... I've been working on a C implementation of YIN and PYIN (a real GPL minefield for someone wanting to provide the end result as MIT/public domain!), and am wondering if it's a good choice for real time, cpu-bound speech pitch detection, or if there's better ways. May I ask what your thoughts are on this?
  - ronsor 3y ago
    Have you also considered implementing the Nebula[1] algorithm? [1] https://github.com/Sleepwalking/nebula https://github.com/Sleepwalking/nebula
    - jansommer 3y ago
      I need non-GPL libraries as a reference. The problem with YIN and especially PYIN is that the MIT-code I've found sometimes looks a bit too similar to earlier code in GPL. Rewriting that into the same but in different code is fairly hard. Here I'm assuming that translating eg. GPL Python or C++ into C would mean the license is retained
      - ska 3y ago
        Can you not just write it from the paper(s)? Or is that more effort than value to you? > that translating eg. GPL Python or C++ into C would mean the license is retained It depends a bit on what exactly "translating" means but you could easily be a derivative work. Honestly in that situation I wouldn't even look at the code. You might use in to test equivalent behavior after you have your own implementation, but only in a gross sense.
rrherr 3y ago
How does the accuracy of this compare to CREPE? https://github.com/marl/crepe https://github.com/marl/crepe https://github.com/maxrmorrison/torchcrepe https://github.com/maxrmorrison/torchcrepe Does anyone know what the current state of the art is, within the Music Information Retrieval community?
- CMLab 3y ago
  CREPE generally has high latency and error rates in instrument pitch recognition, especially for guitar instruments. Our team will release benchmark test data and results later.
  - rrherr 3y ago
    Thanks. I’d love to try TuneNN! Are you releasing a pretrained model? How do I run it on a wav file?
  - xavriley 3y ago
    High latency - agreed but it depends on whether a GPU is available or not. If it is then theoretically CREPE could be real-time. The error rates for pitch recognition are still quite good though for the full CREPE model. I’m interested to see the data on this claim.
ranting-moth 3y ago
To the dev: the tuner gives me an incredibly high error window with the following message. It doesn't prompt to access the mic (I think that's related). Ubuntu/KDE/Firefox: An error occurred running the Unity content on this page. See your browser JavaScript console for more info. The error was: TypeError: 'microphone' (value of 'name' member of PermissionDescriptor) is not a valid value for enumeration PermissionName. checkPermission@https://aifasttune.com/public/web/microphone/microphone.js:33:25 https://aifasttune.com/public/web/microphone/microphone.js:3... _Microphone_checkPermission@https://aifasttune.com/public/web/Build/web.framework.js:10:77159 https://aifasttune.com/public/web/Build/web.framework.js:10:... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[43854]:0xd61a26 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[45105]:0xdc6908 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[43988]:0xd69da2 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[53123]:0xfa26ef https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[55089]:0xfc865a https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[68505]:0x161cb61 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... invoke_iiii@https://aifasttune.com/public/web/Build/web.framework.js:10:408499 https://aifasttune.com/public/web/Build/web.framework.js:10:... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[55073]:0xfc7799 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[55119]:0xfc9785 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[67675]:0x15c04b3 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[65108]:0x15614fd https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[56559]:0x10e07f4 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[65094]:0x15609bd https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[68305]:0x15ee195 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... unityFramework/Module._SendMessageString@https://aifasttune.com/public/web/Build/web.framework.js:10:340414 https://aifasttune.com/public/web/Build/web.framework.js:10:... ccall@https://aifasttune.com/public/web/Build/web.framework.js:10:11768 https://aifasttune.com/public/web/Build/web.framework.js:10:... SendMessage@https://aifasttune.com/public/web/Build/web.framework.js:10:5552 https://aifasttune.com/public/web/Build/web.framework.js:10:... SendMessage@https://aifasttune.com/public/web/Build/web.loader.js:1:3343 https://aifasttune.com/public/web/Build/web.loader.js:1:3343 loadURL@https://aifasttune.com/public/web/game/fastGameController.js:35:26 https://aifasttune.com/public/web/game/fastGameController.js... i@https://aifasttune.com/assets/index-64322640.js:1:777 https://aifasttune.com/assets/index-64322640.js:1:777 setup/<@https://aifasttune.com/assets/index-64322640.js:1:611 https://aifasttune.com/assets/index-64322640.js:1:611
joonatan 3y ago
Could someone fill me in why would machine learning be necessary for pitch detection? Isn't it something that could just be solved with FFT or it's a much more complicated task?
- ks2048 3y ago
  Pitch is a *subjective* property, inherently tied to the complex processing humans use to perceive sounds. “Simple” physical measures like fundamental frequency of a periodic signal are very closely related, but for real-world audio (aren’t really periodic), the relationship is more complicated.
  - chrisshroba 3y ago
    Could you elaborate a bit more? It seems to me like the note being played would always correspond to the fundamental frequency observed. When is this not the case? Maybe as the note rings out, the fundamental frequency and first few overtones lose power, and all that's still audible are the higher overtones?
    - gexaha 3y ago
      That's actually not true, perceived pitch can be different from fundamental frequency, because of psychoacoustics. E. g. you can have "missing fundamental" - https://en.wikipedia.org/wiki/Missing_fundamental https://en.wikipedia.org/wiki/Missing_fundamental - or other effects like "sum and difference tones", which are quite popular in spectralism / spectral music
      - xavriley 3y ago
        Simple techniques like autocorrelation can still recover a missing fundamental. To answer the GP post, using neural networks for this task is overkill for simple, clean signals but it can be desirable if you need a) extremely high accuracy or b) robust results when there are signal degradations like background noise
      - 3y ago
bravura 3y ago
What's the license? What are your thoughts on PESTO which learns pitch-prediction very well with a small network, and uses a self-supervised objective? https://arxiv.org/abs/2309.02265 https://arxiv.org/abs/2309.02265 https://github.com/SonyCSLParis/pesto https://github.com/SonyCSLParis/pesto
filterfiber 3y ago
Does anyone know where I should look if I want to detect specific sounds? Like a smoke alarm, food bowl dispenser (its very distinct), cat meowing, 3d printer collision, that sort of thing?
- deleted 3y ago
  [deleted]
- mistercheph 3y ago
  You would learn how to do this in the first & second chapters of the fast.ai course.
- UncleEntity 3y ago
  This one seems interesting -> https://paperswithcode.com/paper/joint-audio-and-speech-understanding https://paperswithcode.com/paper/joint-audio-and-speech-unde...
- tristanc 3y ago
  Use any model trained on the AudioSet dataset. There is one called EfficientAT i think that I use regularly and is pretty reliable

112 points by CMLab 3y ago

It might be worth pointing out that the banjo model is for a four string banjo, given a five string banjo is the more common instrument.

CMLab 3y ago

A transformer-based network model, pitch tracking for musical instruments. The timbre of musical notes is the result of various combinations and transformations of harmonic relationships, harmonic strengths and weaknesses, instrument resonant peaks, and structural resonant peaks over time. It utilizes the transformer-based tuneNN network model for abstract timbre modeling, supporting tuning for 12+ instrument types.

azinman2 3y ago
This smells like an automated summary.
- advisedwang 3y ago
  It is from the submitter. I think it is intended as a submission statement.
  - azinman2 3y ago
    That’s the first time I’ve seen/noticed that. I think it makes me feel better?
    - willy_k 3y ago
      Not sure what you were feeling bad about, but it’s also directly from the readme, so OP just saw something cool on GitHub and posted it along with a comment with the project’s own description, which is pretty standard on HN (I think, that’s how I’ve done it at least).

vessenes 3y ago

This is cool! The very best software-based tuning tech out there is probably in piano tuning apps; they cost hundreds of dollars+ and are specifically made to report on harmonics and other piano nuances. Do you have any comparisons against other pitch detection tech? Accuracy? Delay/Responsiveness? I assume it's much more compute work than a handcoded FFT type pitch detector. I think it's possible this would find utilization in the piano world if the output offers something new / something that can analyze what a piano tuning maestro can hear and make it accessible to a mid-tier tuner.

jansommer 3y ago
Sounds like you know a thing or two about pitch detection... I've been working on a C implementation of YIN and PYIN (a real GPL minefield for someone wanting to provide the end result as MIT/public domain!), and am wondering if it's a good choice for real time, cpu-bound speech pitch detection, or if there's better ways. May I ask what your thoughts are on this?
- ronsor 3y ago
  Have you also considered implementing the Nebula[1] algorithm? [1] https://github.com/Sleepwalking/nebula https://github.com/Sleepwalking/nebula
  - jansommer 3y ago
    I need non-GPL libraries as a reference. The problem with YIN and especially PYIN is that the MIT-code I've found sometimes looks a bit too similar to earlier code in GPL. Rewriting that into the same but in different code is fairly hard. Here I'm assuming that translating eg. GPL Python or C++ into C would mean the license is retained
    - ska 3y ago
      Can you not just write it from the paper(s)? Or is that more effort than value to you? > that translating eg. GPL Python or C++ into C would mean the license is retained It depends a bit on what exactly "translating" means but you could easily be a derivative work. Honestly in that situation I wouldn't even look at the code. You might use in to test equivalent behavior after you have your own implementation, but only in a gross sense.

rrherr 3y ago

How does the accuracy of this compare to CREPE? https://github.com/marl/crepe https://github.com/marl/crepe https://github.com/maxrmorrison/torchcrepe https://github.com/maxrmorrison/torchcrepe Does anyone know what the current state of the art is, within the Music Information Retrieval community?

CMLab 3y ago
CREPE generally has high latency and error rates in instrument pitch recognition, especially for guitar instruments. Our team will release benchmark test data and results later.
- rrherr 3y ago
  Thanks. I’d love to try TuneNN! Are you releasing a pretrained model? How do I run it on a wav file?
- xavriley 3y ago
  High latency - agreed but it depends on whether a GPU is available or not. If it is then theoretically CREPE could be real-time. The error rates for pitch recognition are still quite good though for the full CREPE model. I’m interested to see the data on this claim.

ranting-moth 3y ago

To the dev: the tuner gives me an incredibly high error window with the following message. It doesn't prompt to access the mic (I think that's related). Ubuntu/KDE/Firefox: An error occurred running the Unity content on this page. See your browser JavaScript console for more info. The error was: TypeError: 'microphone' (value of 'name' member of PermissionDescriptor) is not a valid value for enumeration PermissionName. checkPermission@https://aifasttune.com/public/web/microphone/microphone.js:33:25 https://aifasttune.com/public/web/microphone/microphone.js:3... _Microphone_checkPermission@https://aifasttune.com/public/web/Build/web.framework.js:10:77159 https://aifasttune.com/public/web/Build/web.framework.js:10:... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[43854]:0xd61a26 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[45105]:0xdc6908 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[43988]:0xd69da2 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[53123]:0xfa26ef https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[55089]:0xfc865a https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[68505]:0x161cb61 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... invoke_iiii@https://aifasttune.com/public/web/Build/web.framework.js:10:408499 https://aifasttune.com/public/web/Build/web.framework.js:10:... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[55073]:0xfc7799 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[55119]:0xfc9785 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[67675]:0x15c04b3 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[65108]:0x15614fd https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[56559]:0x10e07f4 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[65094]:0x15609bd https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... @https://aifasttune.com/public/web/Build/web.wasm:wasm-function[68305]:0x15ee195 https://aifasttune.com/public/web/Build/web.wasm:wasm-functi... unityFramework/Module._SendMessageString@https://aifasttune.com/public/web/Build/web.framework.js:10:340414 https://aifasttune.com/public/web/Build/web.framework.js:10:... ccall@https://aifasttune.com/public/web/Build/web.framework.js:10:11768 https://aifasttune.com/public/web/Build/web.framework.js:10:... SendMessage@https://aifasttune.com/public/web/Build/web.framework.js:10:5552 https://aifasttune.com/public/web/Build/web.framework.js:10:... SendMessage@https://aifasttune.com/public/web/Build/web.loader.js:1:3343 https://aifasttune.com/public/web/Build/web.loader.js:1:3343 loadURL@https://aifasttune.com/public/web/game/fastGameController.js:35:26 https://aifasttune.com/public/web/game/fastGameController.js... i@https://aifasttune.com/assets/index-64322640.js:1:777 https://aifasttune.com/assets/index-64322640.js:1:777 setup/<@https://aifasttune.com/assets/index-64322640.js:1:611 https://aifasttune.com/assets/index-64322640.js:1:611

joonatan 3y ago

Could someone fill me in why would machine learning be necessary for pitch detection? Isn't it something that could just be solved with FFT or it's a much more complicated task?

ks2048 3y ago
Pitch is a *subjective* property, inherently tied to the complex processing humans use to perceive sounds. “Simple” physical measures like fundamental frequency of a periodic signal are very closely related, but for real-world audio (aren’t really periodic), the relationship is more complicated.
- chrisshroba 3y ago
  Could you elaborate a bit more? It seems to me like the note being played would always correspond to the fundamental frequency observed. When is this not the case? Maybe as the note rings out, the fundamental frequency and first few overtones lose power, and all that's still audible are the higher overtones?
  - gexaha 3y ago
    That's actually not true, perceived pitch can be different from fundamental frequency, because of psychoacoustics. E. g. you can have "missing fundamental" - https://en.wikipedia.org/wiki/Missing_fundamental https://en.wikipedia.org/wiki/Missing_fundamental - or other effects like "sum and difference tones", which are quite popular in spectralism / spectral music
    - xavriley 3y ago
      Simple techniques like autocorrelation can still recover a missing fundamental. To answer the GP post, using neural networks for this task is overkill for simple, clean signals but it can be desirable if you need a) extremely high accuracy or b) robust results when there are signal degradations like background noise
    - 3y ago

bravura 3y ago

What's the license? What are your thoughts on PESTO which learns pitch-prediction very well with a small network, and uses a self-supervised objective? https://arxiv.org/abs/2309.02265 https://arxiv.org/abs/2309.02265 https://github.com/SonyCSLParis/pesto https://github.com/SonyCSLParis/pesto

filterfiber 3y ago

Does anyone know where I should look if I want to detect specific sounds? Like a smoke alarm, food bowl dispenser (its very distinct), cat meowing, 3d printer collision, that sort of thing?

deleted 3y ago
[deleted]
mistercheph 3y ago
You would learn how to do this in the first & second chapters of the fast.ai course.
UncleEntity 3y ago
This one seems interesting -> https://paperswithcode.com/paper/joint-audio-and-speech-understanding https://paperswithcode.com/paper/joint-audio-and-speech-unde...
tristanc 3y ago
Use any model trained on the AudioSet dataset. There is one called EfficientAT i think that I use regularly and is pretty reliable