27 ms·
Voc: a physical model of the vocal tract, written in ANSI C
- mrstone 9y agoThis is super cool. I seem to remember a JS version of this floating around earlier this year. Does anyone have it?
- ajacksified 9y agoIt's the first link on the page
- traverseda 9y agoTitle says "Vox". Page says "Voc".
- adamnemecek 9y agofixed
- macawfish 9y agoWould this be a good target for deep learning? It's low level in some sense, but still nicely parametric. It strikes me that this could be a good synth for some neural nets to learn how to play.
- vortico 9y agoYes, speech recognition and speech generation would be easier to implement if you used neural networks that were trained on these vocal cord inputs rather than audio samples. In either case, you'd need to solve the inverse problem to generate vocal cord parameters given an audio sample. This seems difficult but I'd imagine some commercial software packages do it to some extent.
- smaddox 9y agoOr you could let the neural network solve the inverse problem for you, at least for speech generation.
- skykooler 9y agoI would imagine a neural network that fed parameter values to Voc would be far faster (real-time?) than something like WaveNet which needs to sample the output thousands of times a second.
- zebproj 9y agoCorrect. WaveNet is a very brute force approach to speech synthesis.
- zebproj 9y agoIt seems like it would be, right? There are 44 tract-diameters you can modify to shape the vocal tract, and these can be used to generated specific vowel formants. I can imagine you can build a system using deep learning that can find the best parameters to match a steady state periodic pitch. It's a bit how some speech codecs work, like LPC10.
- woodson 9y agoProblem is, the output would sound pretty metallic, just like LPC10.
- zebproj 9y agoMaybe, maybe not. LPC10 is a 8kHz speech codec optimized for low-bandwith signals. The Kelley-Lochbaum is a full-blown physical model of the tract. What you put into the filter is important. The LF glottal pulse model used here is a pretty good excitation signal... aspiration noise REALLY makes a difference. It would still sound artificial, but it definitely wouldn't sound metallic.
- jarmitage 9y agoGreat! Also inspired by Pink Trombone, we ported the same model to our maker platform and then to a modular synth: https://www.youtube.com/watch?v=bo5ZEgBEapk https://www.youtube.com/watch?v=bo5ZEgBEapk https://twitter.com/BelaPlatform/status/856110345332674561 https://twitter.com/BelaPlatform/status/856110345332674561 https://github.com/giuliomoro/pink-trombone https://github.com/giuliomoro/pink-trombone
- vortico 9y agoNice, are you planning on releasing a Eurorack form-factor kit for your platform? I'd imagine this can be a good alternative to other digital modules like nw2s::b and Rebel's OWL.
- jarmitage 9y agoHello, yes we are going to do a limited edition run, it is actually a collaboration with Rebel Tech! From our latest Kickstarter update: "The Bela Modular breaks out the audio, analog and digital I/Os to jacks, and handles voltage scaling for Eurorack-compatible CV levels, providing a total of 2 audio in, 2 audio out, 8 analog in, 8 analog out, 4 digital in, 4 digital out and 4 LEDs over two modules, 12HP and 10HP wide. [...] We are planning to do a small production run of Bela Modular units later this year. Please contact us[0] directly if you think you would like one of these, or stay tuned here and on the forum.[1]" [0] info at bela dot io [1] http://forum.bela.io http://forum.bela.io
- zebproj 9y agoHey yeah! I actually came across that when I first set out to make my project. I was going to actually fork off your project, but decided it would be cleaner/faster to do it off the original code since I wanted to write it in ANSI C.
- jarmitage 9y agoI think you probably made a good choice, since our fork of the original was worked on very quickly to get a basic demo going :)
- userbinator 9y agoInteresting use of Knuth's literate programming concept. I remember reading about it and the source code of TeX in PDF format a long time ago, but now that I have a chance to read another piece of code written in it, I'm finding it less readable because of the proportional fonts than if it were equally-commented but in a more conventional monospace programming font. Also, some samples of making it talk would be good, like: https://en.wikipedia.org/wiki/Voder https://en.wikipedia.org/wiki/Voder
- zebproj 9y ago> I'm finding it less readable because of the proportional fonts than if it were equally-commented but in a more conventional monospace programming font. My initial motivations for using literate programming with Voc was to take advantage of TeX's math mode to express what was happening numerically in code, as well as to have the ability to use BibTex inside the code. > Also, some samples of making it talk would be good, like: https://en.wikipedia.org/wiki/Voder https://en.wikipedia.org/wiki/Voder At the bottom of the page, there are music examples on Vimeo, with plots of the 44 vocal tract diameters being manipulated in realtime: https://vimeo.com/220091107 https://vimeo.com/220091107 https://vimeo.com/220091290 https://vimeo.com/220091290 https://vimeo.com/220091487 https://vimeo.com/220091487 My goal was really to build vocalizations, and not necessarily to produce speech. This engine is a bit more low level than that. It could be possible to build a speech engine on top of Voc though... next steps perhaps?
- Jtsummers 9y agoThe font issue should be fixable with some configuration. I personally do literate programming in org mode and generate HTML files from it for my colleagues and myself. With it, code blocks output to HTML use a fixed width font. I haven't checked the tex output (low priority, the HTML output is good enough for me).
- zebproj 9y agoHey! I'm the author of this thing. Let me know if you have any questions about it. FYI, the generated C code is now part of the dev branch of Soundpipe here: http://pbat.ch/proj/soundpipe.html http://pbat.ch/proj/soundpipe.html, my music DSP library, and it has also made it's way into the develop branch of AudioKit: http://audiokit.io http://audiokit.io. Also, check out the original implementation Pink Trombone: https://dood.al/pinktrombone/ https://dood.al/pinktrombone/. It's the perfect interface for this kind of model.
- TheNewAndy 9y agoFunnily enough, I also made a C90 port of the same code, and also named it "voc". Here is an oldish version: http://ultra-premium.com/scratch/voc.zip http://ultra-premium.com/scratch/voc.zip I would completely expect yours to be better in all ways (mine was basically a line for line transcription of all the audio and stripping out the UI stuff, where as you actually went to the effort of understanding the thing being ported).
- zebproj 9y agoI found your code last night, and it really freaked me out that there would be another person in the world who would want to make a C port of pink trombone and call it Voc as well. Voc is pretty much a line-for-line port of PT as well, but I removed some bits like the simplex noise. I also wrote some small utilities to go with Voc, like small plotting programs and some plugins for my audio language for both the whole source-filter model and just the filter. Still sinking my teeth into the literature. Voice synthesis has a very rich history!
- FigBug 9y agoWhat do you mean oldish version? Looks like it's from 2017 , or from an old version of pink trombone. Do you have the latest code in source control anywhere online?
- microcolonel 9y agoVery cool (soundpipe). Since I was a wee lad, I've been thinking about colonizing all major open source DAW/sequencer/audio editor programs with a single processing core, to the extent which would be acceptable. I think a lot more could be done if these individual projects (like Audacity, Hydrogen, Lmms, etc.) were just basically "shells" on the audio processing pipeline.
- anirudt 9y agoThis is quite cool! On a sidenote, can this be used to train or obtain voice parameters of oneself for using it in software programs like Espeak?
- zebproj 9y agoNot directly, no. IIRC, programs like Espeak and Festival use formant synthesis, which would require explicit formant values. Voc models the tract itself... the main parameters are diameters in the vocal tract (which implicitly produce vowel sounds). It may be possible to go the other way around and analytically derive parameters for Voc that match target formant frequencies. Not sure though...
- marmaduke 9y agoThere are other ways though e.g. if features like formants produced by the model can be differentiated with respect to vocal tract parameters, the latter could be estimated based on real data maybe
- adamnemecek 9y agoHere's an example of it in action https://vimeo.com/221310975 https://vimeo.com/221310975
- ruste 9y agoThat was... very strange. I'm not sure how else you'd demonstrate a physical vocal tract model though.
- MrBuddyCasino 9y agoIs there something that does the opposite? Feed vocals, get jaw movements? I've got a weird little project, think "talking skeleton".
- coldtea 9y agoThere are numerous programs that translate spoken words to mouth/jaw movements -- usually for animation (to match the 3D /2D model with the right movements for what it's supposed to say). IIRC, Adobe Animator does that too.
- MrBuddyCasino 9y agoThank. Unfortunately, I need something suitable for embedded hardware.
- coldtea 9y agoCheck this maybe: https://github.com/DanielSWolf/rhubarb-lip-sync https://github.com/DanielSWolf/rhubarb-lip-sync
- Keyframe 9y agoWouldn't you also need to model tongue and mouth chamber (with teeth) and all of their movements?
- zebproj 9y agoYes and no. Perceptually, you don't really need to model everything to get convincing speech sounds. Most of the realism actually comes from performance, and not the mathematical model. In a way, lips and mouth are accounted for here, but in a more abstract away. The KL model approximates the vocal tract as a series of cylindrical tubes with varying diameters. Segments of the tubes actually correspond to things like the tongue and mouth somewhat. In this model there is a really neat tongue control that manipulates these segments. It's quite expressive! This model is a 1d waveguide, so it doesn't account for things like the curvature of the tract. More modern vocal modelling techniques include implementing a 2-dimensional waveguide, which does allow for this control.