12 ms·
The Mathematical Genius of Auto-Tune
- bllguo 9y agoI thought from the title that it goes into the mathematics. Unfortunately all the article says w.r.t math is a hand-wavy explanation of autocorrelation. But it was an interesting story about the life of Auto-Tune's creator.
- biggieshellz 9y agoSee https://www.google.com/patents/US5973252 https://www.google.com/patents/US5973252 -- it looks like the trick is somehow simplifying the autocorrelation operation by fitting a quadratic curve.
- jsjohnst 9y agoYeah, the “mathematical genius” is describing the author of Auto-Tune, not Auto-Tune itself. Through me off a bit too.
- pizza 9y agoI've worked with Hildebrand myself - he's one smart cookie, for sure. edit/tidbit: he told me that the interview process for hiring new developers goes like this: all the questions are straight out of K&R's C Programming Language, you have to get all of them correct, and so far only one programmer has :P
- jsjohnst 9y agoI have an eidetic memory and have read that book multiple times, so there’s a fair chance I’d pass. That said, if I was given that interview as you described, I’d get up and walk out. Interviews like that prove absolutely nothing useful to evaluating a candidate and thus are a waste of time.
- the_d00d 9y agoWhy read it multiple times? Just close your eyes
- mikestew 9y agoBecause the eidetic memory only lasts for a few minutes. The commentor might have meant to use the phrase “photographic memory”, but that has thus far been shown to be myth. But, really, the post was about “I’m smart, and I’d just march right out of that interview”, ignoring that it was just a cute anecdote, so I guess it’s irrelevant. Nice catch, though. :-)
- jsjohnst 9y agoMostly just a cute anecdote is fair, but wasn’t what I was replying to one too? ;) Seriously though, I’m tired of folks doing stupid counterproductive interviews and then parading them as a good thing.
- jsjohnst 9y agoThe other reply was right, because it’s not permanent. I remember a lot just reading, but for full eidetic it lasts a very short time.
- hyperbovine 9y agoSo he's hired one programmer.
- narvind 9y ago0 He's obviously referring to himself.
- pizza 9y agono, he's hired one (other than himself), and she had been working there for ten years by the time we talked
- PhasmaFelis 9y agoSo how long has been trying to hire another one? After 10 years I'd be getting desperate.
- DonHopkins 9y ago>"Seismic data processing involves the manipulation of acoustic data in relation to a linear time varying, unknown system (the Earth model) for the purpose of determining and clarifying the influences involved to enhance geologic interpretation. Coincident (similar) technologies include correlation (statics determination), linear predictive coding (deconvolution), synthesis (forward modeling), formant analysis (spectral enhancement), and processing integrity to minimize artifacts. All of these technologies are shared amongst music and geophysical applications." That is dramatic hand-waving, but it does help convey to non-programmers how dramatic an improvement the right algorithm can implement. It's true what the article says that pitch tracking was considered a difficult "holy grail" in 1995. I attended a talk about pitch tracking at Interval in 1996 by somebody whose name I don't remember any more. The speaker made the point that pitch is a perceptual -- not a mathematical -- concept, and it was hard but possible to do it in real time on a typical PC at the time (i.e. 90 MHZ IBM ThinkPad 760C). But he had done it, and everyone seemed impressed by his demo! ;) The article mentioned formant analysis. Maybe that's related to cepstral analysis, which is another way of tracking the pitch and formants of voice, and has its own cool nomenclature of secret code words. "Cepstral liftering" is basically two FFT's followed by an inverse FFT. If you take the complex FFT of a voice signal, the formants show up as two or three large "hills" in the spectrum, but the pitch manifests as higher frequency repeating furrows in the hills. (Which you want to filter out so you can analyze just the formants, or synthesize a different pitch into them (i.e. auto-tune), so you need to know the frequency of the change in the spectrum magnitude over time: another FFT!) http://www.phon.ucl.ac.uk/courses/spsci/matlab/lect10_files/image001.jpg http://www.phon.ucl.ac.uk/courses/spsci/matlab/lect10_files/... So you take a second FFT of the log of the first FFT, to get a "reverse spectrum" or "cepstrum" in the "quefrency" domain. The fundamental pitch shows up as one big spike in the cepstrum (with smaller spikes for its harmonics). Just "lifter" out the pitch spikes, then perform an inverse FFT to get back to the smooth low frequency formant hills with the high frequency pitch furrows removed. I'm sure there's a lot of "special sauce" in getting the math tweaked and tuned right so it actually sounds good and runs fast. https://stackoverflow.com/questions/4583950/cepstral-analysis-for-pitch-detection https://stackoverflow.com/questions/4583950/cepstral-analysi... The patent seems to be about autocorrelation, which is something different than cepstral analysis, but maybe they could be used together to get even better results. I don't know but would love to learn what the trade-offs and limitations the two techniques have. https://en.wikipedia.org/wiki/Cepstrum https://en.wikipedia.org/wiki/Cepstrum >The name "cepstrum" was derived by reversing the first four letters of "spectrum". Operations on cepstra are labelled quefrency analysis (aka quefrency alanysis), liftering, or cepstral analysis. https://surveillance7.sciencesconf.org/conference/surveillance7/01_a_history_of_cepstrum_analysis_and_its_application_to_mechanical_problems.pdf https://surveillance7.sciencesconf.org/conference/surveillan... "The original application was to the detection of echoes in seismic signals, where it was shown to be greatly superior to the autocorrelation function, because it was insensitive to the colour of the signal."
- MichailP 9y agoAre there some examples of Auto-Tune used with more subtle settings? Famous Cher song used it at setting 0, robotic sounding one, but there are 10 more to go, and they are increasingly more subtle. Also related to this, at least I think, is the issue of turning recording into midi or sheet music. Now that would be a killer app... There is some good software out there, such as Melodyne, but it requires a lot of manual work and tweaking.
- taco_emoji 9y agoI suspect this is one example: https://www.youtube.com/watch?v=BYmn4E-kPSo https://www.youtube.com/watch?v=BYmn4E-kPSo The lead singer's voice just sounds so thin, and unnaturally on-key (no vibrato). I think you'd be hard-pressed to find verifiable examples because those higher settings are probably mostly used to cover up poor singing, and so nobody involved is going to volunteer that information.
- superpope99 9y agoThis is definitely auto-tuned, and I would guess that 90% of auto-tune goes unnoticed as it can be applied a lot more subtly than this.
- criddell 9y agoAs someone who isn't a fan of vibrato, I've always suspected that some singers use it strategically to cover up pitch issues. FWIW, I don't think modern auto-tune plugins have a problem pitch correcting singers using vibrato.
- vkjv 9y agoMelondyne is great with this. Complete lack of vibrato is usually an obvious sign. I think auto-tuners get a bad rap. As an amateur musician, subtle corrections in post-production can save me days of recording to get the right take.
- vonseel 9y agoI didn’t listen to much of this, but the two seconds I heard “I guess it’s all the..” there’s definitely clear artifacting from either auto-tune or compression on the word “all”. And I’m listening on an iPhone. Not a pro editor, but I know my shit.
- jeremysalwen 9y ago>The equations that do autocorrelation are computationally exhaustive: for every one point of autocorrelation (each line on the chart above, right), it might’ve been necessary for Hildebrand to do something like 500 summations of multiply-adds... >Hildebrand realized he was limited by the technology, and instead of giving up, he found a way to work within it using math. “I realized that most of the arithmetic was redundant, and could be simplified,” he says. “My simplification changed a million multiply adds into just four. It was a trick — a mathematical trick.” I think this is the article's way of dramatizing the standard way of calculating the autocorrelation using the convolution theorem: https://en.wikipedia.org/wiki/Autocorrelation#Efficient_computation https://en.wikipedia.org/wiki/Autocorrelation#Efficient_comp...
- thearn4 9y agoFFT is a pretty cool thing. But yeah, this "trick" has been used for correlation-based computations for quite awhile. Assuming that it's the trick that the author was referring to, the article is actually a bit sparse on details.
- soundwave106 9y agoThe patent for Autotune can be found here: https://www.google.com/patents/US5973252?dq=5,973,252 https://www.google.com/patents/US5973252?dq=5,973,252 My (very non-expert) reading of the patent is that it uses sample reduction and some specific features of the periodic form of the human voice to simplify the math involved in the auto-correlation routine. Although this routine does seem to be unique as far as I know, I wouldn't be surprised if it shares similarities to other techniques to reduce correlation complexity.
- hamiltonkibbe 9y agoPolyphase filters maybe? It's an approach used primarily in resampling, and its an easy way to skip a LOT of Multiply-by-zero-and-Accumulate-ing
- nejenendn 9y ago
- purequest 9y agowondering if anyone has listened to Guns N Roses live this past year or so. Axl Rose's voice sounds amazing, considering his age, style of singing, and his well documented hard living. Having listened live at coachella and watched online, im wondering if its possible they are doing it live in real time now.
- marssaxman 9y agoIt is certainly possible - Tascam has been selling a live effects processor based on the autotune algorithm since 2011.
- renaudg 9y agoOf course it's being done live : http://www.antarestech.com/products/detail.php?product=Auto-Tune_Live_2 http://www.antarestech.com/products/detail.php?product=Auto-... Also as a $349 rack device : https://www.amazon.com/Tascam-Producer-Processor-Antares-Autotune/dp/B004OA6JU2 https://www.amazon.com/Tascam-Producer-Processor-Antares-Aut...
- notaboutdave 9y agoSinging live with autotune is standard practice for most pop singers. https://www.youtube.com/watch?v=rczu_Qc7CkA https://www.youtube.com/watch?v=rczu_Qc7CkA
- mcguire 9y agoTl;dr: He mathed the crap out of that thing. Tl;dr #2: Digital signal processing. Is there nothing it can't do?
- pizza 9y agoWhat he (Andy, or was it someone else at Antares? I forget..) told me is that raspy heavy metal vocals are the one type of vocal that doesn't really work with Auto-Tune that nonetheless constantly get requested to get to work
- pcsanwald 9y agoThe ubiquity of auto-tune is particularly interesting because most people don't seem to have a lot of pitch sensitivity. Tons of very famous recordings are slightly out of tune and no one seems to complain, ever. I'd imagine its popularity is more due to convenience rather than demand.
- amelius 9y agoI think slightly out of tune often sounds better (more natural) in all genres except perhaps classical.
- pcsanwald 9y agoagreed, but even in classical music it doesn't really bother me that much sometimes, some of my very favorite string players have occasional intonation issues (who doesn't?). And think of all the jazz records with an out of tune piano!
- contravariant 9y agoDepends what you mean by "out of tune". How good a chord sounds is determined by the ratio of frequencies, not if all the notes fit 'perfectly' in some scale. In fact you can't even create a scale where all chords are perfectly in tune.
- pcsanwald 9y agoall tuning is relative, of course, but my comment is thinking about singers or horn players that play consistently sharp, for example.
- mrob 9y agoThe one genre I can think of where perfect tuning is essential is barbershop music. The defining feature of the genre is chords sung with just intonation and no vibrato, which makes it easy to hear if any of the singers are out of tune. See: https://en.wikipedia.org/wiki/Barbershop_music#Ringing_chords https://en.wikipedia.org/wiki/Barbershop_music#Ringing_chord...
- renaudg 9y agoIt's worth mentioning that even if Auto-Tune has pioneered the field and is now almost a mainstream househould name (much like Photoshop), nowadays all the cool kids are on Melodyne, which is frankly black magic : https://www.youtube.com/watch?v=9FScFKuXXM0 https://www.youtube.com/watch?v=9FScFKuXXM0
- YouKnowBetter 9y agoThank you for that! As someone completly blank about what is happening in this industry, I was impressed and learned a lot.
- jcims 9y agoShould check out some of their newer videos, they are still at it.
- renaudg 9y agoAnother crazy application of Melodyne : turning minor key "sad" songs into "happy" major key ones http://theweek.com/articles/467109/sad-songs-made-happy-amazing-art-turning-minorkey-songs-major http://theweek.com/articles/467109/sad-songs-made-happy-amaz...
- svantana 9y agoOh dear, this guy has no limit to his self-aggrandizement, and the interviewer surely does very little to fact-check his outrageous statements. The facts are more like: auto-tune is the combination of pitch detection and pitch shifting, two problems that were extensively researched already. Even the details, like using autocorrelation via FFT, was standard in the field at the time. This type of pitch correction had already been done in academia, but passed off as a curiosity. The truth is, the guy was at the right place at the right time, nothing more than that. Computers were just becoming fast and cheap enough, and plugin formats making this type of product practical were just being deployed.
- Nition 9y agoDetecting pitch may be easy enough, but a lot of programs still seem to have trouble adjusting pitch transparently without weird artifacts (without simply changing the speed). That's something Auto-Tune does seem to manage better than most.
- svantana 9y agoWell there's a huge difference between monophonic and polyphonic pitch shifting. Monophonic pitch shifting (especially in the sub-semitone range that Auto-Tune does) has been done with high quality since the 1970's.
- smelterdemon 9y agoI had always assumed that Auto-Tune had evolved from or was an advanced form of Vocoder (likely from the similar "robot" effect extreme applications of Auto-tune gives). I've been working on a relatively simple real time music/audio processing project on an Arduino (identifying tempo and using it to create interesting lighting effects for a Halloween costume) and it's an interesting challenge. Extracting any kind of useful information about the underlying musical structure from polyphonic audio is an incredibly hard problem. Add to that limited hardware and the kind of sampling rate you need to capture music (upwards of 40kHz if you want to capture everything you can hear) and you have to get creative.
- soylentcola 9y agoIt's definitely a challenge with something like a microcontroller. My coding skills are borderline nonexistent (but then again, that's part of why I'm experimenting with Arduino in the first place) but I've messed around with using some FFT code published by others to get audio to affect some LED strips. So far the results have been mixed.