10 ms·
Show HN: Sound visualisation, better than FFT (iOS)
- avaku 6y agoI also thought this might be useful for people who have problems perceiving some frequencies (maybe all?), to just see the sound. I am not sure what is the correct way to call people who have trouble hearing, but I am really wondering if this could help someone, and I'd like to help make it better. (edit: typos)
- andai 6y agoI had the same idea. I was looking at some spectrograms of a podcast in Audacity and I began to recognize different people's voices and frequently used words. So I realized that with enough effort it would be possible to learn to read in this way. I also like looking at spectrograms of birdsong. I had a setup where I would see the spectrogram of sound coming through my microphone in realtime. I would place it on the windowsill and a few times I would see very faint birdsong I hadn't noticed with my ears, and then I would "tune in" my ears to that frequency range and I would hear it. Thinking about it the ideal form factor for a realtime spectrogram for the hard of hearing (or the curious) would be something like Google Glass. (Originally thought of a smartwatch but that doesn't make much sense for an additional sense). There's a project that's the opposite of this where a colorblind man attached a sensor to his head that converts color into sine waves he can hear. Adam Montandon. Eventually this electronic sense became a natural part of his experience and he began to dream in "color".
- avaku 6y agoI am not sure it would be possible to visually recognise speech, but might help people with hearing difficulties to perceive a concert better or something. Funny think about birds! Yesterday, as I was preparing to release the app, birds were singing outside of my window (coronavirus lockdown, so quiet), and I noticed how fun to look at their songs on my app! I've spent literally 30 minutes looking at that :) I've posted a link to birdwatchers reddit :)
- avaku 6y agoOn the topic of recognition though, I think it would be much easier to implement speech recognition using the core algorithm used here, rather than from FFT for example. I'm only at the beginning of trying to use the frequency detection algo for something else apart from visualisation... The difficulty is that the dat it generates is much more detailed for higher frequencies, so it's hard to adapt modern ML algorithm inputs to it, I need to think more...
- avaku 6y agoInteresting idea about transforming colour to sound! P.S. I am red-green colourblind too :) Although, I don't find it a problem.
- nmfisher 6y agoOne of my favourite (electronic music) producers mentioned recently that he has been developing tinnitus. Normally this would affect his ability to hear certain frequencies, but he’s so accustomed to working with raw waveforms, he’s not too concerned. He knows exactly what the sound he wants should look like.
- avaku 6y agoThanks for comment. I think this can be useful to music producers (in some way), but the market for plugins is so tough, basically hard to survive even for one developer on revenue from one plugin. I’ll be thinking about this.
- avaku 6y agoHi! I made this app in my spare time off work... I described the motivation on the website. Please let me know any feedback! Would really appreciate it.
- ablerman 6y agoI'm curious how this differs from wavelet analysis? Is this a wavelet application with the wavelet function optimized by AI?
- avaku 6y agoI figured out that I've implemented a real-time wavelet transform while I was doing this... :) I tried and couldn't find any good apps that would use it to display real-time analysis of sound, so I made one. Actually, I was working on the AI to generate music, but I haven't got to the AI bit yet, and decided to release this quickly, just for fun, see what people think. I think much better visualisations can be made from the output of this algo...
- fluffything 6y agoCould you describe more in detail how your algorithm differs from a wavelet transform, if at all? The website doesn't say. You mention that your wavelet transform is "real-time", but the normal wavelet transform already is real time.. or what am I missing ?
- avaku 6y agoI guess you're missing the detail of how it's actually implemented in practice. The short description would be just "wavelet transform" I guess. The details is in how to implement it efficiently so that it can run on the phone in real-time, and draw the visualisation at the same time, without drowning the CPU (plus the additional feature of high-precision detection, which is not solved by wavelet transform itself). I will describe it in more details when I'll be open sourcing the algorithm (after I've implemented something on top of it that I could potentially start selling).
- givinguflac 6y agoThis is really cool, I think the linear and circle visualizations are my favorite. I love how much more detail this provides vs fft! You can really see the “feel” of the music. Nice work.
- avaku 6y agoThank you! I am really happy!!
- dubcanada 6y agoIs this Windows Media Player from 2005? I used to love playing songs in it just to watch the random shapes it would generate.
- avaku 6y agoHa ha, is it? No :) It's just my simple attempt on visualisation. The core feature is the better frequency detection. I want to make better visualisations later :)
- siverson914 6y agoSuper cool. This seems like it could be really good data to build some better visualizations. Just playing with it now, but some quick feedback: - Disable the lock screen while watching the visualization. I have to keep tapping it while music is playing. - Is it possible to select the input source as the "speaker out" (or whatever ios calls it) rather than the microphone so only the music playing is analyzed rather than the microphone. My clicky keyboard is messing up the music visualization. Nice work!
- undershirt 6y agoA question comes to mind: What would be preserved if converting the visuals back into audio? This may help answer another question: What type of visual would allow feeling the music without hearing?
- avaku 6y agoAs a matter of fact, I've converted the frequency representation that the core algo extracts back to audio without noticeable quality loss. Although in the app I am just showing the amplitude, but the phases are also important (people usually say that the phase of the note is not important for perception, but actually in complex sounds it's important to preserve relative phases of different frequencies). I think this intermediate representation that I've extracted using this algo would be much better for machine learning on the sound data than either 1) raw sound or 2) frequencies extracted with FFT. But there is an engineering difficulty to overcome: the frequency data that the algo extracts doubles in its amount (more frequent frequency samples) with each octave... It's like wavelet data... It's a challenge to feed this data to standard ML algorithms, need to think of how to configure the inputs, it would have to be highly hierarchical. My initial goal was to "learn" different instruments from raw sound data, and this intermediate representation is good, because it allows "translation invariance" across frequencies. I've described it here: https://vsound.app/high-precision-in-frequency-domain.html https://vsound.app/high-precision-in-frequency-domain.html It's a work in progress... This app is just for me to see if people are generally interested in this area, I don't want to do something for months only to discover nobody wants it (although it's been a few months I've worked on this app LOL).
- undershirt 6y agoI don’t really understand your descriptions. Maybe you can ground it for people better by comparing visuals produced by normal FFT and your novel method. I might go further by letting people hear the difference between them (by transposing the visuals back into audio, to ground our sense of quality loss in the original medium, sound).
- 6y ago
- zelienople 6y agoI wanted to try it, but apparently not compatible with IOS 9. I wonder if this app has any features that require a later OS, or did you just pick the OS compatibility at random in XCode when you created the project? Why am I using IOS 9, you ask? Because that is the latest my device supports. Why not upgrade you ask? Because my device works fine and is in great shape, so I'd rather not destroy the planet for for that sake of my own greed, vanity and stupidity.
- mwfunk 6y agoSo anyone who has a device capable of running > iOS 9 is greedy, vain, and stupid. Got it.
- avaku 6y agoI am sorry, but I am using JUCE framework [1] which has some C++ features (aligned allocation) that are only supported from iOS 11. I'm constrained by the framework, I've set the minimum iOS version it supports. Sorry again! [1] https://juce.com/ https://juce.com/
- dewey 6y agoSomeone is creating a free app in their spare time and that's how you are going to thank them for it? If it doesn't work for your setup then just move on instead of telling people to spend more of their time to make it work for your edge case.
- drawbars 6y agoWould love to be able to export videos using this, is that possible in the future? I have a bunch of original music that I'd like to upload with visuals like this. Nice work
- avaku 6y agoThanks! I'm also planning to release the desktop app (currently for Apple only first), it would be quite easy to make actually, since I have decided not to charge for it anything :) I've actually made videos that are on the front page by doing screen capture on my laptop! I just need to figure out how to sign the desktop app properly, so it can be distributed for Apple... I might be even able to do a Windows app, since I am using JUCE framework. If you want, feel free to subscribe to my email list (link on the app website), so that I can send updates. Sorry, didn't make twitter account for this :)
- drawbars 6y agoThanks, look forward to this. I think there is a market for 'drag your music into the app and export a music video' using your visualisation.
- avaku 6y agoI've done this initial app just to gauge if there is any interest in this at all (friends were telling me "no", but they don't know anything about music :)). I am thinking that I could do some visualisations for free, and add the fancy ones as a paid option. There are many visualisation apps out there, but none have the same frequency detection algorithm... Many people asking to open source it now, but I don't know how can I do it, and also get paid for making something people can pay for on top of it... Just trying to make something that would allow me to leave my (already high paying!) job... I'll need to think :)
- psalminen 6y agoAny plans on releasing an android or, more preferably, a desktop app using this?
- avaku 6y agoYes, soon! Please subscribe to my email list on the app page. I will try to post on HN again, but I am not sure it'll reach the front page again :) P.S. Please upvote.
- jmenter 6y agoThis app is really neat, and I'm curious about what your app is doing beyond FFT analysis. (I spent like 15 minutes playing a 'simulated' church organ and watching the frequency pattern with the dots. It appears that each line is an octave with the leftmost dot in each row tuned to a G. Super fun!) Edit: I'm thinking how cool would it be to add some kind of visualization like you see here: https://paveldogreat.github.io/WebGL-Fluid-Simulation/ https://paveldogreat.github.io/WebGL-Fluid-Simulation/
- avaku 6y agoThanks so much for looking! You’re right, all the visualisations apart from the first one arrange the notes in octaves (12 notes). I’m starting to work on better 2D and 3D visualisations. Unfortunately Apple recently deprecated OpenGL support, so I’m trying something that’s only supported by Apple now (Metal), which means it would have to be for Apple only. I’m keen to make some 3D visualisations... I’m imagining something like a 3D environment, where you could “walk” and watch things react to music like tree leaves and waterfalls! :) The app is basically just “better” at detecting notes (frequencies) than FFT, it’s basically doing an online wavelet transform. Thanks for your feedback!
- jmenter 6y agoWould you consider releasing the core frequency detection module? (I'm assuming it's in C++?) I'm also an iOS developer and it would be fun to play around with it. Is the chromatic scale you align your visualizations to just or equal tempered?
- avaku 6y agoIt’s equal temperament scale (each note’s frequency is 2^1/12 above the lower one) with 12 notes per octave, have a look at user guide for more details. Many people asked for open sourced version of the algo. I want to do it. Just need to think a little about how to do it properly, so that I can still build something on tops of it to “monetise” what I’ve been doing for a year now... (Everyone wants to do smth enjoyable in their life.) I’ll try to do it soon! Subscribe to email list, I’ll let everyone know.
- ssfrr 6y agoReally nice visualizations - they feel really snappy. I really like in the linear visualization how you can see the kick drum hits move down the spectrum!
- avaku 6y agoYeah, this is the first one I noticed, and thought was really cool :)
- bloomer 6y agoThis is really cool. Have you looked at all at the constant Q transform https://en.m.wikipedia.org/wiki/Constant-Q_transform https://en.m.wikipedia.org/wiki/Constant-Q_transform which is basically a Fourier Transform with logarithmically spaces bins so that it can be aligned with traditional musical notes. Miller Puckette (Max/Pure Data) among others, has a fast algorithm for its calculation.
- avaku 6y agoNo I haven't seen this, it's very related. I guess if you wanted to use this for real time, you would still need to solve the question of window size...
- panpanna 6y agoThe part titled "the science bit" on their homepage is completely unscientific. Am I looking bat the wrong place? Can someone explain the math used here?
- avaku 6y agoI haven’t published the details of the algorithm. The science bit is supposed to give some idea, but not details. I know people want the details straight away, but I don’t work in academia, so my goal is not to make a publication, but to gauge interest in applications of this.
- panpanna 6y agoI was more hopping for something like "we use DFT instead of FFT because reasons" No need to publish a scientific report.
- anotheryou 6y agoSo it's just different windows sizes faded in to one and than displaying them as bands?
- avaku 6y agoThere are different ways to describe it in one sentence :) You can say it’s just a wavelet transform. Or that it’s just vocoder. There are many internal implementation details actually. I haven’t published details of the algorithm (yet).
- avaku 6y agoWho wants to get NOTIFIED about the DESKTOP app, so you can take screen recordings to make videos with your music, please SUBSCRIBE to email list on the app page! I will make the same version as on iOS available as DESKTOP apps for free as well. (Plus the visuals look much better if you feed sound directly from the sound interface, not microphone.)
- kitotik 6y agoDoes it show the actual frequencies in HZ / kHZ anywhere? edit: wording
- avaku 6y agoDo you mean the numbers? I was going to add that feature, will add it in another release to turn this off/on. Sorry, I’ve been working on this for a few months, so I decided to release what I have now, and then get on with improvements... Thanks for suggestion, now I know people want it.
- avaku 6y agoDear HN! While the post is still somewhere around the front pages, I wanted to ask what do you think about the following potential applications for monetisation: 1) Desktop app with various visualisations suitable for displaying on screens at bars / clubs. Charge for subscription for extra (good) visualisations (made with proper graphics, not what I did). 1.a) Same but for making videos for YouTube etc. 2) Phone/Tablet app that can consume sound from other apps. Charge for extra 2D/3D dynamic visualisations (without user being able to move around), one time fee per extra visualisation. 3) 3D environments on Phone/Tablet, where things "vibrate to music", like leaves on the trees, waterfalls, etc. Charge for extra 3D environments. Very hard to make, but possible. 4) Apply machine learning to extract components of the tracks, and allow mixing different components from different songs using a new kind of user interface. Extremely hard, but potentially could lead to a DJ revolution, if successful. 5) Any other ideas? Thank you!
- orbifold 6y agoThere was a post here a while ago, where someone made money from creating preview videos for podcasts, which contained a FFT like visualisation.
- gargs 6y agoI’ve been looking for visualizations for music app on macOS for a long time. If you could make one that works seamlessly with apps I’d be willing to pay for it. But, please don’t make it a subscription. Charge as much as you want for a perpetual license.
- avaku 6y agoThanks a lot for your reply!
- royosherove 6y agoI'd (and I think many audio pros) would pay for this as a VST plugin in my DAW to visualize things as a reference (like magic A/B or MCompare - put a reference track and switch visualizations between the two references). The visualizations could be more details to show various frequencies or zoom in on specific areas etc..
- yoda_sl 6y agoPretty cool! Any chance the app could import some audio file to visualize ?
- avaku 6y agoNote at the moment, but I will work on this. Thanks for feedback! P.S. It looks much better with actual audio rather than microphone, see Instagram videos on app website.
- dr_dshiv 6y agoI'm interested in scalograms and continuous wavelet transforms. Here's something from a few years back http://stevehanov.ca/blog/?id=22 http://stevehanov.ca/blog/?id=22
- avaku 6y agoThe pictures you have there look similar to what I see when I visualise the spectrum over time (some examples here: https://vsound.app/high-precision-in-frequency-domain.html https://vsound.app/high-precision-in-frequency-domain.html). I found that white noise was quite interesting too: https://www.instagram.com/p/BzF1f80AEvH/?utm_source=ig_web_copy_link https://www.instagram.com/p/BzF1f80AEvH/?utm_source=ig_web_c...
- canada_dry 6y agoTip: you should credit (note) the artists used in your demos.
- gok 6y agoDid you consider visualizing cepstrum instead of spectrum?
- avaku 6y agoNever heard that word before :) I'll google!
- dchyrdvh 6y agoIt's a topic that's been fascinating me for a while. Sounds, speech and music "produce" images in some sense, so visualizing sound on a flat screen is a meaningful thing to do. That parallel with the Heisenberg principle is a deep observation, I think. A few random thoughts. Colors correspond to notes: a clear red is like a clear "do" note. Volume is like size of the shape. Repetitive patterns correspond to symmetry. And that observation that there's no such thing as a "moment" if sound, and it's always about the past few seconds of sound that make it look like a music. I think this corresponds to multiple image layers with exponential decay: a sound pattern of past would look like a faint decaying background image that eventually disappears. I wish I could study this topic full time, but in our society you either make money or work on interesting things: the heinsenberg principle gas influence even here.
- avaku 6y agoThese are very interesting thoughts, they remind me of when I’m deeply thinking about how a track/song is represented in my mind and trying to visualise it. (With the exception of red being a “do” note, I think because I’m red-green colourblind). Sometimes I’m drawing to try to imagine much more complex visualisations that leave imprints of past music that then get activated again. At this stage it’s more like dreaming than actual possibility that I could already implement. But it’s nice to dream. I agree about society... That’s why I did this app in my free time off work. I could only spend 1 hour per every 2 days on it. Imagine the possibilities if I (and other people) had more time for creative work...
- fg6hr 6y agoCymatics. It's sound interference drawing some peculiarly complex, but cool looking patterns. Edit: I've tried to find actual math behind those pictures and only found piles of pseudoscience. There's a CymaScope app that draws them, but they are super secretive about it. I suspect that it's just the interference pattern in a tibetian bowl or a cup of tea. It's almost suspicious that wikipedia has detailed scientific articles with hardcore math on dumb topics, but not only wikipedia, but the entire math community seems to carefully avoid this topic. Edit2: I think the math behind this is https://en.wikipedia.org/wiki/Normal_mode https://en.wikipedia.org/wiki/Normal_mode. In this case, the picture can likely be derived by numerically solving the diff equation of a sound wave with a boundary condition on a circle. Edit3: This lead me to https://en.wikipedia.org/wiki/Vibrations_of_a_circular_membrane https://en.wikipedia.org/wiki/Vibrations_of_a_circular_membr... with Bessel equations and all the good stuff. Solving it (numerically) would supposedly create the cool picture of the 432 Hz note. Edit4: And I've returned to where I began: cymatics, cymaglyphs and that cymascope. It gives an impression of a lazy pseudoscience at first, because of the somewhat sloppy language they use, but after watching a John Stuart Reid's presentation (watch it, it's only 40 mins), I had to change my opinion. The indeed capture the interference patterns created by sound in a bowl of water and a picture of that rapidly moving pattern is called a "cymaglyphs". I'd say, visualizing sound and music is a solved problem and the solution is called cymaglyphs. Obviously, an app can't use a bowl of water to solve the diff equations, but it can do that numerically with a decent precision, and apply some smoothing techniques to deal with the rapid evolution of these cymaglyphs.
- avaku 6y agoHi! Thanks for your comment! It’s interesting. I don’t think there is any magic about cymatics per se: it depends on how sound interacts with physical objects, and these objects have certain resonant frequencies, which is when the start to vibrate “interestingly”. I like it as a potential visualisation idea though!
- dchyrdvh 6y agoThat above isn't just an idea, it's the idea of sound visualization. It's a bijective projection between sound and shape that turns nice sound into nice shapes and vice versa. The analog sound visualizer would be an air filled glass sphere put next to speakers, if only we could see air. All you need is a performant software simulation of it. The only problem to solve here is visualizing the rapidly changing patterns in a comprehensible way. P.S. actually, we can see air patterns if we fill that glass sphere with dust. Edit: In fact, that's been done already. Find the "Yantara Jiro voice made visible". They used a cymascope (a water bowl in lab conditions) and captured the formed patterns with a fast camera. A water bowl is indeed a great music visualizer.