5 ms·

Building a Signal Analyzer with Modern Web Tech

138 points by Ameo 3y ago

amelius 3y ago
You really need a realtime-OS for this if you want to prevent buffer underruns from happening. Otherwise, it is certainly not audiophile-safe.
- tonyarkles 3y ago
  I mean… Pro Tools, Reaper, Ableton, Logic, Cubase… these all run just fine without an RTOS. But you did say “audiophile” specifically and not “audio professional” so you might be right, but it likely doesn’t matter in practice.
  - amelius 3y ago
    I guess I'll stop dealing with exceptions in my code because they don't happen in practice anyway.
    - tonyarkles 3y ago
      Does your audio input device run an RTOS? Probably. Audio output device? Also likely. Does the intermediate processing need to be an RTOS? Or does it just need to be able to operate fast enough to consume the input buffers and keep the output buffers from starving?
reaperman 3y ago
The overall article is extremely clearly written and well-diagrammed, as you'd expect from someone who writes software for test/metrology equipment. It does look pretty different from your usual web-dev retrospective, and shares more in common with "$BROWSER Internals" style articles. The thesis: > I recently spent some time building a browser-based signal analyzer (spectrogram + oscilloscope) as part of one of my projects. I ended up using some very modern browser APIs and technologies that I'd not worked with before, and I discovered a lot of really interesting patterns and techniques that I'd never seen before in a web app. > Building this application has made it clear to me that the modern web is extremely well-suited to building complex multi-threaded and graphics-intensive applications.
- CamperBob2 3y ago
  Demo doesn't work on FF in Windows 10, not clear that it's meant to run anywhere other than on Chrome or Safari. So I'm not really sure the author's thesis has been validated, as interesting and useful as the concept appears to be.
  - onion2k 3y ago
    Firefox is lagging behind on a lot of the more esoteric web APIs. I think it's a choice by the Firefox team, and a valid one given the effort required, Mozilla's relatively limited resources, and how few devs actually want or need these APIs. Firefox is more like a "website viewer" than a "web app host" like Chrome and Safari. I think that's what a lot of Firefox users want, and why they choose that browser. It might even be useful to differentiate between them instead of calling everything a 'web browser' in the future. To that end I don't think it's a fair criticism of web app developers or Firefox that apps like this don't work in your browser.
    - pfg_ 3y ago
      The specs that firefox chooses not to implement are stuff like web midi for security concerns. Firefox does not try to market itself as a "website viewer" that does not support "web apps"
      - jacquesm 3y ago
        You must have missed the announcement. Firefox now supports web midi, their security concerns where BS all along anyway.
        mindslight 3y ago
        I can't speak on the specific details of web midi, but after the spate of poorly designed browser features with intrinsic security vulnerabilities (eg canvas fingerprinting), it's reasonable to be wary of things coming down the pipeline.
cushychicken 3y ago
This doesn't seem like something you should be able to do with a web browser. Very dope. Glad I saw this. Great work.
- dist-epoch 3y ago
  People were running full 3D engines in the browser 10 years ago (the original asm.js).
  - _trampeltier 3y ago
    3D in browsers was even in the mid 90s possible. But the hardware was not ready, so VRML never really became a thing. https://www.howtogeek.com/778554/remembering-vrml-the-metaverse-of-1995/ https://www.howtogeek.com/778554/remembering-vrml-the-metave...
jcparkyn 3y ago
Very cool project and great write-up, nice work OP.
bigger_cheese 3y ago
Being able to run an FFT on audio signal in real time in a background thread seems really cool. Not sure how feasible it is but could be the basis for speech recognition engine running in browser (I'm not expert but I know Speech detection machine learning models are trained on range of FFT derived signals) - if you can pipe the FFT output to ML endpoint you could potentially have speech to text generation running in the browser - or something like automatic closed caption generation from an incoming audio signal.
- Ameo 3y ago
  Definitely possible to run high-quality speech-to-text in realtime in the browser: https://whisper.ggerganov.com/ https://whisper.ggerganov.com/ Made by the same guy who created the popular llama.cpp LLM library. The model uses log-mel spectrograms as input. Using modern algorithms, FFT is actually really fast to compute. Definitely dwarfed by the evaluation of the model itself, even when using many threads and Wasm SIMD.
- ioseph 3y ago
  Isn't that already offered out of the box via WebAudio's `AnalyserNode` ? https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNod...
  - bigger_cheese 3y ago
    Yep looks like it - wasn't expecting browsers to have it baked in. I'm not all that familiar with what is available in the world of audio. I've done some rudimentary signal analysis on other types of sensors - accelerometer (for detecting vibrations of specific frequency) and millivolt probes on a hardware instrument - I've used stuff like Matlab or Python to do the signal analysis.
password4321 3y ago
Related: Show HN: I've built a spectrogram analyzer web app https://news.ycombinator.com/item?id=35781959 https://news.ycombinator.com/item?id=35781959 (19 days ago) https://webfft.net/dft/ https://webfft.net/dft/
- totetsu 3y ago
  That Webfft is fun. Try recording yourself making a few long vowel sounds A I U E O AE UH , and take a look at how the harmonics/formants/ the pattern of peeks at different frequencies is distributed between sounds.
- ssgh 3y ago
  FYI, it supports decoding ultrasound wav files. I use a Dodotronic UM250K mic to capture ultrasound up to 125 kHz. LCD monitor and other electronics emits some interesting sounds. For some reason, getUserMedia doesn't allow high sample rates, but ffmpeg works: ffmpeg -f alsa -channels 1 -sample_rate 250000 -i hw:CARD=r4,DEV=0 -t 5 mic.wav
Kiro 3y ago
I think the biggest problem with SharedArrayBuffer is that you need to set the Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy headers for it to work at all (due to Spectre), which means you need a host or a web server where you can configure that.
hulitu 3y ago
> Building a Signal Analyzer with Modern Web Tech No info about HW ? Does he uses the soundcard ? Then it is nothing new.

138 points by Ameo 3y ago

You really need a realtime-OS for this if you want to prevent buffer underruns from happening. Otherwise, it is certainly not audiophile-safe.

tonyarkles 3y ago
I mean… Pro Tools, Reaper, Ableton, Logic, Cubase… these all run just fine without an RTOS. But you did say “audiophile” specifically and not “audio professional” so you might be right, but it likely doesn’t matter in practice.
- amelius 3y ago
  I guess I'll stop dealing with exceptions in my code because they don't happen in practice anyway.
  - tonyarkles 3y ago
    Does your audio input device run an RTOS? Probably. Audio output device? Also likely. Does the intermediate processing need to be an RTOS? Or does it just need to be able to operate fast enough to consume the input buffers and keep the output buffers from starving?

reaperman 3y ago

The overall article is extremely clearly written and well-diagrammed, as you'd expect from someone who writes software for test/metrology equipment. It does look pretty different from your usual web-dev retrospective, and shares more in common with "$BROWSER Internals" style articles. The thesis: > I recently spent some time building a browser-based signal analyzer (spectrogram + oscilloscope) as part of one of my projects. I ended up using some very modern browser APIs and technologies that I'd not worked with before, and I discovered a lot of really interesting patterns and techniques that I'd never seen before in a web app. > Building this application has made it clear to me that the modern web is extremely well-suited to building complex multi-threaded and graphics-intensive applications.

CamperBob2 3y ago
Demo doesn't work on FF in Windows 10, not clear that it's meant to run anywhere other than on Chrome or Safari. So I'm not really sure the author's thesis has been validated, as interesting and useful as the concept appears to be.
- onion2k 3y ago
  Firefox is lagging behind on a lot of the more esoteric web APIs. I think it's a choice by the Firefox team, and a valid one given the effort required, Mozilla's relatively limited resources, and how few devs actually want or need these APIs. Firefox is more like a "website viewer" than a "web app host" like Chrome and Safari. I think that's what a lot of Firefox users want, and why they choose that browser. It might even be useful to differentiate between them instead of calling everything a 'web browser' in the future. To that end I don't think it's a fair criticism of web app developers or Firefox that apps like this don't work in your browser.
  - pfg_ 3y ago
    The specs that firefox chooses not to implement are stuff like web midi for security concerns. Firefox does not try to market itself as a "website viewer" that does not support "web apps"
    - jacquesm 3y ago
      You must have missed the announcement. Firefox now supports web midi, their security concerns where BS all along anyway.
      - mindslight 3y ago
        I can't speak on the specific details of web midi, but after the spate of poorly designed browser features with intrinsic security vulnerabilities (eg canvas fingerprinting), it's reasonable to be wary of things coming down the pipeline.

cushychicken 3y ago

This doesn't seem like something you should be able to do with a web browser. Very dope. Glad I saw this. Great work.

dist-epoch 3y ago
People were running full 3D engines in the browser 10 years ago (the original asm.js).
- _trampeltier 3y ago
  3D in browsers was even in the mid 90s possible. But the hardware was not ready, so VRML never really became a thing. https://www.howtogeek.com/778554/remembering-vrml-the-metaverse-of-1995/ https://www.howtogeek.com/778554/remembering-vrml-the-metave...

jcparkyn 3y ago

Very cool project and great write-up, nice work OP.

bigger_cheese 3y ago

Being able to run an FFT on audio signal in real time in a background thread seems really cool. Not sure how feasible it is but could be the basis for speech recognition engine running in browser (I'm not expert but I know Speech detection machine learning models are trained on range of FFT derived signals) - if you can pipe the FFT output to ML endpoint you could potentially have speech to text generation running in the browser - or something like automatic closed caption generation from an incoming audio signal.

Ameo 3y ago
Definitely possible to run high-quality speech-to-text in realtime in the browser: https://whisper.ggerganov.com/ https://whisper.ggerganov.com/ Made by the same guy who created the popular llama.cpp LLM library. The model uses log-mel spectrograms as input. Using modern algorithms, FFT is actually really fast to compute. Definitely dwarfed by the evaluation of the model itself, even when using many threads and Wasm SIMD.
ioseph 3y ago
Isn't that already offered out of the box via WebAudio's `AnalyserNode` ? https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNod...
- bigger_cheese 3y ago
  Yep looks like it - wasn't expecting browsers to have it baked in. I'm not all that familiar with what is available in the world of audio. I've done some rudimentary signal analysis on other types of sensors - accelerometer (for detecting vibrations of specific frequency) and millivolt probes on a hardware instrument - I've used stuff like Matlab or Python to do the signal analysis.

password4321 3y ago

Related: Show HN: I've built a spectrogram analyzer web app https://news.ycombinator.com/item?id=35781959 https://news.ycombinator.com/item?id=35781959 (19 days ago) https://webfft.net/dft/ https://webfft.net/dft/

totetsu 3y ago
That Webfft is fun. Try recording yourself making a few long vowel sounds A I U E O AE UH , and take a look at how the harmonics/formants/ the pattern of peeks at different frequencies is distributed between sounds.
ssgh 3y ago
FYI, it supports decoding ultrasound wav files. I use a Dodotronic UM250K mic to capture ultrasound up to 125 kHz. LCD monitor and other electronics emits some interesting sounds. For some reason, getUserMedia doesn't allow high sample rates, but ffmpeg works: ffmpeg -f alsa -channels 1 -sample_rate 250000 -i hw:CARD=r4,DEV=0 -t 5 mic.wav

Kiro 3y ago

I think the biggest problem with SharedArrayBuffer is that you need to set the Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy headers for it to work at all (due to Spectre), which means you need a host or a web server where you can configure that.

hulitu 3y ago

> Building a Signal Analyzer with Modern Web Tech No info about HW ? Does he uses the soundcard ? Then it is nothing new.