9 ms·

Signal processing is key to embedded machine learning

104 points by janjongboom 7y ago

kbumsik 7y ago
It's not the key but the fundamentals that we have done for decades...
taeric 7y ago
My concern with a lot of signal processing techniques used in ml is that sometimes they presuppose things that may not be true. That is, signal processing had Nyquist's rates. And typically knows there is an underlying signal. Does ml have either?
- deleted 7y ago
  [deleted]
- mlazos 7y ago
  You can also use the continuous Fourier transforms and not sample so there won’t be a nyquist rate. I also think neural networks have some really interesting properties when viewed from a signal processing perspective. In fact you can view the correct outputs of a neural network as a signal and the incorrect outputs as noise, and use conventional SP techniques to narrow down what parts of a network are informative.
  - funklute 7y ago
    Practically speaking, how would you use a continuous Fourier transform? Do you mean passing it through a "Fourier transform analog circuit" (does that even exist?), and then sample the output?
    - taeric 7y ago
      I was curious on this. Still am. Tried thinking of several things it could mean, but so far I'm coming up short.
      - radarsat1 7y ago
        I think what he means is that in signal processing you can define the signal analytically and perform the Fourier transform analytically, without sampling. The Nyquist rate only comes in when you talk about digital signal processing.
        mlazos 7y ago
        This was what I meant initially, but I realized in a lot of NNs that it might be pretty difficult if not impossible to define the signal analytically.
        taeric 7y ago
        Ah, if I take your meaning of analytically correctly, I think you are basically arguing for symbolic reasoning. I do still see the appeal of that. Seems solidly not a promising path, at this point. :(
    - mlazos 7y ago
      See my other answer - I meant applying it analytically. Your idea is interesting though, and I do know that it should be possible to build a Fourier transform analog circuit. Once you sample though you will be subject to having a nyquist frequency and introducing noise to your signal and the advantage of performing it with an analog circuit will be lost.
      - funklute 7y ago
        Yea, actually I thought a bit more about it as well, and any real (analog) circuit will have a frequency response that tapers of to zero as the frequency goes to infinity. So that basically determines a Nyquist frequency for any analog circuit, even if that frequency is so high that it has no practical implications.
    - dreamcompiler 7y ago
      Build a bank of bandpass filters with different center frequencies, and you have an analog Fourier transform. (Or at least the power spectrum equivalent; it's trickier to get the phase info.) Analog vocoders are one example of this and they were invented long before discrete FTs.
- etrk 7y ago
  > That is, signal processing had Nyquist's rates. And typically knows there is an underlying signal. Does ml have either? What does this question mean? Every band-limited signal has a Nyquist rate. Most signals of interest are well-contained within some finite bandwidth (e.g., human voice). Sampling above this rate will get you very little. If you're building an ML model to process a certain class of sampled signal and you know, for example, 99% of the signal energy falls within a certain frequency range, that should guide your choice of sample rate. If you're sampling at too high a rate, your input layers may have far more parameters than are needed or useful. Whether or not a given ML input actually contains a signal of interest doesn't seem relevant to how you sample and preprocess the signal.
  - taeric 7y ago
    Most machine learning is not on a band limited signal. I've literally seen these tactics applied to demand forecasting. And I just can't square that they should.
    - etrk 7y ago
      Ah, I see what you mean. Yes, if you're not dealing with approximately bandlimited and sampled signals, then this wouldn't apply. The article is about embedded devices processing sensor data (microphones, motion/light sensors, accelerometers, etc.), and in those cases the signal of interest will often be bandlimited.
      - taeric 7y ago
        Completely agreed. In those cases, these tactics are required.
    - srean 7y ago
      Well, natural signals that have an end arent band-limited either. It is a mathematical abstraction that approximates many real world scenarios well enough
      - 7y ago
alephnil 7y ago
Actually signal processing is already used for most machine learning of audio signals, including speech recognition. The reason is that ML algorithms, including deep learning has a hard time learning the information you can get from a discrete Fourier transform. Audio data in time domain are just too noisy for most machine learning, and doing some signal processing as a preprocessor step often helps a lot. Here it seems like he works with non-audio data, where this is less common.
- taeric 7y ago
  This is just saying that signal processing is vital to the input sensors. Which, doesn't seem new. Yes, ml is dependent on getting data. Signal processing is vital to that. I thought this was saying a new application of signal processing in ml.
ganzuul 7y ago
The Kalman filter is basically an ML algo. The key here is to implement already known linear optimization approximation versions of it in common libraries.
- srean 7y ago
  As an ML practitioner I am curious about how you decide what is ML vs what is signal processing. I really cant tell. Its the same freaking problem, be it information theory, machine learning, signal processing. All of us are stuck at the same impasse -- do optimal rate-distortion of a continuous valued signal efficiently and uniformly.
  - ganzuul 7y ago
    Probably a matter of tradition, not substance. Same with control theory. (Curious who downvoted me, when we are in agreement... Maybe the buttons are too small and people just mis-click.)
    - srean 7y ago
      Dont worry about downvotes much. They can be quite erratic at times. I upvoyed both of yours.
proverbialbunny 7y ago
This is an awesome topic, but I'm somewhat annoyed they didn't dive into what kind of DSP and instead turned the article into an advertisement. Does anyone have any good further reading on the topic? (Books, articles, classes, anything really.)
- bsmitty5000 7y ago
  https://terpconnect.umd.edu/~toh/spectrum/IntroToSignalProcessing.pdf https://terpconnect.umd.edu/~toh/spectrum/IntroToSignalProce... https://allsignalprocessing.com/category/lesson-categories/ https://allsignalprocessing.com/category/lesson-categories/ https://www.digikey.com/en/resources/edu/arm-education-media/dsp https://www.digikey.com/en/resources/edu/arm-education-media... http://www.eas.uccs.edu/~mwickert/ece5650/ http://www.eas.uccs.edu/~mwickert/ece5650/ http://www.eas.uccs.edu/~mwickert/ece5615/ http://www.eas.uccs.edu/~mwickert/ece5615/ http://www.naiant.com/SM/SM%20Forum%20Attachments/Other%20documents/926_Music%20DSP.pdf http://www.naiant.com/SM/SM%20Forum%20Attachments/Other%20do... Sorry I don't know if any of these are good. They're in my dsp bookmarks directory, I think I found these a while back when I was trying to relearn these topics.
- Optimal_Persona 7y ago
  The best DSP intro I've found, especially for the non-math whiz is Steven W. Smith's DSP Guide: http://www.dspguide.com/pdfbook.htm http://www.dspguide.com/pdfbook.htm
  - proverbialbunny 7y ago
    I own a copy of this book. It is quite good as a reference once one has some DSP experience. I wouldn't recommend it as an intro. The book is quite good in that it is to the point and provides a good roadmap, but to do so it often omits concrete examples and instead writes out the algorithms in mathese. Knowing calculus should be enough.
- janjongboom 7y ago
  Author here. Our main focus right now is on vibration and non-voice audio. For vibration we use spectral analysis through butterworth filters, FFT over the result, looking at the peaks and locations of the FFT result, and then at spectral power buckets. For audio we look at MFCC (sometimes paired with a bandpass filter). The source for these are in https://github.com/edgeimpulse/processing-blocks https://github.com/edgeimpulse/processing-blocks (Python) and https://github.com/edgeimpulse/inferencing-sdk-cpp https://github.com/edgeimpulse/inferencing-sdk-cpp (optimised C++). What’s though on our end is that we need to run the DSP in real-time on very constrained devices. Unfortunately I have very little good resources on learning more. I’ve done lots of experimentation in Matlab, just getting raw sensor data and playing with the signal processing toolkit that they sell.
uoaei 7y ago
It always comes down to representation. If you can use a deterministic, efficient algorithm to represent the data in a more amenable manner, then the ML system will have a much easier time "making sense" of the patterns inherent in the data compared to a system that has to learn some abstract transformation from raw data to useful representations.

104 points by janjongboom 7y ago

It's not the key but the fundamentals that we have done for decades...

My concern with a lot of signal processing techniques used in ml is that sometimes they presuppose things that may not be true. That is, signal processing had Nyquist's rates. And typically knows there is an underlying signal. Does ml have either?

deleted 7y ago
[deleted]
mlazos 7y ago
You can also use the continuous Fourier transforms and not sample so there won’t be a nyquist rate. I also think neural networks have some really interesting properties when viewed from a signal processing perspective. In fact you can view the correct outputs of a neural network as a signal and the incorrect outputs as noise, and use conventional SP techniques to narrow down what parts of a network are informative.
- funklute 7y ago
  Practically speaking, how would you use a continuous Fourier transform? Do you mean passing it through a "Fourier transform analog circuit" (does that even exist?), and then sample the output?
  - taeric 7y ago
    I was curious on this. Still am. Tried thinking of several things it could mean, but so far I'm coming up short.
    - radarsat1 7y ago
      I think what he means is that in signal processing you can define the signal analytically and perform the Fourier transform analytically, without sampling. The Nyquist rate only comes in when you talk about digital signal processing.
      - mlazos 7y ago
        This was what I meant initially, but I realized in a lot of NNs that it might be pretty difficult if not impossible to define the signal analytically.
        taeric 7y ago
        Ah, if I take your meaning of analytically correctly, I think you are basically arguing for symbolic reasoning. I do still see the appeal of that. Seems solidly not a promising path, at this point. :(
  - mlazos 7y ago
    See my other answer - I meant applying it analytically. Your idea is interesting though, and I do know that it should be possible to build a Fourier transform analog circuit. Once you sample though you will be subject to having a nyquist frequency and introducing noise to your signal and the advantage of performing it with an analog circuit will be lost.
    - funklute 7y ago
      Yea, actually I thought a bit more about it as well, and any real (analog) circuit will have a frequency response that tapers of to zero as the frequency goes to infinity. So that basically determines a Nyquist frequency for any analog circuit, even if that frequency is so high that it has no practical implications.
  - dreamcompiler 7y ago
    Build a bank of bandpass filters with different center frequencies, and you have an analog Fourier transform. (Or at least the power spectrum equivalent; it's trickier to get the phase info.) Analog vocoders are one example of this and they were invented long before discrete FTs.
etrk 7y ago
> That is, signal processing had Nyquist's rates. And typically knows there is an underlying signal. Does ml have either? What does this question mean? Every band-limited signal has a Nyquist rate. Most signals of interest are well-contained within some finite bandwidth (e.g., human voice). Sampling above this rate will get you very little. If you're building an ML model to process a certain class of sampled signal and you know, for example, 99% of the signal energy falls within a certain frequency range, that should guide your choice of sample rate. If you're sampling at too high a rate, your input layers may have far more parameters than are needed or useful. Whether or not a given ML input actually contains a signal of interest doesn't seem relevant to how you sample and preprocess the signal.
- taeric 7y ago
  Most machine learning is not on a band limited signal. I've literally seen these tactics applied to demand forecasting. And I just can't square that they should.
  - etrk 7y ago
    Ah, I see what you mean. Yes, if you're not dealing with approximately bandlimited and sampled signals, then this wouldn't apply. The article is about embedded devices processing sensor data (microphones, motion/light sensors, accelerometers, etc.), and in those cases the signal of interest will often be bandlimited.
    - taeric 7y ago
      Completely agreed. In those cases, these tactics are required.
  - srean 7y ago
    Well, natural signals that have an end arent band-limited either. It is a mathematical abstraction that approximates many real world scenarios well enough
    - 7y ago

alephnil 7y ago

Actually signal processing is already used for most machine learning of audio signals, including speech recognition. The reason is that ML algorithms, including deep learning has a hard time learning the information you can get from a discrete Fourier transform. Audio data in time domain are just too noisy for most machine learning, and doing some signal processing as a preprocessor step often helps a lot. Here it seems like he works with non-audio data, where this is less common.

taeric 7y ago
This is just saying that signal processing is vital to the input sensors. Which, doesn't seem new. Yes, ml is dependent on getting data. Signal processing is vital to that. I thought this was saying a new application of signal processing in ml.

ganzuul 7y ago

The Kalman filter is basically an ML algo. The key here is to implement already known linear optimization approximation versions of it in common libraries.

srean 7y ago
As an ML practitioner I am curious about how you decide what is ML vs what is signal processing. I really cant tell. Its the same freaking problem, be it information theory, machine learning, signal processing. All of us are stuck at the same impasse -- do optimal rate-distortion of a continuous valued signal efficiently and uniformly.
- ganzuul 7y ago
  Probably a matter of tradition, not substance. Same with control theory. (Curious who downvoted me, when we are in agreement... Maybe the buttons are too small and people just mis-click.)
  - srean 7y ago
    Dont worry about downvotes much. They can be quite erratic at times. I upvoyed both of yours.

proverbialbunny 7y ago

This is an awesome topic, but I'm somewhat annoyed they didn't dive into what kind of DSP and instead turned the article into an advertisement. Does anyone have any good further reading on the topic? (Books, articles, classes, anything really.)

bsmitty5000 7y ago
https://terpconnect.umd.edu/~toh/spectrum/IntroToSignalProcessing.pdf https://terpconnect.umd.edu/~toh/spectrum/IntroToSignalProce... https://allsignalprocessing.com/category/lesson-categories/ https://allsignalprocessing.com/category/lesson-categories/ https://www.digikey.com/en/resources/edu/arm-education-media/dsp https://www.digikey.com/en/resources/edu/arm-education-media... http://www.eas.uccs.edu/~mwickert/ece5650/ http://www.eas.uccs.edu/~mwickert/ece5650/ http://www.eas.uccs.edu/~mwickert/ece5615/ http://www.eas.uccs.edu/~mwickert/ece5615/ http://www.naiant.com/SM/SM%20Forum%20Attachments/Other%20documents/926_Music%20DSP.pdf http://www.naiant.com/SM/SM%20Forum%20Attachments/Other%20do... Sorry I don't know if any of these are good. They're in my dsp bookmarks directory, I think I found these a while back when I was trying to relearn these topics.
Optimal_Persona 7y ago
The best DSP intro I've found, especially for the non-math whiz is Steven W. Smith's DSP Guide: http://www.dspguide.com/pdfbook.htm http://www.dspguide.com/pdfbook.htm
- proverbialbunny 7y ago
  I own a copy of this book. It is quite good as a reference once one has some DSP experience. I wouldn't recommend it as an intro. The book is quite good in that it is to the point and provides a good roadmap, but to do so it often omits concrete examples and instead writes out the algorithms in mathese. Knowing calculus should be enough.
janjongboom 7y ago
Author here. Our main focus right now is on vibration and non-voice audio. For vibration we use spectral analysis through butterworth filters, FFT over the result, looking at the peaks and locations of the FFT result, and then at spectral power buckets. For audio we look at MFCC (sometimes paired with a bandpass filter). The source for these are in https://github.com/edgeimpulse/processing-blocks https://github.com/edgeimpulse/processing-blocks (Python) and https://github.com/edgeimpulse/inferencing-sdk-cpp https://github.com/edgeimpulse/inferencing-sdk-cpp (optimised C++). What’s though on our end is that we need to run the DSP in real-time on very constrained devices. Unfortunately I have very little good resources on learning more. I’ve done lots of experimentation in Matlab, just getting raw sensor data and playing with the signal processing toolkit that they sell.

uoaei 7y ago

It always comes down to representation. If you can use a deterministic, efficient algorithm to represent the data in a more amenable manner, then the ML system will have a much easier time "making sense" of the patterns inherent in the data compared to a system that has to learn some abstract transformation from raw data to useful representations.