4 ms·
My concern with a lot of signal processing techniques used in ml is that sometimes they presuppose things that may not be true. That is, signal processing had
by taeric 7y ago
My concern with a lot of signal processing techniques used in ml is that sometimes they presuppose things that may not be true.
That is, signal processing had Nyquist's rates. And typically knows there is an underlying signal. Does ml have either?
- deleted 7y ago[deleted]
- mlazos 7y agoYou can also use the continuous Fourier transforms and not sample so there won’t be a nyquist rate. I also think neural networks have some really interesting properties when viewed from a signal processing perspective. In fact you can view the correct outputs of a neural network as a signal and the incorrect outputs as noise, and use conventional SP techniques to narrow down what parts of a network are informative.
- funklute 7y agoPractically speaking, how would you use a continuous Fourier transform? Do you mean passing it through a "Fourier transform analog circuit" (does that even exist?), and then sample the output?
- taeric 7y agoI was curious on this. Still am. Tried thinking of several things it could mean, but so far I'm coming up short.
- radarsat1 7y agoI think what he means is that in signal processing you can define the signal analytically and perform the Fourier transform analytically, without sampling. The Nyquist rate only comes in when you talk about digital signal processing.
- mlazos 7y agoThis was what I meant initially, but I realized in a lot of NNs that it might be pretty difficult if not impossible to define the signal analytically.
- taeric 7y agoAh, if I take your meaning of analytically correctly, I think you are basically arguing for symbolic reasoning. I do still see the appeal of that. Seems solidly not a promising path, at this point. :(
- mlazos 7y agoSee my other answer - I meant applying it analytically. Your idea is interesting though, and I do know that it should be possible to build a Fourier transform analog circuit. Once you sample though you will be subject to having a nyquist frequency and introducing noise to your signal and the advantage of performing it with an analog circuit will be lost.
- funklute 7y agoYea, actually I thought a bit more about it as well, and any real (analog) circuit will have a frequency response that tapers of to zero as the frequency goes to infinity. So that basically determines a Nyquist frequency for any analog circuit, even if that frequency is so high that it has no practical implications.
- dreamcompiler 7y agoBuild a bank of bandpass filters with different center frequencies, and you have an analog Fourier transform. (Or at least the power spectrum equivalent; it's trickier to get the phase info.) Analog vocoders are one example of this and they were invented long before discrete FTs.
- etrk 7y ago> That is, signal processing had Nyquist's rates. And typically knows there is an underlying signal. Does ml have either? What does this question mean? Every band-limited signal has a Nyquist rate. Most signals of interest are well-contained within some finite bandwidth (e.g., human voice). Sampling above this rate will get you very little. If you're building an ML model to process a certain class of sampled signal and you know, for example, 99% of the signal energy falls within a certain frequency range, that should guide your choice of sample rate. If you're sampling at too high a rate, your input layers may have far more parameters than are needed or useful. Whether or not a given ML input actually contains a signal of interest doesn't seem relevant to how you sample and preprocess the signal.
- taeric 7y agoMost machine learning is not on a band limited signal. I've literally seen these tactics applied to demand forecasting. And I just can't square that they should.
- etrk 7y agoAh, I see what you mean. Yes, if you're not dealing with approximately bandlimited and sampled signals, then this wouldn't apply. The article is about embedded devices processing sensor data (microphones, motion/light sensors, accelerometers, etc.), and in those cases the signal of interest will often be bandlimited.
- taeric 7y agoCompletely agreed. In those cases, these tactics are required.
- srean 7y agoWell, natural signals that have an end arent band-limited either. It is a mathematical abstraction that approximates many real world scenarios well enough
- 7y ago