5 ms·

So, dumb question... If you were training some deep learning model... ...should you be trying a few Wavelet transforms on your inputs, and feeding those in to

So, dumb question... If you were training some deep learning model... ...should you be trying a few Wavelet transforms on your inputs, and feeding those in to your model, too, to see if your model performs better with wavelet inputs?

nickff 5y ago
It very much depends on your inputs, but it’s likely worth a shot. Note that wavelet transforms are generally much slower than fast Fourier transforms.
actusual 5y ago
Instead of saying "I'm going to try this, maybe it will work", you should instead be asking if wavelet transforms are appropriate given the domain you are building a model for. Don't just transform data in the hopes that it will magically work.
- ericjang 5y ago
  Hypothetically, if the computer could try both of these experiments at no cost to you, and tell you whether it improved things or not, does asking whether wavelets are appropriate for the domain even matter?
- taneq 5y ago
  Or if you do, write down the results and publish them even if they’re negative. That’s how science is meant to work.
- cinntaile 5y ago
  I guess the overarching question is... How do you determine if they are good for your application and how do you choose which family of wavelets to apply?
- aaaaaaaaaaab 5y ago
  >Don't just transform data in the hopes that it will magically work. You’ve just described modern machine learning.
- VikingCoder 5y ago
  Do you know how we got penicillin? Alexander Fleming didn't keep a clean lab. Do you know how we discovered X-Rays? Henri Becquerel realized his photographic plates had been darkened after being left in a drawer with uranium sulfate. Do you know how electric guitar distortion was discovered? Willie Kizart dropped his Fender amp. Worse things have happened than experimenting by throwing one more transform on your inputs before processing them.
  - actusual 5y ago
    Sure, and I understand this sentiment. But in practice/industry, it's best not to build/deploy models you don't understand. Edge cases in ML models that are fully automating business decisions can be pretty dangerous. I think the likelihood of a penicillin level discovery happening when I'm trying to train a model to make better marketing budget decisions for a company is quite low.
deleted 5y ago
[deleted]
kortex 5y ago
The first few layer of CNNs are basically learned wavelets. I've had the thought to use basis functions, based on wavelets and/os Legendre polynomials, to perform the first few layers of image processing (more regular and don't need backprop). Haven't been in that space in a while though and don't have much time to mess with it. CNNs have gotten so good though it seems a little moot.
- SubiculumCode 5y ago
  Not in the field, but that seems to me a potentially interesting intuition.
  - tubby12345 5y ago
    it's almost universally known that the whole point of NNs is that they've obviated the need for feature engineering (for some tasks)
    - kortex 5y ago
      Right, exactly. This is manually engineering the lower features, which CNNs just learn. The chief advantage of this technique is a) certain regular wavelets have performance benefits over convolutions, especially when talking low level/FPGA/ASIC space b) being not learned can be beneficial, as these is nothing to overfit. I can see it being handy for embedded/rasppi like applications. In particular, you can festoon a crude face detector with a dozen Haar filters. End to end training is just soooo convenient though.
ad404b8a372f2b9 5y ago
It's already a thing, check out Kymatio. Currently it underperforms compared to a standard CNN but it's an interesting avenue of research.
shenberg 5y ago
Not a dumb question at all. There has been some research on using a wavelet scattering transform as a feature extractor instead of learned convolutions in the first layers of deep neural networks. It works, but isn't as good, _given enough data_. For low-data regimes it makes sense to do this.

nickff 5y ago

It very much depends on your inputs, but it’s likely worth a shot. Note that wavelet transforms are generally much slower than fast Fourier transforms.

actusual 5y ago

Instead of saying "I'm going to try this, maybe it will work", you should instead be asking if wavelet transforms are appropriate given the domain you are building a model for. Don't just transform data in the hopes that it will magically work.

ericjang 5y ago
Hypothetically, if the computer could try both of these experiments at no cost to you, and tell you whether it improved things or not, does asking whether wavelets are appropriate for the domain even matter?
taneq 5y ago
Or if you do, write down the results and publish them even if they’re negative. That’s how science is meant to work.
cinntaile 5y ago
I guess the overarching question is... How do you determine if they are good for your application and how do you choose which family of wavelets to apply?
aaaaaaaaaaab 5y ago
>Don't just transform data in the hopes that it will magically work. You’ve just described modern machine learning.
VikingCoder 5y ago
Do you know how we got penicillin? Alexander Fleming didn't keep a clean lab. Do you know how we discovered X-Rays? Henri Becquerel realized his photographic plates had been darkened after being left in a drawer with uranium sulfate. Do you know how electric guitar distortion was discovered? Willie Kizart dropped his Fender amp. Worse things have happened than experimenting by throwing one more transform on your inputs before processing them.
- actusual 5y ago
  Sure, and I understand this sentiment. But in practice/industry, it's best not to build/deploy models you don't understand. Edge cases in ML models that are fully automating business decisions can be pretty dangerous. I think the likelihood of a penicillin level discovery happening when I'm trying to train a model to make better marketing budget decisions for a company is quite low.

deleted 5y ago

[deleted]

kortex 5y ago

The first few layer of CNNs are basically learned wavelets. I've had the thought to use basis functions, based on wavelets and/os Legendre polynomials, to perform the first few layers of image processing (more regular and don't need backprop). Haven't been in that space in a while though and don't have much time to mess with it. CNNs have gotten so good though it seems a little moot.

SubiculumCode 5y ago
Not in the field, but that seems to me a potentially interesting intuition.
- tubby12345 5y ago
  it's almost universally known that the whole point of NNs is that they've obviated the need for feature engineering (for some tasks)
  - kortex 5y ago
    Right, exactly. This is manually engineering the lower features, which CNNs just learn. The chief advantage of this technique is a) certain regular wavelets have performance benefits over convolutions, especially when talking low level/FPGA/ASIC space b) being not learned can be beneficial, as these is nothing to overfit. I can see it being handy for embedded/rasppi like applications. In particular, you can festoon a crude face detector with a dozen Haar filters. End to end training is just soooo convenient though.

ad404b8a372f2b9 5y ago

It's already a thing, check out Kymatio. Currently it underperforms compared to a standard CNN but it's an interesting avenue of research.

shenberg 5y ago

Not a dumb question at all. There has been some research on using a wavelet scattering transform as a feature extractor instead of learned convolutions in the first layers of deep neural networks. It works, but isn't as good, _given enough data_. For low-data regimes it makes sense to do this.