5 ms·

Sharpened Cosine Distance as an Alternative for Convolutions

72 points by gk1 4y ago

jschveibinz 4y ago
Interesting. However, a matched filter is the optimum detector wrt maximizing SNR of the output signal (the features). https://www.sciencedirect.com/topics/engineering/matched-filter https://www.sciencedirect.com/topics/engineering/matched-fil... https://en.m.wikipedia.org/wiki/Matched_filter https://en.m.wikipedia.org/wiki/Matched_filter I wonder if this technique has been tested in “noisy” signal environments? Or is this just useful in infinite SNR applications?
- krapht 4y ago
  optimal among linear functions, and the noise is gaussian... of course noise is not gaussian in many interesting applications and nothing is stopping a nonlinear function from getting better results than a matched filter, we just have no theory to discover what such a function would be so yes, like you, i'd need to see results on actual data before spending time to learn more about it...
  - fxtentacle 4y ago
    In my opinion, this looks promising: https://github.com/stevenwalton/SCS-CCT https://github.com/stevenwalton/SCS-CCT 4% improvement over Compact Transformers, which is considered SOTA EDIT: As meragrin_ pointed out, I read the graph the wrong way. It's actually 4% worse. Sorry.
    - meragrin_ 4y ago
      Where do you see improvement? According to the graph and text, there is a 4% decrease in performance. It is slightly worse.
- gnramires 4y ago
  I tend to think of a matched filter as an "optimum denoiser" (in the mean squared error sense with white gaussian noise), but it's not an optimum detector in the general case, or an optimum "de-interferer". The case shown here should be more resilient to changes in signal amplitude and non-matching interference. Of course, because the kernel is actually learned, it can adapt to interference and try to be a "matched filter to what we want, plus anti-matched filter to interference" (i.e. give near-0 response to common interference signals). But SCS does seem like a more natural way to achieve a similarity metric and feature detection. (although this non-flexibility might work against it)
stefanpie 4y ago
I remember the original twitter thread on the topic but haven't heard much about it since then. It would be interesting to see a nice arxiv submissions with some initial experiments across common tasks and architectures. Would also be interested in hardware efficient versions for fixed point data. Here is some background and more thoughts/references on this on this topic: https://github.com/brohrer/sharpened-cosine-similarity https://github.com/brohrer/sharpened-cosine-similarity
karmakaze 4y ago
The Sharpened Cosine Distance animation uses a different input. How would it respond to the reverse pattern in the input or the square pulse?
- meatmanek 4y ago
  Indeed. If the problem with convolution is that the rectangular pulse gives a stronger signal than the ramped pulses (which match the convolution kernel), then please show how sharpened cosine distance would behave differently in this problem case.
  - nomel 4y ago
    I can't understand why they used completely different input signals as a comparison. Is convolving using some sort of smooth derivative of each useful? Or, is that too sensitive to noise?
- frankus 4y ago
  This twitter thread (https://twitter.com/_brohrer_/status/1232063619657093120 https://twitter.com/_brohrer_/status/1232063619657093120) delivers.
fxtentacle 4y ago
Cosine features tend to measure if two things point in the same direction. Convolutional features tend to measure how much of a template was mixed into a signal. As such, one can expect cosine features to produce very similar results to a convolutional feature followed by a Sigmoid or Tanh. Remember what the first style transfer papers used for feature extraction? Conv+Tanh. After that, cosine features have been used with great success for generating the attention masks in transformer architectures. If I remember correctly, the original Transformer paper also suggested to normalize the features and to pre-scale the result with sqrt(1/N). So the new part in this formula is mostly the pow. The other parts (cosine distance & dividing by the norm) are already pretty much battle-tested in speech recognition. And the sign is just needed to restore what pow with an even exponent loses. In short, this suggests that applying a gamma-like transform to features increases contrast.
- deleted 4y ago
  [deleted]
freemint 4y ago
This thread seems to summarize the state of this method well: https://twitter.com/_clashluke/status/1497092162193793028 https://twitter.com/_clashluke/status/1497092162193793028
DarkSucker 4y ago
Not being familiar with cosine distance, I found an NIST link: https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/cosdist.htm https://www.itl.nist.gov/div898/software/dataplot/refman2/au.... It seems like the main difference between convolution and cosine distance is the normalization. For example, if I associate vector A with the windowed data set and B with the cosine difference "kernel", then cosine difference will have a larger magnitude than a convolution for small A. Cool idea. I plan to give it a whirl some day.
newuser4321 4y ago
I definitely buy that there is a use case for this kernel. Whenever I see this kind of optimization, I feel like it's at least flirting with classical feature engineering. Not that there's necessarily anything wrong with that. You could probably say the same thing about Relu if you wanted. I just think the "it responds better to certain features" argument, which is how I'm understanding this, can quickly throw you back into some classical computer vision work where you're trying to hand optimize instead of relying on gradient descent to find your features
- feoren 4y ago
  > I feel like it's at least flirting with classical feature engineering I, for one, would welcome more of a bridge between intentional human engineering and machine learning. Right now we have inscrutable billion-node language processing models that cause endless arguments about whether they actually know English or not, but don't actually integrate well with anything humans are actually doing. I'd love to see AI get mixed more fluently with manual solutions. It's too either-or right now; the best solutions will come from a mix of smart engineers working seamlessly with focused AI.
Imnimo 4y ago
>it does not give the highest activation for a signal that matches the kernel This has the feeling of being an important property, but it's not clear to me whether it truly is. Do I actually care whether the output of my feature is extractor is "how well does the signal match the kernel" as opposed to some other function of the signal and kernel?
MauranKilom 4y ago
> A feature is detected independent of its sign or scale as long as the scale is significantly larger than p. Pretty sure that is supposed to end in "... than q." instead. p is the sharpness, q is the noise scale (more or less).

72 points by gk1 4y ago

Interesting. However, a matched filter is the optimum detector wrt maximizing SNR of the output signal (the features). https://www.sciencedirect.com/topics/engineering/matched-filter https://www.sciencedirect.com/topics/engineering/matched-fil... https://en.m.wikipedia.org/wiki/Matched_filter https://en.m.wikipedia.org/wiki/Matched_filter I wonder if this technique has been tested in “noisy” signal environments? Or is this just useful in infinite SNR applications?

krapht 4y ago
optimal among linear functions, and the noise is gaussian... of course noise is not gaussian in many interesting applications and nothing is stopping a nonlinear function from getting better results than a matched filter, we just have no theory to discover what such a function would be so yes, like you, i'd need to see results on actual data before spending time to learn more about it...
- fxtentacle 4y ago
  In my opinion, this looks promising: https://github.com/stevenwalton/SCS-CCT https://github.com/stevenwalton/SCS-CCT 4% improvement over Compact Transformers, which is considered SOTA EDIT: As meragrin_ pointed out, I read the graph the wrong way. It's actually 4% worse. Sorry.
  - meragrin_ 4y ago
    Where do you see improvement? According to the graph and text, there is a 4% decrease in performance. It is slightly worse.
gnramires 4y ago
I tend to think of a matched filter as an "optimum denoiser" (in the mean squared error sense with white gaussian noise), but it's not an optimum detector in the general case, or an optimum "de-interferer". The case shown here should be more resilient to changes in signal amplitude and non-matching interference. Of course, because the kernel is actually learned, it can adapt to interference and try to be a "matched filter to what we want, plus anti-matched filter to interference" (i.e. give near-0 response to common interference signals). But SCS does seem like a more natural way to achieve a similarity metric and feature detection. (although this non-flexibility might work against it)

stefanpie 4y ago

I remember the original twitter thread on the topic but haven't heard much about it since then. It would be interesting to see a nice arxiv submissions with some initial experiments across common tasks and architectures. Would also be interested in hardware efficient versions for fixed point data. Here is some background and more thoughts/references on this on this topic: https://github.com/brohrer/sharpened-cosine-similarity https://github.com/brohrer/sharpened-cosine-similarity

karmakaze 4y ago

The Sharpened Cosine Distance animation uses a different input. How would it respond to the reverse pattern in the input or the square pulse?

meatmanek 4y ago
Indeed. If the problem with convolution is that the rectangular pulse gives a stronger signal than the ramped pulses (which match the convolution kernel), then please show how sharpened cosine distance would behave differently in this problem case.
- nomel 4y ago
  I can't understand why they used completely different input signals as a comparison. Is convolving using some sort of smooth derivative of each useful? Or, is that too sensitive to noise?
frankus 4y ago
This twitter thread (https://twitter.com/_brohrer_/status/1232063619657093120 https://twitter.com/_brohrer_/status/1232063619657093120) delivers.

fxtentacle 4y ago

Cosine features tend to measure if two things point in the same direction. Convolutional features tend to measure how much of a template was mixed into a signal. As such, one can expect cosine features to produce very similar results to a convolutional feature followed by a Sigmoid or Tanh. Remember what the first style transfer papers used for feature extraction? Conv+Tanh. After that, cosine features have been used with great success for generating the attention masks in transformer architectures. If I remember correctly, the original Transformer paper also suggested to normalize the features and to pre-scale the result with sqrt(1/N). So the new part in this formula is mostly the pow. The other parts (cosine distance & dividing by the norm) are already pretty much battle-tested in speech recognition. And the sign is just needed to restore what pow with an even exponent loses. In short, this suggests that applying a gamma-like transform to features increases contrast.

deleted 4y ago
[deleted]

freemint 4y ago

This thread seems to summarize the state of this method well: https://twitter.com/_clashluke/status/1497092162193793028 https://twitter.com/_clashluke/status/1497092162193793028

DarkSucker 4y ago

Not being familiar with cosine distance, I found an NIST link: https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/cosdist.htm https://www.itl.nist.gov/div898/software/dataplot/refman2/au.... It seems like the main difference between convolution and cosine distance is the normalization. For example, if I associate vector A with the windowed data set and B with the cosine difference "kernel", then cosine difference will have a larger magnitude than a convolution for small A. Cool idea. I plan to give it a whirl some day.

newuser4321 4y ago

I definitely buy that there is a use case for this kernel. Whenever I see this kind of optimization, I feel like it's at least flirting with classical feature engineering. Not that there's necessarily anything wrong with that. You could probably say the same thing about Relu if you wanted. I just think the "it responds better to certain features" argument, which is how I'm understanding this, can quickly throw you back into some classical computer vision work where you're trying to hand optimize instead of relying on gradient descent to find your features

feoren 4y ago
> I feel like it's at least flirting with classical feature engineering I, for one, would welcome more of a bridge between intentional human engineering and machine learning. Right now we have inscrutable billion-node language processing models that cause endless arguments about whether they actually know English or not, but don't actually integrate well with anything humans are actually doing. I'd love to see AI get mixed more fluently with manual solutions. It's too either-or right now; the best solutions will come from a mix of smart engineers working seamlessly with focused AI.

Imnimo 4y ago

>it does not give the highest activation for a signal that matches the kernel This has the feeling of being an important property, but it's not clear to me whether it truly is. Do I actually care whether the output of my feature is extractor is "how well does the signal match the kernel" as opposed to some other function of the signal and kernel?

MauranKilom 4y ago

> A feature is detected independent of its sign or scale as long as the scale is significantly larger than p. Pretty sure that is supposed to end in "... than q." instead. p is the sharpness, q is the noise scale (more or less).