5 ms·

Even after taking a college level signal processing course, I didn't really grok why we use sines and cosines. Any explanation of Fourier theory just starts wit

Even after taking a college level signal processing course, I didn't really grok why we use sines and cosines. Any explanation of Fourier theory just starts with the idea to break up a signal into sine and cosine components. Perhaps they'd say those are the simplest, purest instantiation of a frequency. But the idea still just falls out of the sky. I believe I know now: it's because of the convolution theorem. When performing a Fourier transform, we represent the signal in a new basis, where each component will independently get transformed by linear shift invariant systems. Basically the Fourier basis diagonalizes the convolution operator. And the even deeper reason for that is that a complex exponential function can be shifted by simply multiplying it with a constant. In simpler words, it comes down to the barber pole illusion. If you rotate a spring-shaped 3d curve, it looks as if it was traveling upwards. And the 2d projection of the spring are sines and cosines. And it turns out that linear, shift-invariant systems are really common (or are at least a good approximation of many common natural phenomena), so it's very helpful to break up a signal into pieces that each get independently transformed, without any interaction effects.

sbeller 6y ago
> Perhaps they'd say those are the simplest, purest instantiation of a frequency. Actually I thought it is the other way round: A frequency is defined as a sine basically, it has an * amplitude * frequency * phase shift The phase shift is why we need cosines and sines in the Fourier transform as cosine is just a shifted sine. The frequency is the constant in the argument of the sine. And here is where the tail might chase the dog: it's the definition, not an observation I'd think. > When performing a Fourier transform, we represent the signal in a new basis, where each component ... each component is a sine, that is described by the 3 constants above. > In simpler words, it comes down to the barber pole illusion. If you rotate a spring-shaped 3d curve, it looks as if it was traveling upwards. And the 2d projection of the spring are Sines and cosine. Exactly, a complex exponential function is just that spring. And if the absolute value of the argument is not exactly one, the spring spirals outwards or inwards.
- bonoboTP 6y ago
  > Actually I thought it is the other way round: A frequency is defined as a sine basically, it has an * amplitude * frequency * phase shift But this is also true of any periodic wave. The beginner question is, why don't we use triangle waves of square waves? Those also have amplitude, frequency and phase. Frequency just means the reciprocal of the period. To which my answer is that the magical property of complex exponential functions is that they can be shifted by constant (pointwise) multiplication. Which is a really non-obvious fact at first but is crucial in the machinery. The complex exponentials constitute an orthogonal basis which diagonalizes the convolution.
  - kmill 6y ago
    I think it's that exponentials simultaneously diagonalize time shifts, derivatives, and integrals. Convolution follows from these -- though there's something to be said for putting convolution above derivatives and integrals in importance. (A deeper thing going on is that a periodic function can be thought of as a function whose domain is a circle. Circles have obvious rotational symmetry, and when you have symmetry you can use representation theory to decompose things into an (orthogonal) basis. In this case, rotations commute with each other so by some theory the decomposition is going to be entirely through eigenvectors of rotation, which happen to precisely be the exponential functions e^(n theta i) for n an integer. This decomposition is also an isomorphism that carries convolutions to point-wise products in both directions. Also: if you make it so the circle is the complex unit circle, a Fourier transform is the idea that you can create a Laurent polynomial that extends the function to the complex plane minus the origin.)
    - bonoboTP 6y ago
      Nice, indeed at the deeper level it's that if you want to make time shifts equivalent to rotation, you need to move in a circular fashion. Minor point: to me derivatives are just one specific linear time invariant operator, a kind of convolution (with a generalized function) so I think LTI is the thing we really care about.
    - bollu 6y ago
jacobolus 6y ago
Sine (or cosine) is just the projection of uniform circular motion onto a line segment. Why we break arbitrary periodic functions into a sum of sines and cosines (or circular motion of different integer frequencies) is because uniform circular motion is very well understood and studied, so we have many tools for working with it. It’s very easy and convenient to isolate particular terms, and each term has a pretty simple shape, and is infinitely differentiable. These bases are conveniently orthogonal relative to a uniform weight.
- kmill 6y ago
  It's because the eigenvectors of the collection of all time shift operators are the exponential functions, and if you want to stick with real functions, then sin(ax) and cos(ax) together form a basis for the sum of the ai and -ai eigenspaces. (In the non-periodic case, you also have to deal with e^(cx) and products of this with sin and cos for all the "transients"). If you are doing something that is time-invariant, that means it commutes with time shift operators, and that means it has the same eigenvectors. Importantly, this implies such an operator can be analyzed (more or less) by how it scales these eigenvectors.
deleted 6y ago
[deleted]
im3w1l 6y ago
> I didn't really grok why we use sines and cosines. As has been pointed out elsewhere in this thread, they have nice mathematical properties. But another important thing is that they typically work "well enough" for applications. Consider audio. Tones clearly have frequency, but they also have a position in time. Doing a sine-cosine decomposition of a whole song doesn't really make sense, since it has no way of saying that a tone on the piano is played at a given time. So you would think that it would make sense to break the signal down into stuff with frequency and time. Some kind of wavelet probably. Maybe something that very accurately models what a human hears. The thing is that chopping the audio stream up into windows, and decomposing those windows into sines and cosines, while a bit ad hoc, just works well enough.
- zrav 6y ago
  > So you would think that it would make sense to break the signal down into stuff with frequency and time. Some kind of wavelet probably. Maybe something that very accurately models what a human hears. Interestingly wavelet based compression went nowhere because although they have nice mathematical properties, when applied in a lossy compression scheme, they did not fit well with how humans perceive detail/quality, both in terms of psychoaccoustics and psychovisuals, i.e. PSNR vs subjective quality diverged more than with other systems. Not surprisingly none of the state of the art lossy compression algorithms use wavelets.
  - aesthesia 6y ago
    Oh, interesting. I always thought JPEG2000 failed to catch on mainly because of patent encumbrances, not for quality reasons.
    - zrav 6y ago
      Yes, that didn't help of course. JPEG2000 was a bit more efficient than JPEG by virtue of being a newer, more computationally intensive format, not because of wavelets. A modern format like the h.265 derived HEIF still uses DCT/DST based transforms. For video the situation is worse, as AFAIK no-one has been able to come up with a decent wavelet based motion estimation algorithm.
dataflow 6y ago
It's also easier to motivate them (and get an intuition for them) if you have a physical motivation for them. Like if you imagine being in Fourier's shoes (or Heaviside's) and needing to come up with techniques for solving differential equations more easily. Fourier series let you solve PDEs with linear algebra, and Laplace transforms turn ODEs into polynomial equations, both of which are easier to deal with. Or to put it another way, if you imagine being faced with diff eqs and thinking "Hmm, is there a way I can turn these into problems I already know how to solve... like, say, polynomials?", then the idea that there even exists an invertible transformation that lets you do such a thing is itself quite fascinating in its own right—regardless of any elegance the transformation itself might have from a theoretical mathematical standpoint. And having such a motivation and application in mind (and knowing the physical nature of the problem, where we know we'll end up with waves) helps ground the idea in something very concrete and avoids making it look like it's just a random transformation we're studying because we're bored.
analog31 6y ago
One reason for sines and cosines is that they are the solutions for a particular differential equation that seems to show up all over the place in physics and engineering.
jlokier 6y ago
A neat bit of history is that Fourier analysis was originally developed by Fourier to solve a problem - the heat equation - which has no waves in the solution, just diffusion!
vladimirralev 6y ago
I can't recommend enough this lecture https://www.youtube.com/watch?v=iTMn0Kt18tg https://www.youtube.com/watch?v=iTMn0Kt18tg It's about FFT but it goes through the (less known) polynomial multiplication foundation which seems more practical and related to the convolutional applications.
andi999 6y ago
There is no why. The cos and sin form a basis of your function pace. If it aligns with what you want to achieve you use sin and cos, if not you use a different basis (special case of this is e.g. the wavelet transformation)
guyomes 6y ago
Here are some properties that make the cosine base popular. It has the property that among a large set of bases, it roughly provides the best approximation of a given function [0]. Moreover, this approximation can be efficiently obtained by evaluating the function on Chebyshev nodes [1], followed by computing a discrete cosine transform [2]. [0]: https://doi.org/10.1093/comjnl/9.4.404 https://doi.org/10.1093/comjnl/9.4.404 [1]: https://en.wikipedia.org/wiki/Chebyshev_nodes https://en.wikipedia.org/wiki/Chebyshev_nodes [2]: https://en.wikipedia.org/wiki/Discrete_cosine_transform#DCT-II https://en.wikipedia.org/wiki/Discrete_cosine_transform#DCT-...
enriquto 6y ago
> I didn't really grok why we use sines and cosines. We use sines and cosines, but not only sines and cosines. There are many other interesting sets of functions. For example, polynomials, or derivatives of the gaussian functions (called Hermite functions). Bonus point: sines and cosines are polynomials evaluated on the complex unit circle. > Basically the Fourier basis diagonalizes the convolution operator. It also diagonalizes the second derivative, which is the linear operator that governs many physical processes, like wave propagation and heat diffusion.
- EmilioMartinez 6y ago
  >sines and cosines are polynomials evaluated on the complex unit circle Could you explain further?
  - enriquto 6y ago
    If you evaluate the complex function f(x)=z^n on the unit circle z=exp(it), you obtain cos(nt)+i sin(nt). Thus Fourier series are "just" Taylor series evaluated on the unit circle.
dreamcompiler 6y ago
Sinusoids have nice mathematical properties, but there's no law of nature that says you have to use them. Walsh transforms use square waves (actually binary rather than purely square) as basis functions and they have other handy mathematical properties. https://en.m.wikipedia.org/wiki/Hadamard_transform https://en.m.wikipedia.org/wiki/Hadamard_transform

sbeller 6y ago

> Perhaps they'd say those are the simplest, purest instantiation of a frequency. Actually I thought it is the other way round: A frequency is defined as a sine basically, it has an * amplitude * frequency * phase shift The phase shift is why we need cosines and sines in the Fourier transform as cosine is just a shifted sine. The frequency is the constant in the argument of the sine. And here is where the tail might chase the dog: it's the definition, not an observation I'd think. > When performing a Fourier transform, we represent the signal in a new basis, where each component ... each component is a sine, that is described by the 3 constants above. > In simpler words, it comes down to the barber pole illusion. If you rotate a spring-shaped 3d curve, it looks as if it was traveling upwards. And the 2d projection of the spring are Sines and cosine. Exactly, a complex exponential function is just that spring. And if the absolute value of the argument is not exactly one, the spring spirals outwards or inwards.

bonoboTP 6y ago
> Actually I thought it is the other way round: A frequency is defined as a sine basically, it has an * amplitude * frequency * phase shift But this is also true of any periodic wave. The beginner question is, why don't we use triangle waves of square waves? Those also have amplitude, frequency and phase. Frequency just means the reciprocal of the period. To which my answer is that the magical property of complex exponential functions is that they can be shifted by constant (pointwise) multiplication. Which is a really non-obvious fact at first but is crucial in the machinery. The complex exponentials constitute an orthogonal basis which diagonalizes the convolution.
- kmill 6y ago
  I think it's that exponentials simultaneously diagonalize time shifts, derivatives, and integrals. Convolution follows from these -- though there's something to be said for putting convolution above derivatives and integrals in importance. (A deeper thing going on is that a periodic function can be thought of as a function whose domain is a circle. Circles have obvious rotational symmetry, and when you have symmetry you can use representation theory to decompose things into an (orthogonal) basis. In this case, rotations commute with each other so by some theory the decomposition is going to be entirely through eigenvectors of rotation, which happen to precisely be the exponential functions e^(n theta i) for n an integer. This decomposition is also an isomorphism that carries convolutions to point-wise products in both directions. Also: if you make it so the circle is the complex unit circle, a Fourier transform is the idea that you can create a Laurent polynomial that extends the function to the complex plane minus the origin.)
  - bonoboTP 6y ago
    Nice, indeed at the deeper level it's that if you want to make time shifts equivalent to rotation, you need to move in a circular fashion. Minor point: to me derivatives are just one specific linear time invariant operator, a kind of convolution (with a generalized function) so I think LTI is the thing we really care about.
  - bollu 6y ago

jacobolus 6y ago

Sine (or cosine) is just the projection of uniform circular motion onto a line segment. Why we break arbitrary periodic functions into a sum of sines and cosines (or circular motion of different integer frequencies) is because uniform circular motion is very well understood and studied, so we have many tools for working with it. It’s very easy and convenient to isolate particular terms, and each term has a pretty simple shape, and is infinitely differentiable. These bases are conveniently orthogonal relative to a uniform weight.

kmill 6y ago
It's because the eigenvectors of the collection of all time shift operators are the exponential functions, and if you want to stick with real functions, then sin(ax) and cos(ax) together form a basis for the sum of the ai and -ai eigenspaces. (In the non-periodic case, you also have to deal with e^(cx) and products of this with sin and cos for all the "transients"). If you are doing something that is time-invariant, that means it commutes with time shift operators, and that means it has the same eigenvectors. Importantly, this implies such an operator can be analyzed (more or less) by how it scales these eigenvectors.

deleted 6y ago

[deleted]

im3w1l 6y ago

> I didn't really grok why we use sines and cosines. As has been pointed out elsewhere in this thread, they have nice mathematical properties. But another important thing is that they typically work "well enough" for applications. Consider audio. Tones clearly have frequency, but they also have a position in time. Doing a sine-cosine decomposition of a whole song doesn't really make sense, since it has no way of saying that a tone on the piano is played at a given time. So you would think that it would make sense to break the signal down into stuff with frequency and time. Some kind of wavelet probably. Maybe something that very accurately models what a human hears. The thing is that chopping the audio stream up into windows, and decomposing those windows into sines and cosines, while a bit ad hoc, just works well enough.

zrav 6y ago
> So you would think that it would make sense to break the signal down into stuff with frequency and time. Some kind of wavelet probably. Maybe something that very accurately models what a human hears. Interestingly wavelet based compression went nowhere because although they have nice mathematical properties, when applied in a lossy compression scheme, they did not fit well with how humans perceive detail/quality, both in terms of psychoaccoustics and psychovisuals, i.e. PSNR vs subjective quality diverged more than with other systems. Not surprisingly none of the state of the art lossy compression algorithms use wavelets.
- aesthesia 6y ago
  Oh, interesting. I always thought JPEG2000 failed to catch on mainly because of patent encumbrances, not for quality reasons.
  - zrav 6y ago
    Yes, that didn't help of course. JPEG2000 was a bit more efficient than JPEG by virtue of being a newer, more computationally intensive format, not because of wavelets. A modern format like the h.265 derived HEIF still uses DCT/DST based transforms. For video the situation is worse, as AFAIK no-one has been able to come up with a decent wavelet based motion estimation algorithm.

dataflow 6y ago

It's also easier to motivate them (and get an intuition for them) if you have a physical motivation for them. Like if you imagine being in Fourier's shoes (or Heaviside's) and needing to come up with techniques for solving differential equations more easily. Fourier series let you solve PDEs with linear algebra, and Laplace transforms turn ODEs into polynomial equations, both of which are easier to deal with. Or to put it another way, if you imagine being faced with diff eqs and thinking "Hmm, is there a way I can turn these into problems I already know how to solve... like, say, polynomials?", then the idea that there even exists an invertible transformation that lets you do such a thing is itself quite fascinating in its own right—regardless of any elegance the transformation itself might have from a theoretical mathematical standpoint. And having such a motivation and application in mind (and knowing the physical nature of the problem, where we know we'll end up with waves) helps ground the idea in something very concrete and avoids making it look like it's just a random transformation we're studying because we're bored.

analog31 6y ago

One reason for sines and cosines is that they are the solutions for a particular differential equation that seems to show up all over the place in physics and engineering.

jlokier 6y ago

A neat bit of history is that Fourier analysis was originally developed by Fourier to solve a problem - the heat equation - which has no waves in the solution, just diffusion!

vladimirralev 6y ago

I can't recommend enough this lecture https://www.youtube.com/watch?v=iTMn0Kt18tg https://www.youtube.com/watch?v=iTMn0Kt18tg It's about FFT but it goes through the (less known) polynomial multiplication foundation which seems more practical and related to the convolutional applications.

andi999 6y ago

There is no why. The cos and sin form a basis of your function pace. If it aligns with what you want to achieve you use sin and cos, if not you use a different basis (special case of this is e.g. the wavelet transformation)

guyomes 6y ago

Here are some properties that make the cosine base popular. It has the property that among a large set of bases, it roughly provides the best approximation of a given function [0]. Moreover, this approximation can be efficiently obtained by evaluating the function on Chebyshev nodes [1], followed by computing a discrete cosine transform [2]. [0]: https://doi.org/10.1093/comjnl/9.4.404 https://doi.org/10.1093/comjnl/9.4.404 [1]: https://en.wikipedia.org/wiki/Chebyshev_nodes https://en.wikipedia.org/wiki/Chebyshev_nodes [2]: https://en.wikipedia.org/wiki/Discrete_cosine_transform#DCT-II https://en.wikipedia.org/wiki/Discrete_cosine_transform#DCT-...

enriquto 6y ago

> I didn't really grok why we use sines and cosines. We use sines and cosines, but not only sines and cosines. There are many other interesting sets of functions. For example, polynomials, or derivatives of the gaussian functions (called Hermite functions). Bonus point: sines and cosines are polynomials evaluated on the complex unit circle. > Basically the Fourier basis diagonalizes the convolution operator. It also diagonalizes the second derivative, which is the linear operator that governs many physical processes, like wave propagation and heat diffusion.

EmilioMartinez 6y ago
>sines and cosines are polynomials evaluated on the complex unit circle Could you explain further?
- enriquto 6y ago
  If you evaluate the complex function f(x)=z^n on the unit circle z=exp(it), you obtain cos(nt)+i sin(nt). Thus Fourier series are "just" Taylor series evaluated on the unit circle.

dreamcompiler 6y ago

Sinusoids have nice mathematical properties, but there's no law of nature that says you have to use them. Walsh transforms use square waves (actually binary rather than purely square) as basis functions and they have other handy mathematical properties. https://en.m.wikipedia.org/wiki/Hadamard_transform https://en.m.wikipedia.org/wiki/Hadamard_transform