4 ms·

Great, my mathematical nemesis is now a part of LLM functionality as well. Are people trying to make this stuff harder?

mananaysiempre 2y ago
(Almost-)linear models do linear things, it seems, and the Fourier transform is the quintessential linear thing. It is also an extremely neat piece of the real world, but I’m hesitant to guess your background and offer an explanation because your phrasing makes me suspect an engineering one. With concepts usually being the first to be culled in a course targeted at engineers, there could be quite a bit of concept debt to pay off before I could really offer something I could honestly call an explanation. Have you tried the 3Blue1Brown video on the topic[1]? It does not AFAIR offer any answers as to why the Fourier transform should exist or be useful, but it does show very well what it does in the immediate sense. [1] https://www.youtube.com/watch?v=spUNpyF58BY https://www.youtube.com/watch?v=spUNpyF58BY
- globalnode 2y ago
  Thanks for your reply. I think I'm going to have to start reading up on physics/differential equations. My linalg is ok but quite a bit of my computing background has been "here, this is how you calculate it" instead of concepts. I really feel theres something about Fourier that seems pretty important.
TeMPOraL 2y ago
IDK, the more I learn the more it seems to me that Fourier transform is reality's cheat code. It keeps showing up everywhere. Like, the other day I learned[0] that if you shine a light through a small opening, the diffraction pattern you get on the other side is basically the Fourier transform of the aperture outline. (Yes, this also implies that if you take a Fourier transform of an image and make a diffraction grating off the resulting pattern, projecting light through it should paint you the original image.) -- [0] - https://www.youtube.com/watch?v=Y9FZ4igNxNA https://www.youtube.com/watch?v=Y9FZ4igNxNA
- ruined 2y ago
  this kind of interference recording is typically termed a 'hologram'. it works in full 3d
  - danadam 2y ago
    3Blue1Brown has a very interesting video about how holograms are made: https://www.youtube.com/watch?v=EmKQsSDlaa4 https://www.youtube.com/watch?v=EmKQsSDlaa4
- smallmancontrov 2y ago
  Right, physics runs on differential equations and sinusoids/exponentials are eigenfunctions of differential equations. You can project reality onto any complete basis of functions you like, but this one tends to diagonalize the physics of our universe, which is an overpowered ability inside of our universe.
  - mananaysiempre 2y ago
    > tends to diagonalize the physics of our universe Because it diagonalizes all good translationally invariant operators, and our universe is fond of translation invariance until you get into general relativity. (This sounds less mysterious once you learn that all good translationally invariant operators are essentially convolutions. Neither of these statements is often taught at the elementary level, probably because of the difficulty and ambiguity in defining “all good” and “are essentially”.)
    - krackers 2y ago
      > good translationally invariant operators are essentially convolutions. I first learned this seemingly obvious in hindsight corollary from another comment on HN [1] and it blew my mind. I wish it was included in the usual descriptions of why we choose complex exponential basis for things like the Laplace transform. It's all well and good that they're eigenfunctions of translations, but it still left me wondering why we care about that in the first place. (If your first exposure is instead from a physics or EE perspective, I suspect the framing would be more obvious, as compared to how it's usually introduced in DiffEq when the choice of basis just seems like a "neat" trick given that it behaves well under differentiation). [1] https://news.ycombinator.com/item?id=37915848 https://news.ycombinator.com/item?id=37915848
almostgotcaught 2y ago
You gotta be in the in-crowd to understand that this paper, like so many others, is one of those dumb posthoc analogy/metaphor papers. These papers are where they just ran a bunch of experiments (ie just ran the training script over and over) and formulated a hypothesis empirically. Of course in order to lend the hypothesis some credibility they have to make an allusion to something formal/mathematical: > Fourier features -- dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain Brilliant and very rigorous!

(Almost-)linear models do linear things, it seems, and the Fourier transform is the quintessential linear thing. It is also an extremely neat piece of the real world, but I’m hesitant to guess your background and offer an explanation because your phrasing makes me suspect an engineering one. With concepts usually being the first to be culled in a course targeted at engineers, there could be quite a bit of concept debt to pay off before I could really offer something I could honestly call an explanation. Have you tried the 3Blue1Brown video on the topic[1]? It does not AFAIR offer any answers as to why the Fourier transform should exist or be useful, but it does show very well what it does in the immediate sense. [1] https://www.youtube.com/watch?v=spUNpyF58BY https://www.youtube.com/watch?v=spUNpyF58BY

globalnode 2y ago
Thanks for your reply. I think I'm going to have to start reading up on physics/differential equations. My linalg is ok but quite a bit of my computing background has been "here, this is how you calculate it" instead of concepts. I really feel theres something about Fourier that seems pretty important.

TeMPOraL 2y ago

IDK, the more I learn the more it seems to me that Fourier transform is reality's cheat code. It keeps showing up everywhere. Like, the other day I learned[0] that if you shine a light through a small opening, the diffraction pattern you get on the other side is basically the Fourier transform of the aperture outline. (Yes, this also implies that if you take a Fourier transform of an image and make a diffraction grating off the resulting pattern, projecting light through it should paint you the original image.) -- [0] - https://www.youtube.com/watch?v=Y9FZ4igNxNA https://www.youtube.com/watch?v=Y9FZ4igNxNA

ruined 2y ago
this kind of interference recording is typically termed a 'hologram'. it works in full 3d
- danadam 2y ago
  3Blue1Brown has a very interesting video about how holograms are made: https://www.youtube.com/watch?v=EmKQsSDlaa4 https://www.youtube.com/watch?v=EmKQsSDlaa4
smallmancontrov 2y ago
Right, physics runs on differential equations and sinusoids/exponentials are eigenfunctions of differential equations. You can project reality onto any complete basis of functions you like, but this one tends to diagonalize the physics of our universe, which is an overpowered ability inside of our universe.
- mananaysiempre 2y ago
  > tends to diagonalize the physics of our universe Because it diagonalizes all good translationally invariant operators, and our universe is fond of translation invariance until you get into general relativity. (This sounds less mysterious once you learn that all good translationally invariant operators are essentially convolutions. Neither of these statements is often taught at the elementary level, probably because of the difficulty and ambiguity in defining “all good” and “are essentially”.)
  - krackers 2y ago
    > good translationally invariant operators are essentially convolutions. I first learned this seemingly obvious in hindsight corollary from another comment on HN [1] and it blew my mind. I wish it was included in the usual descriptions of why we choose complex exponential basis for things like the Laplace transform. It's all well and good that they're eigenfunctions of translations, but it still left me wondering why we care about that in the first place. (If your first exposure is instead from a physics or EE perspective, I suspect the framing would be more obvious, as compared to how it's usually introduced in DiffEq when the choice of basis just seems like a "neat" trick given that it behaves well under differentiation). [1] https://news.ycombinator.com/item?id=37915848 https://news.ycombinator.com/item?id=37915848

almostgotcaught 2y ago

You gotta be in the in-crowd to understand that this paper, like so many others, is one of those dumb posthoc analogy/metaphor papers. These papers are where they just ran a bunch of experiments (ie just ran the training script over and over) and formulated a hypothesis empirically. Of course in order to lend the hypothesis some credibility they have to make an allusion to something formal/mathematical: > Fourier features -- dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain Brilliant and very rigorous!