8 ms·

Whenever this kind of stuff comes up I feel like a bit of a fraud... I’ve written a bunch of scientific data analysis code. I have a science PhD. Written large

by xevb3k 8y ago

Whenever this kind of stuff comes up I feel like a bit of a fraud... I’ve written a bunch of scientific data analysis code. I have a science PhD. Written large image analysis pipelines that worked as well as the state of the art... been published etc. For the most part I’ve found basic math and heuristics to be good enough. Every so often I go relearn calculus. But honestly, none of this stuff ever seems to come in handy. Maybe it’s because most of what I encounter is novel datasets where there’s no established method? I reasonably regularly pick up new discrete methods, but the numerical stuff never seems super useful... I don’t know, just a confession I guess... it never comes up on interviews either for what it’s worth.

rxhernandez 8y ago
Just my anecdotal experience, I've seen it come up in interviews for an algorithms group at a medium sized biomed company.
tzahola 8y ago
What kind of novel dataset are we talking about? Just from the top of my head, you could have encountered eigenvectors/eigenvalues: - if you ever used spectral graph algorithms - if you ever done dimensionality reduction via principal component analysis - if you ever calculated the steady state distribution of a Markov chain
- xevb3k 8y ago
  Image analysis and signal corrections on time series data. Extracting features from microscope images for example, correcting for signal convolutions.
jdonaldson 8y ago
Eigenvectors and Eigenvalues show up everywhere, although sometimes it's in the form of an iterative estimate (PageRank is basically the power method estimation of the first eigenvector of a connected graph of web pages). They're in the same class as logarithms and Fourier transforms IMHO. You won't need to calculate them by hand, but you should know what they do and why they're important.
- xevb3k 8y ago
  Interesting, perhaps that’s it... I’ve used FFTs a few times, but even FFTs have never been make or break in terms of getting a pipeline or analysis working well.
  - cowboysauce 8y ago
    The Fourier transform is actually linked with linear algebra. You can think of it as taking a vector in an infinite dimensional Hilbert space (your signal) and decomposing it into its components (the amplitudes of the frequencies).
    - stephencanon 8y ago
      The Fourier transform as used in practice in signal processing is ~always discrete, hence finite-dimensional. It's literally "just" a rotation with an especially nice decomposition that allows efficient multiplication.
      - danharaj 8y ago
        A particular fft in practice may be discrete but the relation between fft's of different resolutions of the "same" signal hints at the infinite structure which bundles up all the finite subspaces into one conceptual object.
- WhompingWindows 8y ago
  Why should you know what they do and why they're important? How does that practically change my R code?
  - angersock 8y ago
    Because, say, knowing about Fourier transforms can help you write more efficient filtering or open up new ways to view your data--perhaps there's a really interesting behavior in the frequency domain you'd miss otherwise. If you just want to be a statistical script kiddie you do you. :)
electricslpnsld 8y ago
It comes up all the time when trying to build second order optimization methods. With the eigensystem of your objective in hand you have a complete understanding of the (non-) convexity of your energy landscape, which is useful to ensure you always have good search directions, etc.
aje403 8y ago
This is frightening but believable. I've worked with a few "quants" who stared at me doe eyed explaining eigen* and basic calculus concepts to them in the context of why their calculations don't add up. You mention you've used fourier transforms before - if you don't understand an eigenbasis then you don't have a fundamental understanding the math you're deploying.
- xevb3k 8y ago
  In the context I used it, FFT wasn’t really of fundamental importance. Using FFT (and in particular FFTW) gave a performance improvement (execution speed), but no real advantage in terms of accuracy over an alternative naive method... So... yes I guess I’ve just not seen anywhere in my work where this stuff has proved useful...
  - aje403 8y ago
    It's pretty much ubiquitous in any quantitative field... would not even know where to start
- ajross 8y ago
  > You mention you've used fourier transforms before - if you don't understand an eigenbasis then you don't have a fundamental understanding the math you're deploying. That's a bit uncharitable. A fourier decomposition can absolutely be understood as an explicit bag of calculus tricks, with no loss of precision or generality. And an awful lot can be done with just those tools -- you don't need to explain JPEG compression or VLBI astronomy in terms of eigenvectors, for example. Obviously (heh, "obviously") it's true that the space of decomposed functions form an orthogonal basis, so technically we're "really" operating in a linear space and that has expressive power too. But there are lots of ways of looking at problems. To wit, you're not wrong. You're just... Well, you know.
  - aje403 8y ago
    Dude your post is orthogonal
soVeryTired 8y ago
For a large fraction of probability theory, you only need two main facts from linear algebra. First, linear transforms map spheres to ellipsoids. The axes of the ellipsoid are the eigenvectors. Second, linear transforms map (hyper) cubes to parallelpipeds. If you start with a unit cube, the volume of the parallelpiped is the determinant of the transform. That more or less covers covariances, PCA, and change of variables. Whenever I try to understand or re-derive a fact in probability, I almost always end up back at one or the other fact. They're also useful in multivariate calculus, which is really just stitched-together linear algebra.
- sakuronto 8y ago
  Not a very useful addition but hypercube is to cube as parallelotope is to parallelepiped.
- zawerf 8y ago
  I use the 2nd point a lot for debugging 3d transforms. To expand upon it, for example in three dimensions the three axes are: (1, 0, 0) (0, 1, 0) (0, 0, 1) To find out where those axes are after a 3x3 matrix transform, you just read off the first, second, and third columns of the matrix respectively. Then you can mentally visualize another unit cube in the new coordinate system using those three vectors as the edges of the cube. Really basic change-of-basis stuff but academic lectures don't emphasize how useful it is to be able to look at a matrix of numbers immediately know what it does.
  - aidos 8y ago
    This concept totally changed my intuitive understanding of matrices. Beautifully illustrated in the below 3blue1brown video. https://youtu.be/kYB8IZa5AuE?t=3m15s https://youtu.be/kYB8IZa5AuE?t=3m15s
- qmalzp 8y ago
  I think the first point is only true for symmetric matrices (which includes those that show up in multivariable calc). In general, the eigenvectors need not be orthogonal.
Bukhmanizer 8y ago
It's one of those things that you don't notice when it's missing, but probably would help a bit if you knew it. That being said, I have to deal with linear algebra every day, and aside from proofs (which obviously they help with), there have been maybe a handful of times that having a deep knowledge of eigenvectors and eigenvalues has helped significantly. Once or twice though, I've got massive speedups (>500x) just by knowing how to do the same thing in a more efficient way. My feeling is having a basic knowledge of testing/caching/memory management is way more useful when you're doing large image analysis.
- xevb3k 8y ago
  Interesting, well I’ll try to keep reviewing this stuff and hoping I find an application. I really would like to find an application in my work, because without that I find new techniques don’t really stick and after a few months I forget them...
  - Bukhmanizer 8y ago
    I'm the exact same way, my job is really heavy in linear algebra, so it sticks more easily for me. Usually I go through the code and ask "what am I trying to do here" and "can I do this a better way". A lot of the aforementioned speedups have come because the previous developer was obviously trying to do something, like create a linear projector, but were following some sort of math formula, so made a bunch of extraneous matrices that were huge. It's a simple fix, but adds up when you're dealing with massive datasets.
  - gh02t 8y ago
    It depends on the field you're in. For example, if you're in an area that heavily uses differential equations (many engineering disciplines) then you're probably gonna be using eigenvectors a lot, as they are important for solving a lot of problems. Other areas may not need them at all. It also depends on your depth in the field. A rank and file engineer may not need to know anything about them - they underpin a lot of numerical methods, but get hidden away in software packages. Someone developing those software packages likely will, though. Techniques based on eigenvectors and eigenvalues are extremely important in my field (nuclear engineering... you've probably heard the term "critical", that refers to an eigenvalue), but I know someone who is an excellent civil engineer and knows next to nothing about them (or linear algebra in general) because they aren't that important for what he works on. Forgetting stuff you don't use is pretty normal, the important thing is to be able to recognize when a technique you don't remember the details of might be applicable, and to know where to look to refresh your memory.
wglb 8y ago
It is certainly frequent in engineering. If you need to analyze the stability of an electrical grid, there isn't much alternative. A fun book on this is https://openlibrary.org/books/OL2398351M/The_algebraic_eigenvalue_problem https://openlibrary.org/books/OL2398351M/The_algebraic_eigen...

by xevb3k 8y ago

rxhernandez 8y ago

Just my anecdotal experience, I've seen it come up in interviews for an algorithms group at a medium sized biomed company.

tzahola 8y ago

What kind of novel dataset are we talking about? Just from the top of my head, you could have encountered eigenvectors/eigenvalues: - if you ever used spectral graph algorithms - if you ever done dimensionality reduction via principal component analysis - if you ever calculated the steady state distribution of a Markov chain

xevb3k 8y ago
Image analysis and signal corrections on time series data. Extracting features from microscope images for example, correcting for signal convolutions.

jdonaldson 8y ago

Eigenvectors and Eigenvalues show up everywhere, although sometimes it's in the form of an iterative estimate (PageRank is basically the power method estimation of the first eigenvector of a connected graph of web pages). They're in the same class as logarithms and Fourier transforms IMHO. You won't need to calculate them by hand, but you should know what they do and why they're important.

xevb3k 8y ago
Interesting, perhaps that’s it... I’ve used FFTs a few times, but even FFTs have never been make or break in terms of getting a pipeline or analysis working well.
- cowboysauce 8y ago
  The Fourier transform is actually linked with linear algebra. You can think of it as taking a vector in an infinite dimensional Hilbert space (your signal) and decomposing it into its components (the amplitudes of the frequencies).
  - stephencanon 8y ago
    The Fourier transform as used in practice in signal processing is ~always discrete, hence finite-dimensional. It's literally "just" a rotation with an especially nice decomposition that allows efficient multiplication.
    - danharaj 8y ago
      A particular fft in practice may be discrete but the relation between fft's of different resolutions of the "same" signal hints at the infinite structure which bundles up all the finite subspaces into one conceptual object.
WhompingWindows 8y ago
Why should you know what they do and why they're important? How does that practically change my R code?
- angersock 8y ago
  Because, say, knowing about Fourier transforms can help you write more efficient filtering or open up new ways to view your data--perhaps there's a really interesting behavior in the frequency domain you'd miss otherwise. If you just want to be a statistical script kiddie you do you. :)

electricslpnsld 8y ago

It comes up all the time when trying to build second order optimization methods. With the eigensystem of your objective in hand you have a complete understanding of the (non-) convexity of your energy landscape, which is useful to ensure you always have good search directions, etc.

aje403 8y ago

This is frightening but believable. I've worked with a few "quants" who stared at me doe eyed explaining eigen* and basic calculus concepts to them in the context of why their calculations don't add up. You mention you've used fourier transforms before - if you don't understand an eigenbasis then you don't have a fundamental understanding the math you're deploying.

xevb3k 8y ago
In the context I used it, FFT wasn’t really of fundamental importance. Using FFT (and in particular FFTW) gave a performance improvement (execution speed), but no real advantage in terms of accuracy over an alternative naive method... So... yes I guess I’ve just not seen anywhere in my work where this stuff has proved useful...
- aje403 8y ago
  It's pretty much ubiquitous in any quantitative field... would not even know where to start
ajross 8y ago
> You mention you've used fourier transforms before - if you don't understand an eigenbasis then you don't have a fundamental understanding the math you're deploying. That's a bit uncharitable. A fourier decomposition can absolutely be understood as an explicit bag of calculus tricks, with no loss of precision or generality. And an awful lot can be done with just those tools -- you don't need to explain JPEG compression or VLBI astronomy in terms of eigenvectors, for example. Obviously (heh, "obviously") it's true that the space of decomposed functions form an orthogonal basis, so technically we're "really" operating in a linear space and that has expressive power too. But there are lots of ways of looking at problems. To wit, you're not wrong. You're just... Well, you know.
- aje403 8y ago
  Dude your post is orthogonal

soVeryTired 8y ago

For a large fraction of probability theory, you only need two main facts from linear algebra. First, linear transforms map spheres to ellipsoids. The axes of the ellipsoid are the eigenvectors. Second, linear transforms map (hyper) cubes to parallelpipeds. If you start with a unit cube, the volume of the parallelpiped is the determinant of the transform. That more or less covers covariances, PCA, and change of variables. Whenever I try to understand or re-derive a fact in probability, I almost always end up back at one or the other fact. They're also useful in multivariate calculus, which is really just stitched-together linear algebra.

sakuronto 8y ago
Not a very useful addition but hypercube is to cube as parallelotope is to parallelepiped.
zawerf 8y ago
I use the 2nd point a lot for debugging 3d transforms. To expand upon it, for example in three dimensions the three axes are: (1, 0, 0) (0, 1, 0) (0, 0, 1) To find out where those axes are after a 3x3 matrix transform, you just read off the first, second, and third columns of the matrix respectively. Then you can mentally visualize another unit cube in the new coordinate system using those three vectors as the edges of the cube. Really basic change-of-basis stuff but academic lectures don't emphasize how useful it is to be able to look at a matrix of numbers immediately know what it does.
- aidos 8y ago
  This concept totally changed my intuitive understanding of matrices. Beautifully illustrated in the below 3blue1brown video. https://youtu.be/kYB8IZa5AuE?t=3m15s https://youtu.be/kYB8IZa5AuE?t=3m15s
qmalzp 8y ago
I think the first point is only true for symmetric matrices (which includes those that show up in multivariable calc). In general, the eigenvectors need not be orthogonal.

Bukhmanizer 8y ago

It's one of those things that you don't notice when it's missing, but probably would help a bit if you knew it. That being said, I have to deal with linear algebra every day, and aside from proofs (which obviously they help with), there have been maybe a handful of times that having a deep knowledge of eigenvectors and eigenvalues has helped significantly. Once or twice though, I've got massive speedups (>500x) just by knowing how to do the same thing in a more efficient way. My feeling is having a basic knowledge of testing/caching/memory management is way more useful when you're doing large image analysis.

xevb3k 8y ago
Interesting, well I’ll try to keep reviewing this stuff and hoping I find an application. I really would like to find an application in my work, because without that I find new techniques don’t really stick and after a few months I forget them...
- Bukhmanizer 8y ago
  I'm the exact same way, my job is really heavy in linear algebra, so it sticks more easily for me. Usually I go through the code and ask "what am I trying to do here" and "can I do this a better way". A lot of the aforementioned speedups have come because the previous developer was obviously trying to do something, like create a linear projector, but were following some sort of math formula, so made a bunch of extraneous matrices that were huge. It's a simple fix, but adds up when you're dealing with massive datasets.
- gh02t 8y ago
  It depends on the field you're in. For example, if you're in an area that heavily uses differential equations (many engineering disciplines) then you're probably gonna be using eigenvectors a lot, as they are important for solving a lot of problems. Other areas may not need them at all. It also depends on your depth in the field. A rank and file engineer may not need to know anything about them - they underpin a lot of numerical methods, but get hidden away in software packages. Someone developing those software packages likely will, though. Techniques based on eigenvectors and eigenvalues are extremely important in my field (nuclear engineering... you've probably heard the term "critical", that refers to an eigenvalue), but I know someone who is an excellent civil engineer and knows next to nothing about them (or linear algebra in general) because they aren't that important for what he works on. Forgetting stuff you don't use is pretty normal, the important thing is to be able to recognize when a technique you don't remember the details of might be applicable, and to know where to look to refresh your memory.

wglb 8y ago

It is certainly frequent in engineering. If you need to analyze the stability of an electrical grid, there isn't much alternative. A fun book on this is https://openlibrary.org/books/OL2398351M/The_algebraic_eigenvalue_problem https://openlibrary.org/books/OL2398351M/The_algebraic_eigen...